REVIEW 4 major objections 5 minor 1 cited by
Towards cosmological inference on unlabeled out-of-distribution HI observational data
T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read A neural network trained on simulated 21-centimetre maps can be adapted to unlabeled maps from a different simulation and still recover the matter density $\Omega_{\rm m}$ with $R^2 \ge 0.9$, without ever seeing target labels.
desk verdict Solid proof-of-concept for unsupervised domain adaptation in cosmology, but the headline R² may depend on an unstated map-level train/test split. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is two-step feature alignment in the encoder's latent space. A source encoder plus a regressor are pre-trained supervised on in-distribution maps; a target encoder with identical architecture is initialized with the source weights; then, with the source encoder and regressor frozen, either (a) an adversarial discriminator (ADDA) tries to tell source codes from target codes while the target encoder tries to fool it, or (b) the Wasserstein distance between the two code distributions is minimized by optimal transport. The regressor trained on the source is then applied to the aligned target codes. The argument relies on the two domains sharing the same label space and the same conditional map-to-cosmology mapping, so that aligning features does not destroy the cosmological information.
What would settle it
A direct test: bin both simulation suites by ground-truth $\Omega_{\rm m}$ and train a classifier to tell SIMBA from IllustrisTNG maps within each bin. If the classifier succeeds well above chance within bins, then $p_S(y|x) \ne p_T(y|x)$ and the covariate-shift premise is false; adaptation could then be discarding rather than preserving cosmological information.
Extended reading notes
Core claim
The central claim is that a target encoder adapted without access to the target labels can match the source encoder's supervised performance on the matter density, provided only covariate shift separates the domains. In the SIMBA-to-IllustrisTNG direction, the adapted target network achieves $R^2 = 0.945$ on $\Omega_{\rm m}$ and $0.735$ on $\sigma_8$, nearly identical to the source network's in-distribution $R^2 = 0.948$, while the un-adapted source network collapses to $R^2 = -0.474$. The paper further shows that the same recovery works with only 100 target instances, reaching $R^2 \approx 0.89$ for $\Omega_{\rm m}$ in that direction, and documents that the reverse direction (IllustrisTNG to SIMBA) recovers $\Omega_{\rm m}$ but not $\sigma_8$, which it attributes to the loss of small-scale information in the blurrier SIMBA maps.
Load-bearing premise
The load-bearing premise is that the two datasets differ only in the appearance of the maps, not in how a map is related to the cosmology that produced it; if that map-to-cosmology relation differs between simulations, aligning the features could erase the information the frozen source regressor needs.
Editorial extensions
If this is right
- A model trained on simulated HI maps can be adapted to unlabeled maps from a different simulation and still infer $\Omega_{\rm m}$ with $R^2 \ge 0.9$, close to the supervised source performance.
- Only 100 out-of-distribution maps, about 100 times fewer than the full training set, are enough to adapt the target encoder and recover $\Omega_{\rm m}$ with $R^2 \approx 0.89$ in the SIMBA-to-IllustrisTNG setup.
- The success is asymmetric: recovering $\sigma_8$ from the blurrier SIMBA maps fails even after adaptation, suggesting that the target maps' small-scale content limits which parameters can be transferred.
- The same two-step adaptation recipe can in principle be applied to other large-scale observables whose simulated training data differ from the data to which the model is deployed.
Reading between the lines
- If the covariate-shift assumption carries over to real surveys, the same pipeline could be run with a simulation as the source and survey maps as the target; a natural first test would be on semi-realistic mock observations with injected foregrounds and noise, where the true cosmology is still known.
- The observed asymmetry (blurry-to-detailed succeeds, detailed-to-blurry fails for $\sigma_8$) suggests that domain adaptation cannot manufacture information that is absent in the target maps; for real surveys this would mean resolution and foreground cleaning determine which cosmological parameters are recoverable after adaptation.
- Feature alignment in latent space is only a proxy for label alignment. A diagnostic that directly tests whether $p_S(y|x) = p_T(y|x)$ holds, for example checking whether source and target maps with the same $\Omega_{\rm m}$ remain separable, would tell whether the aligned features preserve cosmological information or simply discard it.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents an unsupervised domain adaptation (UDA) framework for field-level cosmological inference from 21-cm HI maps. Using the CAMELS IllustrisTNG and SIMBA simulation suites as source and target domains, the authors pre-train a source encoder and regressor on labeled in-distribution maps, then adapt a target encoder on unlabeled target maps using either adversarial discriminative domain adaptation (ADDA) or optimal transport (Wasserstein distance). The adapted target encoder is evaluated with the frozen source regressor on target test maps, achieving reported R² values around 0.9 for Ωm in the full-data experiments and degrading performance in the 100-instance adaptation experiments. The authors conclude that UDA can transfer cosmological inference across simulation suites without target labels, with implications for upcoming HI surveys.
Significance. If the central claim holds, this is a useful proof of concept for applying simulation-trained networks to unlabeled survey data, an important practical problem for upcoming HI intensity mapping experiments. The method is clearly described, the code is publicly linked, and the adaptation losses do not use target labels, so there is no obvious circularity in the label usage. However, the significance depends critically on the evaluation protocol: the reported generalization must be measured on simulations whose labels and initial conditions were not seen in training. The paper also includes a reproducible public-data setup, which is a strength, but the current evidence is weakened by the ambiguous train/test split description and by the absence of repeated-run statistics.
major comments (4)
- [Section II and Section IV] The train/test split is not defined at the simulation level. Section II states that 12,000 pairs of {map, label} from each domain are used for training and 1,500 unseen instances for testing, but it does not say whether the test maps come from simulations that also contributed training maps. In CAMELS, each simulation has a unique (Ωm, σ8, ASN1, ASN2, AAGN1, AAGN2) combination and a fixed initial condition, with 2D maps being projections of the same volume. If training and test maps share simulations, the encoder and frozen regressor could exploit a per-simulation fingerprint (shared large-scale structure and label), inflating the reported R² and undermining the out-of-distribution generalization claim. Please clarify the split, and if simulations overlap, re-run the experiments with a simulation-exclusive split (train on some simulations, test on unseen simulations).
- [Abstract, Table II, and Section IV.C] The abstract's claim that 'in all scenarios considered ... the target encoder ... is able to retrieve the underlying Ωm ... to a great accuracy of R² ≥ 0.9' is contradicted by Table II: for TNG→SIMBA with 100 instances, Ωm R² is 0.698 (ADDA) and 0.692 (OT). In addition, σ8 values in Table I are far below 0.9 in several rows (e.g., 0.394 for SIMBA→TNG/OT, 0.205 and -0.215 for TNG→SIMBA). The claim should be restricted to full-data Ωm results, or the experiments and conclusions should be revised accordingly.
- [Section IV and Tables I-II] All reported R² values come from single runs without error bars or multiple random seeds. Since the comparison between ADDA and OT (e.g., SIMBA→TNG σ8: 0.735 vs 0.394) and between full-data and 100-instance settings (e.g., TNG→SIMBA Ωm: 0.903/0.924 vs 0.698/0.692) is central to the paper's conclusions, the stability of these numbers is important. Please provide repeated runs (e.g., 5–10 seeds) with mean ± standard deviation for at least the headline results in Tables I and II.
- [Section III.A] The paper assumes p_S(y|x) = p_T(y|x) on the grounds that both suites share the same parameter priors and that topology is 'equally sensitive' to cosmology, but this concept-shift assumption is not directly tested. If the different baryonic physics changes the map-to-cosmology mapping, aligning feature marginals (Eqs. 2–3 and Eq. 9) can distort or discard information needed by the frozen source regressor. A concrete diagnostic would be to compare the adapted target encoder with a supervised oracle trained on target labels (used only for assessment) to measure how much label-relevant information is preserved after alignment; at minimum, the assumption should be discussed as an additional limitation.
minor comments (5)
- [Section III.A] The phrase 'join probability' should be 'joint probability'.
- [Section IV and Figures 4, 6] The vertical bars in Figures 4 and 6 are labeled 'error bars' but they represent the absolute difference between the ground truth and the predicted parameter, not an uncertainty estimate. Please clarify the caption.
- [Section II] The sentence '1,500 of unseen instances from both domains are used for testing' is ambiguous; state whether this is 1,500 instances per domain or 1,500 total.
- [Table II] Define TNADDA and TNOT in the caption or in the text near the table, since these abbreviations are not self-explanatory outside the main text.
- [Section IV.C] The phrase 'about 100 times smaller than the original target domain dataset' is imprecise; 100 instances versus 12,000 is 120 times smaller, which is two orders of magnitude as stated in the text.
Circularity Check
No circular step found; self-citations are non-load-bearing reuse and the central Omega_m result is empirically evaluated on held-out target labels.
full rationale
The derivation chain is not circular. The source encoder and regressor are pre-trained on labeled source maps, the target encoder is aligned to source features via adversarial losses (Eqs. 2-3) or the Wasserstein distance (Eq. 9), and the frozen source regressor is applied to the aligned target features. This is the intended transfer mechanism, not a construction that forces the reported result. The R^2 values for Omega_m and sigma_8 are measured on target labels used only for evaluation; the paper states that target labels are 'not accessed during training phases' and 'only used to assess the performance' (Section II). The fact that sigma_8 transfer fails in several scenarios shows the evaluation is not trivially forced. The covariate-shift assumption p_S(y|x)=p_T(y|x) is an explicit modeling assumption, not a derived conclusion, so any concern about its validity is a correctness risk rather than circularity. The self-citations are to the authors' earlier loss function and hyperparameter prescription [2] (Eq. 1 and Section III B) and to background work [28, 29]; these are component reuse or contextual references, not load-bearing support for the adaptation claim. No equation reduces to another by construction, and no fitted target label is renamed as a prediction.
Assumptions & free parameters
free parameters (3)
- Adaptation epochs per scenario =
200 (SIMBA→TNG), 800 (TNG→SIMBA)
- Adaptation learning rates =
encoder: 1e-5, discriminator: 1e-4
- Source pre-training learning rate and batch size =
lr=0.0015, batch=50
assumptions (4)
- domain assumption The mapping from HI map to cosmology is the same in both simulation suites: p_S(y|x) = p_T(y|x), so only covariate shift is present.
- domain assumption The pre-trained source regressor remains valid on aligned target features.
- domain assumption The label priors are identical across domains: p_S(y) = p_T(y).
- standard math Standard ML background: adversarial training, optimal transport (Wasserstein) minimization, and UMAP work as described in the cited literature.
Cite this review
Pith. "Pith review of Towards cosmological inference on unlabeled out-of-distribution HI observational data." pith.science (2026). https://pith.science/paper/IINQPQFU
@misc{pith2026241110515,
author = {Pith},
title = {Pith review of: Towards cosmological inference on unlabeled out-of-distribution HI observational data},
year = {2026},
howpublished = {\url{https://pith.science/paper/IINQPQFU}},
note = {Machine review of arXiv:2411.10515}
}
abstract
We present an approach that can be utilized in order to account for the covariate shift between two datasets of the same observable with different distributions. This helps improve the generalizability of a neural network model trained on in-distribution samples (IDs) when inferring cosmology at the field level on out-of-distribution samples (OODs) of {\it unknown labels}. We make use of HI maps from the two simulation suites in CAMELS, IllustrisTNG and SIMBA. We consider two different techniques, namely adversarial approach and optimal transport, to adapt a target network whose initial weights are those of a source network pre-trained on a labeled dataset. Results show that after adaptation, salient features that are extracted by source and target encoders are well aligned in the embedding space. This indicates that the target encoder has learned the representations of the target domain via the adversarial training and optimal transport. Furthermore, in all scenarios considered in our analyses, the target encoder, which does not have access to any labels ($\Omega_{\rm m}$) during adaptation phase, is able to retrieve the underlying $\Omega_{\rm m}$ from out-of-distribution maps to a great accuracy of $R^{2}$ score $\ge$ 0.9, comparable to the performance of the source encoder trained in a supervised learning setup. We further test the viability of the techniques when only a few out-of-distribution instances are available for training and find that the target encoder still reasonably recovers the matter density. Our approach is critical in extracting information from upcoming large scale surveys.
Figures
Figures from the paper (6 more)
Forward citations
Cited by 1 Pith paper
-
Reproducibility of machine learning analyses of 21 cm reionization maps
Convolutional networks trained on 21 cm reionization images often memorize simulation boxes rather than physics, yielding high same-box test scores but poor performance on unseen simulations.
Reference graph
Works this paper leans on
-
[1]
Alonso D., Bull P., Ferreira P. G., Santos M. G., 2015, Monthly Notices of the Royal Astronomical Society, 447, 400
work page 2015
-
[2]
Andrianomena S., Hassan S., 2023, Journal of Cosmology and Astroparticle Physics, 2023, 051
work page 2023
-
[3]
Bandura K., et al., 2014, in Ground-based and Airborne Tele- scopes V . pp 738–757
work page 2014
-
[4]
A., Davies R
Battye R. A., Davies R. D., Weller J., 2004, Monthly Notices of the Royal Astronomical Society, 355, 1339
2004
-
[5]
Battye R., Browne I., Dickinson C., Heron G., Maffei B., Pourt- sidou A., 2013, Monthly Notices of the Royal Astronomical Society, 434, 1239
work page 2013
-
[6]
K., 2005, Monthly Notices of the Royal Astronomical Society, 358, 968
Bharadwaj S., Pandey S. K., 2005, Monthly Notices of the Royal Astronomical Society, 358, 968
2005
-
[7]
Bigot-Sazy M.-A., et al., 2015, arXiv:1511.03006
arXiv 2015
-
[8]
Bonneel N., Van De Panne M., Paris S., Heidrich W., 2011, in Proceedings of the 2011 SIGGRAPH Asia conference. pp 1–12
work page 2011
Show all 55 references
-
[9]
Bousmalis K., Trigeorgis G., Silberman N., Krishnan D., Erhan D., 2016, Advances in neural information processing systems, 29
2016
-
[10]
G., Patel P., Santos M
Bull P., Ferreira P. G., Patel P., Santos M. G., 2015, The Astro- physical Journal, 803, 21
2015
-
[11]
B., McDonald P., 2008, Physical Review Letters, 100, 091303
Chang T.-C., Pen U.-L., Peterson J. B., McDonald P., 2008, Physical Review Letters, 100, 091303
2008
-
[12]
N., Madireddy S., Johnston T., Nord B., 2020, arXiv:2011.03591
´Ciprijanovi´c A., Kafkes D., Jenkins S., Downey K., Per- due G. N., Madireddy S., Johnston T., Nord B., 2020, arXiv:2011.03591
2020 arXiv
-
[13]
Ciprijanovic A., et al., 2022, Mach. Learn. Sci. Technol., 3, 35007
2022
-
[14]
N., Wild S
´Ciprijanovi´c A., Lewis A., Pedro K., Madireddy S., Nord B., Perdue G. N., Wild S. M., 2023, Machine Learning: Science and Technology, 4, 025013
2023
-
[15]
Courty N., Flamary R., Tuia D., Rakotomamonjy A., 2016, IEEE transactions on pattern analysis and machine intelligence, 39, 1853
2016
-
[16]
H., Appleby S., 2019, Monthly Notices of the Royal 11 Astronomical Society, 486, 2827
Dav ´e R., Angl ´es-Alc´azar D., Narayanan D., Li Q., Rafiefer- antsoa M. H., Appleby S., 2019, Monthly Notices of the Royal 11 Astronomical Society, 486, 2827
2019
-
[17]
R., et al., 2017, Publications of the Astronomical Society of the Pacific, 129, 045001
DeBoer D. R., et al., 2017, Publications of the Astronomical Society of the Pacific, 129, 045001
2017
-
[18]
pp 133–139
Dell’Aiera M., Vuillaume T., Jacquemont M., Benoit A., 2023, in Proceedings of the 20th International Conference on Content-based Multimedia Indexing. pp 133–139
2023
-
[19]
R., 2021, Ad- vances in data science and information engineering: proceed- ings from ICDATA 2020 and IKE 2020, pp 877–894
Farahani A., V oghoei S., Rasheed K., Arabnia H. R., 2021, Ad- vances in data science and information engineering: proceed- ings from ICDATA 2020 and IKE 2020, pp 877–894
2021
-
[20]
Faucher-Giguere C.-A., Lidz A., Zaldarriaga M., Hernquist L., 2009, The Astrophysical Journal, 703, 1416
2009
-
[21]
Flamary R., et al., 2021, Journal of Machine Learning Re- search, 22, 1
2021
-
[22]
pp 1180–1189
Ganin Y ., Lempitsky V ., 2015, in International conference on machine learning. pp 1180–1189
2015
-
[23]
Gilda S., de Mathelin A., Bellstedt S., Richard G., 2024, Un- supervised domain adaptation for constraining star formation histories
2024
-
[24]
Gillet N., Mesinger A., Greig B., Liu A., Ucci G., 2019, Monthly Notices of the Royal Astronomical Society, 484, 282
2019
-
[25]
Greig B., Mesinger A., 2015, Monthly Notices of the Royal Astronomical Society, 449, 4246
2015
-
[26]
Haardt F., Madau P., 2012, The Astrophysical Journal, 746, 125
2012
-
[27]
G., 2017, Monthly Notices of the Royal Astronomical Society, 468, 122
Hassan S., Dav ´e R., Finlator K., Santos M. G., 2017, Monthly Notices of the Royal Astronomical Society, 468, 122
2017
-
[28]
Hassan S., Andrianomena S., Doughty C., 2020, Monthly No- tices of the Royal Astronomical Society, 494, 5761
2020
-
[29]
Hassan S., et al., 2022, The Astrophysical Journal, 937, 83
2022
-
[30]
pp 1989–1998
Hoffman J., Tzeng E., Park T., Zhu J.-Y ., Isola P., Saenko K., Efros A., Darrell T., 2018, in International conference on ma- chine learning. pp 1989–1998
2018
-
[31]
Kantorovitch L., 1958, Management science, 5, 1
1958
-
[32]
Li Y ., et al., 2023, The Astrophysical Journal, 954, 139
2023
-
[33]
Liu A., Tegmark M., 2011, Physical Review D—Particles, Fields, Gravitation, and Cosmology, 83, 103006
2011
-
[34]
Liu A., Tegmark M., Bowman J., Hewitt J., Zaldarriaga M., 2009, Monthly Notices of the Royal Astronomical Society, 398, 401
2009
-
[35]
R., Mondal R., Watkinson C
Majumdar S., Pritchard J. R., Mondal R., Watkinson C. A., Bharadwaj S., Mellema G., 2018, Monthly Notices of the Royal Astronomical Society, 476, 4007
2018
-
[36]
maps whose dominant contributions were removed using PCA
built a denoising U-net model to predict the cosmological signal from PCA-reduced maps, i.e. maps whose dominant contributions were removed using PCA. Despite various noise prescriptions adopted when simulat- ing HI maps for a given survey, the mock datasets for train- arXiv:2...
2025 arXiv
-
[37]
L., Lancaster L., Villaescusa-Navarro F., Melchior P., Ho S., Perreault-Levasseur L., Spergel D
Makinen T. L., Lancaster L., Villaescusa-Navarro F., Melchior P., Ho S., Perreault-Levasseur L., Spergel D. N., 2021, Journal of Cosmology and Astroparticle Physics, 2021, 081
2021
-
[38]
McInnes L., Healy J., Melville J., 2018, arXiv preprint arXiv:1802.03426
2018 arXiv
-
[39]
Mellema G., et al., 2013, Experimental Astronomy, 36, 235
2013
-
[40]
Nelson D., et al., 2019, Computational Astrophysics and Cos- mology, 6, 1
2019
-
[41]
Papamakarios G., Pavlakou T., Murray I., 2017, Advances in neural information processing systems, 30
2017
-
[42]
Pourtsidou A., Bacon D., Crittenden R., 2017, Monthly Notices of the Royal Astronomical Society, 470, 4251
2017
-
[43]
F., Karagiannis D., Maartens R., 2024, Physics of the Dark Universe, 45, 101530
Randrianjanahary L. F., Karagiannis D., Maartens R., 2024, Physics of the Dark Universe, 45, 101530
2024
-
[44]
J., 2000, International journal of computer vision, 40, 99
Rubner Y ., Tomasi C., Guibas L. J., 2000, International journal of computer vision, 40, 99
2000
- [45]
-
[46]
Simonyan K., Zisserman A., 2014, arXiv preprint arXiv:1409.1556
2014 arXiv
-
[47]
J., Gretton A., Borgwardt K., 2006, in 13th interna- tional conference, ICONIP
Smola A. J., Gretton A., Borgwardt K., 2006, in 13th interna- tional conference, ICONIP. pp 3–6
2006
-
[48]
The architecture of the encoder considered in our anal- yses
Sugiyama M., Nakajima S., Kashima H., Buenau P., Kawanabe M., 2007, Advances in neural information processing systems, TABLE III. The architecture of the encoder considered in our anal- yses. ”BN + ReLU” denotes a batch normalization followed by a ReLU activation. The output s...
2007
-
[49]
pp 443–450
Sun B., Saenko K., 2016, in Computer Vision–ECCV 2016 Workshops: Amsterdam, The Netherlands, October 8-10 and 15-16, 2016, Proceedings, Part III 14. pp 443–450
2016
-
[50]
pp 7167–7176
Tzeng E., Hoffman J., Saenko K., Darrell T., 2017, in Proceed- ings of the IEEE conference on computer vision and pattern recognition. pp 7167–7176
2017
-
[51]
Villaescusa-Navarro F., et al., 2021, The Astrophysical Journal, 915, 71
2021
-
[52]
Villaescusa-Navarro F., et al., 2022a, arxiv:2201.01300
-
[53]
Villaescusa-Navarro F., et al., 2022b, The Astrophysical Journal Supplement Series, 259, 61
-
[54]
Wang Z., et al., 2021, The Astrophysical Journal, 907, 4
2021
-
[55]
12 TABLE IV
Wold S., Esbensen K., Geladi P., 1987, Chemometrics and in- telligent laboratory systems, 2, 37 Appendix A: Architecture of the encoder To provide more information, we present in the Table III the architecture of the encoder which is in both source and target networks. 12 TABL...
1987
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.