REVIEW 2 major objections 4 minor 52 references
How to set up your first machine learning project in astronomy
T0 review · 2 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read Machine learning astronomy projects fail less when they start from a specific objective, trivial baselines, uncertainty handling, and a test harness.
desk verdict Useful best-practices guide for astronomy ML, but a wrong definition of precision and a purity-based model-selection example undermine its own reliability and need correction. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the test harness together with the baseline ladder. A test harness is an external validation setup, such as simulated data, hand-crafted inputs like a two-dimensional Gaussian on noise, or a small independent ground-truth sample, that the pipeline must pass before being trusted on real data. The baseline ladder runs from the dummy predictor that always returns the most common class, through the historical method the project aims to replace, to simple standard models with default settings. The paper uses these two devices to turn machine learning from open-ended tinkering into a measurable engineering process: the baselines define what counts as improvement, and the test harness defines what counts as understanding. Calibrated uncertainty output, judged by the distribution of true values on the cumulative predicted distribution, is the third pillar that makes the results usable in downstream science.
What would settle it
A systematic audit of published astronomy ML projects would falsify the paper's causal claim if it found no association between using a test harness, baselines, and uncertainty calibration and later independent verification or adoption of the model; the check would be whether models from workflow-following projects are more often confirmed by new observations.
Extended reading notes
Core claim
The paper's central claim is that the success of a machine-learning project in astronomy is largely decided before any model is trained, by the clarity of the objective and the honesty of the validation setup. It is a recommendation document, not a new algorithm. The authors contend that an objective should be specific enough to determine the required true-positive and false-positive rates, that trivial baselines must be reported beside any sophisticated method, that the full pipeline should be exercised on simulated or independently known data before real application, and that a scientific model is incomplete unless it produces uncertainties that have been checked for calibration. The intended result is a literature in which successes, failures, and equally-good variants are reported, so the community can learn which methods suit which problems.
Load-bearing premise
The workflow assumes that a high-quality simulation or independent ground-truth data set exists to serve as the test harness; for rare or poorly understood phenomena, no such external check may be available, and the validation step cannot be fully applied.
Editorial extensions
If this is right
- Projects with an explicit finish line can stop when the target is reached instead of tuning indefinitely.
- Every model comparison becomes more meaningful when dummy and historical baselines appear in the same plot.
- Calibrated per-object uncertainties let downstream Bayesian analyses ingest ML predictions as probability distributions rather than point estimates.
- Reporting failed variations and equally-good hyperparameters would save the community from repeating the same searches.
- Covariate-shift corrections, such as weighting training samples by their probability of belonging to the training set, are needed whenever labeled sources are brighter or nearer than the application sample.
Reading between the lines
- The test-harness step could be operationalized as a mandatory submission check for astronomy ML papers: a rank-histogram plot and baseline table would let reviewers see calibration at a glance.
- The same workflow transfers to other sciences with expensive labels, where the labeled sample is systematically different from the unlabeled population.
- A quantitative prediction of the paper is that teams following the workflow should have their models independently verified or adopted more often; this could be tested by auditing whether published models are later checked against new observations.
- Framing the test harness as an engineering gate suggests a sharper criterion than 'the model works': a model should not be deployed until it passes adversarial hybrid inputs, such as two galaxy images summed and still classified as a galaxy.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper is a perspective/recommendation document for astronomers and data scientists starting machine learning projects. It argues that ML projects should begin with a specific, measurable, and useful objective; establish dummy, historical, and initial baselines; create a test harness with simulated or independent ground-truth data; ingest and produce uncertainties; address covariate shift; and report ablation studies and failed trials. It also includes a dictionary that maps ML terms to astronomy terms, discusses non-technical aspects of the work environment, and provides workflow diagrams. The paper contains no new empirical results or derivations; its claims are heuristic and grounded in external references and well-known ML methodology.
Significance. If followed, the proposed workflow could reduce common sources of error and frustration in astronomy ML projects, particularly through its emphasis on explicit objectives, baselines, uncertainty handling, and out-of-distribution validation. The paper is clearly written, well-structured, and uses concrete examples (e.g., the eFEDS covariate-shift illustration) to make its points accessible. Its strengths include an unusually explicit glossary, a strong recommendation to report failures as well as successes, and practical advice on K-fold cross-validation and ablation studies. However, a formal error in the definition of precision—repeated in the dictionary and in two key sections—undermines the paper's central model-selection recommendation and must be corrected before the paper can serve as a reliable reference.
major comments (2)
- [Dictionary; §2.1; §2.3.1] The Dictionary defines precision as '1 - false positive rate'. This is incorrect: precision = TP/(TP+FP), whereas 1 - FPR = TN/(FP+TN), which is specificity (true negative rate). The same conflation appears in §2.1, where the paper suggests that reporting 'true positive rate (completeness) and false positive rate (purity)' allows the user to assess the objective, and in §2.3.1, where calibrating both classifiers to a 1% false positive rate is described as encoding 'pure samples'. Purity is precision, not 1 - FPR. Under class imbalance, a fixed FPR can correspond to very different precision values, so an FPR-calibrated comparison does not generally select the classifier that yields the purest sample. This is load-bearing because the paper's recommended model-selection criterion (calibrate to a desired FPR then compare TPR) is presented as equivalent to choosing a pure sample. Please correct the glossary definition and revise the relevant passages to use precision (or explicitly discuss the connection between FPR and precision under the assumed operating point).
- [§2.4 (Create a test harness)] The test-harness recommendation assumes that high-quality simulations or independent ground-truth datasets are available for the problem under study. The paper does not address the common case where no such external validation exists, such as classification of rare transient phenomena whose physical classes are not yet known, or problems where simulations are unavailable or not faithful. Without an alternative strategy (e.g., internal consistency checks, researcher-declared diagnostic experiments, or semi-supervised agreement metrics), the universal claim that a test harness 'should' be created may not be actionable for a substantial fraction of astronomy projects. I recommend adding a brief discussion of fallback validation strategies when no simulation or independent ground truth is available.
minor comments (4)
- [§5.3] The sentence 'Similarly, in classification balanced training samples or imbalance corrections are needed' is grammatically incomplete; it appears to intend 'in classification, either balanced training samples or imbalance corrections are needed'.
- [Dictionary] The entry for 'unbiased (regression)' contains a typo: 'on on the training sample' should read 'on the training sample'.
- [§3.3 (Unsupervised learning)] The text reads 'also known aspareidolia' with a missing space; it should be 'also known as pareidolia'.
- [Figure 1 caption] The caption states that 'Any confusion matrix corresponds to a position on the receiver operator curve (ROC)'; this is true only for a confusion matrix at a specified threshold, and the phrasing could be tightened to avoid implying that every confusion matrix lies on the ROC curve.
Circularity Check
No circularity: the paper is a guidelines document with no derivation chain whose conclusions reduce to its inputs.
full rationale
This paper is a recommendations-style guide for setting up machine learning projects in astronomy. It contains no fitted parameters, no predictive model, and no derivation that could reduce to its own inputs. The central claims—that projects should have measurable objectives, baselines, uncertainty handling, and test-harness validation—are supported by external references to general ML practice, methodology literature, and standard statistical concepts. Self-references, such as 'unsupervised learning (Fotopoulou, sub.)' and '(Singhal, Fotopoulou, et al., in prep)', are mentions of ongoing or related work, not load-bearing evidence for any derived result. The only notable issue is a definitional error in the Dictionary ('precision: 1 - false positive rate'), which is a correctness concern, not a circularity concern: it does not make any conclusion equivalent to its premises. No step in the paper is self-definitional, no prediction is fitted and then reported as validated, and no uniqueness claim is imported from the authors' prior work. The circularity burden is therefore zero.
Assumptions & free parameters
assumptions (2)
- domain assumption Standard machine learning evaluation practices, including train/test splits, K-fold cross-validation, and dummy baselines, are applicable to astronomy datasets without modification.
- domain assumption The target audience has sufficient ML background to implement the recommendations, as stated in Section 1 ('It is assumed you are familiar with ML fundamentals').
Cite this review
Pith. "Pith review of How to set up your first machine learning project in astronomy." pith.science (2026). https://pith.science/paper/YHSEA4W7
@misc{pith2026250208222,
author = {Pith},
title = {Pith review of: How to set up your first machine learning project in astronomy},
year = {2026},
howpublished = {\url{https://pith.science/paper/YHSEA4W7}},
note = {Machine review of arXiv:2502.08222}
}
read the original abstract
Large, freely available, well-maintained data sets have made astronomy a popular playground for machine learning projects. Nevertheless, robust insights gained to both machine learning and physics could be improved by clarity in problem definition and establishing workflows that critically verify, characterize and calibrate machine learning models. We provide a collection of guidelines to setting up machine learning projects to make them likely useful for science, less frustrating and time-intensive for the scientist and their computers, and more likely to lead to robust insights. We draw examples and experience from astronomy, but the advice is potentially applicable to other areas in science.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
Mandt, S., Hoffman, M. D. & Blei, D. M. Stochastic Gradient Descent as Approximate Bayesian Inference. arXiv e-prints arXiv:1704.04289, DOI: 10.48550/arXiv.1704.04289 (2017). 1704.04289
-
[2]
The crisp-dm model: the new blueprint for data mining
Shearer, C. The crisp-dm model: the new blueprint for data mining. J. data warehousing 5, 13–22 (2000)
work page 2000
-
[3]
Saltz, J. S. The need for new processes, methodologies and tools to support big data teams and improve big data project effectiveness. In 2015 IEEE International Conference on Big Data (Big Data), 2066–2071 (IEEE, 2015)
work page 2015
-
[4]
Data Science Methodologies: Current Challenges and Future Approaches
Martinez, I., Viles, E. & Olaizola, I. G. Data Science Methodologies: Current Challenges and Future Approaches. arXiv e-prints arXiv:2106.07287, DOI: 10.48550/arXiv.2106.07287 (2021). 2106.07287
work page Pith review arXiv doi:10.48550/arxiv.2106.07287 2021
-
[5]
Artrith, N. et al. Best practices in machine learning for chemistry. Nat. Chem. 13, 505–508, DOI: 10.1038/ s41557-021-00716-z (2021)
work page 2021
-
[6]
Garofalo, M., Botta, A. & Ventre, G. Astrophysics and big data: Challenges, methods, and tools. Proc. Int. Astron. Union 12, 345–348 (2016). 17/20
work page 2016
-
[7]
Zhang, Y . & Zhao, Y . Astronomy in the Big Data Era.Data Sci. J. 14, 11, DOI: 10.5334/dsj-2015-011 (2015)
-
[8]
Deep Machine Learning in Cosmology: Evolution or Revolution?
Lahav, O. Deep Machine Learning in Cosmology: Evolution or Revolution? arXiv e-prints arXiv:2302.04324, DOI: 10.48550/arXiv.2302.04324 (2023). 2302.04324
work page Pith review arXiv doi:10.48550/arxiv.2302.04324 2023
Show all 52 references
- [9]
- [10]
-
[11]
Fluke, C. J. & Jacobs, C. Surveying the reach and maturity of machine learning and artificial intelligence in astronomy. WIREs Data Min. Knowl. Discov. 10, e1349, DOI: 10.1002/widm.1349 (2020). 1912.02934
2020 arXiv
-
[12]
J., VanderPlas, J
Ivezi´c, Ž., Connolly, A. J., VanderPlas, J. T. & Gray, A. Statistics, Data Mining, and Machine Learning in Astronomy: A Practical Python Guide for the Analysis of Survey Data, Updated Edition (2019)
2019
- [13]
-
[14]
Mastering Machine Learning with scikit-learn (Packt, 2017)
Hackeling, G. Mastering Machine Learning with scikit-learn (Packt, 2017)
2017
-
[15]
G., Mahabal, A
Graham, M., Drake, A., Djorgovski, S. G., Mahabal, A. & Donalek, C. Challenges in the automated classification of variable stars in large databases. In European Physical Journal Web of Conferences, vol. 152 of European Physical Journal Web of Conferences, 03001, DOI: 10.1051/e...
2017
-
[16]
Yang, H. et al. Data mining techniques on astronomical spectra data - II. Classification analysis. MNRAS 518, 5904–5928, DOI: 10.1093/mnras/stac3292 (2023). 2212.09286
2023 arXiv
-
[17]
Yang, H. et al. Data mining techniques on astronomical spectra data - I. Clustering analysis. MNRAS 517, 5496–5523, DOI: 10.1093/mnras/stac2975 (2022). 2212.08419
2022 arXiv
-
[18]
Active learning literature survey
Settles, B. Active learning literature survey. Computer Sciences Technical Report 1648, University of Wisconsin– Madison (2009)
2009
-
[19]
& Bassett, B
Lochner, M. & Bassett, B. A. ASTRONOMALY: Personalised active anomaly detection in astronomical data. Astron. Comput. 36, 100481, DOI: 10.1016/j.ascom.2021.100481 (2021). 2010.11202
2021
-
[20]
The mnist database of handwritten digit images for machine learning research
Deng, L. The mnist database of handwritten digit images for machine learning research. IEEE Signal Process. Mag. 29, 141–142 (2012)
2012
-
[21]
Doran, G. T. et al. There’sa smart way to write management’s goals and objectives. Manag. review 70, 35–36 (1981)
1981
-
[22]
Bausell, R. B. & Li, Y .-F. Power analysis for experimental research: a practical guide for the biological, medical and social sciences (Cambridge University Press, 2002)
2002
-
[23]
operational definition, DOI: 10.1093/oi/authority.20110803100251503
-
[24]
Spectra of Supernovae
Minkowski, R. Spectra of Supernovae. PASP 53, 224, DOI: 10.1086/125315 (1941)
1941 doi
-
[25]
Thornton, C., Hutter, F., Hoos, H. H. & Leyton-Brown, K. Auto-weka: Combined selection and hyperparameter optimization of classification algorithms. In Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’13, 847–855, DOI: 10...
- [26]
-
[27]
Lieu, M. et al. Deep Learning of Astronomical Features with Big Data. In Teuben, P. J., Pound, M. W., Thomas, B. A. & Warner, E. M. (eds.)Astronomical Data Analysis Software and Systems XXVII, vol. 523 of Astronomical Society of the Pacific Conference Series, 49 (2019). 18/20
2019
-
[28]
Interpretable Machine Learning (2022), 2 edn
Molnar, C. Interpretable Machine Learning (2022), 2 edn
2022
-
[29]
Rudin, C. et al. Interpretable machine learning: Fundamental principles and 10 grand challenges. Stat. Surv. 16, 1–85 (2022)
2022
-
[30]
& Galstyan, A
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K. & Galstyan, A. A survey on bias and fairness in machine learning. ACM computing surveys (CSUR) 54, 1–35 (2021)
2021
-
[31]
& Salakhutdinov, R
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I. & Salakhutdinov, R. Dropout: A simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15, 1929–1958 (2014)
2014
-
[32]
Hamill, T. M. Interpretation of Rank Histograms for Verifying Ensemble Forecasts. Mon. Weather. Rev. 129, 550, DOI: 10.1175/1520-0493(2001)129<0550:IORHFV>2.0.CO;2 (2001)
2001 doi
-
[33]
Ghosh, A. et al. GaMPEN: A Machine-learning Framework for Estimating Bayesian Posteriors of Galaxy Morphological Parameters. ApJ 935, 138, DOI: 10.3847/1538-4357/ac7f9e (2022). 2207.05107
2022 arXiv
-
[34]
Salvato, M. et al. The eROSITA Final Equatorial-Depth Survey (eFEDS). Identification and characterization of the counterparts to point-like sources. Astron. Astrophys. 661, A3, DOI: 10.1051/0004-6361/202141631 (2022). 2106.14520
2022 arXiv
-
[35]
Rosenbaum, P. R. & Rubin, D. B. Reducing bias in observational studies using subclassification on the propensity score. J. Am. statistical Assoc. 79, 516–524 (1984)
1984
-
[36]
A., Trotta, R
Revsbech, E. A., Trotta, R. & van Dyk, D. A. STACCATO: a novel solution to supernova photometric classification with biased training sets. MNRAS 473, 3969–3986, DOI: 10.1093/mnras/stx2570 (2018). 1706. 03811
2018 doi
-
[37]
Domain-adversarial training of neural networks
Ganin, Y .et al. Domain-adversarial training of neural networks. The journal machine learning research 17, 2096–2030 (2016)
2016
-
[38]
Perdue, G. N. et al. Reducing model bias in a deep learning classifier using domain adversarial neural networks in the MINERvA experiment. J. Instrumentation 13, P11020, DOI: 10.1088/1748-0221/13/11/P11020 (2018). 1808.08332
2018 arXiv
-
[39]
& Stitelman, O
Kaufman, S., Rosset, S., Perlich, C. & Stitelman, O. Leakage in data mining: Formulation, detection, and avoidance. ACM Trans. Knowl. Discov. Data6, DOI: 10.1145/2382577.2382579 (2012)
2012
-
[40]
& Narayanan, A
Kapoor, S. & Narayanan, A. Leakage and the reproducibility crisis in machine-learning-based science. Patterns DOI: 10.1016/j.patter.2023.100804 (2023). Publisher: Elsevier
2023
-
[41]
The logic of scientific discovery (Routledge, 2005)
Popper, K. The logic of scientific discovery (Routledge, 2005)
2005
-
[42]
Hopkins, P. F. A new class of accurate, mesh-free hydrodynamic simulation methods. MNRAS 450, 53–110, DOI: 10.1093/mnras/stv195 (2015). 1409.7395
2015 arXiv
-
[43]
Carleo, G. et al. Machine learning and the physical sciences. Rev. Mod. Phys. 91, 045002, DOI: 10.1103/ RevModPhys.91.045002 (2019). 1903.10563
2019 arXiv
-
[44]
& Tang, K
Zhang, Y ., Tiˇno, P., Leonardis, A. & Tang, K. A survey on neural network interpretability. IEEE Transactions on Emerg. Top. Comput. Intell.5, 726–742, DOI: 10.1109/TETCI.2021.3100641 (2021)
2021
-
[45]
& Wang, G
Fan, F.-L., Xiong, J., Li, M. & Wang, G. On interpretability of artificial neural networks: A survey. IEEE Transactions on Radiat. Plasma Med. Sci. 5, 741–760, DOI: 10.1109/TRPMS.2021.3066428 (2021)
2021
-
[46]
J., Shlens, J
Goodfellow, I. J., Shlens, J. & Szegedy, C. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572 (2014)
2014 arXiv
-
[47]
Wilkinson, M. D. et al. The FAIR Guiding Principles for scientific data management and stewardship. Sci. Data 3, 160018, DOI: 10.1038/sdata.2016.18 (2016)
2016 doi
-
[48]
Chamba, N., Knapen, J. H. & Black, D. How to plan your astronomy research paper in ten steps. Nat. Astron. 6, 1015–1020, DOI: 10.1038/s41550-022-01757-1 (2022). 2207.12959. 19/20
2022 arXiv
-
[49]
H., Chamba, N
Knapen, J. H., Chamba, N. & Black, D. How to write and develop your astronomy research paper. Nat. Astron. 6, 1021–1026, DOI: 10.1038/s41550-022-01759-z (2022). 2110.05503
2022 arXiv
-
[50]
Writing a Scientific Paper I
Sterken, C. Writing a Scientific Paper I. The writing process. In Sterken, C. (ed.) EAS Publications Series, vol. 50 of EAS Publications Series, 1–63, DOI: 10.1051/eas/1150001 (2011)
2011
-
[51]
Writing a Scientific Paper II
Sterken, C. Writing a Scientific Paper II. Communication by Graphics. In Sterken, C. (ed.) EAS Publications Series, vol. 50 of EAS Publications Series, 65–170, DOI: 10.1051/eas/1150002 (2011)
2011
-
[52]
Writing a Scientific Paper III
Sterken, C. Writing a Scientific Paper III. Ethical Aspects. In Sterken, C. (ed.) EAS Publications Series, vol. 50 of EAS Publications Series, 173–282, DOI: 10.1051/eas/1150003 (2011). Acknowledgements The thoughts laid out here have been heavily influenced by conversations wi...
2011
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.