REVIEW 3 major objections 6 minor 46 references
Data challenges as a tool for time-domain astronomy
T0 review · 3 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read Data challenges are emerging as powerful tools to prepare time-domain astronomy for the large-volume survey era, and PLAsTiCC is the flagship case.
desk verdict A solid, accurate review of time-domain data challenges centered on PLAsTiCC, whose advocacy for challenges as LSST preparation would be stronger if it treated transferability to real data as an open question. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the data challenge itself, defined as a common real or simulated dataset released to the community with a target product or classification task. Within that frame, the mechanism the review emphasizes is the deliberately non-representative train/test split, exemplified by PLAsTiCC: training on 8,000 objects and testing on roughly three million forces classifiers to contend with selection effects similar to those expected for LSST. The evaluation machinery is the class-weighted log-loss, $L_n \equiv -\sum_{m=1}^{M}\tau_{n,m}\ln p(m\mid d_n)$, which rewards a classifier for returning calibrated probabilities across all classes rather than for maximizing purity or efficiency on a single class. Around this core, the review positions deterministic versus probabilistic classifiers, ROC curves and AUC as diagnostics, and Bayesian anomaly detection as the complementary task.
What would settle it
When real LSST alerts with spectroscopic labels become available, check whether the ranking of PLAsTiCC winning classifiers on weighted log-loss matches their leaderboard ranking; any large inversion would show that simulated non-representativity was not a faithful proxy for the survey's selection effects. A simpler pre-registration: run a comparison challenge whose training set is representative rather than skewed and see if it beats the non-representative design on a held-out LSST-like test set.
Extended reading notes
Core claim
The central claim, stated in the abstract and summary, is that data challenges are powerful tools with which to answer fundamental astronomical questions. In the time domain, classification and anomaly detection are the two tasks that benefit most: classification challenges test whether objects can be sorted into known classes, and anomaly detection challenges test whether genuinely new objects can be flagged without labels. The paper argues that PLAsTiCC in particular succeeded as a proof of concept by being non-representative on purpose—the training data contained 8,000 objects while the test data contained closer to three million—and by being open to participants without astronomical domain knowledge. The result was broad participation (1,085 teams) and, the paper contends, methodology development aimed at sparse, imbalanced, and heterogeneous data.
Load-bearing premise
The case stands on the assumption that a deliberately skewed training set—8,000 objects, with roughly three million in the test set—trains classifiers that transfer to the real LSST sky, and that optimizing class-weighted log-loss is the right objective for that transfer.
Editorial extensions
If this is right
- If data challenges like PLAsTiCC work as claimed, LSST-era brokers will enter operations already tested against non-representative training data and highly imbalanced test samples, rather than being debugged on live alerts.
- A metric that rewards calibrated probabilities across all classes should push the community toward classifiers that output meaningful probabilities for rare object types, not just labels for the most common classes.
- Releasing simulated models and truth tables after a challenge ends, as PLAsTiCC and SNPhotCC did, extends the value of the exercise far beyond the official competition period.
- The same challenge structure can be moved into the live-streaming regime, which the paper identifies as essential for fast wide-field surveys that already classify alerts in real time.
- Anomaly detection challenges complement classification by flagging objects that do not fit known classes, helping to decide which rare transients deserve scarce spectroscopic follow-up.
Reading between the lines
- A testable extension the review does not explore: run the same simulation with two tracks—one using PLAsTiCC's non-representative split and one using a representative split—and compare leaderboard rankings on a held-out LSST-like sample; this would directly measure whether induced non-representativity helps or hurts transfer.
- Since weighted log-loss combines discrimination and calibration, future challenges could include reliability diagrams or scaled Brier scores as secondary metrics; the review does not propose these diagnostics.
- The platform choice may matter as much as the data: PLAsTiCC's open, no-domain-knowledge design drew 1,085 teams, so an experiment varying platform accessibility across otherwise identical challenges could test how participation breadth changes solution diversity.
- A natural ensemble, only implicit in the review, is to use a trained classifier to pre-filter known classes and then run anomaly detection on the residual objects, which could boost sensitivity to rare transients in the LSST stream.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This manuscript is a review-style advocacy paper arguing that data challenges, and time-domain challenges in particular, are powerful tools for preparing astronomy for large-volume surveys. It describes the motivations for challenges, gives examples (Galaxy Zoo, GREAT3, strong-lens finding, SNPhotCC, PLAsTiCC), discusses classification versus anomaly detection, and explains evaluation metrics including ROC/AUC, the PLAsTiCC weighted log-loss, and the GREAT3 quality factor. The paper's main concrete focus is PLAsTiCC, including its deliberately non-representative training set, and the final sections look toward future image-based and streaming challenges.
Significance. The paper is useful as a concise survey and clearly identifies a real trend: structured competitions on simulated data are increasingly used to drive methodology development. Its strengths are the accessible descriptions of SNPhotCC and PLAsTiCC, the correct statement of the AUC interpretation, and the honest acknowledgment of non-representativity as a known challenge. The mathematical descriptions are mostly accurate (Eq. 3 matches the GREAT3 source), and the reference list will be useful to newcomers. The significance is moderate: the advocacy claim is plausible but not empirically demonstrated, and the paper does not provide any quantitative evidence that challenge rankings predict performance on real survey data.
major comments (3)
- [Section 3.1 and Section 6] The central advocacy claim that PLAsTiCC-style challenges prepare the community for LSST data is not supported by evidence of transfer to real survey data. Section 2 states that the PLAsTiCC training set is deliberately non-representative (8000 training objects versus roughly three million test objects), and Section 3.1 concedes that supervised learning is highly dependent on how representative the training set is of the test data; yet the paper reports no test of the winning classifiers against spectroscopically confirmed transients or real alert streams. The participation count (1085 teams) and simulation validation in Narayan et al. (2019) establish engagement and internal consistency, not that challenge rankings transfer. I recommend either adding such a validation discussion or explicitly labeling the transferability claim as an open question in Section 6.
- [Section 4.2, Eq. (1)] The formula as written is not the weighted log-loss described in the text. In Eq. (1), tau_{n,m} is defined to be 1 only when n=m, which conflates the object index n with the class index m; the object-level truth should be a class indicator y_{n,m}. In addition, the sentence immediately after the equation says the metric is weighted over classes, but no class weights appear in Eq. (1). Please correct the equation to match Malz et al. (2018) or clearly show where the weights enter.
- [Section 4.1] The validation of the PLAsTiCC simulations is attributed to Narayan et al. (2019), which is cited as 'in prep.' Because this validation is the only evidence offered that the challenge data faithfully represent LSST-like observations, the review should cite the published model paper (Kessler et al. 2019b) or otherwise provide a verifiable reference for this load-bearing step.
minor comments (6)
- [Section 2] There is a duplicated article in 'the the Australian SKA Pathfinder (ASKAP)'; it should read 'the Australian SKA Pathfinder.'
- [Section 2] The sentence about the recent discovery of gravitational-wave sources and their electromagnetic counterparts cites Palaversa (2015), which is a LINEAR light-curve analysis and is not the appropriate reference; the relevant GW170817/AT2017gfo discovery papers should be cited instead.
- [Figure 2 caption] The caption writes 'support vector machine (SVN)', while the legend in the same figure says SVM; this should be '(SVM)'.
- [Section 3.1.1] The phrase 'ROC curves should be computed at a range of different classification thresholds, to accurately compute a classification probability' is imprecise: ROC curves summarize the TPR/FPR trade-off across thresholds but do not compute a classification probability. Consider rephrasing.
- [Section 3.1.1] The notation '0≥Pij≤ 1' should read '0 ≤ P_ij ≤ 1'.
- [References] Author-name formatting is inconsistent across the reference list (e.g., 'Alejandro F. Saez, D. E. H. 2016' versus full author lists elsewhere); please normalize all entries to the journal style.
Circularity Check
No circularity: the review's advocacy claim is descriptive and supported by external examples, not derived from its own inputs.
full rationale
This is a review/advocacy paper, not a derivation. Its central claim that data challenges are powerful tools for time-domain astronomy is supported by examples of past challenges (GREAT, SNPhotCC, Galaxy Zoo, LISA, ASKAP, PLAsTiCC), participation counts, and descriptions of challenge logistics. No equation is derived from an input in a way that would make an output equal to an input by construction. The weighted log-loss metric in Eq. (1) is presented as a design choice of the PLAsTiCC team, citing Malz et al. (2018); it is not used to predict a result from a fitted parameter. The self-citations to PLAsTiCC team papers are descriptive references to a community challenge and are not load-bearing: the paper's claim that challenges are useful does not reduce to those citations. The paper itself acknowledges the key limitation that supervised learning depends on how representative the training set is of the test data (Section 3.1.2), and it does not claim to have demonstrated transferability to real survey data. There is no fitted input called a prediction, no uniqueness theorem imported from the authors, and no ansatz smuggled in via citation. The finding is therefore no significant circularity.
Assumptions & free parameters
assumptions (3)
- domain assumption Simulated data challenges with deliberately non-representative training sets train classifiers that improve performance on future survey data.
- domain assumption Evaluation metrics such as weighted log-loss and AUROC/TPR metrics capture the science goals of the challenge.
- domain assumption LSST will produce of order 10^7 alerts per night and petabytes of data, motivating the need for challenge-driven methods.
Cite this review
Pith. "Pith review of Data challenges as a tool for time-domain astronomy." pith.science (2026). https://pith.science/paper/ZRGP7IJF
@misc{pith2026190810540,
author = {Pith},
title = {Pith review of: Data challenges as a tool for time-domain astronomy},
year = {2026},
howpublished = {\url{https://pith.science/paper/ZRGP7IJF}},
note = {Machine review of arXiv:1908.10540}
}
read the original abstract
Data challenges are emerging as powerful tools with which to answer fundamental astronomical questions. Time-domain astronomy lends itself to data challenges, particularly in the era of classification and anomaly detection. With improved sensitivity of wide-field surveys in optical and radio wavelengths from surveys like the Large Synoptic Survey Telescope (LSST) and the Canadian Hydrogen Intensity Mapping Experiment (CHIME), we are entering the large-volume era of transient astronomy. I highlight some recent time-domain challenges, with particular focus on the Photometric LSST Astronomical Time series Classification Challenge (PLAsTiCC), and describe metrics used to evaluate the performance of those entering data challenges.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
Alejandro F. Saez, D. E. H. 2016, Detecting anomalies in astronomical signals using machine learning algorithms embedded in an FPGA, doi: 10.1117/12.2231491
-
[2]
2017, ArXiv e-prints, arXiv:1702.00786
Amaro-Seoane, P., Audley, H., Babak, S., et al. 2017, ArXiv e-prints, arXiv:1702.00786. https://arxiv.org/abs/1702.00786
arXiv 2017
-
[3]
2014, in The Third Hot-wiring the Transient Universe Workshop, ed
Bellm, E. 2014, in The Third Hot-wiring the Transient Universe Workshop, ed. P. R. Wozniak, M. J. Graham, A. A. Mahabal, & R. Seaman, 27–33. https://arxiv.org/abs/1410.8185
arXiv 2014
-
[4]
2009, MNRAS, 399, 1191, doi: 10.1111/j.1365-2966.2009.15383.x
Cardamone, C., Schawinski, K., Sarzi, M., et al. 2009, MNRAS, 399, 1191, doi: 10.1111/j.1365-2966.2009.15383.x
arXiv 2009
-
[5]
Chambers, K. C., Magnier, E. A., Metcalfe, N., et al. 2016, ArXiv e-prints, arXiv:1612.05560. https://arxiv.org/abs/1612.05560
arXiv 2016
-
[6]
Charnock, T., & Moss, A. 2017, ApJ, 837, L28, doi: 10.3847/2041-8213/aa603d CHIME/FRB Collaboration, Amiri, M., Bandura, K., et al. 2018, ApJ, 863, 48, doi: 10.3847/1538-4357/aad188
-
[7]
Dai, M., Kuhlmann, S., Wang, Y., & Kovacs, E. 2018, MNRAS, 477, 4142, doi: 10.1093/mnras/sty965 D’Andrea, C. B., Smith, M., Sullivan, M., et al. 2018, ArXiv e-prints, arXiv:1811.09565. https://arxiv.org/abs/1811.09565 de Jong, J. T. A., Verdoes Kleijn, G. A., Kuijken, K. H., &
work page Pith review arXiv 2018
-
[8]
Valentijn, E. A. 2013, Experimental Astronomy, 35, 25, doi: 10.1007/s10686-012-9306-1
Show all 46 references
-
[9]
R., Gough, R
DeBoer, D. R., Gough, R. G., Bunton, J. D., et al. 2009, IEEE Proceedings, 97, 1507, doi: 10.1109/JPROC.2009.2016516
2009
-
[10]
J., Djorgovski, S
Drake, A. J., Djorgovski, S. G., Mahabal, A., et al. 2012, in New Horizons in Time Domain Astronomy, ed. E. Griffin, R. Hanisch, & R. Seaman, Vol. 285, 306–308, doi: 10.1017/S1743921312000889
2012 doi
-
[11]
2016, A&A, 593, A5, doi: 10.1051/0004-6361/201628672
Dumusque, X. 2016, A&A, 593, A5, doi: 10.1051/0004-6361/201628672
2016 doi
-
[12]
2017, A&A, 598, A133, doi: 10.1051/0004-6361/201628671
Dumusque, X., Borsa, F., Damasso, M., et al. 2017, A&A, 598, A133, doi: 10.1051/0004-6361/201628671
2017 doi
-
[13]
A., Armstrong, R., et al
Fender, R., Woudt, P. A., Armstrong, R., et al. 2017, ArXiv e-prints, arXiv:1711.04132. https://arxiv.org/abs/1711.04132
2017 arXiv
-
[14]
F., Lawrence, K
Helin, E. F., Lawrence, K. J., Pravdo, S. H., Rabinowitz, D. L., & NEAT Team. 1998, in Bulletin of the American Astronomical Society, Vol. 30, 1036
1998
-
[15]
J., Hand, D
Henrion, M., Mortlock, D. J., Hand, D. J., & Gandy, A. 2013, Classification and Anomaly Detection for Astronomical Survey Data, ed. J. M. Hilbe (New York, NY: Springer New York), 149–184, doi: 10.1007/978-1-4614-3508-2 8
2013 doi
-
[16]
F., Sesar, B., et al
Hernitschek, N., Schlafly, E. F., Sesar, B., et al. 2016, ApJ, 817, 73, doi: 10.3847/0004-637X/817/1/73
2016 doi
-
[17]
M., Whiting, M
Hopkins, A. M., Whiting, M. T., Seymour, N., et al. 2015, Publications of the Astronomical Society of Australia, 32, e037, doi: 10.1017/pasa.2015.37
2015 doi
-
[18]
M., Paech, K., et al
Hoyle, B., Rau, M. M., Paech, K., et al. 2015, MNRAS, 452, 4183, doi: 10.1093/mnras/stv1551
2015 doi
-
[19]
Ishida, E. E. O., & de Souza, R. S. 2013, MNRAS, 430, 509, doi: 10.1093/mnras/sts650
2013 doi
-
[20]
2010a, ArXiv e-prints, arXiv:1001.5210
Kessler, R., Conley, A., Jha, S., & Kuhlmann, S. 2010a, ArXiv e-prints, arXiv:1001.5210. https://arxiv.org/abs/1001.5210
-
[21]
2010b, Publications of the Astronomical Society of the Pacific, 122, 1415, doi: 10.1086/657607
Kessler, R., Bassett, B., Belov, P., et al. 2010b, Publications of the Astronomical Society of the Pacific, 122, 1415, doi: 10.1086/657607
-
[22]
2019a, arXiv e-prints, arXiv:1903.11756
Kessler, R., Narayan, G., Avelino, A., et al. 2019a, arXiv e-prints, arXiv:1903.11756. https://arxiv.org/abs/1903.11756 —. 2019b, Publications of the Astronomical Society of the Pacific, 131, 094501, doi: 10.1088/1538-3873/ab26f1
1903 arXiv
-
[23]
Winter, M. K. 2016, The Astrophysical Journal Supplement Series, 225, 31, doi: 10.3847/0067-0049/225/2/31 11
2016 doi
-
[24]
2017, ArXiv e-prints, arXiv:1709.06257
Mahabal, A., Sheth, K., Gieseke, F., et al. 2017, ArXiv e-prints, arXiv:1709.06257. https://arxiv.org/abs/1709.06257
2017 arXiv
-
[25]
2018, ArXiv e-prints, arXiv:1809.11145
Malz, A., Hloˇ zek, R., Allam, Tarek, J., et al. 2018, ArXiv e-prints, arXiv:1809.11145. https://arxiv.org/abs/1809.11145
2018 arXiv
-
[26]
2014, The Astrophysical Journal Supplement Series, 212, 5, doi: 10.1088/0067-0049/212/1/5
Mandelbaum, R., Rowe, B., Bosch, J., et al. 2014, The Astrophysical Journal Supplement Series, 212, 5, doi: 10.1088/0067-0049/212/1/5
2014 doi
-
[27]
B., Meneghetti, M., Avestruz, C., et al
Metcalf, R. B., Meneghetti, M., Avestruz, C., et al. 2018, ArXiv e-prints, arXiv:1802.03609. https://arxiv.org/abs/1802.03609
2018 arXiv
-
[28]
2016, MNRAS, 455, 626, doi: 10.1093/mnras/stv2333 M¨ oller, A., Ruhlmann-Kleider, V., Leloup, C., et al
Parley, N. 2016, MNRAS, 455, 626, doi: 10.1093/mnras/stv2333 M¨ oller, A., Ruhlmann-Kleider, V., Leloup, C., et al. 2016, Journal of Cosmology and Astro-Particle Physics, 2016, 008, doi: 10.1088/1475-7516/2016/12/008
2016 doi
-
[29]
D., et al
Narayan, G., Zaidi, T., Soraisam, M. D., et al. 2018, The Astrophysical Journal Supplement Series, 236, 9, doi: 10.3847/1538-4365/aab781
2018 doi
-
[30]
2019, in prep
Collaboration, LSST Transients, T., & Variable Stars Science Collaboration. 2019, in prep
2019
-
[32]
2014, ApJ, 793, 23, doi: 10.1088/0004-637X/793/1/23
Nun, I., Pichara, K., Protopapas, P., & Kim, D.-W. 2014, ApJ, 793, 23, doi: 10.1088/0004-637X/793/1/23
2014 doi
-
[33]
2015, ArXiv e-prints, arXiv:1505.02082
Palaversa, L. 2015, ArXiv e-prints, arXiv:1505.02082. https://arxiv.org/abs/1505.02082
2015 arXiv
-
[34]
2007, AJ, 134, 1285, doi: 10.1086/520956
Poznanski, D., Maoz, D., & Gal-Yam, A. 2007, AJ, 134, 1285, doi: 10.1086/520956
2007 doi
-
[35]
A., Trotta, R., & van Dyk, D
Revsbech, E. A., Trotta, R., & van Dyk, D. A. 2018, MNRAS, 473, 3969, doi: 10.1093/mnras/stx2570
2018 doi
-
[36]
M., & Poznanski, D
Schafer, C. M., & Poznanski, D. 2012, MNRAS, 419, 1121, doi: 10.1111/j.1365-2966.2011.19768.x
2012
-
[37]
R., Winn, J
Ricker, G. R., Winn, J. N., Vanderspek, R., et al. 2014, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 9143, Space Telescopes and Instrumentation 2014: Optical, Infrared, and Millimeter Wave, 914320, doi: 10.1117/12.2063489
2014 doi
-
[38]
A., & Lochner, M
Roberts, E., Bassett, B. A., & Lochner, M. 2019, arXiv e-prints, arXiv:1902.08627. https://arxiv.org/abs/1902.08627
2019 arXiv
-
[39]
Rowe, B. T. P., Jarvis, M., Mandelbaum, R., et al. 2015a, Astronomy and Computing, 10, 121, doi: 10.1016/j.ascom.2015.02.002 —. 2015b, Astronomy and Computing, 10, 121, doi: 10.1016/j.ascom.2015.02.002
2015 doi
-
[40]
2016, The Astronomical Journal, 153, doi: 10.3847/1538-3881/aa661b Square Kilometre Array Cosmology Science Working
Sesar, B., Hernitschek, N., Mitrovi, S., et al. 2016, The Astronomical Journal, 153, doi: 10.3847/1538-3881/aa661b Square Kilometre Array Cosmology Science Working
2016 doi
-
[41]
J., Battye, R
Group, Bacon, D. J., Battye, R. A., et al. 2018, ArXiv e-prints, arXiv:1811.02743. https://arxiv.org/abs/1811.02743 The PLAsTiCC team, Allam, Tarek, J., Bahmanyar, A., et al. 2018, ArXiv e-prints, arXiv:1810.00001. https://arxiv.org/abs/1810.00001
2018 arXiv
-
[42]
Tyson, J. A. 2002, in Survey and Other Telescope Technologies and Discoveries, ed. J. A. Tyson & S. Wolff, Vol. 4836, 10–20, doi: 10.1117/12.456772
2002 doi
-
[43]
Bassett, B. A. 2015, MNRAS, 453, 2848, doi: 10.1093/mnras/stv1816
2015 doi
-
[44]
W., Lintott, C
Willett, K. W., Lintott, C. J., Bamford, S. P., et al. 2013, MNRAS, 435, 2835, doi: 10.1093/mnras/stt1458
2013 doi
-
[45]
W., Galloway, M
Willett, K. W., Galloway, M. A., Bamford, S. P., et al. 2017, MNRAS, 464, 4176, doi: 10.1093/mnras/stw2568
2017 doi
-
[46]
2010, doi: 10.1184/R1/6475475.v1
Xiong, L., Poczos, B., Connolly, A., & Schneider, J. 2010, doi: 10.1184/R1/6475475.v1
2010 doi
-
[47]
2018, ArXiv e-prints, arXiv:1811.08055
Zhang, C., Song, D., Chen, Y., et al. 2018, ArXiv e-prints, arXiv:1811.08055. https://arxiv.org/abs/1811.08055
2018 arXiv
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.