{"id":"2b3a78c6-036c-4591-8fbc-58e25a26ae92","arxiv_id":"1908.04210","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Machine-learning classification of simulated photometric supernovae retains up to 75 percent of cosmological information with SALT2 features and roughly one third with Newling or wavelet features.","lead":"This paper tests how much cosmological information survives when supernovae are classified by machine learning instead of spectroscopy, using simulated data from the Supernova Photometric Classification Challenge. It reports that a SALT2-based classifier preserves about 75 percent of the type Ia supernova information, while model-independent features preserve about one third.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline 75%/33% information-retention and half-sigma constraints are obtained by choosing per-redshift-bin purity thresholds using the true distance moduli of the same test catalogs, so these numbers are in-sample oracle results rather than a validated real-survey selection method.","rationale":"The reader's weakest assumption correctly identifies the most load-bearing concern: the per-redshift-bin purity thresholds are chosen by minimizing the RMSE on the same test catalogs whose cosmological constraints are later reported, using true distance moduli. Eq. (7) defines MSE from Delta mu = mu - mu_fid(z_sim), so the optimization uses ground truth that is unavailable in real surveys. The abstract's two headline numbers, the 75% SALT2 information retention and roughly 33% for Newling/Wavelets, are therefore best-case in-sample values rather than a demonstrated end-to-end method for real photometric data. The paper is otherwise careful and honest: it acknowledges the SALT2 bias, provides explicit TPOT pipelines in Appendix B, uses realistic SNPCC simulations, and the qualitative ranking among feature sets and classifiers is supported by multiple metrics. Those qualitative conclusions may well survive an out-of-sample check; what is not established is the absolute level of information retention and the claim of a robust threshold-selection method. A held-out validation split, as described in the concrete test, would settle whether the oracle selection is the source of the performance. Given this gap, the existing CONDITIONAL verdict remains appropriate; no change is needed beyond what the reader already recommended.","tokens_in":26663,"tokens_out":8979,"duration_ms":99667,"concrete_test":"For each of the 10,000 realizations, split the 20,219-object test set into two disjoint halves: use the first half (with true distance moduli and labels) only to choose the per-redshift-bin purity thresholds that minimize RMSE, freeze them, and evaluate the LambdaCDM/wCDM constraints and C_eff from Eq. (17) on the second half; repeat with roles swapped. If the held-out SALT2 C_eff drops by more than about 10 percentage points below the reported 75% (or Newling/Wavelets below about 33%), the headline numbers are in-sample artifacts rather than expected performance.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5.2 selects, in each redshift bin, the purity threshold that minimizes the RMSE defined in Eq. (7), where Delta mu is computed from the true simulated distance modulus and the fiducial model. The same test catalogs are then used for the cosmological fits (Eq. 15) and for the effective completeness in Table 6 via Eq. (17). This is an oracle selection: enforcing a targeted purity in a bin requires knowing which test objects are actually SNeIa, and minimizing RMSE requires knowing their true distances. Real photometric surveys have neither. The paper does not propose a way to select these thresholds without ground truth, and the abstract presents the resulting numbers as the method's performance. The MB(z) calibration in Appendix A adds a second oracle: it uses the pure SNeIa catalog to measure and remove per-bin distance biases, so the comparison to a perfect classifier is partly by construction. The qualitative ranking of feature sets may survive, but the specific 75%/33% figures and the half-sigma statement are best-case in-sample numbers, not expected out-of-sample performance.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"Using the SNPCC simulated supernova catalog, the authors train supervised classifiers (AdaBoost, Random Forest, Extra-Trees, Gradient Boost, XGBoost, and TPOT) on four feature sets (SALT2 light-curve fits, Newling, Karpenka, and wavelet coefficients), with and without host-galaxy photometric redshift. They compare AUC and Average Precision scores and then perform cosmological fits of Omega_m0 and w on catalogs selected by purity thresholds. They propose a 'binned purity' selection that minimizes the RMSE of distance moduli in each redshift bin. They report that the best ML catalog with SALT2 features retains roughly 75% of the cosmological information of a perfect SNeIa classifier, while Newling and wavelet features retain around 33%, and that all fitted parameters agree within half a sigma of the fiducial model. The best TPOT pipelines are provided in Appendix B.","tokens_in":26810,"tokens_out":4847,"duration_ms":53581,"significance":"If the 75%/33% information-retention figures and the half-sigma agreement describe expected out-of-sample performance, this would be a useful guide for photometric SN surveys such as LSST. The paper's strengths include a systematic comparison of many classifiers and feature sets on a public benchmark, a clear presentation of the bias-variance tradeoff, and explicit reproducible pipelines in Appendix B. However, as detailed in the major comments, the headline numbers are obtained through in-sample optimization on the same test catalogs used to report the final constraints, so they are best interpreted as upper bounds rather than validated on-sky performance. The qualitative ranking of feature sets is likely robust, but the specific percentages in the abstract should not be taken as expected real-survey performance.","major_comments":[{"comment":"The binned-purity threshold is chosen by minimizing the RMSE defined in Eq. (7), with Delta-mu computed from the true simulated distance modulus and the fiducial model, on the same test catalogs that are then used for the cosmological fits (Eq. 15) and for the effective completeness in Table 6 (Eq. 17). This is an oracle selection that requires knowing the true SNeIa types and true distances of the objects whose purity is being optimized. Consequently, the headline results - roughly 75% (SALT2) and 33% (Newling/wavelets) information retention and the 'within half a sigma' agreement - are in-sample, best-case numbers, not expected performance for a real photometric survey. Please demonstrate the selection method on a validation set that is not used for the final constraint (for instance, split the 20,219 test light curves into a threshold-optimization set and a separate cosmology set), or explicitly state in the abstract and conclusions that these figures are upper bounds assuming perfect knowledge of the test truth.","section":"Sec. 5.2, Eq. (7), Eq. (15), Table 6"},{"comment":"The MB(z) calibration uses the pure SNeIa catalog to measure and subtract, per redshift bin, the mean offset between the SALT2 distance moduli and the fiducial mu(z). This removes light-curve fitting bias by construction, and the statement in Appendix A that this procedure 'guarantees that any cosmological bias in the classified catalogs would be a result of the ML classification code' overstates what is shown: the correction also absorbs any distance-estimator bias common to the perfect-classifier and ML catalogs, and it is derived using the true class labels. Please clarify that the comparison is conditional on perfect distance calibration, and discuss whether a per-bin MB(z) correction computed from the pure SNeIa catalog could itself remove part of the classifier-induced bias if the contamination varies with redshift.","section":"Appendix A"},{"comment":"The paper appropriately notes that the Bias-Variance tradeoff assumes the bias cannot be modeled. However, this caveat does not address the fact that the per-bin purity thresholds themselves are selected using the true Delta-mu of the test set. Please add a discussion of how the thresholds would be chosen in practice (for example, using a spectroscopically confirmed subset as a validation set) and state how the reported information retention would change if the thresholds were fixed a priori rather than optimized on the same test catalogs.","section":"Sec. 6, Conclusions"}],"minor_comments":[{"comment":"The sentence 'such as the The Rubin Observatory Legacy Survey of Space and Time' contains a duplicated 'the'; please correct.","section":"Abstract"},{"comment":"The vertical axis label 'Regression Scores' is unclear; consider labeling it 'RMSE components' or define the plotted quantities more explicitly in the caption.","section":"Fig. 3"},{"comment":"The notation C_{ij} is used both for the ensemble-averaged covariance and for a single-catalog quantity Delta-mu_i Delta-mu_j; please clarify the averaging being performed.","section":"Eq. (16)"},{"comment":"The reference 'Möller & Deboissì Ere (2018)' contains a typo; the correct name appears to be 'de Boissière' and the reference entry should be checked for accuracy.","section":"Table 5"},{"comment":"The text uses RMSE and MSE interchangeably; since Eq. (7) defines both, please state explicitly which quantity is minimized in the binned-purity selection.","section":"Sec. 5.2"}],"recommendation":"major_revision","confidential_remarks":"The central claim is interesting but the in-sample oracle optimization is a serious issue for the abstract's quantitative statements. If the authors can re-run the threshold selection on a separate validation set, or clearly reframe the 75%/33% and half-sigma results as upper bounds, the paper could be salvageable. The ranking of feature sets and the comparison of ML methods appear robust, so I would not reject outright. The editor may wish to ask for a revised abstract that does not overstate the performance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid simulation study that extends Lochner et al. with more ML pipelines, the AP metric, and an end-to-end look at cosmological constraints after classification. The genuinely new pieces are the effective-completeness calculation (Table 6) and the binned purity threshold selection. That is worth having.\n\nWhat it does well: the ML comparison is careful—repeated training sets, 5-fold CV, and TPOT pipelines supplied in an appendix. The authors are also honest that SALT2 has an advantage because the simulated lightcurves were generated with SALT2. The qualitative ranking (SALT2 > Newling/Wavelets > Karpenka; boosting > averaging) is robust and consistent with L16. The effective completeness definition is a useful way to translate classification performance into cosmological information.\n\nThe soft spot is the one the stress-test hits: the binned purity thresholds are chosen to minimize the RMSE computed on the very same test catalogs that produce the cosmological constraints (Section 5.2, Figures 10/11). That makes the 75% and 33% numbers in-sample oracle results. In a real survey you do not know true distance moduli, so you cannot select thresholds this way. The MB(z) correction in Appendix A is also an oracle in the same sense—it uses the pure SNeIa catalog to zero out the distance bias, so the comparison to a perfect classifier is partly by construction. The qualitative ranking may survive, but the specific numbers should not be quoted as expected performance without a held-out or independent validation.\n\nThe lack of released code is a minor extra limitation; the TPOT pipelines are listed but not the full classification code, so exact reproducibility is limited. That said, the simulation is public (SNPCC), so a determined reader could reimplement.\n\nOverall: the central method is plausible, the comparison is informative, and the paper is a reasonable addition to the literature. It deserves a serious referee; the main request would be to validate threshold selection on a separate test set or via cross-validation, and to present the oracle numbers alongside non-oracle numbers. Who it is for: anyone working on photometric SN classification for DES/LSST, and people building ML classification pipelines.","headline":"Useful benchmark for photometric SN classification, but the headline 75%/33% information-retention numbers are selected using the test set's true distance moduli, so treat them as best-case in-sample results.","tokens_in":27413,"tokens_out":2644,"would_cite":true,"duration_ms":25233,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Using simulated photometric supernovae, this paper shows that machine-learning classification with per-redshift purity thresholds chosen by a bias-variance tradeoff can keep roughly 75% of the type-Ia distance information and recover the…","keywords":["supernova classification","photometric supernovae","machine learning","bias-variance tradeoff","effective completeness","cosmological constraints","SALT2","SNPCC"],"falsifier":"Re-run the pipeline on the same simulated catalog but with the test set split into a threshold-tuning portion and a held-out portion: choose the per-bin purities from the tuning portion only, then measure cosmology on the held-out portion; if the effective completeness falls well below 75% or the recovered parameters shift by more than half a sigma, the claimed transfer to real surveys fails.","tokens_in":26390,"feed_emoji":"🔭","tokens_out":8443,"duration_ms":80073,"temperature":0.7,"pith_summary":"The paper asks how much cosmological information survives when supernova types are assigned by machine-learning classifiers instead of spectroscopy, since upcoming surveys will observe far more supernovae than they can confirm spectroscopically. Using the simulated SNPCC light-curve catalog, it shows that if one chooses the classification threshold in each redshift bin by minimizing the mean squared error of the distance modulus, the resulting photometrically classified catalogs recover the fiducial cosmology within half a sigma. With SALT2 light-curve features the catalogs retain about 75% of the type-Ia distance information, and with Newling-model or wavelet features about 33%. The result matters because it quantifies how much dark-energy constraining power remains in a purely photometric sample.","feed_headline":"ML-classified supernovae retain up to 75% of cosmology info","feed_subtitle":"With SALT2 light-curve features, photometric catalogs recover dark-energy parameters within half a sigma.","key_machinery":"The machinery is the bias-variance decomposition of the distance-modulus error, MSE = Var[Delta_mu] + $b^{2}$, applied to the choice of classification threshold. A threshold maps to a catalog purity; raising purity usually lowers contamination bias but raises variance as the catalog shrinks, so the paper selects the purity in each redshift bin that minimizes the total MSE rather than maximizing purity alone. The other load-bearing object is the effective completeness, C_eff = (sigma_Ia / $\\sigma$)^2, which converts the ratio of parameter errors between a perfect and a photometric classifier into an effective number of type-Ia supernovae; this is the quantity behind the 75% and 33% information-retention numbers.","core_discovery":"The central claim is that a machine-learning-selected supernova catalog is cosmologically usable without spectroscopic confirmation. In the paper's implementation, each light curve receives a probability of being type Ia; raising the probability threshold raises catalog purity but discards objects and, surprisingly, can increase rather than decrease distance bias because the contaminants are not symmetric in absolute magnitude. The paper therefore proposes choosing, separately in each redshift bin, the purity threshold that minimizes the total MSE (variance plus squared bias) of the SALT2 distance modulus. With this binned-purity selection, all feature sets recover the fiducial Omega_m0 and w within half a sigma, and the effective completeness reaches about 75% for SALT2 features, 30-35% for Newling and wavelet features, and about 20% for Karpenka features.","pith_inferences":["If the same pipeline were run on a data set where the purity thresholds are tuned only on a training subset and then frozen before any cosmology is measured, the 75% figure would likely drop; the paper tunes thresholds on the true distances of the same test catalogs, so the headline number should be read as an upper bound.","The fact that high-purity samples are dominated by type-Ibc contaminants similar in magnitude to type Ia suggests a natural extension: instead of hard cuts, feed the classifier's continuous probabilities into a Bayesian sample-combination estimator, which could recover some of the lost information.","The effective-completeness ratio could serve as a standardized figure of merit for comparing photometric classifiers across surveys, since it is directly tied to cosmological parameter errors rather than to AUC scores.","Because SALT2 was among the models used to generate the simulated SNeIa, a fair test would rerun the comparison on simulations built from independent explosion models; the true retention may sit closer to the 33% figure of the other feature sets."],"forward_implications":["If the claim holds, spectroscopic follow-up is not a prerequisite for competitive dark-energy constraints: a purely photometric catalog can carry most of the type-Ia distance information.","SALT2-style features are worth the extra fitting cost, since they preserve roughly twice as much information as parametric or wavelet alternatives.","Threshold choice should be treated as a statistical decision, not a fixed purity cut; optimizing per redshift bin keeps biases below 0.03 mag in the range 0.4 < z < 1.1.","The sign change of the classification bias at high purity means that pushing purity upward can hurt cosmology; MSE-based selection protects against that.","Even in the pessimistic case that SALT2's advantage is partly an artifact of the simulation, the other feature sets still retain about a third of the information, so photometric classification remains viable."],"supporting_citations":[{"why":"Releases the simulated SNPCC light-curve catalog that all training, testing, and cosmology runs use.","marker":"Kessler et al. (2010b)"},{"why":"Reports the original challenge and its purity-completeness figure of merit, the framework this paper extends.","marker":"Kessler et al. (2010a)"},{"why":"Provides the baseline ML comparison, the wavelet feature pipeline, and the AUC metric that the paper reproduces and expands.","marker":"Lochner et al. (2016)"},{"why":"Defines the SALT2 spectral template used both as the best feature set and as the distance estimator.","marker":"Guy et al. (2007)"},{"why":"Supplies the parametric Newling light-curve model used as one of the feature sets.","marker":"Newling et al. (2011)"},{"why":"Supplies the parametric Karpenka light-curve model used as the remaining feature set.","marker":"Karpenka et al. (2012)"},{"why":"Documents the light-curve fitting biases in magnitude-limited surveys that motivate the per-bin absolute-magnitude calibration in Appendix A.","marker":"Kessler & Scolnic (2017)"},{"why":"Quantifies distance biases from selection and fitting that the paper removes by hand before evaluating classification bias.","marker":"Kessler et al. (2019)"}],"fun_headline_variants":["Supernova ML: 75% cosmology info without spectra","Bias-variance trick boosts photometric supernova cosmology","Photometric SN classification: optimal thresholds yield 75% info","No spectra needed: ML keeps 75% of supernova cosmology"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The result assumes that purity thresholds chosen using the true distances of the same simulated test catalogs will transfer to real surveys where those distances are unknown, and that the per-bin absolute-magnitude calibration applied beforehand does not erase the very bias being measured.","fun_headline_variants_meta":{"raw":{"variants":["Supernova ML: 75% cosmology info without spectra","Bias-variance trick boosts photometric supernova cosmology","Photometric SN classification: optimal thresholds yield 75% info","No spectra needed: ML keeps 75% of supernova cosmology"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00084,"raw_usage":{"total_tokens":3653,"prompt_tokens":933,"completion_tokens":2720,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":549,"completion_tokens_details":{"reasoning_tokens":2650}},"tokens_in":549,"tokens_out":2720,"duration_ms":16720,"temperature":1.0,"reasoning_tokens":2650,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:48:28.846336+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the pipeline on the same simulated catalog but with the test set split into a threshold-tuning portion and a held-out portion: choose the per-bin purities from the tuning portion only, then measure cosmology on the held-out portion; if the effective completeness falls well below 75% or the recovered parameters shift by more than half a sigma, the claimed transfer to real surveys fails.","supporting_citations":[{"cited_title":"Photometric Supernova Classification With Machine Learning","cited_arxiv_id":"1603.00882","evidence_quote":"Provides the baseline ML comparison, the wavelet feature pipeline, and the AUC metric that the paper reproduces and expands."},{"cited_title":"Statistical Classification Techniques for Photometric Supernova Typing","cited_arxiv_id":"1010.1005","evidence_quote":"Supplies the parametric Newling light-curve model used as one of the feature sets."},{"cited_title":"V., Feroz F., Hobson M","cited_arxiv_id":null,"evidence_quote":"Supplies the parametric Karpenka light-curve model used as the remaining feature set."}],"review_version":1}