{"id":"f08f88a3-7580-477f-bd38-df71e243527b","arxiv_id":"2607.05393","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":7,"one_line_summary":"A dual-network asymmetric co-teaching framework trained on injected transients and contaminated survey data achieves human-label-free real-bogus classification with calibrated uncertainty.","lead":"The paper trains a real-bogus transient classifier without human labels by pairing simulated supernova injections with contaminated survey detections, using an asymmetric co-teaching method robust to label noise. A smart generalist might read it because the approach could let upcoming sky surveys like LSST auto-filter false detections without costly manual labeling.","discovery_kind":"unclear","skeptic_critique":{"model":"glm-5.2","headline":"UQ superiority and 99.35% accuracy claims rest on a small, curated evaluation set (4,820 sources / 306 objects) with no error bars or significance testing on the reported metrics.","rationale":"The reader's CONDITIONAL verdict with MODERATE confidence is appropriate. The concern I raise — small curated evaluation set with no significance testing — is already noted in the reader's rationale ('evaluation is limited by a small, highly curated evaluation set... UQ comparisons lack statistical significance testing'). The reader chose the training-side assumption as the weakest link, but the evaluation-side concern is more load-bearing because it directly undermines the evidence for the UQ superiority claim, which is a headline result. However, this does not change the verdict: CONDITIONAL is the right call. The methodology is sound in principle, the training-side robustness is tested, and the paper is honest about limitations. The missing piece is statistical rigor on the evaluation side, which is a fixable gap rather than a fundamental flaw. Moving to REJECT would be unjustified — the method is well-motivated and the results are promising. Moving to ACCEPT would require the bootstrap analysis I propose. CONDITIONAL with a note that the UQ superiority claim needs statistical validation is the correct posture. The paper has genuine independent support: reproducible code (ML4transients library, Zenodo deposit with interactive visualizations), a physically motivated injection pipeline integrated with LSST Science Pipelines, and a clear experimental design for contamination testing. These are real strengths that warrant a conditional rather than negative verdict.","tokens_in":26257,"tokens_out":2781,"duration_ms":74934,"concrete_test":"Bootstrap-resample the 4,820-source evaluation set (e.g., 2000 iterations with replacement) and compute 95% confidence intervals for the pairwise differences in NLL, Brier score, and ECE between 'Our method' (Ensemble-MC-Dropout for Co-teaching) and Deep Ensemble. If the 95% CI for any metric difference includes zero, the claim of superior calibration is not statistically supported at the current sample size. Additionally, evaluate on a less-curated subset (e.g., relaxing the ≥6-night and flux-ratio cuts in Table 1) to test whether UQ advantages persist on harder, more representative cases.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that the hybrid UQ approach achieves 'competitive calibration relative to more expensive ensemble baselines' is supported by differences on a single evaluation set of 4,820 source detections (Table 6). The margins are small: NLL 0.292 vs 0.314 (Deep Ensemble), Brier 0.063 vs 0.065, ECE 0.0375 vs 0.0535. No confidence intervals, bootstrap analysis, or significance tests are reported. On a curated set of this size, these differences could easily fall within sampling noise. The object-level accuracy of 99.35% (304/306) is even more fragile — two misclassifications separate near-perfect from merely good. Furthermore, the evaluation set is explicitly described as 'a high-purity evaluation subset rather than a representative sample of the full alert stream' (Section 2.3), with aggressive filtering (SNR>5, ≥6 nights, host-galaxy cuts, flux-ratio cuts) that removes exactly the ambiguous cases where UQ matters most. So the UQ method is evaluated on the easiest cases and claimed to be superior without uncertainty on the superiority itself. The reader identified this concern in the rationale but prioritized the training-side assumption (bogus-dominated survey data) as the weakest link. That training assumption is real but explicitly tested up to 35% contamination and acknowledged as a limitation. The evaluation-side concern is more load-bearing because it undermines the evidence base for all three pillars of the central claim: classification performance, robustness, and UQ competitiveness.","agreement_with_reader":"partial"},"referee_report":{"model":"glm-5.2","summary":"This paper presents a human-label-free approach to Real-Bogus classification for transient candidates, using injected supernova-like sources as a clean positive class and raw survey detections as a noisy negative class. The authors introduce Asym-Co-teaching, a class-dependent extension of co-teaching, to handle the asymmetric label noise inherent in this setup. They evaluate robustness under controlled contamination (up to 35%), compare uncertainty quantification methods (MC dropout, deep ensembles, repulsive ensembles, and a proposed hybrid), and provide latent-space visualizations via UMAP. The method is evaluated on a manually curated set of 306 objects (4,820 source detections) from HSC-UDEEP data, and extended to object-level light-curve classification.","tokens_in":26474,"tokens_out":2274,"duration_ms":119868,"significance":"The human-label-free training paradigm is a genuine and timely contribution for the LSST era, where labeled training data is a bottleneck. The asymmetric co-teaching adaptation is well-motivated by the class-dependent noise structure of the problem. The controlled noise experiments (Table 5) provide useful evidence of robustness. The authors provide reproducible code and interactive visualizations on Zenodo, which is commendable. The hybrid UQ strategy (Section 5.5) that reuses the co-teaching dual-network pair is a practical and computationally efficient idea. The latent-space analysis (Section 6, Fig. 4) provides qualitative insight into model behavior and bogus substructure.","major_comments":[{"comment":"§7.2, Table 6: The UQ comparison reports small margins between methods (e.g., NLL 0.292 vs 0.314 for Deep Ensemble; Brier 0.063 vs 0.065; ECE 0.0375 vs 0.0535) on a single evaluation set of 4,820 sources, with no confidence intervals, bootstrap analysis, or significance tests. The text in §7.2 states the method 'consistently outperforms' ensemble baselines, but without uncertainty estimates on the metrics themselves, this claim is not statistically supported. This is load-bearing for the UQ superiority claim and should be addressed with bootstrap confidence intervals or similar.","section":null},{"comment":"§2.3, Table 1: The evaluation set is constructed with aggressive filtering (SNR>5, ≥6 nights, host-galaxy cuts, flux-ratio >1.4) that explicitly removes ambiguous cases. The paper acknowledges this defines 'a high-purity evaluation subset rather than a representative sample of the full alert stream,' but the 99.35% object-level accuracy reported in §7.4 and the UQ calibration results in Table 6 are all measured on this easy subset. Since the paper claims the method is 'suited for transfer to forthcoming surveys' (Abstract, §9), the absence of any evaluation on harder or more representative cases weakens the generalizability evidence. At minimum, the authors should explicitly scope their performance and UQ claims to high-purity SN-like events and discuss how the evaluation bias might affect the relative UQ comparison.","section":null},{"comment":"§4.2, Appendix C: The forget rates for Asym-Co-teaching are set using prior knowledge of the noise levels (e.g., (0.05, 0.01) for the baseline). In the controlled-noise experiments (Table 5), this is reasonable since the true contamination is known. However, for the baseline dataset representing real survey data, the 5% noise estimate for the survey class is an assumption whose sensitivity is not tested. The paper acknowledges this in §8 ('noise-rate estimation is an interesting direction for future work'), but the practical applicability of the method to a new survey depends on choosing appropriate forget rates without ground truth. A brief sensitivity analysis showing how performance degrades if the forget rate is misspecified would strengthen the transfer claim.","section":null}],"minor_comments":[{"comment":"§3.3: The Bayesian optimization is performed on a 30% subset of the data. It would be useful to state whether the selected hyperparameters (Table 3) were validated on the full dataset or held-out data.","section":null},{"comment":"Table 5: The 'B' column header is ambiguous — it presumably means 'Bogus specificity' but is not explicitly defined in the caption.","section":null},{"comment":"§7.4: The 99.35% accuracy (304/306) is reported without noting that it corresponds to only 2 misclassifications. This should be stated explicitly to give readers proper context on the statistical fragility.","section":null},{"comment":"Fig. 4 caption: The cutouts are described as 'approximate with respect to exact UMAP locations.' This is fine for illustration but should be noted more prominently in the figure itself, not just the caption.","section":null},{"comment":"§5.1, Eq. (6): The Spearman correlation formula assumes no tied ranks. With discrete or binned data, ties are likely; the standard tie-corrected formula should be used or the assumption noted.","section":null},{"comment":"Appendix C: The forget-rate values for Asym-Co-teaching use (forget_rate_0, forget_rate_1) but it is not immediately clear which rate corresponds to which class. This should be stated explicitly.","section":null},{"comment":"§2.2: The injection magnitude prior m_inj ~ U(m_host-1, m_host+3) is stated without justification for these particular bounds. A brief motivation would help.","section":null},{"comment":"The paper would benefit from a comparison table placing this method alongside existing Real-Bogus approaches (e.g., Reyes et al. 2018, Carrasco-Davis et al. 2021) in terms of training data requirements and performance, even if direct metric comparison is not possible due to different datasets.","section":null}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid methodological contribution with a genuinely novel training paradigm. The main concern is not about correctness of the central human-label-free claim but about the statistical strength of the UQ superiority claim, which is presented more confidently than the evidence supports. The evaluation set limitations are acknowledged but not fully reckoned with in the conclusions. I would encourage the editor to ask the authors to either add error bars on the UQ metrics or soften the superiority language to 'comparable' rather than 'outperforms.' The paper fits the journal scope well."},"author_rebuttal":{"model":"glm-5.2","summary":"We thank the referee for a careful and constructive report. The referee correctly identifies the core contributions of the paper and raises three major comments, all of which are substantive and well-targeted. We address each below. In brief: (1) we agree that bootstrap confidence intervals are needed for the UQ comparison and will add them; (2) we agree that the evaluation-set scoping needs to be made more explicit throughout the paper, including in the abstract and conclusions; (3) we agree that a sensitivity analysis on forget-rate misspecification would strengthen the transfer claim and will add one. We detail our planned revisions for each point.","responses":[{"response":"The referee is correct. The margins in Table 6 are small (e.g., NLL 0.292 vs. 0.314 for Deep Ensemble; ECE 0.0375 vs. 0.0535), and without confidence intervals or significance tests, the claim of consistent outperformance is not statistically justified. We will revise the manuscript to include bootstrap confidence intervals (with at least 1,000 resamples) for all metrics in Table 6 (NLL, Brier score, ECE). We will also add paired bootstrap tests to assess whether the differences between our method and each baseline are statistically significant. The text in §7.2 will be revised accordingly: where the confidence intervals overlap, we will describe the result as 'competitive with' rather than 'consistently outperforms,' and we will explicitly state which differences are significant at a given level. We agree this is load-bearing for the UQ claim and the revision is necessary. We note that the ECE difference (0.0375 vs. 0.0535 for Deep Ensemble, a 29.9% relative reduction) is the largest margin and may survive a significance test, but we will let the data determine this rather than asserting it a priori.","revision_made":"yes","referee_comment":"§7.2, Table 6: The UQ comparison reports small margins between methods on a single evaluation set of 4,820 sources, with no confidence intervals, bootstrap analysis, or significance tests. The claim that the method 'consistently outperforms' ensemble baselines is not statistically supported."},{"response":"The referee raises a valid concern about evaluation bias. We agree that the current evaluation set, by construction, favors high-purity SN-like events and that the 99.35% object-level accuracy and the UQ calibration metrics should not be interpreted as representative of performance on the full alert stream. We will make the following revisions: (1) In §2.3, we will add an explicit statement that all quantitative results in the paper apply to the high-purity SN-like evaluation subset and that performance on a more representative or harder sample is not assessed. (2) In the Abstract and §9 (Conclusions), we will scope the transfer claim more carefully, replacing 'suited for transfer to forthcoming surveys' with language that acknowledges the method is transferable in principle (the training pipeline is survey-agnostic) but that performance characterization is limited to the curated high-purity subset. (3) In §7.4, we will add a caveat that the 99.35% accuracy is measured on an easy subset and should not be extrapolated. (4) We will add a paragraph in §8 discussing how the evaluation bias likely affects the relative UQ comparison: since all methods are evaluated on the same subset, the relative ranking may be less sensitive to the bias than the absolute values, but we cannot rule out that harder cases would shift the comparison. We acknowledge that we cannot, within the scope of the current revision, construct a new representative evaluation set, as this would require additional manual labeling or a different evaluation methodology. We will state this limitation explicitly.","revision_made":"partial","referee_comment":"§2.3, Table 1: The evaluation set is constructed with aggressive filtering that explicitly removes ambiguous cases. The 99.35% object-level accuracy and UQ calibration results are all measured on this easy subset. The paper claims the method is 'suited for transfer to forthcoming surveys' but the absence of evaluation on harder or more representative cases weakens the generalizability evidence."},{"response":"This is a fair and constructive suggestion. The practical applicability of Asym-Co-teaching to a new survey does depend on choosing appropriate forget rates without ground truth, and the current manuscript does not test sensitivity to misspecification. We will add a sensitivity analysis in which we train Asym-Co-teaching on the baseline dataset with forget rates that are systematically misspecified (e.g., setting the survey-class forget rate to 0%, 10%, 15%, 20% when the assumed true noise is ~5%, and similarly varying the injected-class forget rate). We will report the resulting accuracy, ROC AUC, and calibration metrics in a new table or figure. This will allow readers to assess how much performance degrades under misspecification and provide practical guidance for applying the method to a new survey. We expect that moderate overestimation of the forget rate will lead to some loss of training signal but should not catastrophically degrade performance, since co-teaching is designed to be robust to the discarded samples being clean rather than noisy; however, we will let the empirical results speak. We will also add a brief discussion in §8 on strategies for estimating forget rates in practice for a new survey (e.g., using training dynamics or a small validation set), while noting that automatic noise-rate estimation remains future work as already stated.","revision_made":"yes","referee_comment":"§4.2, Appendix C: The forget rates for Asym-Co-teaching are set using prior knowledge of the noise levels. For the baseline dataset, the 5% noise estimate for the survey class is an assumption whose sensitivity is not tested. A sensitivity analysis showing how performance degrades if the forget rate is misspecified would strengthen the transfer claim."}],"tokens_in":26444,"tokens_out":1259,"duration_ms":134810,"standing_objections":[]},"desk_editor":{"model":"glm-5.2","letter":"The main thing to know: this paper proposes training a real-bogus classifier without human labels by using injected transients as a clean positive class and raw survey detections as a noisy negative class, handled via a class-dependent variant of co-teaching (Asym-Co-teaching). That combination is genuinely new and well-motivated. The injection scheme is physically grounded — host-galaxy-associated positions, magnitude priors tied to host brightness — and the robustness experiments under controlled contamination (up to 35%) are done properly, with provenance labels tracked separately from training labels so the analysis is honest about what the model actually recovers. The asymmetric co-teaching formulation is a clean adaptation: class-specific forget rates make sense when noise is concentrated in one class, and the formalism in Section 4.2 is correct. The hybrid UQ strategy — using the two co-teaching networks as a mini-ensemble augmented with MC dropout at inference — is a reasonable low-cost approach, and the Bayesian mixture approximation in Section 5.5 is at least principled even if heuristic. Code and interactive visualizations are shipped on Zenodo, which is good practice. The latent-space UMAP analysis is qualitative but genuinely useful for understanding failure modes. The stress-test concern about the evaluation set is the real soft spot, and it lands. The eval set is 306 objects / 4,820 sources, aggressively filtered for high-purity SN-like events (SNR > 5, ≥6 nights, host-galaxy cuts, flux-ratio cuts). The authors explicitly say this is a high-purity subset, not a representative sample. That is fine for proving the method works on easy cases, but the UQ comparison in Table 6 shows margins of NLL 0.292 vs 0.314 (deep ensemble), Brier 0.063 vs 0.065, ECE 0.0375 vs 0.0535 — differences that could easily be sampling noise on ~4,800 sources with no bootstrap, no confidence intervals, no significance test. The 99.35% object-level accuracy is 304/306, which is two objects away from merely good. The UQ method is evaluated on exactly the cases where it should matter least (high-purity, high-SNR, well-sampled), and the claim of superiority over ensembles is not statistically supported. The reader's concern about the training-side assumption (bogus-dominated survey data) is less load-bearing than the eval-side concern. The authors test contamination up to 35% and acknowledge the limitation explicitly. That assumption is reasonable for current DIA pipelines and the robustness experiments address it. The bigger issue is that all three pillars of the central claim — classification performance, robustness, and UQ competitiveness — are evaluated on the same small curated set. A reader who works on time-domain pipelines will get value from the methodological framework: the injection-based training loop, the asymmetric co-teaching adaptation, and the hybrid UQ strategy are all transferable ideas. The paper deserves a serious referee. The referee should push hard for (1) bootstrap or jackknife error bars on the UQ metrics, (2) evaluation on a less filtered set that includes ambiguous cases where UQ actually matters, and (3) a more honest framing of the UQ comparison as 'comparable' rather than 'superior' given the evidence. The core method is sound; the evidence base for the UQ claims needs strengthening.","headline":"Human-label-free real-bogus via asymmetric co-teaching is a solid methodological contribution, but UQ superiority claims rest on a small curated eval set with no significance testing.","tokens_in":27272,"tokens_out":800,"would_cite":false,"duration_ms":72410,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"glm-5.2","headline":"Train Real-Bogus Classifiers Without Human Labels","keywords":["real-bogus classification","weakly supervised learning","co-teaching","uncertainty quantification","transient detection","difference image analysis","label noise","deep learning"],"falsifier":"If the transient fraction among raw DIA detections exceeds roughly 35%, the asymmetric noise assumption breaks down and the classifier would be trained to mislabel real transients as bogus.","tokens_in":26395,"feed_emoji":"🔭","tokens_out":1025,"duration_ms":69284,"temperature":0.7,"pith_summary":"The paper claims that transient-bogus classification for time-domain astronomical surveys can be trained without any human-labeled data by combining simulated source injections (clean positive examples) with raw survey detections (a noisy negative class dominated by artifacts but containing some real transients). The central mechanism is Asymmetric Co-teaching: two neural networks trained simultaneously, each selecting low-loss samples from the other network for training, but with class-specific discard rates that account for the fact that label noise is concentrated in one class. This setup treats the problem as weakly supervised learning with asymmetric, class-dependent label noise. The paper shows the method remains stable up to 35% contamination of the bogus-labeled class by genuine transients, and proposes a hybrid uncertainty strategy that combines the two co-teaching networks with MC dropout at inference time, achieving calibration competitive with 50-member deep ensembles at a fraction of the training cost.","feed_headline":"Train Real-Bogus Classifiers Without Human Labels","feed_subtitle":"Injected transients plus noisy survey data, held together by asymmetric co-teaching, match ensemble calibration at a fraction of the cost.","key_machinery":"Asymmetric Co-teaching: a dual-network training procedure where each network filters training samples for the other, but with separate forget rates per class. The injected-transient class has a low forget rate (preserving nearly all samples) while the survey-bogus class has a higher forget rate (discarding likely-mislabeled examples). This asymmetry matches the noise structure: injections have clean labels, survey detections have corrupted labels.","core_discovery":"Asym-Co-teaching enables a classifier to separate real transients from bogus detections using only injected simulated sources and unlabeled survey data, maintaining ROC AUC around 0.95 even when over a third of the bogus-labeled training class is secretly real transients. The hybrid uncertainty method (Ensemble-MC-Dropout for Co-teaching) achieves the best calibration among all tested methods (NLL 0.292, ECE 0.0375) while requiring only two trained networks plus dropout sampling at inference, instead of 50 independently trained models. When source-level predictions are grouped by sky position into light-curve-level object classifications, the method achieves 99.35% accuracy on a curated 306-","pith_inferences":["If the transient fraction in raw survey detections rises substantially (as DIA pipelines improve and produce fewer artifacts), the asymmetric noise assumption weakens and the method may need explicit noise-rate estimation rather than fixed forget rates.","The finding that the two-network co-teaching pair plus dropout matches 50-member ensembles suggests that training-time coupling between networks may produce more useful functional diversity than independent initialization alone, which has implications for ensemble design beyond astronomy.","The latent-space visualization revealing structured subclasses within the bogus population (dipoles, negative residuals, low-SNR artifacts) suggests that the model learns physically meaningful artifact categories without being told they exist, which could inform targeted pipeline improvements."],"forward_implications":["Future surveys like LSST could deploy real-bogus classifiers by re-running an injection pipeline on their own data, eliminating the need for survey-specific human-labeled training sets that are expensive and slow to produce.","The hybrid uncertainty approach (2 networks + MC dropout) could replace large ensembles in resource-constrained survey pipelines, cutting training cost by roughly 25x while improving calibration.","Epoch-level uncertainty from single-image classification could be used to down-weight ambiguous measurements in downstream light-curve fitting, propagating classification confidence into photometric analysis.","The asymmetric co-teaching framework could be applied to other weakly supervised problems where one class has reliable labels (from simulation) and the other is a noisy real-world sample."],"fun_headline_variants":["Label-Free Real-Bogus Sorting via Asymmetric Co-Teaching","Calibrated Real-Bogus Detection Without Human Labels","Real-Bogus Sorting Resists High Contamination Without Labels","Two-Network UQ Matches Deep Ensembles for Real-Bogus Sorting","Transient Injections Train Real-Bogus Classifiers Without Labels"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The method assumes that unlabeled survey detections are overwhelmingly bogus, so they can serve as a noisy negative class. If a large fraction of survey detections are actually real transients, the training labels become too corrupted and the model would learn to suppress genuine signals. The paper tests robustness up to 35% contamination but relies on real surveys having a much lower transient fraction.","fun_headline_variants_meta":{"raw":{"variants":["Label-Free Real-Bogus Sorting via Asymmetric Co-Teaching","Calibrated Real-Bogus Detection Without Human Labels","Real-Bogus Sorting Resists High Contamination Without Labels","Two-Network UQ Matches Deep Ensembles for Real-Bogus Sorting","Transient Injections Train Real-Bogus Classifiers Without Labels","Label-Free Real-Bogus Classifiers Reach 99% Light-Curve Accuracy","Dual-Network Uncertainty for Label-Free Real-Bogus Sorting"]},"model":"glm-5.2","effort":"high","cost_usd":0.0,"raw_usage":{"total_tokens":1681,"prompt_tokens":629,"completion_tokens":1052,"prompt_tokens_details":null},"tokens_in":629,"tokens_out":1052,"duration_ms":16162,"temperature":1.0,"reasoning_tokens":869,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-07T12:21:04.758235+00:00","model_set":{"reader":"glm-5.2"},"falsifier":"If the transient fraction among raw DIA detections exceeds roughly 35%, the asymmetric noise assumption breaks down and the classifier would be trained to mislabel real transients as bogus.","supporting_citations":[],"review_version":1}