{"id":"c45b2731-6e30-4059-8928-020730d40a71","arxiv_id":"2412.02409","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"RB-C1000, a pipeline combining active learning and semi-supervised pseudo-labeling, achieves roughly 98.8% real/bogus classification accuracy on new ZTF datasets using only 1,000 labels.","lead":"A deep learning pipeline that combines active learning with pseudo-labeling classifies real versus bogus astronomical transients using only 1,000 human labels, reaching about 98.8% accuracy on three newly built ZTF datasets. If it transfers to WFST and LSST, the method could cut the human labeling effort needed when a large time-domain survey starts.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The constructed labels define real as TNS-reported SNe and bogus as unvetted random candidates, so the 98% accuracy is not demonstrably a real-vs-bogus classifier for the general alert stream; this unverified label representativeness is the load-bearing weakness.","rationale":"The paper is a clear, honest application of active learning and pseudo-labeling to a real-bogus classification task. The experiments are repeated and report standard deviations, the method is described in detail (Algorithm 1, Figure 6), and the authors explicitly flag the dataset representativeness limitation in Section 2. The most load-bearing condition for the central claim is that the training and test labels represent the real/bogus decision of an actual alert stream. That condition is unmet by construction: real is a subset of SNe, bogus is a random sample of candidates from one epoch/field set with no evidence of cross-matching against real-source catalogs. This undermines the external validity of the headline accuracy numbers and the WFST deployment claim. I considered the alternative concern that the headline efficiency comparison is not statistically significant—RB-C1000's summed indicator on ZTF-NEWm (587.5±0.8) vs fully supervised 2500 labels (586.3±2.0) overlap within uncertainty, and R and unlabeled-sample size were selected using test results—but this is secondary: even if the efficiency comparison were definitive, it would only show efficiency on the compiled SNe-vs-random dataset, not on the general alert stream. The paper's own caveat prevents this from being an internal contradiction, so the reader's CONDITIONAL verdict is appropriate and no change is needed.","tokens_in":20469,"tokens_out":10570,"duration_ms":113085,"concrete_test":"Build an external validation set of ZTF alerts that have been labeled by human experts (or by cross-matching with TNS and known variable-star/AGN/moving-object catalogs), containing both artifacts and real sources of multiple classes including non-SNe. Run the trained RB-C1000 (or retrained under the paper's protocol) on this set and compute class-wise accuracy for the non-SN real sources. If the accuracy on non-SN real sources is materially below the reported ~98.6%, or if a cross-match of the paper's bogus stamps finds a non-negligible fraction of real astrophysical sources, the label-construction concern lands and the central claim should be re-scoped to 'SNe detection' rather than general real/bogus classification.","verdict_should_be":"UNCHANGED","load_bearing_attack":"For the central practical claim—that RB-C1000 can build a competitive real/bogus classifier in the early WFST survey with 1000 labels—it must be true that the labels measure real vs bogus in the alert stream. Section 2 states the opposite: real sources are only TNS supernovae, and bogus detections are randomly drawn detection candidates that 'might contain a few real detections.' Thus the 98.8% / 98.8% / 98.6% accuracies measure separation between SNe and unvetted artifacts from one epoch/field set, not performance on general transients (stellar flares, AGN variability, moving objects, faint SNe, etc.). A model trained on these labels can be rewarded for labeling non-SN astrophysical sources as bogus, since such sources are absent from the real class and may appear in the bogus class. The relative efficiency comparison against a fully supervised ResNet-18 (Table A.6) uses the same label definitions, so the efficiency gain may survive, but the absolute performance claimed for survey deployment is not established. The authors explicitly and appropriately caveat this in Sections 2 and 5; the concern is that a central, headline claim of the paper is precisely the one the caveat disqualifies.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RB-C1000, a deep-learning pipeline for real/bogus transient classification that combines active learning with pseudo-labeling-based semi-supervised learning. The method starts from M=900 labeled samples, has experts label K=100 hard samples, then iteratively pseudo-labels V high-confidence unlabeled samples over R rounds. Using three newly compiled ZTF datasets (ZTF-NEWg, ZTF-NEWr, ZTF-NEWm), each with 13,000 real sources (TNS supernovae) and 30,000 unvetted bogus candidates, the authors report mean accuracies of 98.8%, 98.8%, and 98.6% with only 1,000 labels, and argue that this surpasses a fully supervised ResNet-18 trained on 2,500 random labels. The paper includes repeated experiments (five trials), confidence intervals on metrics, and publicly available code and data. The authors repeatedly and explicitly caution that the datasets are not representative of the general alert stream.","tokens_in":20781,"tokens_out":3139,"duration_ms":36476,"significance":"If the reported efficiency gain is real, the method is a useful contribution to time-domain astronomy: it would allow early-survey operations to build a competitive classifier with far fewer manual labels. The algorithmic pipeline is coherent, the experiments are repeated five times with reported standard deviations, and the code and datasets are public, which are strengths. The central limitation is that the constructed labels do not match the stated deployment target: real is defined as TNS-reported supernovae, while bogus is a random, unvetted sample of detection candidates possibly containing real astrophysical sources. The paper acknowledges this in Section 2 and again in Sections 4 and 5, but the abstract and conclusion nevertheless frame the results as enabling general real-source discovery in WFST. The method's internal comparisons (active learning vs. random selection, SSL vs. no SSL) are meaningful on the constructed datasets, but the absolute performance and the headline efficiency claim require either representative labels or a considerably more cautious interpretation.","major_comments":[{"comment":"The label definitions undermine the central deployment claim. Real sources are only TNS supernovae from 2019–2022, and bogus sources are randomly drawn detection candidates that, as the text acknowledges, 'might contain a few real detections.' The reported 98.8%/98.8%/98.6% accuracies therefore measure separation between TNS supernovae and unvetted artifacts, not performance on the general alert stream (stellar flares, AGN variability, faint SNe, moving objects, etc.). The paper already states this caveat, but the abstract and conclusion still claim the method can 'discover real sources that vary in brightness over time in the early stage of WFST.' Please either provide a validation set with representative, vetted real and bogus labels (e.g., a human-vetted alert sample or a set including known non-SN real sources), or explicitly restrict the performance claims to 'SNe-like real sources' and remove the implication of general WFST readiness.","section":"Section 2, Section 5"},{"comment":"Several hyperparameters were selected using the same test sets on which final performance is reported. The text states that R=3 was chosen because it 'often shows the best performance on the overall indicator on these three datasets'; Table A.3 selects 10,000 unlabeled samples partly from test-set performance; Table A.4 selects M/K ratio 0.9 based on the highest sum on ZTF-NEWm. This makes the reported numbers optimistic and the efficiency comparison in Table A.6 circular in the sense that the configuration was tuned on the same test set. Please use a separate validation split for hyperparameter selection and report test-set results only for the selected configuration, or present the full set of configurations with an appropriate correction for multiple comparisons.","section":"Section 4.3, Table A.3, Table A.4, Table A.5"},{"comment":"The headline efficiency claim is not statistically established. RB-C1000 (ZTF-NEWm, sum = 587.5 ± 0.8) is compared with a fully supervised model using 2,500 random labels (sum = 586.3 ± 2.0). The difference in means is 1.2 points; using the reported standard deviations and five repeats, the difference is within approximately 1.3 standard errors, i.e., not significant at the usual 0.05 level. Please report a paired comparison across the five repeats (e.g., paired t-test or Wilcoxon signed-rank test) or otherwise quantify the uncertainty of the improvement, and adjust the claim 'significantly higher efficiency' accordingly.","section":"Table A.6, Section 4.3"},{"comment":"The active-learning component's standalone benefit is not consistently demonstrated. On ZTF-NEWr, adding the K hard samples degrades the ALS result relative to ITS (sum 581.4 vs. 581.7), and the final gain appears only after semi-supervised learning. The current separate attribution of '1.4%, 3.4%, and 7.2%' to active learning and '4.1%, 8.7%, and 7.3%' to semi-supervised learning is a post-hoc decomposition of a pipeline with interacting components. Please present an ablation that isolates each component's marginal contribution under the same hyperparameter selection procedure, or interpret the gains as joint effects.","section":"Section 4.3, Table A.2"}],"minor_comments":[{"comment":"In the concluding paragraph, 'newly complied datasets' should be 'newly compiled datasets.'","section":"Section 5"},{"comment":"The figure labels contain 'T rainingCNN' and 'InferenceCNN' with irregular spacing; please clean up the figure text.","section":"Figure 6"},{"comment":"The validation set is taken from the (M+K) labeled samples rather than from a separate pool; this is acceptable for model selection but should be stated more explicitly, since it means the 'validation' performance is not an unbiased estimate of generalization to the unlabeled pool.","section":"Section 4.1"},{"comment":"The bogus sample construction uses only fields 774–780 on a single date (January 2, 2021). The paper should note whether any real sources from those fields on that date were excluded from the bogus pool, and how the random selection avoids overlap with the real-source set from the same fields.","section":"Section 2"},{"comment":"The 'overall performance' metric (sum of six indicators) is not a standard evaluation measure and mixes quantities with different scales; consider reporting it as a secondary diagnostic or justifying its use.","section":"Table A.1"}],"recommendation":"major_revision","confidential_remarks":"The authors are unusually honest about the label-representativeness problem, which is the main scientific risk. The paper's contribution could be repositioned as a methodological demonstration on a challenging benchmark dataset, but the abstract and conclusion currently overstate readiness for WFST. The hyperparameter selection on the test set and the marginal statistical significance of the efficiency gain are fixable with additional analysis and are the main reasons I am not recommending acceptance in the current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis is a solid, honest engineering paper. The new bits are the three ZTF datasets (g, r, mixed) and the AL+SSL pipeline—initial supervised model, expert labeling of K uncertain samples, then R rounds of pseudo-label retraining—that reaches ~98.8% accuracy with 1000 labels, beating a supervised ResNet-18 trained on 2500 random labels on the same task. Experiments are repeated five times, the ablations (active learning, pseudo-label rounds, unlabeled pool size, M/K split) are genuinely useful, and the citation pattern looks normal. Code and data are promised on GitHub, though I could not verify them.\n\nThe main soft spot is the one the stress-test flags: the labels define real as TNS-listed supernovae and bogus as randomly drawn candidates that may include real sources. The 98.8% accuracy is therefore a separation between SNe and unvetted artifacts from one epoch/field set, not a real-vs-bogus classifier for a general alert stream. The authors state this clearly in Sections 2 and 5, but the abstract and conclusion still push the WFST deployment angle, which the evidence does not support. The relative label-efficiency gain survives, because the fully supervised baseline uses the same labels; the absolute number should not be quoted outside the controlled setting.\n\nSecondary issues are minor: R=3, tau=0.95, and the unlabeled pool size are chosen by test-set performance, which adds some optimism; there is no direct comparison against existing AL or SSL methods for this task, so the novelty of the combination is not benchmarked.\n\nThese issues are manageable. The paper is clear, structurally reproducible, and unusually candid about its limitations. I would send it to peer review, with two requests: validate on an existing curated real/bogus benchmark or a small manually vetted alert sample, and soften the deployment framing. It will be useful to time-domain survey teams and ML practitioners. I'd bring it to a reading group, and I'd cite it for the datasets or as a label-efficient baseline.","headline":"Solid, honest AL+SSL real/bogus paper whose accuracy claims are limited by knowingly non-representative labels; deserves review but not the deployment framing.","tokens_in":21289,"tokens_out":3840,"would_cite":true,"duration_ms":40111,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A real/bogus transient classifier built from only 1000 labels can match fully supervised models trained on 2500.","keywords":["real-bogus classification","active learning","semi-supervised learning","pseudo-labeling","Zwicky Transient Facility","time-domain surveys","convolutional neural networks","WFST"],"falsifier":"Take a fresh sample of alert streams from a running survey, label every candidate by expert visual inspection (including non-supernova real sources such as AGN variability and moving objects), and run the RB-C1000 model trained on the paper's recipe; if accuracy on this expert-labeled test set falls well below 98% or the false-negative rate on non-supernova real sources is much higher than the reported ~1.8%, the claimed generalization does not hold.","tokens_in":20303,"feed_emoji":"🔭","tokens_out":6867,"duration_ms":66464,"temperature":0.7,"pith_summary":"This paper aims to show that a transient real/bogus classifier competitive with fully supervised models can be trained using only 1000 human-annotated examples. The proposed method, RB-C1000, first trains a network on 900 labels, asks experts to label the 100 most ambiguous remaining samples, then repeatedly pseudo-labels the most confident unlabeled samples and retrains. On three newly compiled ZTF datasets, RB-C1000 reaches average accuracies of 98.8%, 98.8%, and 98.6%, and it matches or outperforms a fully supervised ResNet-18 trained on 2500 random labels on the hardest dataset. The result matters because large upcoming surveys such as WFST and LSST need real-time filtering of transient candidates at a time when labeled data are scarce. The authors caution that their datasets contain only supernovae as real sources and random detections as bogus, so the reported numbers may not transfer directly to a real alert stream.","feed_headline":"1000 labels match a 2500-label real-bogus classifier","feed_subtitle":"Active learning plus pseudo-labeling hits 98.8 percent accuracy on ZTF data, easing early-survey alert filtering.","key_machinery":"The central mechanism is the three-stage RB-C1000 pipeline built on a ResNet-18 backbone taking three input channels (science, reference, and difference image stamps of 63x63 pixels). The initial training stage (ITS) trains on M=900 labeled samples and ranks unlabeled data by prediction confidence to identify the K=100 hardest samples. The active learning stage (ALS) retrains on the labeled set of 1000 samples and selects V pseudo-labeled samples whose confidence exceeds a threshold of τ=0.95. The semi-supervised learning stage (SSLS) retrains from scratch on labeled plus pseudo-labeled data and repeats R=3 times. This combination lets the model exploit both expert-selected hard examples and abundant unlabeled data without extra human annotation.","core_discovery":"The paper's central claim is that a real/bogus classifier built with only 1000 labeled samples can perform as well as a fully supervised model trained on thousands of random labels. In the RB-C1000 pipeline the initial supervised model selects the K=100 hardest unlabeled samples for expert annotation; the active model then picks V high-confidence samples and assigns them pseudo-labels; and the semi-supervised model retrains on the expanded set for R=3 iterations. On the ZTF-NEWg, ZTF-NEWr, and ZTF-NEWm datasets this yields average accuracies of 98.8%, 98.8%, and 98.6%, with the mixed-band ZTF-NEWm result (98.6%) exceeding the fully supervised ResNet-18 baseline trained on 2500 random labels (98.4%). The authors frame this as a practical route to deploying deep-learning real/bogus classification at the start of a survey, when annotation resources are limited.","pith_inferences":["The same two-stage annotation strategy (label hardest, then pseudo-label confident rest) could be transferred to other label-scarce astronomical classification problems such as anomaly detection or supernova typing, where the gains from active selection on harder data appear largest.","Because the mixed-band dataset showed the largest active-learning gain, the method is likely most valuable when the data distribution is heterogeneous; a direct test would be to run RB-C1000 on multi-band or multi-telescope inputs with different noise properties.","The reported band asymmetry (r-to-g transfer fails while g-to-r works) implies that a single cross-band model is not safe; an inference is that WFST could train per-band RB-C1000 models using 1000 labels each, which the pipeline's cost makes feasible.","If the pseudo-label threshold or iteration count is tuned on one telescope's stamps, the optimal values may shift for another telescope; a testable extension is to re-scan τ and R on LSST-like simulated stamps."],"forward_implications":["Surveys like WFST and LSST can deploy a deep-learning real/bogus filter early in operations with only about 1000 human labels, reducing annotation cost and enabling rapid follow-up.","The RB-C1000 recipe improves over random labeling by 1.4-7.2 percentage points in summed performance and over labeled-only training by 4.1-8.7 points, depending on the band.","A fully supervised model needs roughly 2500 random labels to match RB-C1000's 1000-label performance on the mixed-band dataset, so the method is more label-efficient.","The pseudo-labeling iterations saturate around R=3, and using 5000-10000 unlabeled samples balances performance and training time.","The classifier is intended to be integrated into the WFST data processing pipeline for early-survey transient classification."],"supporting_citations":[{"why":"Presents BRAAI, a supervised real-bogus classifier for ZTF that requires large labeled training sets, serving as the contrast case for the paper's low-label approach.","marker":"Duev et al. 2019"},{"why":"Introduces MeerCRAB and notes that models do not transfer between telescopes because of i.i.d. violations, motivating training on target survey data.","marker":"Hosenie et al. 2021"},{"why":"Deep-HiTS, a supervised deep learning real-bogus classifier whose dependence on many labels the paper aims to reduce.","marker":"Cabrera-Vives et al. 2017"},{"why":"Surveys active learning strategies, providing the basis for selecting the most informative (hard) samples for annotation.","marker":"Settles 2009"},{"why":"Surveys semi-supervised learning, providing the pseudo-labeling idea used to exploit unlabeled data.","marker":"Zhu 2005"},{"why":"Defines ResNet-18, the backbone architecture of the classifier, which the paper modifies for two-class output.","marker":"He et al. 2016"},{"why":"Describes the real-source selection and stamp extraction procedure that the paper adapts to compile the ZTF datasets.","marker":"Reyes-Jainaga et al. 2023"},{"why":"Describes the ZTF survey that provides the science, reference, and difference images used in the three datasets.","marker":"Bellm et al. 2018"}],"fun_headline_variants":["1000 labels match 2500 in real-bogus transient classification","Active learning cuts label needs by 60% for transient classification","98.8% accuracy from just 1000 labeled samples in transient AI","Semi-supervised boost: 1000 labeled samples rival 2500"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results depend on the labels being trustworthy: real sources are only supernovae reported to TNS, and the bogus set is a random sample of detection candidates that the authors say may contain a few real detections, so label contamination would inflate the measured accuracy.","fun_headline_variants_meta":{"raw":{"variants":["1000 labels match 2500 in real-bogus transient classification","Active learning cuts label needs by 60% for transient classification","98.8% accuracy from just 1000 labeled samples in transient AI","Semi-supervised boost: 1000 labeled samples rival 2500"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000358,"raw_usage":{"total_tokens":1927,"prompt_tokens":919,"completion_tokens":1008,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":535,"completion_tokens_details":{"reasoning_tokens":929}},"tokens_in":535,"tokens_out":1008,"duration_ms":10932,"temperature":1.0,"reasoning_tokens":929,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T23:28:58.841549+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a fresh sample of alert streams from a running survey, label every candidate by expert visual inspection (including non-supernova real sources such as AGN variability and moving objects), and run the RB-C1000 model trained on the paper's recipe; if accuracy on this expert-labeled test set falls well below 98% or the false-negative rate on non-supernova real sources is much higher than the reported ~1.8%, the claimed generalization does not hold.","supporting_citations":[{"cited_title":"2009, Active learning literature survey https://minds.wisconsin.edu/handle/1793/60660","cited_arxiv_id":null,"evidence_quote":"Surveys active learning strategies, providing the basis for selecting the most informative (hard) samples for annotation."},{"cited_title":"MeerCRAB: MeerLICHT Classification of Real and Bogus Transients using Deep Learning","cited_arxiv_id":"2104.13950","evidence_quote":"Introduces MeerCRAB and notes that models do not transfer between telescopes because of i.i.d. violations, motivating training on target survey data."},{"cited_title":"Deep-HiTS: Rotation Invariant Convolutional Neural Network for Transient Detection","cited_arxiv_id":"1701.00458","evidence_quote":"Deep-HiTS, a supervised deep learning real-bogus classifier whose dependence on many labels the paper aims to reduce."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Surveys semi-supervised learning, providing the pseudo-labeling idea used to exploit unlabeled data."},{"cited_title":"Multi-scale stamps for real-time classification of alert streams","cited_arxiv_id":"2304.13080","evidence_quote":"Describes the real-source selection and stamp extraction procedure that the paper adapts to compile the ZTF datasets."}],"review_version":1}