{"id":"87c14734-5e31-4935-b583-832017484978","arxiv_id":"2412.00077","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Selfish Evolution detects and corrects corrupted labels by training a secondary network on the temporal evolution of a primary network overfitting to individual samples.","lead":"This paper introduces Selfish Evolution, a method that watches a neural network as it overfits to individual training samples and uses those overfitting patterns to spot and fix wrong labels. It reports recovering hundreds of previously missed supernovae in a simulated survey and improving label quality on MNIST under extreme noise.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Gold evolution cubes may be contaminated by support-batch self-inclusion, and transfer to train cubes is unvalidated; this threatens the 817-supernova headline result.","rationale":"The reader identified transferability of the E2L mapping as the weakest assumption. I sharpen this into a concrete, testable contamination path: the support batch used during evolution is drawn from the same small gold set used to train E2L, so a gold sample's own clean image can enter its own cube, creating a signal that train cubes cannot have. This is a genuine internal-consistency risk, not merely a lack of error bars. It is especially load-bearing because the abstract's headline number, 817 missed supernovae, comes from the supernova experiment, which is exactly the experiment using the support-batch protocol. The MNIST experiment does not use a support batch, so it is not subject to this particular confound; that result (80% noise reduced to 21.2%) provides some independent support for the core idea. However, the supernova claim is the main scientific contribution and remains unverified under the uncontaminated protocol. The proposed test is simple and decisive: regenerate gold cubes with the selfish sample excluded from the support batch and hold out part of the gold set. If performance survives, the concern is resolved and the conditional verdict can be upgraded; if not, the headline claim should be revised. I therefore keep the verdict CONDITIONAL, aligned with the reader, but with a more specific condition to satisfy.","tokens_in":10183,"tokens_out":10102,"duration_ms":93195,"concrete_test":"Regenerate gold evolution cubes with support batches sampled from G after excluding the selfish sample (or from an independent clean set). Retrain E2L on these uncontaminated cubes, then (a) evaluate correction accuracy on a held-out subset of gold cubes and (b) re-run the training-set correction to count discovered objects. If held-out accuracy or the 817 count drops materially, the original result is confounded by self-inclusion in the support batch.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim requires that the E2L network, trained on gold evolution cubes, transfers to the full noisy training set. The protocol in §4.1.3 generates every cube by first training on a random clean 'support batch' and then overfitting to the selfish sample. The clean dataset here is the gold subset G (507 images), and §4.1.4 trains E2L on cubes from that same G. Because G is small, when generating a gold sample's cube, the support batch can contain that same sample's clean image. The cube then reveals the true label during the support-batch half, a strong artificial signal that is never present for training-set samples, since G and D are disjoint by construction. E2L can exploit this artifact instead of learning a general mapping from overfitting dynamics to clean labels. The paper neither excludes the selfish sample from the support batch nor reports held-out accuracy on gold cubes. The reported 817 recovered objects and 50.1% cosine similarity may therefore be inflated by this train/gold distribution mismatch rather than by the proposed mechanism.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Selfish Evolution, a label-noise correction method that works by taking a primary network trained (for any duration) on a noisy dataset, resetting it to a fixed checkpoint, and then generating per-sample 'evolution cubes' by first overfitting to a clean support batch and then to the single sample, recording the network's outputs over epochs. A secondary Evolution-to-Label (E2L) network is trained on evolution cubes from a small clean gold subset, with clean labels as targets, and then applied to the main noisy training set to produce corrected labels; the process can be iterated in closed-loop 'super-epochs'. The method is evaluated on a supernova-detection image-generation task using DESC DC2 cutouts (reporting 817 recovered supernovae at 50% noise) and on MNIST classification with 50% and 80% label noise (reducing final noise from 41.3% to 21.2% after 10 super-epochs, comparable to Co-teaching).","tokens_in":10461,"tokens_out":5614,"duration_ms":54280,"significance":"The core idea is original and potentially valuable: instead of avoiding overfitting, the method explicitly exploits the dynamics of overfitting to individual samples as a signal for detecting and correcting corrupted labels. The application to supernova discovery is scientifically motivated, and the closed-loop scheme for iteratively cleaning a training set is a sensible design. The MNIST comparison to Co-teaching provides a useful sanity check. However, the evidence supporting the central claims is currently incomplete: the headline supernova result comes from a single split with no error bars or specified threshold, the gold/evolution-cube generation has a plausible contamination artifact that is not addressed, and the claimed network-state-agnostic property is never tested. The paper also promises CIFAR results in a supplementary that is not present and provides no code or data, so reproducibility cannot be assessed.","major_comments":[{"comment":"The protocol for generating E2L training cubes does not exclude the selfish sample from the support batch when both are drawn from the gold subset G. Since the support batch is sampled from the clean dataset and the selfish sample for a gold cube is also in G, a cube can contain the same sample's clean image in its first half, leaking the true label through a signal that never occurs for the disjoint training-set samples D used in evaluation. The paper neither excludes this self-overlap nor reports held-out accuracy of E2L on gold cubes (or on D cubes with known clean labels), so the reported 817 recovered objects and 50.1% cosine similarity may be inflated by this train/gold distribution mismatch rather than by the proposed overfitting-dynamics mechanism. Please modify the generation so the support batch excludes the selfish sample, and validate E2L on held-out gold cubes and on D cubes.","section":"§4.1.3–4.1.4"},{"comment":"The headline result of 817 recovered supernovae at 50% noise rests on a single run without error bars, and the 'Hard. Discovered objects' column depends on an unspecified threshold for hard similarity. The text states that only 'three main representative' experiments out of several are shown, but the selection protocol is not given, so the reader cannot tell whether 817 is typical, best-case, or a lucky split. Please report mean and standard deviation over multiple seeds, define the hard-similarity and discovery thresholds, and provide results for all experiments rather than a representative subset. This is load-bearing for the paper's central discovery claim.","section":"§4.1.5, Table 1"},{"comment":"The claimed 'network-state agnostic' property and 'automatic convergence' of the closed loop are not tested. The method is described as not presuming any particular state of the network (early stopping, fully trained, etc.), but all experiments intervene after an initial training phase of fixed duration, and the MNIST results report only 1 and 10 super-epochs. There is no experiment that interrupts the primary model at different training checkpoints, nor a trajectory of the final noise level over a longer closed-loop run. Without such measurements, the state-agnostic and convergence claims are assertions rather than demonstrated results; please add experiments varying the intervention point and reporting the noise level per super-epoch.","section":"§5, Table 3"}],"minor_comments":[{"comment":"The text says CIFAR results are in the supplementary material, but no supplementary material is included with the manuscript; please add it or remove the claim.","section":"§4.2"},{"comment":"The method is described as 'weakly supervised', but E2L is trained on clean gold labels, which is external supervision; please clarify the intended sense of 'weak' or revise the terminology.","section":"§3, Step 3"},{"comment":"There are several presentation issues: 'the idea output' should be 'the ideal output'; Table 2 lacks a caption; the affiliations contain spacing oddities; and 'noised' is used inconsistently with 'noisy'.","section":"§4.1"},{"comment":"The paper states that source code and data will be made available upon acceptance, but no code or data are provided now; given the centrality of the quantitative claims, providing at least the evaluation protocol and thresholds would improve verifiability.","section":"§5"}],"recommendation":"major_revision","confidential_remarks":"The support-batch contamination concern is real and must be addressed before the supernova results can be trusted. The paper's framing as a computer-vision contribution is somewhat thin because the evaluation is limited to one astronomy dataset and MNIST, with no CIFAR results present; the missing supplement and lack of error bars make the empirical core hard to assess. The idea is novel enough to deserve a revision rather than rejection, but the current evidence does not support the strong claims made in the abstract."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea is worth taking seriously: instead of early-training dynamics, the authors force the model to overfit individual samples and mine the evolution of outputs for label noise and correction. That is a real shift from Dataset Cartography and similar work, and applying it to image-like labels (astronomical cutouts) rather than classification opens a useful niche. The MNIST experiments, while modest, show the mechanism works at least to some degree. The paper deserves credit for aiming at correction, not just detection, since in the supernova context a missed object is a lost discovery.\n\nThe soft spots are substantial. The stress-test concern is valid: the support batch used in the first half of each evolution cube is drawn from the same gold subset G that supplies the E2L training cubes. For a gold sample, that support batch can contain the sample itself, so the clean label leaks into the cube during the first half. E2L can exploit a trivial cue that never exists for the train-set samples. The paper does not exclude the selfish sample from the support batch, and it reports no held-out accuracy on gold cubes. That alone makes the 817-recovered-object headline unreliable. Also, the paper is thin on evidence: three representative runs, no error bars, no code or data, the promised CIFAR supplement absent, and a claim of network-state agnosticism that is never actually tested across different stopping points.\n\nI do not think the method is dead. The MNIST results are less affected by the artifact and show a real signal, so the underlying idea has legs. But as presented, the paper overstates what is proven. A serious referee should ask for code and data, a variant of the supernova experiment where the support batch explicitly excludes the selfish sample, held-out evaluation on gold cubes, and variance across repeated runs. With those, the method could be solid. Without them, the key numbers are not credible.\n\nThis is a paper for the LNL and astronomy-ML communities, and it should go to peer review — the novelty and application justify referee time. But I would not publish it in its current form. If the authors can address the contamination and release the artifacts, it could be a useful contribution.","headline":"Overfitting dynamics for label correction is a genuinely fresh idea, but the headline supernova result is likely inflated by a support-batch artifact, and the paper currently lacks the code, data, and controls to back its claims.","tokens_in":10939,"tokens_out":1720,"would_cite":false,"duration_ms":17697,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Selfish Evolution uses a model's overfitting history to detect and correct corrupted labels, recovering 817 missed supernovae and cutting MNIST noise from 80% to 21.2%.","keywords":["label noise","overfitting dynamics","evolution cubes","learning with noisy labels","supernova detection","weak supervision","image generation","closed-loop correction"],"falsifier":"Run the same protocol on a noisy dataset with known ground truth but with the gold subset drawn from a different distribution than the training set (for instance, a different region of the sky or a different digit style); if cleaned-label accuracy does not beat the baseline without correction, the transfer assumption is false.","tokens_in":10022,"feed_emoji":"🔭","tokens_out":3909,"duration_ms":31184,"temperature":0.7,"pith_summary":"This paper proposes that the sequence of outputs a network produces while being forced to overfit a single sample—its 'overfitting dynamics'—contains enough information to tell whether that sample's label is wrong and what the correct label should be. The authors train a small clean 'gold' subset, record these evolution cubes for each sample, and train a secondary network to map cubes to clean labels, then apply it to the whole noisy training set in a closed loop. On a supernova-detection task with image-like regression labels, the method recovered 817 supernovae that had been missed under 50% label noise; on MNIST with 80% random label corruption, ten closed-loop iterations reduced final noise from 80% to 21.2%. The motivation is that in scientific discovery, a corrupted label is not just noise to be discarded but a potentially missed object worth recovering.","feed_headline":"Overfitting histories fix corrupted labels, recover 817 supernovae","feed_subtitle":"A secondary network reads each sample's overfitting pattern to correct noisy labels, cutting MNIST noise from 80% to 21.2%.","key_machinery":"The key object is the evolution cube: a 3-D tensor stacking the model's outputs on one sample over the overfitting steps, produced by a 'race' between tuning on a clean support batch and overfitting to the sample's current (possibly noisy) label. The Evolution-to-Label (E2L) network, a thinner version of the same encoder-decoder architecture, is trained on gold-subset cubes to map these dynamics directly to clean labels, implicitly performing both noise detection and correction. The closed-loop protocol re-feeds the E2L-corrected labels as the new training labels in successive super-epochs, which allows the system to converge toward a clean dataset.","core_discovery":"The central claim is that overfitting dynamics, rather than early-training dynamics, provide a recoverable signal about label noise even when the network is already trained on noisy data. For each sample, starting from a fixed pre-trained state, the network is first tuned on a clean support mini-batch and then driven to overfit the sample itself; the per-epoch outputs form a spatiotemporal evolution cube. The paper shows that these cubes carry enough information that a secondary Evolution-to-Label (E2L) network, trained only on cubes from a small gold subset, can generate corrected labels for the full training set, and that iterating the process makes the labels progressively cleaner without assuming any particular state of the primary network.","pith_inferences":["The method may extend to other pixel-level regression tasks, such as medical image segmentation or denoising, where false negatives are costly and a small clean labeled subset exists.","The reliance on a gold subset could potentially be relaxed: if evolution cubes from confident high-loss samples are self-labeled and added to the E2L training set, the method might bootstrap with no clean labels at all.","A testable extension would be to replace the E2L network with a sequence model (RNN or transformer) that reads the temporal dimension explicitly, which might improve correction on longer evolution histories.","The 'race' between support batch and selfish sample suggests the method's success depends on the optimizer's behavior under conflicting gradients; changing optimizer or batch composition could make the signal stronger or weaker, which is a direct empirical knob to probe."],"forward_implications":["Corrupted labels can be corrected rather than dropped, turning noise into recoverable discoveries in scientific pipelines.","The method works for image-like regression labels (pixel-wise outputs), a regime most label-noise research does not address.","Because it is network-state agnostic, it can be applied to models that have already been fine-tuned for a long time, not just early in training.","Closed-loop iteration produces progressive denoising: ten super-epochs on MNIST cut 80% noise to 21.2%, matching a much larger Co-teaching network.","On the supernova task with 50% noise, the approach recovered 817 missed objects, a direct gain in discovery yield."],"supporting_citations":[{"why":"Sedaghat and Mahabal's TransiNet defines the image-generation task and the encoder-decoder architecture that the paper reuses for both main and E2L networks.","marker":"[29]"},{"why":"The DESC DC2 simulated sky survey provides the supernova cutout dataset used in the main experiments.","marker":"[1, 2]"},{"why":"Arpit et al.'s memorization analysis motivates the idea that networks fit clean patterns before memorizing noisy labels, the backdrop for studying overfitting behavior.","marker":"[4]"},{"why":"Co-teaching is the strongest comparison baseline on MNIST, using a much larger dual-network architecture.","marker":"[12]"},{"why":"Dataset Cartography is the closest conceptual predecessor, using training dynamics to characterize noisy examples, though it stops at detection.","marker":"[33]"},{"why":"Adam is the optimizer used throughout the initial training and both halves of the evolution process.","marker":"[16]"}],"fun_headline_variants":["Overfitting dynamics expose and correct noisy labels","Selfish Evolution turns overfitting into label correction","Extreme label noise tamed by overfitting histories","Learn to fix corrupted labels from overfitting patterns"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that the overfitting patterns seen on a small gold subset with clean labels transfer to the much larger noisy training set, so that a correction rule learned from a few hundred examples works across all samples.","fun_headline_variants_meta":{"raw":{"variants":["Overfitting dynamics expose and correct noisy labels","Selfish Evolution turns overfitting into label correction","Extreme label noise tamed by overfitting histories","Learn to fix corrupted labels from overfitting patterns"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000191,"raw_usage":{"total_tokens":1296,"prompt_tokens":852,"completion_tokens":444,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":468,"completion_tokens_details":{"reasoning_tokens":385}},"tokens_in":468,"tokens_out":444,"duration_ms":4535,"temperature":1.0,"reasoning_tokens":385,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:51:03.360599+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same protocol on a noisy dataset with known ground truth but with the gold subset drawn from a different distribution than the training set (for instance, a different region of the sky or a different digit style); if cleaned-label accuracy does not beat the baseline without correction, the transfer assumption is false.","supporting_citations":[{"cited_title":"Effective image differencing with convolutional neural networks for real-time transient hunting","cited_arxiv_id":null,"evidence_quote":"Sedaghat and Mahabal's TransiNet defines the image-generation task and the encoder-decoder architecture that the paper reuses for both main and E2L networks."},{"cited_title":"A closer look at memorization in deep networks","cited_arxiv_id":null,"evidence_quote":"Arpit et al.'s memorization analysis motivates the idea that networks fit clean patterns before memorizing noisy labels, the backdrop for studying overfitting behavior."},{"cited_title":"Co-teaching: Robust training of deep neu- ral networks with extremely noisy labels","cited_arxiv_id":null,"evidence_quote":"Co-teaching is the strongest comparison baseline on MNIST, using a much larger dual-network architecture."},{"cited_title":"Smith, and Yejin Choi","cited_arxiv_id":null,"evidence_quote":"Dataset Cartography is the closest conceptual predecessor, using training dynamics to characterize noisy examples, though it stops at detection."}],"review_version":1}