{"id":"e0ccb0c9-7e13-49af-abd4-8170363aeb07","arxiv_id":"2505.03811","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"ScarceGAN adds a leeway class for uncertain negatives to semi-supervised GANs and splits negatives into subclasses, improving recall for rare class detection in weakly labeled longitudinal data.","lead":"This paper introduces ScarceGAN, a modified semi-supervised GAN for finding extremely rare positive samples in longitudinal data when only a few noisy labels are available. It adds an unknown class as a leeway term to keep uncertain negatives away from the positive class, reporting higher recall than vanilla semi-supervised GAN on risky player detection and on KDDCUP99.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The leeway class is defined as negative in both the supervised and unsupervised paths; because every unlabeled real sample is trained toward K or U, ScarceGAN has no mechanism to recover novel positives from the unlabeled set, so the recall claim is only supported for positives that resemble the…","rationale":"The reader's weakest assumption is indeed the most load-bearing. ScarceGAN's leeway class is not a neutral 'unknown' bucket; it is explicitly a negative-class bucket, and the unsupervised loss gives every real unlabeled sample a target of K or U, never R. In a PU setting with 0.05-0.1% positive prevalence, unlabeled positives are likely and novel risky behavior is a primary motivation for unlabeled monitoring, so this is not a minor edge case. The paper's own problem statement acknowledges unseen behavior in the unlabeled set, but only commits to handling it in the negative class; the positive side is left unaddressed. The evaluation compounds this because the confirmed positives in Sets 1/2/3 and SIUs come from the same rule-engine/business flagging process that produced the labeled prior, so high recall may reflect agreement with the label-generation rule rather than generalization to new positive behaviors. The proposed KDD subtype-split experiment would settle this directly: if novel positives are systematically assigned to U, the central claim is overstated and the architecture requires an explicit mechanism for unlabeled positives. This reinforces the reader's conditional verdict rather than overturning it: the idea is plausible and the controlled experiments are worth running, but the current evidence does not establish the framework's ability to identify unseen risky behavior.","tokens_in":16934,"tokens_out":10619,"duration_ms":114836,"concrete_test":"Run a subtype-split experiment on KDDCUP99 R2L: select R2L subtypes or flag patterns that are rare or distinct, hold them out as 'novel' positives, and include them in training only through the unlabeled set. Train ScarceGAN with the remaining R2L subtypes as the labeled positive prior, using the paper's 900/30k protocol. Then measure (i) recall on held-out novel positives and (ii) the fraction of those novel positives assigned to U rather than R. Swap the held-out subtype and repeat. If novel-positive recall is substantially below known-positive recall or the misses concentrate in U, the leeway class absorbs unseen positives and the 85% recall claim must be restricted to the labeled-prior distribution.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Section 3.2, U is explicitly defined as part of the negative class, and in Section 3.3.2 the unsupervised loss is L_US_Real_D = alpha*CCE(x|C_y in K) + (1-alpha)*CCE(x|C_y in U). Every unlabeled real sample is therefore pushed toward K or U; U is the low-confidence sink, and no unlabeled sample is ever pushed toward R. If a genuinely new risky behavior appears only in the unlabeled set, or an existing rare positive is not confidently close to the small labeled prior, the model is trained to assign it to U and it is missed. The paper's problem statement (Section 1) anticipates unseen behavior in the unlabeled set but only commits to handling it in the negative class; no closed-world assumption for positives is stated or argued. The evaluation hides this risk: the confirmed positives in Sets 1/2/3 and SIUs were generated by the same rule-engine/business flagging pipeline that produced the 150 labeled positives, so high recall on them may reflect imitation of the rule engine rather than discovery of novel positives. The 85% recall claim therefore rests on an unstated closed-world positive assumption, which is exactly the kind of assumption that can fail in longitudinal telemetry when risky behavior evolves.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"ScarceGAN is a semi-supervised GAN framework for detecting a rare positive class in longitudinal telemetry data with a small, noisy label prior. The paper's main technical additions are: (i) dividing the negative class into three domain-defined subclasses (D, N, H); (ii) introducing an 'unknown' leeway class U that is treated as part of the negative space; (iii) modifying the supervised and unsupervised discriminator losses to use alpha-weighted cross-entropy between known and unknown negative assignments; and (iv) adopting a 'bad generator' objective for complementary learning. Features are constructed as ten Prophet hyperparameters over longitudinal counters. On a proprietary skill-gaming dataset, the authors report held-out precision/recall of 75.7%/72.5% versus 34%/24% for vanilla SSGAN, recall of roughly 75-91% on small sets of confirmed positives, and reduced verbosity on false-positive users. On KDDCUP99, they report R2L F1=28 and competitive results for imbalanced 'normal' class detection. The paper positions ScarceGAN as a new benchmark for rare-class identification in longitudinal settings.","tokens_in":17263,"tokens_out":10999,"duration_ms":108224,"significance":"If the claims hold, ScarceGAN would be a practically useful recipe: the leeway class is a simple and interpretable way to absorb noisy negative labels, the divided-negative-focus idea is well motivated by the described data, and the ablations (binary versus divided negatives, with/without leeway, with/without bad generator) directly test each design component. The authors also ship source code and evaluate on a real deployment plus a public dataset, which are strengths. The significance is bounded, however, by two issues: the model treats the unlabeled data as containing no novel positives, yet the paper's framing suggests discovery of previously unseen risky behavior; and the quantitative evidence for the headline recall is based on very small positive test sets without confidence intervals. These limitations affect the strength of the central claim rather than the plausibility of the architecture-level ideas.","major_comments":[{"comment":"In Section 3.2, U is explicitly defined as part of the negative class, and in Section 3.3.2 the unsupervised real-sample loss is L_US_Real_D = alpha*CCE(x|C_y in K) + (1-alpha)*CCE(x|C_y in U). Every unlabeled real sample is therefore trained toward K or U, never directly toward the positive class R, so the model has no mechanism to recover a genuinely novel positive behavior that appears only in the unlabeled set. This is load-bearing because the paper frames its contribution as identifying scarce risky players and notes in Section 1 that newer samples redefine the positive distribution; if the claim is restricted to positives resembling the labeled prior, that closed-world assumption must be stated explicitly. The evaluation does not address this risk: the confirmed positives in Sets 1-3 appear to come from the same rule-engine/business flagging pipeline that generated the training positives, so high recall there may reflect imitation of the flagging rules rather than detection of new risk patterns. The SIUs are self-identified, but they are only seven samples.","section":"Section 3.2 and Section 3.3.2"},{"comment":"The quantitative basis for the headline recall is thin. The held-out positive test set has 52 samples, and Table 3 reports mean precision/recall over six runs with no standard deviations or confidence intervals. Table 4's 'over 85% recall' claim is based on denominators of 4, 9, 11, and 7 confirmed positives in Set 1, Set 2, Set 3, and SIUs, respectively; the text also says recall is 'between 85% to 90%' while Set 1 shows 3/4 = 75%. Please report per-run values, confidence intervals, and a paired comparison with vanilla SSGAN so that the stated improvements are statistically supported.","section":"Section 5.2, Tables 3 and 4"},{"comment":"The supervised positive-class loss includes a Percent Recall term (L_S+_D = CCE + %Recall+ve), which is not differentiable and is not accompanied by any explanation of how it enters gradient-based training (for example, as a reward, a validation criterion, or a soft proxy). Without this detail, the exact objective being optimized is unspecified and the ablation results cannot be precisely attributed. Please define the full training objective and the role of the recall term.","section":"Section 3.3.1"},{"comment":"The KDDCUP99 R2L comparison is not controlled. ScarceGAN is given 900 of 999 positive R2L samples as a labeled prior plus 30k labeled negatives, while the supervised baselines in Table 5 are presumably trained on the full KDD training labels under the standard split; the paper does not state that the baselines were re-run under the same label budget or that identical test partitions were used. The claim of a new R2L F1 benchmark (28) therefore needs a controlled comparison. The U2R row, with 52 positives in the entire training set, should not be used to draw conclusions from an F1 difference between 0 and 2 without intervals.","section":"Section 5.4.1, Table 5"},{"comment":"The central new mechanism depends on free parameters that are chosen rather than derived: alpha (0.6-0.7), epsilon (0.75), the definition of the three negative classes D/N/H, and the ten Prophet hyperparameters. No sensitivity analysis is reported for any of these choices, even though alpha directly controls the K/U split and is the key design parameter. Please add sweeps over alpha and epsilon, justify the negative-class definitions with evidence of their temporal stability in the longitudinal setting, and include an ablation comparing the Prophet features with standard longitudinal aggregates inside ScarceGAN itself.","section":"Section 3.2 and Section 5.3.2"}],"minor_comments":[{"comment":"The text says 'Results for Set 1, 2, 3 and unblocked, SIU's are presented in Figure 4', but these results appear in Table 4; the figure/table cross-reference should be corrected.","section":"Section 5.3.1"},{"comment":"The generator loss contains an undefined symbol '∦' and the second term's probability/indicator is not fully specified; please provide a clean equation with explicit expectations and the implementation used in the code.","section":"Section 3.4"},{"comment":"Section 5.1 says 'about 500 samples per negative class' while Section 5.2 describes 500 each with 350 train/150 held-out; the exact counts, and whether the 52 positive held-out samples are all confirmed risky, need to be stated consistently.","section":"Section 5.1 and Section 5.2"},{"comment":"The text uses 'compliment' where 'complement' is intended; please proofread for terminology.","section":"Introduction and Section 2"},{"comment":"The row 'ScarceGAN 2 class' is ambiguous; rename it consistently with Section 5.3.2 ('ScarceGAN with binary predictions') and define all abbreviations in the caption.","section":"Table 3"},{"comment":"Figures 1 and 2 have very short captions; adding axis labels, legends, and a sentence describing each panel would make the qualitative motivation checks usable.","section":"Figures 1 and 2"}],"recommendation":"major_revision","confidential_remarks":"The central architectural ideas are plausible and the ablation structure is a strength, but the manuscript overclaims novel-positive detection without an explicit closed-world assumption, and the headline quantitative results lack confidence intervals and controlled comparisons. In revision, the authors should be required to scope all novelty claims to the closed-world setting, add sensitivity analyses for alpha and epsilon, and provide per-run results or confidence intervals for the small positive test sets."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper to know about: ScarceGAN, a semi-supervised GAN for extremely rare class detection with weak labels. The headline is that the leeway class—an alpha-weighted soft assignment of ambiguous negatives to an 'unknown negative' bucket in both supervised and unsupervised paths—is a genuinely new arrangement of known parts, and the ablations show it works. Vanilla SSGAN gets 34% precision/24% recall on the held-out set; ScarceGAN gets 75.7%/72.5%. That is a real jump, and the KDDCUP99 R2L result (F1 28 on a 0.09% class) is a reasonable new benchmark.\n\nThe ideas are assembled rather than invented from scratch: the divided negative classes (D/N/H) reduce effective imbalance, the 'bad generator' comes from Dai et al., and the Prophet-derived 'third-order statistics' are a sensible way to encode longitudinal patterns. The paper is honest about its component origins, and the ablation isolating the leeway term and the binary-class variant supports the claim that each piece matters.\n\nNow the soft spots, in proportion. The stress-test note is on target: U is defined as negative in both paths, and the unsupervised loss pushes every real sample toward K or U. Nothing ever pushes an unlabeled sample toward the positive class, so a genuinely new risky behavior in the unlabeled set will be absorbed into U and missed. The paper's problem statement anticipates unseen behavior in the negative class but never states the corollary: it assumes no novel positives. For a stable domain like 'sustained extreme play' that may be fine, but it is a closed-world assumption that should be explicit. The evaluation compounds the issue: the confirmed positives in Sets 1/2/3 and SIUs come from the same rule-engine/business pipeline that produced the labeled prior, so high recall partly reflects mimicry. The test sets are tiny—52 held-out positives, then 4, 9, 11, and 7—so the 85-90% recall numbers carry large error bars with no confidence intervals. Alpha and epsilon are hand-tuned on the data, though the held-out evaluation gives some protection. The GitHub link is provided, but we did not attempt to run it, so reproducibility is unverified.\n\nIf read as a domain-specific recipe for detecting known risky behavior from weak negative labels, it is a solid, useful contribution. As a general method for discovering novel rare positives in evolving telemetry, it is not supported. KDDCUP99 results are encouraging but the setup uses 900 of 999 R2L positives as labeled prior, which is not the weak-prior scenario the paper emphasizes.\n\nWho gets value: practitioners in fraud, responsible gaming, intrusion detection who want a working combination of bad-generator SSGAN with a noise-absorbing negative class. Researchers in PU learning should be aware of the closed-world positive limitation. I'd bring it to a reading group but probably not cite it soon.\n\nDesk recommendation: yes, a serious referee would find enough substance to engage with, and the reviewer should push for a stated closed-world assumption, a temporally shifted positive evaluation, and error bars. The core idea is reasonable; the evaluation needs strengthening.","headline":"ScarceGAN's leeway class is a genuinely useful trick for weak negative labels, but its recall claims rest on an unstated closed-world positive assumption and an evaluation tied to the rule engine that produced the labels.","tokens_in":17801,"tokens_out":6859,"would_cite":false,"duration_ms":52962,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ScarceGAN claims that adding an 'unknown' leeway class and splitting negatives into several classes lets a semi-supervised GAN identify extremely rare positives in weakly labeled longitudinal data, reaching over 85% recall on risky…","keywords":["ScarceGAN","rare class identification","semi-supervised GAN","leeway class","positive-unlabelled learning","longitudinal telemetry","weak label prior","KDDCUP99"],"falsifier":"Take a longitudinal dataset with a scarce positive class and weak negative labels, deliberately hold back every sample of a newly introduced positive behavior (one that resembles known negatives in feature space), feed those samples to the model only as unlabeled real data, and measure recall on them. If ScarceGAN sends them to U or to a negative class instead of flagging them as positive, the central claim is falsified.","tokens_in":16735,"feed_emoji":"🎯","tokens_out":7500,"duration_ms":70614,"temperature":0.7,"pith_summary":"ScarceGAN addresses a practical puzzle: how to find extremely rare positive events (risky players, rare network attacks) when the labeled prior is tiny, noisy, and almost entirely negative. The paper claims that the right move is to stop forcing every unlabeled sample into a known negative class. Instead, the discriminator gains an explicit 'unknown' leeway class, so uncertain-but-definitely-not-positive samples have somewhere to go, and the remaining scarcity of the positive class is handled by splitting known negatives into several classes and training a deliberately 'bad' generator. If the reported results hold, this makes rare-class identification feasible in longitudinal telemetry with only weak priors: over 85% recall on risky skill-gaming players, roughly a 60-point jump over vanilla semi-supervised GAN, plus a new F1 benchmark of 28 for the 0.09%-imbalance R2L attack class in KDDCUP99.","feed_headline":"Rare-class recall jumps from 24% to 85% with a leeway class","feed_subtitle":"ScarceGAN's unknown class absorbs uncertain negatives, rescuing scarce positives in weakly labeled telemetry.","key_machinery":"The load-bearing object is the leeway class U, an extra output of the discriminator that absorbs samples the classifier cannot confidently assign to a known negative class but is sure are not positive. Around U, ScarceGAN builds three coordinated mechanisms: divided focus (splitting negatives into D, N, and H to reduce the effective imbalance), a dual cross-entropy supervised loss that mixes $\\alpha$-weighted known-class loss with $(1-\\alpha)$-weighted U-loss, and a complementary ('bad') generator whose low-density, feature-matched samples sharpen decision boundaries. The claimed effect is that U acts as a release valve for label noise and weak priors, letting the discriminator preserve a high-recall positive boundary without overfitting to noisy negative labels.","core_discovery":"On its own terms, the paper's discovery is that a semi-supervised GAN can be re-targeted from 'classify every sample into a known class' to 'classify confidently, and park the rest in a leeway class.' ScarceGAN defines five supervised outputs: three negative classes (dormant, normal, heavy), one positive class (risky), and an unknown negative class U, and it assigns every labeled negative sample a weighted loss that is partly categorical cross-entropy on its stated class and partly categorical cross-entropy on U, with weight $\\alpha$ typically 0.6–0.7. The unsupervised path makes the same split between 'known' and 'unknown' for real unlabeled samples, plus a fake class for generator outputs. The generator is trained to be complementary, producing samples that sit near decision boundaries rather than mimicking the data, which forces the discriminator to refine its boundaries. Together these changes are claimed to lift positive-class precision/recall from 34%/24% on the held-out gaming set to 75.7%/72.5%, and to transfer unchanged to KDDCUP99, where ScarceGAN reports F1 = 28 for R2L (0.09% imbalance) and recall 0.99 when the 'normal' class is the target.","pith_inferences":["Editorial inference: the leeway-class trick should transfer to any scarce-positive setting with noisy negatives—fraud, faults, medical screening—because it only requires the model to say 'not positive but unknown which negative,' which is a weaker and often more honest supervisory signal.","Editorial inference: the paper's alpha value encodes an assumed 30–40% negative-label noise rate; a direct extension would estimate noise per class and adapt alpha, turning a hand-set hyperparameter into a measured quantity.","Editorial inference: because U absorbs every uncertain real sample, the framework's safety depends on the closed-world assumption on positives; a practical deployment should monitor the U class for clusters, since a dense cluster of U could signal a new positive behavior the prior never saw.","Editorial inference: the third-order Prophet features are presented as helpful but not required; an ablation feeding raw longitudinal aggregates into ScarceGAN (which the paper says is possible) would separate the value of the classifier from the value of the feature engineering, and the same comparison could be run on KDDCUP99's raw features."],"forward_implications":["If ScarceGAN's results hold, the leeway mechanism gives a practical template for rare-class detection under weak priors: rather than more positive data, the gain comes from restructuring how negatives are scored.","The held-out gaming evaluation implies that the same network, without re-engineering, can serve as a first-pass filter that flags roughly 1–6% of daily platform traffic while catching 85–90% of confirmed risky players.","The KDDCUP99 result implies that ScarceGAN can beat supervised baselines on a 0.09%-imbalance attack class (R2L, F1 = 28), where conventional classifiers score in single digits.","The ablations imply that both the leeway class and the divided negative classes are individually load-bearing: removing U drops precision to 40.5%, and collapsing to binary negatives drops recall to 60%.","The complementary generator contributes about 10 points of precision/recall over the same architecture without it, implying that generator quality and classifier quality are not in tension here—a deliberately bad generator helps."],"supporting_citations":[{"why":"Defines the GAN min-max game that ScarceGAN re-formulates for the rare-class setting.","marker":"[21]"},{"why":"Supplies the early semi-supervised GAN dual-path discriminator architecture that ScarceGAN modifies.","marker":"[43, 47]"},{"why":"Provides the complementary 'bad generator' objective that ScarceGAN adopts to sharpen decision boundaries.","marker":"[13]"},{"why":"Supplies feature matching and GAN stabilization techniques used in the ScarceGAN generator and discriminator.","marker":"[48]"},{"why":"GenPU is the positive-unlabelled GAN baseline that motivates optimizing the discriminator rather than the generator.","marker":"[25]"},{"why":"CatGAN gives the categorical semi-supervised GAN formulation that ScarceGAN extends to multiple negative classes and a leeway class.","marker":"[52]"},{"why":"Prophet supplies the time-series hyper-parameters used as third-order longitudinal features in the gaming evaluation.","marker":"[55]"},{"why":"Provides the KDDCUP99 rare-class benchmark table that ScarceGAN compares against for R2L and U2R.","marker":"[15]"},{"why":"ALAD is the adversarial anomaly-detection baseline whose recall and F1 ScarceGAN outperforms on KDDCUP99.","marker":"[59]"},{"why":"Supplies the KDDCUP99 intrusion dataset on which the R2L and normal-class results are measured.","marker":"[29]"}],"fun_headline_variants":["Leeway class turns 24% recall into 85% for rare class","GAN's leeway class lifts rare-class recall from 24% to 85%","Unknown class absorbs uncertain negatives, boosting rare recall to 85%","Semi-supervised GAN: leeway class rescues rare positives, recall 85%","Leeway class: from 24% to 85% rare recall"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument assumes the unlabeled pool contains no genuinely new positive (risky) behaviors: every real sample is split between known classes and the leeway class U, and U is always treated as negative, so a novel risky pattern that appears only in the unlabeled set would be absorbed into U and missed.","fun_headline_variants_meta":{"raw":{"variants":["Leeway class turns 24% recall into 85% for rare class","GAN's leeway class lifts rare-class recall from 24% to 85%","Unknown class absorbs uncertain negatives, boosting rare recall to 85%","Semi-supervised GAN: leeway class rescues rare positives, recall 85%","Leeway class: from 24% to 85% rare recall"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001224,"raw_usage":{"total_tokens":5132,"prompt_tokens":1146,"completion_tokens":3986,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":762,"completion_tokens_details":{"reasoning_tokens":3882}},"tokens_in":762,"tokens_out":3986,"duration_ms":27915,"temperature":1.0,"reasoning_tokens":3882,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:23:26.514663+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a longitudinal dataset with a scarce positive class and weak negative labels, deliberately hold back every sample of a newly introduced positive behavior (one that resembles known negatives in feature space), feed those samples to the model only as unlabeled real data, and measure recall on them. If ScarceGAN sends them to U or to a negative class instead of flagging them as positive, the central claim is falsified.","supporting_citations":[{"cited_title":"Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde- Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio","cited_arxiv_id":null,"evidence_quote":"Defines the GAN min-max game that ScarceGAN re-formulates for the rare-class setting."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the complementary 'bad generator' objective that ScarceGAN adopts to sharpen decision boundaries."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies feature matching and GAN stabilization techniques used in the ScarceGAN generator and discriminator."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"GenPU is the positive-unlabelled GAN baseline that motivates optimizing the discriminator rather than the generator."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"CatGAN gives the categorical semi-supervised GAN formulation that ScarceGAN extends to multiple negative classes and a leeway class."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the KDDCUP99 rare-class benchmark table that ScarceGAN compares against for R2L and U2R."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the KDDCUP99 intrusion dataset on which the R2L and normal-class results are measured."}],"review_version":1}