{"id":"1b572cb5-e71b-45ff-be07-9f619fee0861","arxiv_id":"2412.07437","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Balancing after the train-test split avoids leakage-based inflation in a credit card fraud model, but the paper's evidence for the inflation effect is borrowed from other studies rather than measured in a matched experiment.","lead":"Credit card fraud data is so one-sided that models need extra balancing, and this paper asks whether balancing the data before splitting it into training and test sets unfairly inflates performance. It reports that a popular tree-based model still scores high when balancing is done only on training data, but the paper's main comparison is weakened by missing and mismatched experiments.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central comparison is not measured: the pre-split results in Table 1 are copied from another paper, so the claimed inflation from pre-split sampling rests on an uncontrolled cross-study comparison.","rationale":"The reader's weakest_assumption is exactly the load-bearing concern: the paper never runs its own pre-split sampling condition, so the central comparison is between the author's post-split results and external pre-split results that are not matched in models, hyperparameters, splits, or evaluation protocols. I agree with the REJECT verdict. The qualitative claim that pre-split sampling can cause leakage is already established in the cited literature, but the abstract's specific comparative finding is unmeasured here. In addition, I noted a secondary leakage risk in Appendix C: StandardScaler is fit on the full dataset before any split is shown, which would undercut the paper's claim that its own post-split evaluation is leak-free. No formal verification, released full repository, or repeated-trial variance is provided to offset these gaps. Therefore the reader's verdict should remain REJECT; my read does not change it.","tokens_in":10751,"tokens_out":4065,"duration_ms":41389,"concrete_test":"Run one controlled experiment using the Kaggle dataset, the Table 4 baseline XGBoost parameters, and a fixed stratified split: (i) no sampling, (ii) SMOTE applied after the split, and (iii) SMOTE applied before the split. Repeat over 5 random seeds and report mean and standard deviation for precision, recall, and F1. If condition (iii) does not reproduce near-perfect scores comparable to Table 1 while condition (ii) matches Table 3's SMOTE row, the paper's causal attribution is unsupported. As a secondary check, rerun the Appendix C preprocessing with the scaler fitted only on the training fold to see whether the full-data scaling changes the reported post-split results.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's central claim is that pre-split sampling inflates XGBoost performance metrics while post-split sampling produces lower but valid results. The paper's own experiments, Section 9.2 and Table 3, only implement post-split sampling. The pre-split condition appears in Table 1 with near-perfect metrics (99.969% F1), but the text explicitly cites 'Table 1[2]': these numbers are taken from Qasim et al. [2] rather than measured here. Tables 5 and 6 are likewise external results from Mohbey [20] and Chogugudza [10] using different models and protocols. The paper then attributes the gap between Table 1's 99.969% F1 and its own post-split SMOTE row's 92.56% F1 to sampling timing, without controlling for different hyperparameters, data splits, evaluation thresholds, or even dataset versions. This is the load-bearing comparison for the abstract's causal claim, and it is not tested. A secondary internal issue compounds the problem: Appendix C calls fit_transform on Amount and Time over the full dataset before any train-test split is shown, which is itself a potential leakage path and undermines the claim that the post-split results preserve evaluation integrity.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper claims to compare XGBoost performance for credit card fraud detection under three scenarios: no imbalance handling, sampling applied after the train-test split, and sampling applied before the train-test split. It reports its own post-split experiments on the Kaggle credit card dataset, imports pre-split XGBoost results from Qasim et al. [2], and imports additional XGBoost and Random Forest results from Mohbey et al. [20] and Chogugudza [10]. The abstract concludes that pre-split sampling artificially inflates metrics and that post-split sampling yields lower but evaluation-integrity-preserving results, with XGBoost remaining superior. The paper includes code appendices for SMOTE, CTGAN, and preprocessing.","tokens_in":10880,"tokens_out":4237,"duration_ms":42468,"significance":"If the central claim were established, the paper would provide a useful, concrete caution about a common methodological pitfall in fraud-detection benchmarks. The paper does articulate a relevant problem and provides code-level detail for several pipelines, which is a useful start. However, the load-bearing comparison is not actually run: the pre-split condition is imported from another study, the comparability of external results is unverified, and the preprocessing code in Appendix C itself contains a potential leakage path. As a result, the significance of the empirical findings is currently low: the paper does not demonstrate the inflation effect it claims to characterize, and it does not provide a controlled comparison supporting its \"XGBoost remains superior\" conclusion.","major_comments":[{"comment":"The pre-split sampling condition, which is central to the abstract's three-scenario comparison, is not implemented in this paper. The text states \"Table 1[2]\" and the table's near-perfect values are taken from Qasim et al. [2], not from the authors' experiments. Section 9.2 and Table 3 contain only post-split experiments. Therefore the reported gap between Table 1's 99.969% F1-score and Table 3's 92.56% F1-score cannot be attributed to sampling timing: it may reflect differences in hyperparameters, data splits, evaluation thresholds, dataset preprocessing, or the specific dataset version. To support the abstract's causal claim, the authors must run their own pre-split sampling condition under the same protocol, or explicitly reframe the conclusion as a cross-study observation.","section":"Section 9.1, Table 1"},{"comment":"The preprocessing code shown in Appendix C calls StandardScaler().fit_transform on the Amount and Time columns of the full data object (lines 1-6) before any train-test split is shown. If this code reflects the actual experimental protocol, the scaler statistics incorporate test-set information, which is itself a form of data leakage independent of sampling. This undermines the claim in Section 9.2 that the post-split results \"preserve the integrity of the evaluation process.\" The authors must either demonstrate that the split occurs before any fit_transform is applied, or revise the code and rerun the experiments.","section":"Appendix C"},{"comment":"The comparison with external results is uncontrolled. Table 5 reproduces an XGBoost result from Mohbey et al. [20] and Table 6 reproduces a Random Forest result from Chogugudza [10], with no verification that the models, hyperparameters, data splits, evaluation metrics, or dataset preprocessing are commensurable with the authors' own runs. The concluding sentence \"From these results it is now evident that the xgboost model is still superior even without the risk of data leakage\" is not supported by the presented evidence. Additionally, Table 3 lacks essential experimental detail: the train-test split ratio, the number of independent runs, and any measure of variance are not reported, and Table 4 lists only a subset of hyperparameters, omitting the sampling_strategy values for SMOTE and RUS used in the main experiments (those appear only in one pipeline in Appendix A). Without this information, the metric differences in Table 3 could be within run-to-run noise.","section":"Section 9.2, Tables 5 and 6"}],"minor_comments":[{"comment":"There are duplicate references: [7] and [8] are the same SMOTE paper, [14] and [15] are identical Hashemi et al. entries, and [19] and [20] are the same Mohbey et al. entry. These should be consolidated.","section":"References"},{"comment":"The text reads \"quite descent results\" and should read \"quite decent results.\" Also, the name \"Chougugudza\" in the text does not match the spelling \"Chogugudza\" in reference [10].","section":"Section 9.2"},{"comment":"The citation \"Table 1[2]\" interleaves a citation with a table reference; the original source should be credited in the caption and its experimental settings (dataset version, hyperparameters, split procedure) should be described in the text so that readers can judge comparability.","section":"Section 9.1, Table 1"},{"comment":"The comment \"also consider u -> o\" is unclear, and the pipeline order is defined as over-sampling then under-sampling. The comment suggests the reverse order should be considered; please clarify the intended design and justify the chosen order.","section":"Appendix A, Listing 1"}],"recommendation":"reject","confidential_remarks":"The central comparison promised in the abstract and introduction is not executed: the pre-split condition is taken from another paper, and the external comparisons are not controlled. The preprocessing code in Appendix C also appears to leak information through the scaler. This is not a case of minor gaps; the paper's main conclusion is unsupported by its own experiments. The topic itself is worthwhile, so the authors could potentially revise by running the full set of scenarios with a single protocol, but that would be a substantial rewrite rather than a minor revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper's headline claim is not supported by its own experiments. The abstract promises a three-scenario comparison (no sampling, post-split sampling, pre-split sampling). Only the post-split arm is actually run here. The pre-split numbers in Table 1 come from Qasim et al., and the no-sampling and Random Forest comparisons in Tables 5 and 6 come from other papers with different models and protocols. The author then attributes the gap between Table 1's 99.969% F1 and its own post-split SMOTE's 92.56% F1 to sampling timing, without any evidence that the external results would be comparable if rerun with the same hyperparameters, split ratio, or evaluation code. That is the load-bearing comparison for the abstract's causal claim, and it is not tested.\n\nTo give credit where it is due: the paper is readable, and it correctly identifies a real practical pitfall. The post-split XGBoost benchmark (Table 3) is a reasonable sanity check, and the author is transparent about attributing the copied tables to their sources. The qualitative lesson — pre-split sampling leaks test information — is well established in the cited literature, so the paper's contribution is at best a confirmation, not a new finding.\n\nThe soft spots are serious. Besides the central missing experiment, there are no error bars or repeated trials anywhere, so we cannot judge stability. Hyperparameters are listed for the baseline, but the split ratio is not stated. Worse, Appendix C fits StandardScaler on the full dataset before any train-test split, which is itself a leakage path. That directly contradicts the paper's claim that its post-split results preserve evaluation integrity. The stress-test note is right about this.\n\nWho is this for? A practitioner who has never thought about leakage might read it and get the right high-level message. But as a research paper, it does not provide the controlled comparison it promises. The fix is straightforward: run the same XGBoost configuration under all three conditions, report variance across repeated splits, and release the full notebook. Until then, the central claim is not measured.\n\nMy recommendation: do not send this to peer review in its current form. It would be a desk reject. If the author were willing to do the matched experiment and clean up the preprocessing leak, it could become a modest but useful confirmation paper.","headline":"The abstract promises a three-scenario comparison, but only the post-split arm is actually measured; the pre-split numbers are imported from other papers, so the central claim about inflated metrics rests on an uncontrolled cross-study comparison.","tokens_in":11509,"tokens_out":1819,"would_cite":false,"duration_ms":19514,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that pre-split sampling in credit card fraud detection inflates XGBoost's reported performance through data leakage, and that post-split sampling yields lower but trustworthy metrics without sacrificing the model's…","keywords":["credit card fraud detection","XGBoost","class imbalance","data leakage","SMOTE","sampling techniques","train-test split","imbalanced learning"],"falsifier":"Run the same XGBoost configuration on the same Kaggle credit-card data twice, once applying SMOTE before the train-test split and once applying it only to the training set, then compare metrics on an identical held-out test set. If the pre-split model's test scores are not substantially higher than the post-split model's, or if the post-split model's scores match the pre-split ones, the leakage-inflation claim would be falsified.","tokens_in":10382,"feed_emoji":"💳","tokens_out":4573,"duration_ms":43232,"temperature":0.7,"pith_summary":"Credit card fraud detection models are often trained on heavily imbalanced data, and sampling techniques like SMOTE, random over-sampling, and CGAN are used to balance it. The paper's central claim is that if these sampling techniques are applied before the train-test split, the model's reported performance is artificially inflated because the test set has leaked into the training process. Using the standard Kaggle credit card fraud dataset of 284,807 transactions with 0.172% fraud, the paper compares pre-split results from other published studies with its own post-split experiments on XGBoost. It finds that pre-split models reach near-perfect metrics, while post-split models score lower but honestly, and argues that XGBoost remains a strong fraud detector under leak-proof evaluation. This matters because many published benchmarks may overstate real-world performance, and the paper's comparison offers a cautionary template for how evaluation integrity should be preserved.","feed_headline":"Pre-split sampling inflates fraud-detection scores","feed_subtitle":"When balancing leaks test data, XGBoost scores inflate; post-split sampling gives lower, honest numbers.","key_machinery":"The mechanism carrying the argument is the ordering of resampling relative to the train-test split. When sampling precedes the split, synthetic or duplicated minority-class examples are generated from the whole dataset, so test-set information leaks into training and inflates evaluation metrics; when sampling is applied only to the training portion after the split, the test set remains a true holdout. The paper uses this contrast as its analytical lens, pulling pre-split results from other studies for comparison against its own post-split XGBoost runs.","core_discovery":"The central claim is that applying sampling techniques before the train-test split creates data leakage and produces artificially inflated performance metrics for XGBoost in credit card fraud detection. The paper contrasts external pre-split results, such as an XGBoost model with SMOTE scoring 99.969% accuracy, 100% recall, and 99.969% AUC, with its own experiments where sampling is applied only to the training set after the split; those honest models achieve lower but credible scores, with the best configuration reaching 95.00% F1-score using CGAN augmentation and cost-sensitive learning. The conclusion is that XGBoost retains high detection performance without the inflation, so pre-split sampling is both unnecessary and a threat to evaluation validity.","pith_inferences":["A controlled head-to-head experiment with identical data and hyperparameters, changing only the sampling order, would convert the paper's cross-study comparison into a direct test of the leakage effect; the paper itself does not run its own pre-split condition.","The size of the gap between pre-split and post-split scores is likely an upper bound on pure leakage effects, because the compared studies also differ in model configuration, preprocessing, and evaluation protocols.","The same split-order caution generalizes to other heavily imbalanced domains such as medical diagnostics or network intrusion detection, where pre-split resampling carries the same inflation risk.","One could isolate the leakage mechanism by training a model on a dataset where synthetic minority samples are generated from the full data but evaluated on a clean held-out set; if that reproduces the inflated scores, leakage is confirmed as the cause."],"forward_implications":["Published fraud-detection benchmarks that balanced data before splitting likely overstate how well the models will perform in real deployment.","Practitioners should treat reported metrics as trustworthy only when sampling was performed after the split, ideally inside each cross-validation fold.","XGBoost maintains strong performance under honest evaluation, with F1-scores around 94 to 95 percent and accuracy above 99.9 percent on the tested dataset.","Evaluation protocols for any imbalanced classification problem should mandate resampling after the split to avoid leakage-driven inflation.","Hybrid approaches that combine synthetic data generation with cost-sensitive learning appear to be a viable path that improves sensitivity without compromising validity."],"supporting_citations":[{"why":"supplies the pre-split SMOTE XGBoost results in Table 1 that the paper uses as the leakage-inflated baseline.","marker":"[2]"},{"why":"supplies the AE-XGB-SMOTE-CGAN results in Table 2, another pre-split configuration the paper flags as leakage-prone.","marker":"[13]"},{"why":"provides the no-imbalance-handling XGBoost comparison in Table 5 used to show XGBoost performs well even without balancing.","marker":"[20]"},{"why":"provides the Random Forest with pre-split SMOTE results in Table 6 used as a comparison point for the honest post-split XGBoost models.","marker":"[10]"},{"why":"supplies the Kaggle credit card fraud dataset that all of the paper's own experiments run on.","marker":"[22]"},{"why":"provides the empirical grounding that data leakage inflates prediction performance, supporting the paper's central mechanism.","marker":"[23]"},{"why":"defines SMOTE, the primary oversampling technique used in the paper's post-split pipelines.","marker":"[8]"}],"fun_headline_variants":["Sampling before split inflates fraud scores","Leaky sampling exaggerates XGBoost fraud metrics","Post-split sampling reveals honest fraud detection","Data leakage pumps up XGBoost fraud results"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison only works if the external pre-split results and the author's own post-split runs are directly comparable, even though they use different models, hyperparameters, feature engineering, and evaluation protocols.","fun_headline_variants_meta":{"raw":{"variants":["Sampling before split inflates fraud scores","Leaky sampling exaggerates XGBoost fraud metrics","Post-split sampling reveals honest fraud detection","Data leakage pumps up XGBoost fraud results"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000213,"raw_usage":{"total_tokens":1420,"prompt_tokens":943,"completion_tokens":477,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":559,"completion_tokens_details":{"reasoning_tokens":417}},"tokens_in":559,"tokens_out":477,"duration_ms":5353,"temperature":1.0,"reasoning_tokens":417,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:50:56.380247+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same XGBoost configuration on the same Kaggle credit-card data twice, once applying SMOTE before the train-test split and once applying it only to the training set, then compare metrics on an identical held-out test set. If the pre-split model's test scores are not substantially higher than the post-split model's, or if the post-split model's scores match the pre-split ones, the leakage-inflation claim would be falsified.","supporting_citations":[{"cited_title":"Credit Card Fraud Detection Using XGBoost Algorithm","cited_arxiv_id":null,"evidence_quote":"supplies the pre-split SMOTE XGBoost results in Table 1 that the paper uses as the leakage-inflated baseline."},{"cited_title":"A novel method for detecting credit card fraud problems","cited_arxiv_id":null,"evidence_quote":"supplies the AE-XGB-SMOTE-CGAN results in Table 2, another pre-split configuration the paper flags as leakage-prone."},{"cited_title":"Credit-Card-Fraud-Prediction- Using-XGBoost -An-Ensemble-Learning-Approach","cited_arxiv_id":null,"evidence_quote":"provides the no-imbalance-handling XGBoost comparison in Table 5 used to show XGBoost performs well even without balancing."},{"cited_title":"The classification performance of ensemble decision tree classifiers: a case study of detecting fraud in credit card transactions","cited_arxiv_id":null,"evidence_quote":"provides the Random Forest with pre-split SMOTE results in Table 6 used as a comparison point for the honest post-split XGBoost models."},{"cited_title":"Credit Card Fraud Detection","cited_arxiv_id":null,"evidence_quote":"supplies the Kaggle credit card fraud dataset that all of the paper's own experiments run on."},{"cited_title":"SMOTE: Synthetic Minority Over-sampling Technique","cited_arxiv_id":null,"evidence_quote":"defines SMOTE, the primary oversampling technique used in the paper's post-split pipelines."}],"review_version":1}