Pith. sign in

REVIEW 3 major objections 4 minor 25 references

Impact of Sampling Techniques and Data Leakage on XGBoost Performance in Credit Card Fraud Detection

T0 review · 3 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Pith's one-line read This paper argues that pre-split sampling in credit card fraud detection inflates XGBoost's reported performance through data leakage, and that post-split sampling yields lower but trustworthy metrics without sacrificing the model's…

desk verdict The abstract promises a three-scenario comparison, but only the post-split arm is actually measured; the pre-split numbers are imported from other papers, so the central claim about inflated metrics rests on an uncontrolled cross-study comparison. read the letter →

arxiv 2412.07437 v1 pith:GDLMFF4R submitted 2024-12-10 cs.LG

classification cs.LG
keywords creditcardfrauddetectionXGBoostclassimbalancedataleakageSMOTEsamplingtechniquestrain-testsplitimbalancedlearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Credit card fraud detection models are often trained on heavily imbalanced data, and sampling techniques like SMOTE, random over-sampling, and CGAN are used to balance it. The paper's central claim is that if these sampling techniques are applied before the train-test split, the model's reported performance is artificially inflated because the test set has leaked into the training process. Using the standard Kaggle credit card fraud dataset of 284,807 transactions with 0.172% fraud, the paper compares pre-split results from other published studies with its own post-split experiments on XGBoost. It finds that pre-split models reach near-perfect metrics, while post-split models score lower but honestly, and argues that XGBoost remains a strong fraud detector under leak-proof evaluation. This matters because many published benchmarks may overstate real-world performance, and the paper's comparison offers a cautionary template for how evaluation integrity should be preserved.

What carries the argument

The mechanism carrying the argument is the ordering of resampling relative to the train-test split. When sampling precedes the split, synthetic or duplicated minority-class examples are generated from the whole dataset, so test-set information leaks into training and inflates evaluation metrics; when sampling is applied only to the training portion after the split, the test set remains a true holdout. The paper uses this contrast as its analytical lens, pulling pre-split results from other studies for comparison against its own post-split XGBoost runs.

What would settle it

Run the same XGBoost configuration on the same Kaggle credit-card data twice, once applying SMOTE before the train-test split and once applying it only to the training set, then compare metrics on an identical held-out test set. If the pre-split model's test scores are not substantially higher than the post-split model's, or if the post-split model's scores match the pre-split ones, the leakage-inflation claim would be falsified.

Watch

Extended reading notes

Core claim

The central claim is that applying sampling techniques before the train-test split creates data leakage and produces artificially inflated performance metrics for XGBoost in credit card fraud detection. The paper contrasts external pre-split results, such as an XGBoost model with SMOTE scoring 99.969% accuracy, 100% recall, and 99.969% AUC, with its own experiments where sampling is applied only to the training set after the split; those honest models achieve lower but credible scores, with the best configuration reaching 95.00% F1-score using CGAN augmentation and cost-sensitive learning. The conclusion is that XGBoost retains high detection performance without the inflation, so pre-split sampling is both unnecessary and a threat to evaluation validity.

Load-bearing premise

The comparison only works if the external pre-split results and the author's own post-split runs are directly comparable, even though they use different models, hyperparameters, feature engineering, and evaluation protocols.

Editorial extensions

If this is right

  • Published fraud-detection benchmarks that balanced data before splitting likely overstate how well the models will perform in real deployment.
  • Practitioners should treat reported metrics as trustworthy only when sampling was performed after the split, ideally inside each cross-validation fold.
  • XGBoost maintains strong performance under honest evaluation, with F1-scores around 94 to 95 percent and accuracy above 99.9 percent on the tested dataset.
  • Evaluation protocols for any imbalanced classification problem should mandate resampling after the split to avoid leakage-driven inflation.
  • Hybrid approaches that combine synthetic data generation with cost-sensitive learning appear to be a viable path that improves sensitivity without compromising validity.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A controlled head-to-head experiment with identical data and hyperparameters, changing only the sampling order, would convert the paper's cross-study comparison into a direct test of the leakage effect; the paper itself does not run its own pre-split condition.
  • The size of the gap between pre-split and post-split scores is likely an upper bound on pure leakage effects, because the compared studies also differ in model configuration, preprocessing, and evaluation protocols.
  • The same split-order caution generalizes to other heavily imbalanced domains such as medical diagnostics or network intrusion detection, where pre-split resampling carries the same inflation risk.
  • One could isolate the leakage mechanism by training a model on a dataset where synthetic minority samples are generated from the full data but evaluated on a clean held-out set; if that reproduces the inflated scores, leakage is confirmed as the cause.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper claims to compare XGBoost performance for credit card fraud detection under three scenarios: no imbalance handling, sampling applied after the train-test split, and sampling applied before the train-test split. It reports its own post-split experiments on the Kaggle credit card dataset, imports pre-split XGBoost results from Qasim et al. [2], and imports additional XGBoost and Random Forest results from Mohbey et al. [20] and Chogugudza [10]. The abstract concludes that pre-split sampling artificially inflates metrics and that post-split sampling yields lower but evaluation-integrity-preserving results, with XGBoost remaining superior. The paper includes code appendices for SMOTE, CTGAN, and preprocessing.

Significance. If the central claim were established, the paper would provide a useful, concrete caution about a common methodological pitfall in fraud-detection benchmarks. The paper does articulate a relevant problem and provides code-level detail for several pipelines, which is a useful start. However, the load-bearing comparison is not actually run: the pre-split condition is imported from another study, the comparability of external results is unverified, and the preprocessing code in Appendix C itself contains a potential leakage path. As a result, the significance of the empirical findings is currently low: the paper does not demonstrate the inflation effect it claims to characterize, and it does not provide a controlled comparison supporting its "XGBoost remains superior" conclusion.

major comments (3)
  1. [Section 9.1, Table 1] The pre-split sampling condition, which is central to the abstract's three-scenario comparison, is not implemented in this paper. The text states "Table 1[2]" and the table's near-perfect values are taken from Qasim et al. [2], not from the authors' experiments. Section 9.2 and Table 3 contain only post-split experiments. Therefore the reported gap between Table 1's 99.969% F1-score and Table 3's 92.56% F1-score cannot be attributed to sampling timing: it may reflect differences in hyperparameters, data splits, evaluation thresholds, dataset preprocessing, or the specific dataset version. To support the abstract's causal claim, the authors must run their own pre-split sampling condition under the same protocol, or explicitly reframe the conclusion as a cross-study observation.
  2. [Appendix C] The preprocessing code shown in Appendix C calls StandardScaler().fit_transform on the Amount and Time columns of the full data object (lines 1-6) before any train-test split is shown. If this code reflects the actual experimental protocol, the scaler statistics incorporate test-set information, which is itself a form of data leakage independent of sampling. This undermines the claim in Section 9.2 that the post-split results "preserve the integrity of the evaluation process." The authors must either demonstrate that the split occurs before any fit_transform is applied, or revise the code and rerun the experiments.
  3. [Section 9.2, Tables 5 and 6] The comparison with external results is uncontrolled. Table 5 reproduces an XGBoost result from Mohbey et al. [20] and Table 6 reproduces a Random Forest result from Chogugudza [10], with no verification that the models, hyperparameters, data splits, evaluation metrics, or dataset preprocessing are commensurable with the authors' own runs. The concluding sentence "From these results it is now evident that the xgboost model is still superior even without the risk of data leakage" is not supported by the presented evidence. Additionally, Table 3 lacks essential experimental detail: the train-test split ratio, the number of independent runs, and any measure of variance are not reported, and Table 4 lists only a subset of hyperparameters, omitting the sampling_strategy values for SMOTE and RUS used in the main experiments (those appear only in one pipeline in Appendix A). Without this information, the metric differences in Table 3 could be within run-to-run noise.
minor comments (4)
  1. [References] There are duplicate references: [7] and [8] are the same SMOTE paper, [14] and [15] are identical Hashemi et al. entries, and [19] and [20] are the same Mohbey et al. entry. These should be consolidated.
  2. [Section 9.2] The text reads "quite descent results" and should read "quite decent results." Also, the name "Chougugudza" in the text does not match the spelling "Chogugudza" in reference [10].
  3. [Section 9.1, Table 1] The citation "Table 1[2]" interleaves a citation with a table reference; the original source should be credited in the caption and its experimental settings (dataset version, hyperparameters, split procedure) should be described in the text so that readers can judge comparability.
  4. [Appendix A, Listing 1] The comment "also consider u -> o" is unclear, and the pipeline order is defined as over-sampling then under-sampling. The comment suggests the reverse order should be considered; please clarify the intended design and justify the chosen order.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's central comparison is empirical and does not reduce to its own inputs, though it rests on an uncontrolled cross-study comparison.

full rationale

The paper's argument is entirely empirical: it compares XGBoost performance under pre-split sampling, post-split sampling, and no sampling. The load-bearing pre-split results are not derived or fitted by the author; they are taken from another study, as the text states: "As shown in Table 1[2], the XGBoost model achieves near-perfect scores in all metrics." The post-split results come from the author's own experiments in Table 3. There is no equation that defines one result in terms of another, no parameter fitted to a target and then renamed as a prediction, and no self-citation chain supporting a uniqueness claim. The weakness that a reader might identify is that the pre-split and post-split numbers come from different papers, models, and protocols, so attributing the gap to sampling timing is not controlled. That is a validity or correctness concern, not circular reasoning. Even Appendix C's scaling of Amount and Time before an explicit split could constitute leakage, but it does not create a definitional loop between the paper's claims and its evidence. Accordingly, no circular step is present and the score is 0.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The central comparison depends on untested comparability of external tables and on unstated hyperparameter choices. There are no invented physical or theoretical entities; all moving parts are standard ML components.

free parameters (6)
  • learning_rate = 0.4
    Set by hand in Table 4; no tuning or sensitivity analysis is reported, yet it directly affects all post-split results.
  • n_estimators = 1000
    Fixed in Table 4; no convergence check or early stopping is reported.
  • SMOTE sampling_strategy = 0.8
    Chosen in Appendix A without justification; affects the amount of synthetic oversampling.
  • RandomUnderSampler sampling_strategy = 0.9
    Chosen in Appendix A without justification; affects the amount of majority-class removal.
  • scale_pos_weight = 577.27/10
    Used in the CGAN and cost-sensitive learning row of Table 3; the value is derived from the class ratio but divided by 10 with no stated reason.
  • CTGAN epochs = 300
    Set in Appendix B; no convergence check or evaluation of synthetic sample quality is reported.
assumptions (4)
  • domain assumption Pre-split sampling causes test-set information to enter training and inflates performance metrics.
    Invoked from refs [23,24] in Section 3.3 and used in Section 9.1 to interpret near-perfect external scores as leakage.
  • ad hoc to paper External results in Tables 1, 5, and 6 are comparable to the author's own experiments.
    Section 9.2 uses these external tables to draw the central conclusion about pre-split inflation and XGBoost superiority, with no matched protocol.
  • domain assumption The Kaggle credit card dataset's labels and PCA features are reliable ground truth.
    Section 6 treats the public dataset as given, without discussing label noise, drift over the two-day window, or feature semantics.
  • ad hoc to paper The fixed XGBoost hyperparameters in Table 4 represent a fair baseline for all compared techniques.
    Parameters are set by hand and no tuning or sensitivity analysis is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Impact of Sampling Techniques and Data Leakage on XGBoost Performance in Credit Card Fraud Detection." pith.science (2026). https://pith.science/paper/GDLMFF4R

@misc{pith2026241207437,
  author       = {Pith},
  title        = {Pith review of: Impact of Sampling Techniques and Data Leakage on XGBoost Performance in Credit Card Fraud Detection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GDLMFF4R}},
  note         = {Machine review of arXiv:2412.07437}
}
read the original abstract

Credit card fraud detection remains a critical challenge in financial security, with machine learning models like XGBoost(eXtreme gradient boosting) emerging as powerful tools for identifying fraudulent transactions. However, the inherent class imbalance in credit card transaction datasets poses significant challenges for model performance. Although sampling techniques are commonly used to address this imbalance, their implementation sometimes precedes the train-test split, potentially introducing data leakage. This study presents a comparative analysis of XGBoost's performance in credit card fraud detection under three scenarios: Firstly without any imbalance handling techniques, secondly with sampling techniques applied only to the training set after the train-test split, and third with sampling techniques applied before the train-test split. We utilized a dataset from Kaggle of 284,807 credit card transactions, containing 0.172\% fraudulent cases, to evaluate these approaches. Our findings show that although sampling strategies enhance model performance, the reliability of results is greatly impacted by when they are applied. Due to a data leakage issue that frequently occurs in machine learning models during the sampling phase, XGBoost models trained on data where sampling was applied prior to the train-test split may have displayed artificially inflated performance metrics. Surprisingly, models trained with sampling techniques applied solely to the training set demonstrated significantly lower results than those with pre-split sampling, all the while preserving the integrity of the evaluation process.

Figures

Figures reproduced from arXiv: 2412.07437 by the authors.

Figure 1
Figure 1. Class distribution of the credit card transactions dataset, showing a stark [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Boxplot of transaction amounts by class type, showing the distribution and [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Correlation heatmap of features in the dataset, showing relationships between [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Distribution of selected features in the dataset, comparing legitimate and fraud [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

25 extracted references · 20 canonical work pages

  1. [2]

    Credit Card Fraud Detection Using XGBoost Algorithm

    Ahmed Qasim Abdulghani, Osman Nuri UCAN, and Khattab M. Ali Alheeti. “Credit Card Fraud Detection Using XGBoost Algorithm”. In: 2021 14th Interna- tional Conference on Developments in eSystems Engineering (DeSE). 2021, pp. 487–

  2. [20]

    Credit-Card-Fraud-Prediction- Using-XGBoost -An-Ensemble-Learning-Approach

    K. Mohbey, Mohammad Khan, and Ajay Indian. “Credit-Card-Fraud-Prediction- Using-XGBoost -An-Ensemble-Learning-Approach”. In: International Journal of Information Retrieval Research 12 (July 2022). doi: 10.4018/IJIRR.299940

  3. [10]

    The classification performance of ensemble decision tree classifiers: a case study of detecting fraud in credit card transactions

    Mcdonald Chogugudza. “The classification performance of ensemble decision tree classifiers: a case study of detecting fraud in credit card transactions”. Identifier: vital:69317. Master’s thesis. University of Fort Hare, Nov. 2022. url: https : / / vital.seals.ac.za/vital/access/manager/Repository/vital:69317?site_ name=GlobalView

  4. [1]

    Fraud detection sys- tem: A survey

    Aisha Abdallah, Mohd Aizaini Maarof, and Anazida Zainal. “Fraud detection sys- tem: A survey”. In: Journal of Network and Computer Applications 68 (2016), pp. 90–113. issn: 1084-8045. doi: https : / / doi . org / 10 . 1016 / j . jnca . 2016 . 04 . 007. url: https : / / www . sciencedirect . com / science / article / pii / S1084804516300571

  5. [3]

    Survey of Credit Card Anomaly and Fraud Detection Using Sampling Techniques

    M. Alamri and M. Ykhlef. “Survey of Credit Card Anomaly and Fraud Detection Using Sampling Techniques”. In: Electronics 11 (2022), p. 4003

  6. [4]

    Data mining for credit card fraud: A comparative study

    S. Bhattacharyya et al. “Data mining for credit card fraud: A comparative study”. In: Decision Support Systems 50.3 (2011), pp. 602–613. doi: 10.1016/j.dss.2010. 08.008. 14

  7. [5]

    Random forests

    Leo Breiman. “Random forests”. In: Machine learning 45.1 (2001), pp. 5–32. doi: 10.1023/A:1010933404324

  8. [6]

    Lost, stolen or skimmed: Overcoming credit card fraud in South Africa

    Trevor Budhram. “Lost, stolen or skimmed: Overcoming credit card fraud in South Africa”. In: South African Crime Quarterly 40 (2016), pp. 31–37. doi: 10.17159/ 2413-3108/2012/V0I40A843

Show all 25 references
  1. [7]

    SMOTE: Synthetic Minority Over-sampling Technique

    Nitesh V Chawla et al. “SMOTE: Synthetic Minority Over-sampling Technique”. In: Journal of artificial intelligence research 16 (2002), pp. 321–357. doi: 10.1613/ jair.953

  2. [8]

    SMOTE: Synthetic Minority Over-sampling Technique

    Nitesh V. Chawla et al. “SMOTE: Synthetic Minority Over-sampling Technique”. In: Journal of Artificial Intelligence Research 16 (2002), pp. 321–357. doi: 10.1613/ jair.953

  3. [9]

    Xgboost: A scalable tree boosting system

    Tianqi Chen and Carlos Guestrin. “Xgboost: A scalable tree boosting system”. In: Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining . ACM. 2016, pp. 785–794. doi: 10.1145/2939672.2939785

  4. [11]

    Generating multi-label discrete patient records using generative adversarial networks

    E. Choi et al. “Generating multi-label discrete patient records using generative adversarial networks”. In: Proceedings of the Machine Learning for Healthcare Con- ference. 2017, pp. 286–305

  5. [12]

    Effective data generation for imbalanced learning using conditional generative adversarial networks

    Georgios Douzas and Fernando Bacao. “Effective data generation for imbalanced learning using conditional generative adversarial networks”. In: Expert Systems with Applications 91 (2018), pp. 464–471. issn: 0957-4174. doi: https://doi.org/10. 1016/j.eswa.2017.09.030 . url: http...

  6. [13]

    A novel method for detecting credit card fraud problems

    H. Du et al. “A novel method for detecting credit card fraud problems”. In: PLoS ONE 19.3 (2024), e0294537. doi: 10.1371/journal.pone.0294537 . url: https: //doi.org/10.1371/journal.pone.0294537

  7. [15]

    Fraud De- tection in Banking Data by Machine Learning Techniques

    Seyedeh Khadijeh Hashemi, Seyedeh Leili Mirtaheri, and Sergio Greco. “Fraud De- tection in Banking Data by Machine Learning Techniques”. In: IEEE Access 11 (2023), pp. 3034–3043. doi: 10.1109/ACCESS.2022.3232287

  8. [16]

    Fraud detection: Intention- ality and deception in cognition

    Paul E. Johnson, Stefano Grazioli, and Karim Jamal. “Fraud detection: Intention- ality and deception in cognition”. In: Accounting, Organizations and Society 18.5 (1993), pp. 467–488. issn: 0361-3682. doi: https : / / doi . org / 10 . 1016 / 0361 - 3682(93)90042- 5. url: https...

  9. [17]

    Keep it simple: random oversampling for imbalanced data

    Firuz Kamalov, Ho-Hon Leung, and Aswani Kumar Cherukuri. “Keep it simple: random oversampling for imbalanced data”. In: 2023 Advances in Science and En- gineering Technology International Conferences (ASET) . 2023, pp. 1–4. doi: 10. 1109/ASET56582.2023.10180891. 15

  10. [18]

    GAN-based imbalanced data intrusion detection system

    J. Lee and K. Park. “GAN-based imbalanced data intrusion detection system”. In: Personal and Ubiquitous Computing 25 (2021), pp. 121–128. doi: 10.1007/s00779- 019-01332-y

  11. [21]

    A Comparison Study of Credit Card Fraud Detection: Supervised versus Unsupervised

    Xuetong Niu, Li Wang, and Xulei Yang. A Comparison Study of Credit Card Fraud Detection: Supervised versus Unsupervised. 2019. arXiv: 1904.10604 [cs.LG]. url: https://arxiv.org/abs/1904.10604

  12. [22]

    Credit Card Fraud Detection

    Andrea Dal Pozzolo et al. Credit Card Fraud Detection. https://www.kaggle.com/ datasets/mlg- ulb/creditcardfraud. Accessed: 2024-11-11. 2016. url: https: //www.kaggle.com/datasets/mlg-ulb/creditcardfraud

  13. [23]

    Data leakage inflates prediction performance in connectome- based machine learning models

    Matthew Rosenblatt et al. “Data leakage inflates prediction performance in connectome- based machine learning models”. In: Nature Communications 15.1 (2024), p. 1829. doi: 10.1038/s41467-024-46150-w

  14. [24]

    Hazards of data leakage in machine learning: a study on classification of breast cancer using deep neural networks

    Ravi K Samala et al. “Hazards of data leakage in machine learning: a study on classification of breast cancer using deep neural networks”. In: Medical Imaging 2020: Computer-Aided Diagnosis . Vol. 11314. International Society for Optics and Photonics. 2020, p. 1131416. doi: 10...

  15. [25]

    Semi-supervised anomaly detection for EEG waveforms using deep belief nets

    D. Wulsin et al. “Semi-supervised anomaly detection for EEG waveforms using deep belief nets”. In: Proceedings of the 2010 Ninth international conference on machine learning and applications . 2010, pp. 436–441

  16. [26]

    Impact of random oversampling and random undersampling on the performance of prediction models developed using observational health data

    C. Yang, E. A. Fridgeirsson, J. A. Kors, et al. “Impact of random oversampling and random undersampling on the performance of prediction models developed using observational health data”. In: Journal of Big Data 11 (2024), p. 7. doi: 10.1186/s40537-023-00857-7 . 16 Appendix A:...

  17. [492]

    doi: 10.1109/DeSE54285.2021.9719580

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.