REVIEW 3 major objections 5 minor 7 references
On the (Mis)Use of Machine Learning with Panel Data
T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read Randomly splitting panel data into train and test sets leaks information and inflates machine-learning performance.
desk verdict Clear, useful framework for avoiding leakage in panel-data ML; the empirical magnitudes are illustrative, not precise, but the central message is sound and worth publishing. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The paper's central object is a four-way taxonomy of training/testing splits for panel data—random split at the observation (unit-time) level, at the unit level, at the group (e.g., state) level, and non-random split on time—paired with a two-way distinction between cross-sectional prediction policy problems and sequential forecasting policy problems. The taxonomy does the argument's work: each split produces a characteristic combination of temporal and cross-sectional leakage, and the paper shows that the right split is the one whose leakage pattern matches the deployment scenario. The empirical machinery is a controlled comparison of nearly 500 model configurations varying the split, the use of contemporaneous predictors, the inclusion of outcome lags, test-set size, and algorithm, while holding the raw predictors fixed.
What would settle it
Run the same model comparison using real-time historical vintages of the county income data (the values actually available at each forecast origin) rather than the finalized panel; if the correctly time-split models perform as well as the random-split models, the performance gap attributed to leakage is an artifact of data revision rather than leakage.
Extended reading notes
Core claim
The central claim is that there is no single off-the-shelf way to split panel data; the split must follow the goal. A random split at the observation (unit-time) level always creates both temporal leakage (future observations help train models that then 'predict' the past) and cross-sectional leakage (the same units appear in both training and testing), so the model has already seen the testing set in substance. If the goal is cross-sectional prediction, the right split is at the unit or group level, accepting trend leakage but eliminating unit leakage. If the goal is forecasting, the right split is non-random on time, accepting that all units appear in both sets but ensuring the model never trains on the future. The paper shows empirically that leaked models outperform correctly split models—for example, average AUC for Random Forest classification rises from 0.708 to 0.759 under temporal leakage, and in the 2009 recession year the gap is much larger (AUC 0.442 honest vs 0.692 leaked). For regression, the XGBoost leakage ratio exceeds 17 percent.
Load-bearing premise
The empirical demonstration assumes that official county income data are not revised using future information, so the 'honest' time-split forecasting models are truly uncontaminated; if revisions do carry future data, the measured inflation from leakage is biased.
Editorial extensions
If this is right
- If researchers use random observation-level splits on panel data, reported AUC/MSE gains will typically overstate real-world performance, and policy targeting built on those models will be less effective than advertised.
- For cross-sectional targeting problems, unit-level (or group-level) splits should become the default, with higher-level splits when spatial autocorrelation is suspected.
- For forecasting problems, only lagged predictors and non-random time splits are honest; contemporaneous predictors are a form of temporal leakage.
- Reporting practices should shift: applied ML papers with panel data should explicitly state the split strategy and how leakage was prevented, as none of the nine policy-targeting papers cited does.
- A model that performs suspiciously well on post-break data (like 2009) is likely leaking, not forecasting, so unusually strong out-of-sample results on hard-to-predict events should be treated as a warning sign.
Reading between the lines
- Beyond the paper, the same leakage logic implies that default random k-fold cross-validation in standard ML libraries is unsafe for panel data; hyperparameter tuning should use blocked or temporal folds to avoid selecting models by their leakage.
- A testable extension: the gap between county-level and state-level splits should grow with the strength of spatial autocorrelation, so one could use Moran's I or similar measures to predict when group-level splits matter most.
- For readers auditing applied work, the framework suggests a concrete check: demand a holdout on data the model has never seen—either a later time period or a different geography—before trusting policy claims.
- If the data-revision assumption is right, the leakage bias in this application is likely a lower bound for settings with stronger common shocks and more persistent unit heterogeneity, since those amplify both temporal and cross-sectional leakage.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper provides a conceptual framework and empirical illustration of data leakage in supervised machine learning applied to panel data. The authors distinguish two prediction policy problems—cross-sectional prediction and sequential forecasting—and argue that a random split at the unit-time level produces both temporal and cross-sectional leakage, while unit-level and time-level splits each remove one source. They propose practical guidelines for splitting and cross-validation, and illustrate the consequences using a balanced panel of 3,058 U.S. counties (2000–2019) for income growth forecasting and classification, running 480 model configurations. The empirical results show that models affected by leakage achieve better apparent out-of-sample performance, with an XGBoost MSE ratio above 17% in the regression task and an AUC gap of about 0.05 for Random Forest in classification.
Significance. If accepted, the paper fills a real gap in applied econometrics: it provides an accessible organizing framework and concrete recommendations for avoiding leakage in panel ML. The framework is conceptually sound and the distinction between cross-sectional prediction and sequential forecasting is useful. The empirical application is extensive (480 models, public replication package) and demonstrates the direction and rough magnitude of leakage-induced inflation. The claims are falsifiable and the code is available, which strengthens the paper's value for practitioners. The main limitations—lack of uncertainty quantification, a single test window, and reliance on revised BEA data—do not threaten the conceptual core, but they do bound the strength of the quantitative conclusions.
major comments (3)
- [Section 4.1, Figure 2, footnote 19] The headline comparisons of 'leaked' versus 'non-leaked' models (e.g., the Random Forest AUC gap of 0.051 and the XGBoost MSE leakage ratio above 17%) are reported without any confidence intervals or significance tests. Footnote 19 states that no suitable group-level ROC test is known and that uncertainty quantification would be computationally demanding, but with 480 models one could compute bootstrap intervals on the differences for matched configurations (e.g., identical predictors and split type) or use permutation tests. As written, the word 'significantly' in the text is informal and the magnitudes are not statistically grounded. This is a load-bearing point for the empirical demonstration, and the manuscript should either add uncertainty quantification or explicitly reframe these as descriptive differences.
- [Section 4.2, Figure 4, Tables A.7–A.8] The comparison of random, county, and state splits does not isolate cross-sectional leakage because the splits change the composition of the training and test sets. A state split introduces new states into the test set, so the observed performance drop could reflect a distribution shift across states (different economic structures) rather than the removal of spatial leakage. The statement that 'all the leaked models dominate the non-leaked ones, indicating the presence of spatial leakage' (p. 20) overreaches. A more controlled design would hold the test units and periods fixed while varying only whether the same or neighboring units appear in training, e.g., by excluding a spatial buffer around test counties rather than splitting by state. This point affects the quantitative support for the cross-sectional leakage claim, although it does not weaken the conceptual argument.
- [Section 2, footnote 12] The claim that BEA historical revisions do not create temporal leakage because revisions are driven by methodological improvements rather than future data is asserted with a general citation but not tested against real-time vintage data. If the assertion is wrong, the 'non-leaked' time-split forecasting models are themselves contaminated by revised historical values, and the reported leakage gaps (e.g., the XGBoost MSE ratio above 17%) would be biased, most likely conservatively. The authors should either verify the assumption using available vintages for the outcome and key predictors, or explicitly state that all quantitative leakage magnitudes should be interpreted as lower bounds.
minor comments (5)
- [Online Appendix Tables A.4 and A.6] These tables are missing their column headers and appear garbled (e.g., rows begin with '2448 yes yes 0.0421066 yes 34244 yes' with no model or split-type labels), which makes it impossible to verify the reported model configurations from the tables alone.
- [Section 4.1, footnote 22] Footnote 22 warns that the Logit classification results are unstable due to class imbalance, yet the Logit models are included in the main headline comparisons in Figures 2 and 3; consider either removing them from the classification figures or reporting a class-balanced metric such as balanced accuracy alongside AUC.
- [Abstract and Section 4] The abstract says 'nearly 500 models' but Section 4 states 480 models; make the count consistent.
- [Section 4.1, Figure 3] The text says the leakage ratio for the regression problem in 2009 'remains similar' to the all-period ratio, but the reader is not given the numerical value; adding the value to the caption or text would help.
- [Table 2 and Section 4.2] The recommendation in Table 2 to split at the unit level for cross-sectional prediction should be reconciled with the later discussion of spatial leakage; the table mentions stratified sampling at a higher level only as a fallback, but the empirical results in Figure 4 show substantial drops under state splits, which suggests the fallback may often be the primary choice for spatially correlated panels.
Circularity Check
No circularity: the leakage taxonomy and empirical demonstration are structurally defined, and the BEA vintage caveat is an external-validity concern, not a definitional loop.
full rationale
The paper's central claim is that random unit-time splits of panel data generate both temporal and cross-sectional leakage and thereby inflate apparent out-of-sample performance. This claim is not circular: 'leaked' models are identified by structural design choices—the inclusion of contemporaneous predictors or a split that is not time-based—rather than by the performance outcome. The empirical application then measures whether these structurally defined configurations in fact perform better, holding predictors and model families fixed, so the reported AUC/MSE gaps are empirical findings rather than consequences of the definitions. The labeling of models as temporally leaked in Figures 2 and 3 is based on observable pipeline features (contemporaneous variables or non-time splits), not on the performance metric, so the demonstration does not reduce to 'models we call leaked perform better because we define leaked as performing better.' Similarly, the cross-sectional leakage analysis in Section 4.2 compares random, county, and state splits and reports the resulting performance ordering; again, the split types are defined structurally and the performance ordering is measured. Footnote 12 contains a genuine external-validity caveat about BEA data revisions, but this is not a circular step: if the assumption failed, both time-split and random-split forecasting models would be contaminated, which would make the estimated leakage gap conservative rather than definitionally forced. The footnote 11 reference to Cerqua et al. (2024) is a software-availability and novelty claim, not a load-bearing premise of the argument, and the core leakage concepts are independently supported by external citations (Kaufman et al. 2012; Kapoor & Narayanan 2023; Petropoulos et al. 2022). No equation, fitted parameter, or uniqueness theorem is imported from the authors' prior work to forbid alternatives or to define the conclusion into existence. The paper is therefore self-contained in its derivation chain and exhibits no significant circularity.
Assumptions & free parameters
assumptions (4)
- domain assumption Panel data violate the i.i.d. assumption because units and time periods are dependent.
- domain assumption Including future information in the training set inflates out-of-sample performance.
- domain assumption For sequential forecasting, only lagged or time-invariant predictors should be used.
- domain assumption BEA historical data revisions are not driven by future information, so the empirical outcome data have no revision-based leakage.
Cite this review
Pith. "Pith review of On the (Mis)Use of Machine Learning with Panel Data." pith.science (2026). https://pith.science/paper/WRIHMLXZ
@misc{pith2026241109218,
author = {Pith},
title = {Pith review of: On the (Mis)Use of Machine Learning with Panel Data},
year = {2026},
howpublished = {\url{https://pith.science/paper/WRIHMLXZ}},
note = {Machine review of arXiv:2411.09218}
}
read the original abstract
We provide the first systematic assessment of data leakage issues in the use of machine learning on panel data. Our organizing framework clarifies why neglecting the cross-sectional and longitudinal structure of these data leads to hard-to-detect data leakage, inflated out-of-sample performance, and an inadvertent overestimation of the real-world usefulness and applicability of machine learning models. We then offer empirical guidelines for practitioners to ensure the correct implementation of supervised machine learning in panel data environments. An empirical application, using data from over 3,000 U.S. counties spanning 2000-2019 and focused on income prediction, illustrates the practical relevance of these points across nearly 500 models for both classification and regression tasks.
Reference graph
Works this paper leans on
-
[1]
Antulov-Fantulin, N., Lagravinese, R., & Resce, G. (2021). Predicting bankruptcy of local government: A machine learning approach. Journal of Economic Behavior & Organization , 183, 681-699. Apicella, A., Isgrò, F., & Prevete, R. (2024). Don't Push the Button! Exploring Data Leakage Risks in Machine Learning and Transfer Learning. arXiv preprint arXiv:240...
work page Pith review arXiv 2021
-
[21]
Kleinberg, J., Ludwig, J., Mullainathan, S., & Obermeyer, Z. (2015). Prediction policy problems. American Economic Review, 105(5), 491-495. Kuhn, M. (2008). Building predictive models in R using the caret package. Journal of Statistical Software, 28, 1-26. Kuhn, M., & Johnson, K. (2024). Applied Machine Learning for Tabular Data. Available at: https://aml...
work page 2015
-
[246]
Causal inference and policy evaluation without a control group
Battaglini, M., Guiso, L., Lacava, C., Miller, D. L., & Patacchini, E. (2024). Refining public policies with machine learning: The case of tax auditing . Journal of Econometrics , a vailable online 23 September 2024.Bernett, J., Blumenthal, D. B., Grimm, D. G., Haselbeck, F., Joeres, R., Kalinina, O. V., & List, M. (2024). Guiding questions to avoid data ...
work page Pith review arXiv 2024
-
[845]
Foote, A., Grosz, M., & Stevens, A. (2019). Locate your nearest exit: mass layoffs and local labor market response. ILR Review, 72(1), 101-126. Frey, E. (2024). How to cross-validate your panel data in Python. Towards Data Science. Available at : https://github.com/4Freye/panelsplit. Fuhr, J., & Papies, D. (2024). Double Machine Learning meets Panel Data ...
arXiv 2019
-
[871]
Rosenblatt, M., Tejavibulya, L., Jiang, R., Noble, S., & Scheinost, D. (2024). Data leakage inflates prediction performance in connectome -based machine learning models. Nature Communications, 15(1),
work page 2024
-
[1829]
26 Sansone, D., & Zhu, A. (2023). Using machine learning to create an early warning system for welfare recipients. Oxford Bulletin of Economics and Statistics, 85, 959-992. Semenova, V., Goldman, M., Chernozhukov, V., & Taddy, M. (2023). Inference on heterogeneous treatment effects in high‐dimensional dynamic panels under weak dependence. Quantitative Eco...
work page 2023
-
[2009]
– Full Results Model Contemporaneous Outcome Lags Split Type MSE Adj. Test Size Train Size Test Size rforest yes yes random 0.052 no 16696 4710 ols yes yes random 0.053 no 16696 4710 xgboost yes yes random 0.053 no 16696 4710 rforest yes yes random 0.054 yes 14676 612 xgboost yes yes random 0.055 yes 14676 612 rforest no yes random 0.056 no 16696 4710 ols...
work page 2009
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.