Pith. sign in

REVIEW 3 major objections 7 minor 31 references

A Systematic Evaluation of Generative Models on Tabular Transportation Data

T0 review · 3 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Generic tabular generative models, tested on New York City taxi data, do not transfer to transportation data: synthetic trip networks are structurally far from real ones, and standard privacy metrics miss the models that actually leak.

desk verdict Useful transportation-data benchmark with two sensible new diagnostics, but the headline graph-metric result is currently uninterpretable because the paper never explains how the zone-based synthetic trips were produced. read the letter →

arxiv 2502.08856 v1 pith:M3TNQ7MA submitted 2025-02-13 cs.LG

Chengen Wang , Alvaro Cardenas , Gurcan Comert , Murat Kantarcioglu This is my paper · ORCID
classification cs.LG
keywords synthetictabulardatatransportationnetworksgraphsimilaritymetricNYCtaxiprivacyleakagemembershipinferencegenerativemodelevaluationmodecollapse
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tests six off-the-shelf tabular generative models—Gaussian Copula, CTGAN, TVAE, CTABGAN, STaSy, and TabDDPM—on New York City green taxi trip data. The authors want to show that models developed for generic spreadsheets do not transfer to transportation data, where trips collectively form a network. They introduce a graph metric that compares real and synthetic trip networks and an improved privacy metric based on the ratio of training-data to holdout-data distances to the nearest synthetic record. Their results find a large structural gap between synthetic and real networks across all models, plus widespread mode collapse and one privacy vulnerability that standard metrics miss.

What carries the argument

The load-bearing object is the graph similarity metric $S_G$, computed on zone-level graphs whose nodes are NYC taxi zones and whose edges are trips between zones, weighted by the fraction of all trips. $S_G$ subtracts the total variation distance between the real and synthetic edge-fraction distributions from 1, so it measures how well synthetic trips preserve the collective network. The second mechanism is the Distance to Closest Record ratio ($rDCR$), the ratio of the $\alpha$-percentile distance from training data to the nearest synthetic record over the same distance for holdout data; a ratio below 1 flags overfitting and membership-inference risk.

What would settle it

Retrain the same six models on the March 2019 zone-level data—where the zone column is the target—and recompute $S_G$ against real zone graphs. If the scores approach the real train/test reference value 73.17, the reported gap was an artifact of mismatched training and evaluation data rather than a general property of the models.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that the performance of tabular generative models is not portable to transportation data: rankings change with the metric, and no model reproduces the zone-to-zone network structure of real taxi trips. The graph metric scored every model far below the reference similarity between two real samples from the same dataset, and TabDDPM—the best overall on downstream prediction and coverage—scored worst on network structure. The proposed privacy ratio also overturns the ranking that the standard nearest-neighbor distance gives: Gaussian Copula looks safest by raw distance yet is the only model vulnerable to distance-based membership inference at small percentiles.

Load-bearing premise

The paper's headline gap result assumes the zone-level synthetic graphs used for scoring were produced in a way that is valid to compare with the real zone graph; the paper does not state whether the models were retrained on zone-based data or how zone trips were derived from models trained on coordinate data.

Editorial extensions

If this is right

  • If a transport agency publishes synthetic taxi or ride-hailing data from a generic tabular generator, downstream users should not assume the synthetic trip network reflects real flows.
  • Model rankings depend on the evaluation lens: TabDDPM dominates downstream prediction and coverage, while network-structure fidelity is poor for all six models.
  • A privacy report based only on raw distance to the closest real record can be misleading; the percentile-based ratio should accompany it.
  • The zone column with hundreds of categories is a specific failure point, suggesting categorical cardinality is a key challenge for tabular generators.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the graph metric is equally sensitive on other mobility datasets, it could become a standard diagnostic for any synthetic data meant to preserve network flow, not just taxi trips.
  • A direct retraining experiment on the zone-level dataset would tell whether the low graph scores come from the models themselves or from feeding them coordinates and evaluating on zones.
  • The rDCR idea could generalize to other record-level data where overfitting is the main privacy risk, such as medical or financial tables.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper presents a systematic comparison of six tabular generative models (Gaussian Copula, CTGAN, TVAE, CTABGAN, STaSy, TabDDPM) on New York City Green Taxi data. It evaluates downstream-task R², Wasserstein distance, coverage, DCR-based privacy, and two proposed metrics: a graph similarity metric (Section 4.1, Eq. 10) comparing the total-variation distance between real and synthetic edge-fraction distributions over zone pairs, and a Distance-to-Closest-Record ratio (rDCR, Section 4.2) that compares training-data and holdout-data distances to synthetic records. The results indicate TabDDPM has the best downstream and coverage performance, CTABGAN and TabDDPM have the smallest Wasserstein distances, all models show a large graph-similarity gap relative to the real-real reference (73.17), and Gaussian Copula has rDCR<1 at small percentiles. The abstract concludes that existing tabular models do not perform as consistently as claimed and that the graph metric reveals a significant gap between synthetic and real transportation networks.

Significance. The study addresses a timely question and brings two useful diagnostic ideas: a zone-graph structural similarity and a holdout-based distance ratio for privacy. The evaluation is unusually broad (six models, multiple utility and privacy axes) and the code is promised. If the dataset-alignment issue in Table 3 is resolved, the graph metric could be a valuable addition to the tabular-synthesis evaluation toolbox. At present, however, the paper's headline 'significant gap' claim is not interpretable without knowing how synthetic zone trips were generated, and the privacy-leakage conclusion is a proxy-based inference rather than a demonstrated attack. The manuscript is therefore promising but requires a substantial revision.

major comments (3)
  1. [5.1, 5.4 and Table 3] The graph metric experiments use a different dataset than all other experiments, but the paper never specifies how the generative models were trained for Table 3. Section 5.1 introduces the March 2019 Green Taxi zone-based dataset only for the graph metric, and Section 5.4 says only 'here we use the dataset with zone-based locations.' If models are trained on 2015 lat/lon data and synthetic coordinates are mapped to zones, the graph metric confounds model quality with a coordinate-to-zone mapping; if the models are retrained on March 2019 zone data, the training setup is omitted and Table 3 is not comparable with Tables 1, 2, 4, and 5. This is load-bearing because the abstract's 'significant gap' conclusion rests on Table 3. Please state which dataset was used for training and sampling, describe the mapping/retraining procedure, and if the graph metric requires a separate dataset, report the corresponding experimental setup.
  2. [4.2, 5.4, Table 5, Figure 1] The statement that Gaussian Copula 'is vulnerable to membership inference attacks at very small values of α' is not supported by an actual attack. The paper defines vulnerability as rDCR<1 in Section 4.2, so the observed rDCR<1 at small percentiles is a direct consequence of the metric's definition, not evidence of a successful membership inference attack. A distance-ratio proxy can be a useful diagnostic, but the claim should be either reframed as a distance-based proxy finding or supplemented by running a standard membership inference attack and relating rDCR to attack success.
  3. [5.4, Tables 1-4] All model rankings are based on point estimates without significance tests, and several pairwise comparisons have overlapping standard deviations (e.g., Table 1 dwn syn tr: GaussianCopula 77.55±1.20 vs CTGAN 78.05±2.29). Claims such as 'TabDDPM achieves the best downstream task performance' and 'CTABGAN and TabDDPM have the best performance' in Section 5.4 are therefore stronger than the reported statistics justify. Please provide significance tests or explicitly state that the rankings are descriptive and not statistically verified.
minor comments (7)
  1. [4.2] Clarify how dα is computed for the training and holdout sets, including whether the holdout set has the same size as the training set and how ties near the percentile boundary are handled.
  2. [Table 3 caption] Add a note explaining why STaSy entries are N/A and why G tr te is constant across models; the current text mentions these issues only in the body.
  3. [5.1] State whether the 40,000 training rows are a random sample from the 19.2 million rows, whether any stratification or random seed was used, and whether the March 2019 zone dataset was sampled in the same way.
  4. [Eq. (9)] Fix the unmatched parenthesis in the definition of Π(Pr, Pg) in the Wasserstein distance formula.
  5. [Table 1] Make the column headers consistent with the explanations in the text (e.g., 'dwntr tr' vs 'dwn tr tr') and mark the reference-only columns explicitly in the table rather than only in the prose.
  6. [Figure 2] Report training time and sampling time separately for the diffusion-based models, since the current combined running time mixes two different stages and makes cross-model complexity comparisons difficult to interpret.
  7. [5.4 (Graph Similarity Metric)] The claim that TabDDPM 'suffers severe mode collapse' is an inference from a low graph score; a direct diagnostic such as the number of unique generated zones or a cluster-based mode-collapse measure would strengthen this conclusion.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the graph and privacy metrics are applied as explicit definitions and the reported findings are empirical measurements, not conclusions derived from their inputs.

full rationale

This paper is an empirical evaluation rather than a derivation chain: no parameter is fitted to a subset of the data and then reported as a prediction of a closely related quantity, and no load-bearing premise is justified solely by a self-citation (the reference list contains no work by the present authors). The two proposed metrics are explicit definitions: SG in Eq. 10 is defined as 1 minus the total variation distance between edge-fraction distributions of real and synthetic graphs, and rDCR in Section 4.2 is defined as a ratio of nearest-neighbor distances at a chosen percentile. The subsequent statements that the graph metric reveals a significant gap and that Gaussian Copula has rDCR < 1 at small percentiles are direct readings of the computed metric values, not conclusions assumed inside the definitions; the values could in principle have come out differently. The main weakness is a reproducibility gap, not circularity: Table 3's note 'here we use the dataset with zone-based locations' together with Section 5.1's introduction of the March 2019 zone-based dataset leaves unspecified how synthetic zone trips were generated from models otherwise trained on the 2015 latitude/longitude data. This is an omitted experimental detail that affects the interpretation of Table 3, but it is not an equivalence between the metric's input and its output. Accordingly, no circular step is exhibited, and the circularity score is 0.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The paper introduces two evaluation metrics but no new entities or fitted model parameters. The load-bearing assumptions are the validity of the metrics themselves and the ambiguous dataset alignment for the graph metric.

free parameters (3)
  • percentile alpha in rDCR = 5 (default), swept in Figure 1
    Chosen by hand for the privacy metric; affects whether Gaussian Copula is flagged as vulnerable.
  • number of nearest neighbors K for coverage and DCR = not stated
    The paper does not report the K value used in the coverage metric from [21] and in the DCR computations; this affects the numbers in Tables 4 and 5.
  • generative model hyperparameters = not reported
    Learning rates, epochs, batch sizes, and other settings for GaussianCopula, CTGAN, TVAE, CTABGAN, STaSy, and TabDDPM are not specified, making the rankings hard to reproduce.
assumptions (5)
  • domain assumption Total variation distance between edge-fraction distributions is a meaningful measure of structural and functional alignment of transportation networks.
    Invoked in Section 4.1 when defining SG; the entire 'significant gap' conclusion depends on this metric validity.
  • domain assumption rDCR < 1 indicates overfitting and vulnerability to membership inference attacks.
    Stated in Section 4.2 without independent validation against actual attack success rates.
  • ad hoc to paper The March 2019 zone-based Green Taxi dataset can be used to evaluate models for the graph metric, even though the main experiments use 2015 lat/lon data.
    Section 5.1 introduces the 2019 dataset for the graph metric but does not state how the generative models are trained or adapted for it.
  • domain assumption All models were trained under comparable conditions on the same data splits.
    Section 5.3 states three models are trained and five samplings per model, but no tuning or seed control is reported; ranking comparisons assume fair conditions.
  • domain assumption Predicting total fare with gradient boosting is a representative downstream utility measure.
    Section 3.2 defines the downstream task; conclusions about utility rely on this single task.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Systematic Evaluation of Generative Models on Tabular Transportation Data." pith.science (2026). https://pith.science/paper/M3TNQ7MA

@misc{pith2026250208856,
  author       = {Pith},
  title        = {Pith review of: A Systematic Evaluation of Generative Models on Tabular Transportation Data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/M3TNQ7MA}},
  note         = {Machine review of arXiv:2502.08856}
}
read the original abstract

The sharing of large-scale transportation data is beneficial for transportation planning and policymaking. However, it also raises significant security and privacy concerns, as the data may include identifiable personal information, such as individuals' home locations. To address these concerns, synthetic data generation based on real transportation data offers a promising solution that allows privacy protection while potentially preserving data utility. Although there are various synthetic data generation techniques, they are often not tailored to the unique characteristics of transportation data, such as the inherent structure of transportation networks formed by all trips in the datasets. In this paper, we use New York City taxi data as a case study to conduct a systematic evaluation of the performance of widely used tabular data generative models. In addition to traditional metrics such as distribution similarity, coverage, and privacy preservation, we propose a novel graph-based metric tailored specifically for transportation data. This metric evaluates the similarity between real and synthetic transportation networks, providing potentially deeper insights into their structural and functional alignment. We also introduced an improved privacy metric to address the limitations of the commonly-used one. Our experimental results reveal that existing tabular data generative models often fail to perform as consistently as claimed in the literature, particularly when applied to transportation data use cases. Furthermore, our novel graph metric reveals a significant gap between synthetic and real data. This work underscores the potential need to develop generative models specifically tailored to take advantage of the unique characteristics of emerging domains, such as transportation.

Figures

Figures reproduced from arXiv: 2502.08856 by the authors.

Figure 1
Figure 1. Privacy leakage assessment based on rDCR, i.e., DCR ratio [PITH_FULL_IMAGE:figures/full_fig_p012_1.png] view at source ↗
Figure 2
Figure 2. The complexity of the generative models in terms of running time in [PITH_FULL_IMAGE:figures/full_fig_p013_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

31 extracted references · 24 canonical work pages

  1. [1]

    In: International conference on machine learning

    Arjovsky, M., Chintala, S., Bottou, L.: Wasserstein generative adversarial networks. In: International conference on machine learning. pp. 214–223. PMLR (2017) 6

    work page 2017

  2. [2]

    https://agkn.wordpress.com/2014/09/15/ riding-with-the-stars-passenger-privacy-in-the-nyc-taxicab-dataset/ (2014), [Online; accessed 2024-11] 2 14 C

    ATOCKAR: Riding with the stars: Passenger privacy in the nyc taxicab dataset. https://agkn.wordpress.com/2014/09/15/ riding-with-the-stars-passenger-privacy-in-the-nyc-taxicab-dataset/ (2014), [Online; accessed 2024-11] 2 14 C. Wang et al

    work page 2014

  3. [3]

    Bishop, C.M., Nasrabadi, N.M.: Pattern recognition and machine learning, vol. 4. Springer (2006) 4

    work page 2006

  4. [4]

    IEEE transactions on neural networks and learning systems (2022) 2

    Borisov, V., Leemann, T., Seßler, K., Haug, J., Pawelczyk, M., Kasneci, G.: Deep neural networks and tabular data: A survey. IEEE transactions on neural networks and learning systems (2022) 2

    work page 2022

  5. [5]

    In: NeurIPS 2023 Workshop on Synthetic Data Generation with Generative AI (2023) 7

    Boudewijn, A.T.P., Ferraris, A.F., Panfilo, D., Cocca, V., Zinutti, S., De Schepper, K., Chauvenet, C.R.: Privacy measurements in tabular synthetic data: State of the art and future research directions. In: NeurIPS 2023 Workshop on Synthetic Data Generation with Generative AI (2023) 7

    work page 2023

  6. [6]

    In: 2022 IEEE Symposium on Security and Privacy (SP)

    Carlini, N., Chien, S., Nasr, M., Song, S., Terzis, A., Tramer, F.: Membership inference attacks from first principles. In: 2022 IEEE Symposium on Security and Privacy (SP). pp. 1897–1914. IEEE (2022) 8

    work page 2022

  7. [7]

    https://en.wikipedia.org/wiki/ Earth_mover%27s_distance (2024), [Online; accessed 2024-11] 5

    contributors, W.: Earth mover’s distance. https://en.wikipedia.org/wiki/ Earth_mover%27s_distance (2024), [Online; accessed 2024-11] 5

    work page 2024

  8. [8]

    Available at SSRN 4229042 (2017) 8

    Correa, D.: Exploring the taxi and uber demands in new york city: An empirical analysis and spatial modeling. Available at SSRN 4229042 (2017) 8

    work page 2017

Show all 31 references
  1. [9]

    arXiv preprint arXiv:2008.12763 (2020) 6

    Fan, J., Liu, T., Li, G., Chen, J., Shen, Y., Du, X.: Relational data synthesis using generative adversarial networks: A design space exploration. arXiv preprint arXiv:2008.12763 (2020) 6

    2020 arXiv

  2. [10]

    Advances in neural infor- mation processing systems 27 (2014) 2, 3, 6

    Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. Advances in neural infor- mation processing systems 27 (2014) 2, 3, 6

    2014

  3. [11]

    arXiv preprint arXiv:1906.03006 (2019) 5, 6

    Hilprecht, B., H¨ arterich, M., Bernau, D.: Reconstruction and membership inference attacks against generative models. arXiv preprint arXiv:1906.03006 (2019) 5, 6

    2019 arXiv

  4. [12]

    Advances in neural information processing systems 33, 6840–6851 (2020) 4

    Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. Advances in neural information processing systems 33, 6840–6851 (2020) 4

    2020

  5. [13]

    Advances in Neural In- formation Processing Systems 34, 12454–12465 (2021) 4

    Hoogeboom, E., Nielsen, D., Jaini, P., Forr´ e, P., Welling, M.: Argmax flows and multinomial diffusion: Learning categorical distributions. Advances in Neural In- formation Processing Systems 34, 12454–12465 (2021) 4

    2021

  6. [14]

    arXiv preprint arXiv:2210.04018 (2022) 3, 4, 5, 9

    Kim, J., Lee, C., Park, N.: Stasy: Score-based tabular data synthesis. arXiv preprint arXiv:2210.04018 (2022) 3, 4, 5, 9

    2022 arXiv

  7. [15]

    arXiv preprint arXiv:1312.6114 (2013) 2, 3

    Kingma, D.P., Welling, M.: Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013) 2, 3

    2013 arXiv

  8. [16]

    In: International Conference on Machine Learn- ing

    Kotelnikov, A., Baranchuk, D., Rubachev, I., Babenko, A.: Tabddpm: Modelling tabular data with diffusion models. In: International Conference on Machine Learn- ing. pp. 17564–17579. PMLR (2023) 2, 3, 4, 5, 9

    2023

  9. [17]

    Curran Associates Inc., Red Hook, NY, USA (2019) 6

    Kynk¨ a¨ anniemi, T., Karras, T., Laine, S., Lehtinen, J., Aila, T.: Improved precision and recall metric for assessing generative models. Curran Associates Inc., Red Hook, NY, USA (2019) 6

    2019

  10. [18]

    scikit learn: Coefficient of determination. https://scikit-learn.org/stable/ modules/generated/sklearn.ensemble.GradientBoostingRegressor.html# sklearn.ensemble.GradientBoostingRegressor.score (2024), [Online; accessed 2024-11] 6

    2024

  11. [19]

    https://scikit-learn.org/stable/ modules/generated/sklearn.ensemble.GradientBoostingRegressor.html (2024), [Online; accessed 2024-11] 6

    scikit learn: Gradient boosting regressor. https://scikit-learn.org/stable/ modules/generated/sklearn.ensemble.GradientBoostingRegressor.html (2024), [Online; accessed 2024-11] 6

    2024

  12. [20]

    EPJ data science 11(1), 58 (2022) 8 A Systematic Evaluation of Generative Models on Transportation Data 15

    Mauro, G., Luca, M., Longa, A., Lepri, B., Pappalardo, L.: Generating mobility networks with generative adversarial networks. EPJ data science 11(1), 58 (2022) 8 A Systematic Evaluation of Generative Models on Transportation Data 15

    2022

  13. [21]

    In: International Conference on Machine Learning

    Naeem, M.F., Oh, S.J., Uh, Y., Choi, Y., Yoo, J.: Reliable fidelity and diversity metrics for generative models. In: International Conference on Machine Learning. pp. 7176–7185. PMLR (2020) 5, 6

    2020

  14. [22]

    In: 2016 IEEE International Conference on Data Science and Advanced Analytics (DSAA)

    Patki, N., Wedge, R., Veeramachaneni, K.: The synthetic data vault. In: 2016 IEEE International Conference on Data Science and Advanced Analytics (DSAA). pp. 399–410 (2016). https://doi.org/10.1109/DSAA.2016.49 3

    2016 doi

  15. [23]

    Frontiers in big Data 4, 679939 (2021) 8

    Platzer, M., Reutterer, T.: Holdout-based empirical assessment of mixed-type syn- thetic data. Frontiers in big Data 4, 679939 (2021) 8

    2021

  16. [24]

    In: 2017 IEEE Symposium on Security and Pri- vacy (SP)

    Shokri, R., Stronati, M., Song, C., Shmatikov, V.: Membership inference attacks against machine learning models. In: 2017 IEEE Symposium on Security and Pri- vacy (SP). pp. 3–18 (2017). https://doi.org/10.1109/SP.2017.41 3, 6, 7

    2017 doi

  17. [25]

    arXiv preprint arXiv:2011.13456 (2020) 4, 5

    Song, Y., Sohl-Dickstein, J., Kingma, D.P., Kumar, A., Ermon, S., Poole, B.: Score- based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456 (2020) 4, 5

    2020 arXiv

  18. [26]

    In: 31st USENIX Security Symposium (USENIX Security 22)

    Stadler, T., Oprisanu, B., Troncoso, C.: Synthetic data–anonymisation groundhog day. In: 31st USENIX Security Symposium (USENIX Security 22). pp. 1451–1468 (2022) 5

    2022

  19. [27]

    https://www.nyc.gov/site/tlc/ about/tlc-trip-record-data.page (2023), [Online; accessed 2024-11] 8

    Taxi, Commission, L.: Tlc trip record data. https://www.nyc.gov/site/tlc/ about/tlc-trip-record-data.page (2023), [Online; accessed 2024-11] 8

    2023

  20. [28]

    https://data.cityofnewyork

    Taxi, Commission, L.: 2015 green taxi trip data. https://data.cityofnewyork. us/Transportation/2015-Green-Taxi-Trip-Data/gi8d-wdg5/about_data (2024), [Online; accessed 2024-11] 3, 8

    2024

  21. [29]

    arXiv preprint arXiv:2201.09871 (2022) 6, 11

    Thompson, R., Knyazev, B., Ghalebi, E., Kim, J., Taylor, G.W.: On evaluation metrics for graph generative models. arXiv preprint arXiv:2201.09871 (2022) 6, 11

    2022 arXiv

  22. [30]

    Advances in neural information processing systems 32 (2019) 2, 3, 4, 9

    Xu, L., Skoularidou, M., Cuesta-Infante, A., Veeramachaneni, K.: Modeling tabular data using conditional gan. Advances in neural information processing systems 32 (2019) 2, 3, 4, 9

    2019

  23. [31]

    In: Asian Conference on Machine Learning

    Zhao, Z., Kunar, A., Birke, R., Chen, L.Y.: Ctab-gan: Effective table data synthe- sizing. In: Asian Conference on Machine Learning. pp. 97–112. PMLR (2021) 3, 4, 6, 7, 9, 11

    2021

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.