Pith. sign in

REVIEW 4 major objections 8 minor 87 references

TabTreeFormer: Tabular Data Generation Using Hybrid Tree-Transformer

T0 review · 4 major / 8 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read TabTreeFormer claims its hybrid tree-transformer beats eight baselines on utility, fidelity, privacy, and efficiency in synthetic tabular data generation.

desk verdict The tree-prompt and dual-quantization tokenizer are genuinely new and worth knowing, but the 'consistently outperforms' claim rests on a no-mask variant that memorizes, so the central empirical claim does not hold as stated. read the letter →

arxiv 2501.01216 v7 pith:YKMN6KG2 submitted 2025-01-02 cs.LG

Jiayu Li , Bingyin Zhao , Zilong Zhao , Uzair Javaid , Kevin Yee , Biplab Sikdar This is my paper · ORCID
classification cs.LG
keywords tabulardatagenerationsynthetictree-basedinductivebiasautoregressivetransformerdual-quantizationtokenizerordinalembeddingscross-entropylossconditional
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes TabTreeFormer, a hybrid generative model that pushes tree-based inductive biases into an autoregressive transformer to synthesize tabular data. Its central claim is that feeding the transformer the leaf indices from a fitted LightGBM model, as prompts, transfers the non-smooth and non-rotationally invariant structure that makes trees strong on tables, and that this transfer yields better synthetic data than eight established baselines across nine datasets. To handle multimodal continuous columns, the paper adds a dual-quantization tokenizer that maps each number to a K-Means bin plus a quantile ID, and it modifies the embedding and loss to respect the ordinal relation among quantile tokens. The authors report that the masked variants preserve privacy on all datasets, while the unmasked variant delivers the highest utility, with a 44 percent gain over the baseline variant.

What carries the argument

The load-bearing mechanism is the leaf-index prompt matrix: each row of the training table is passed through an ensemble of T LightGBM trees, and the index of the leaf where the row lands in each tree is prepended to the token sequence, so the transformer learns to generate conditionally on these tree-determined clusters. Around it, the paper builds a dual-quantization tokenizer that splits each continuous value into a K-Means cluster ID (capturing multimodality) and a quantile ID (preserving precision), plus a function-generated quantile embedding based on scaled sigmoids and an ordinal cross-entropy loss that penalizes quantile mispredictions less when the predicted ID is close to the target. Position-dependent valid-token sets are used at inference, so every sampled sequence is a valid row without rejection sampling.

What would settle it

Train TabTreeFormer on a dataset with no designated target column (use a random column as target) and compare downstream utility against the same model using the real target; if the gain over non-tree baselines largely vanishes, the target-specific tree fit is the source of the improvement. Alternatively, replace the fitted LightGBM with randomly generated partitions of the same leaf counts and check whether utility drops to baseline levels.

Watch

Extended reading notes

Core claim

On its own terms, TabTreeFormer's discovery is that a tree-based model trained on a tabular dataset can act as a source of multiple overlapping clusterings of the rows, and that prepending the leaf-index matrix of these trees to the token sequence of each row gives the transformer a useful conditional prior for generation. The paper argues that trees' piecewise-constant, axis-aligned decisions match the discrete and weakly correlated nature of tabular features, and that the leaf indices thereby inject inductive biases that pure transformers lack. Combined with a dual-quantization tokenizer and ordinal-aware training, the model is claimed to match or beat the strongest existing generators in downstream utility and fidelity, while the masked version keeps distance-to-closest-record privacy at safe levels on every tested dataset. The utility-first, unmasked variant is reported to outperform all baselines and to achieve a 44 percent improvement in machine-learning efficacy relative to its own baseline variant.

Load-bearing premise

The whole utility gain rests on the assumption that the leaf indices of a LightGBM model fitted to a target column give the transformer a genuinely informative and transferable condition; if a dataset lacks a good target column, or the trees fail to summarize the data, the advantage may disappear.

Editorial extensions

If this is right

  • If the paper is right, the best synthetic tabular data for downstream machine-learning tasks should come from a tree-conditioned autoregressive transformer, with the unmasked variant leading when privacy is not a concern.
  • Masked variants of TabTreeFormer keep DCR-based privacy intact across all nine datasets, while still achieving near-best downstream utility.
  • Because each numeric column consumes only two tokens, TabTreeFormer generation time is sub-second on most test sets and scales better than other auto-regressive baselines as feature count grows.
  • The ordinal embedding and loss are not tied to a particular tree model; the ablation suggests multiple-tree models (LightGBM, XGBoost) work better than a single decision tree.
  • The trend-fidelity improvement over auto-regressive baselines grows with the number of features, consistent with the claim that the inductive bias helps filter out low-correlation features.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If leaf indices are a general conditional prior, the same mechanism could be ported to transformer-based classifiers or regressors on tables, not just generators; the paper notes this possibility but does not test it.
  • The dependency on a designated target column is a practical constraint: on datasets without a natural label, the utility gain may shrink or vanish unless a good proxy column is found.
  • A direct test would be to swap LightGBM for random trees or a forest fitted on shuffled targets; if the utility advantage persists, the benefit may come from the cluster conditioning itself rather than from the tree's predictive structure.
  • The ordinal embedding construction may transfer to any token space with a monotonic, absolute scale (e.g., age, price, ordinal survey responses), beyond tabular generation.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 8 minor

Summary. The paper proposes TabTreeFormer, a hybrid architecture that combines a LightGBM-based tree model, a dual-quantization tokenizer (K-Means bins plus quantile IDs), and an autoregressive GPT-2-style transformer with ordinal-aware embeddings and an ordinal cross-entropy loss. The method is trained with masking and early stopping for privacy, and a no-mask variant (TTF-NM) is proposed for utility-priority settings. The authors evaluate on nine OpenML datasets against eight baselines, reporting utility (TSTR machine learning efficacy), fidelity (Shape/Trend from SDMetrics), privacy (DCR Mann-Whitney tests), and efficiency. The abstract claims consistent outperformance in utility, fidelity, and privacy with competitive efficiency, and reports a 44% utility gain for the best variant.

Significance. If the claims were fully supported, the paper would make a solid contribution: it is, to my knowledge, a genuinely new way to inject tree-based inductive biases into transformer generators, and the compact tokenizer with valid-token constrained sampling gives a plausible efficiency benefit. The paper provides open code, a broad benchmark (9 datasets, 8 baselines, 3 repeats), and some theoretical justification for the ordinal embedding and loss. The privacy-resilient TTF-S/TTF-L variants show good DCR behavior and competitive utility. However, several load-bearing claims in the abstract are contradicted by the paper's own reported numbers, in particular the fidelity claim and the attribution of the utility gain to the proposed method rather than to memorization.

major comments (4)
  1. [Abstract; Table 3] The abstract claims that TabTreeFormer 'consistently outperforms baselines in utility, fidelity, and privacy metrics.' Table 3 directly contradicts the fidelity part: on the averaged Shape metric, TTF-S scores 0.910±0.037 and TTF-L 0.915±0.042, below Forest Diffusion (0.931±0.047) and TabSyn (0.925±0.052). The paper's own text in Section 4.2 correctly says the method achieves 'comparable performance' in Shape, not superiority. The abstract's unqualified claim should be revised to reflect the actual ranking.
  2. [Section 4.2; Table 2; Table 12; Appendix E.2] The main utility advantage is driven by TTF-NM, the no-mask, no-early-stopping variant. Table 12 shows that TTF-NM has DCR p=0.000 on credit, qsar, and wdbc, and Appendix E.2 states that this setting 'makes the model prone to memorizing exact values in the training data.' The 44% gain reported in the abstract is therefore not evidence for the tree-transformer inductive bias; the privacy-preserving variants TTF-S and TTF-L have average RE (0.031 and 0.021) that is worse than or comparable to Forest Diffusion (0.020), TabSyn (0.024), and REaLTabFormer (0.027) in Table 2. A claim of 'consistently outperforms in utility and privacy' cannot be sustained when the utility champion violates privacy and the privacy-preserving derivate does not beat the baselines.
  3. [Section 4.2; Tables 2, 10] No statistical significance tests are reported for the head-to-head comparisons. With only 3 repeats, the differences between TTF-L (RE 0.021±0.035), TabSyn (0.024±0.030), and Forest Diffusion (0.020±0.033) in Table 2 are within one standard deviation, so the claim of consistent utility outperformance is not supported by the evidence as presented. The authors should run paired tests across datasets (e.g., Wilcoxon signed-rank) for utility, fidelity, and privacy, and report the number of datasets on which each method wins.
  4. [Section 3.4, Theorem 2] Theorem 2 states a distance-order preservation property for the quantile embeddings defined by Equation 3. However, Section 3.4 immediately says that 'instead of fixing the embedding values as in Equation 3, we initialize the values by the equations, and they are updated during training.' Thus the theorem only holds for the initialization, not for the trained embeddings used in the final model. The theoretical support for the ordinal behavior after training is actually provided by the ordinal cross-entropy loss (OCEL), not by Theorem 2. The paper should either state the theorem as an initialization property or prove a corresponding statement for the learned embedding.
minor comments (8)
  1. [Abstract / Full text] The abstract header gives the code URL as github.com/li-jiayu-ljy/tabtreeformer, while the main text (page 2) gives anonymous.4open.science/r/tabtreeformer-9585. Please make these consistent.
  2. [Section 3.1] The symbol T is used both for the tree-based model and for the number of trees, which is confusing (e.g., 'a fitted tree model T with T trees'). Consider using a different symbol for the number of trees, such as T_trees.
  3. [Table 1] The 'Format' column for the 'Special' row appears to list all token types rather than the special tokens' format; the table formatting should be fixed for the row to show [BOS], [EOS], and [mask] separately.
  4. [Appendix A.4.5] There is a typo: 'qunatile' should be 'quantile' in 'including bin and qunatile tokens.'
  5. [Section 3.3, Theorem 1] The statement 'sample data from Gi for a probability of |X[i]| / |X|' should read 'with probability |X[i]| / |X|' to be grammatically correct.
  6. [Table 2, footnote 2] The footnote says 'The reported utility improvement in abstract and introduction is computed from this row.' It should specify the reference value: the 44% gain is relative to which baseline variant (e.g., TTF-L)?
  7. [Appendix B.5, proof of Theorem 2] The proof says the inequality |q_id - q_jd| < |q_id - q_kd| holds 'by the monotonicity of q_id with respect to i and triangle inequality.' The triangle inequality is not used here; the step follows directly from the strict monotonicity of the embedding function when j and k are on the same side of i. Please correct the wording.
  8. [Section 4.4] The sentence 'Two core settings of NM different from L version is the mask ratio and temperature' should be 'are' instead of 'is.'

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: TabTreeFormer's claims rest on external benchmarks and independent mathematical lemmas, not on definitions or self-citations.

full rationale

The paper's core derivation chain is self-contained. The tree-based model (LightGBM) is fitted on the training data and its leaf indices are used as conditioning prompts (Section 3.1, Algorithm 2); this is a conditional-generation design, not an analytic identity that forces the reported utility numbers. The dual-quantization tokenizer, ordinal embeddings (Equation 3), and ordinal cross-entropy loss (Equation 5) are stated constructions; Theorems 2-3 and Lemma 1 are proven from monotonicity and softmax properties, independent of experimental results. Theorem 1 is an elementary mixture-of-partitions identity and is not used to define any measured quantity. The 44% utility gain is computed from Table 2's RE row against external baseline scores under TSTR, so it is an empirical comparison rather than a fitted parameter renamed as a prediction. Self-citations (CTAB-GAN+, TabuLa) appear only as baselines and related work, not as load-bearing justification of the method. The reviewer concern that TTF-NM's utility may reflect memorization is a validity and generalization risk, not a circularity: the DCR and MLE metrics are separate measurements, and no equation reduces one to the other.

Assumptions & free parameters 7 free parameters · 5 assumptions · 0 invented entities

The central claim rests on three components: a tree-based model fitted to a target column with hyperparameters tuned on the training set, a dual-quantization tokenizer with hand-set K and Q, and an ordinal loss with hand-set sigma and m. The privacy evaluation assumes DCR is a valid disclosure metric. No new physical or conceptual entities are introduced.

free parameters (7)
  • K-Means bins K = 10
    Chosen by hand as a small number of clusters to capture multimodality in Section 3.2.
  • Quantile count Q = 1000
    Chosen as sufficient precision in Appendix A.3.
  • Weight function scale sigma = 0.005
    Hand-set scaling in the ordinal cross-entropy weight, Equation 6 and Appendix C.3.
  • Weight function minimum m = 0.5
    Hand-set minimum weight at the target class, Equation 6.
  • Temperatures for categorical and numeric tokens = 2.0 and 1.0 for S/L; 0.2 and 0.1 for NM
    Set in Appendix A.4.5; ablation figures 9 to 11 show selection based on utility and privacy trade-off.
  • Mask ratios for tree and value tokens = Uniform [0.5,0.75] and [0.25,0.5]; 0 in NM
    Chosen in Appendix A.4.3; the no-mask variant TTF-NM is used for the headline utility gain.
  • Tree hyperparameters and number of trees T = Tuned by Optuna, variable
    Fitted on training data to predict the target column in Appendix A.2.1.
assumptions (5)
  • domain assumption Tree-based models encode useful inductive biases for tabular data, including non-smoothness and robustness to low-correlation features.
    Invoked in Sections 2.2 and 3.1 to justify prepending leaf indices.
  • domain assumption Leaf indices from a supervised tree fitted on the target column are informative prompts for generating realistic rows.
    Used in Section 3.1 and Algorithm 2; all 9 datasets have a target column, so this is untested on target-free tables.
  • domain assumption The sigmoid-based quantile embeddings and ordinal cross-entropy loss preserve ordinal information after training.
    Claimed in Section 3.4 and Theorem 2; the embeddings are trainable, so the initialization may be washed out.
  • domain assumption DCR with the Mann-Whitney U test is a valid measure of privacy risk.
    Used for privacy evaluation in Section 4.3 and Appendix D.6.
  • standard math Standard probability and calculus results.
    Used in proofs of Theorems 1 to 3 and Lemma 1.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TabTreeFormer: Tabular Data Generation Using Hybrid Tree-Transformer." pith.science (2026). https://pith.science/paper/YKMN6KG2

@misc{pith2026250101216,
  author       = {Pith},
  title        = {Pith review of: TabTreeFormer: Tabular Data Generation Using Hybrid Tree-Transformer},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YKMN6KG2}},
  note         = {Machine review of arXiv:2501.01216}
}
read the original abstract

Transformers have shown impressive results in tabular data generation. However, they lack domain-specific inductive biases which are critical for preserving the intrinsic characteristics of tabular data. They also suffer from poor scalability and efficiency due to quadratic computational complexity. In this paper, we propose TabTreeFormer, a hybrid transformer architecture that integrates inductive biases of tree-based models (e.g., non-smoothness and non-rotational invariance) to effectively handle the discrete and weakly correlated features in tabular datasets. To improve numerical fidelity and capture multimodal distributions, we introduce a novel tokenizer that learns token sequences based on the complexity of tabular values. This reduces vocabulary size and sequence length, yielding more compact and efficient representations without sacrificing performance. We evaluate TabTreeFormer on nine diverse datasets, benchmarking against eight generative models. We show that TabTreeFormer consistently outperforms baselines in utility, fidelity, and privacy metrics with competitive efficiency. Notably, in scenarios prioritizing data utility over privacy and efficiency, the best variant of TabTreeFormer delivers a 44% performance gain relative to its baseline variant. Our code is available at: https://github.com/li-jiayu-ljy/tabtreeformer.

Figures

Figures reproduced from arXiv: 2501.01216 by the authors.

Figure 1
Figure 1. Performance comparison of TabTreeFormer (Ours) [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of TabTreeFormer (data flow: bottom [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Marginal densities of representative multimodal continuous [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (5 more)
Figure 6
Figure 6. Figure 6: Number of features vs. aver￾age generation time per (valid) row of TTF and ART baselines. Fidelity: Pair-wise correlation between columns (Trend). TabTreeFormer-L outperforms all neural network baselines in “Trend”, implying the superior capability of TabTreeFormer in …
Figure 7
Figure 7. Figure 7: Utility of different tree￾based model in TTF. The x-axis shows the downstream model type. 50 100 150 200 # Trees 0.75 0.80 0.85 Utility Score ML LN RF XGB [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 11
Figure 11. Figure 11: Utility & fidelity of trend of different temperatures. Efficiency: Computation Time. Comparing TabTreeFormer with all other ART baselines in terms of computation time, as shown in [PITH_FULL_IMAGE:figures/full_fig_p009_11.png]
Figure 9-11
Figure 9-11. Figure 9-11: The utility and DCR changes with mask ratio and demonstrate demonstrates clearly the [PITH_FULL_IMAGE:figures/full_fig_p009_9-11.png]
Figure 14
Figure 14. Figure 14: The relation of weights wti (normalized so that ∥w∥ = 1 for visualization purpose to token index i, with vocabulary size V = 1000, and target token t = 350. Recall the function of weight given the target token and current token in Equation 6. This section will provide…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

87 extracted references · 52 canonical work pages

  1. [1]

    Optuna: A next- generation hyperparameter optimization framework

    Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next- generation hyperparameter optimization framework. In Ankur Teredesai, Vipin Kumar, Ying Li, Rómer Rosales, Evimaria Terzi, and George Karypis, editors,Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2019, An...

    arXiv 2019

  2. [2]

    Arik and Tomas Pfister

    Sercan Ö. Arik and Tomas Pfister. TabNet: Attentive interpretable tabular learning. Proceedings of the AAAI Conference on Artificial Intelligence, 35(8):6679–6687, May 2021. doi: 10.1609/aaai.v35i8.16826

    doi:10.1609/aaai.v35i8.16826 2021

  3. [3]

    Assefa, Danial Dervovic, Mahmoud Mahfouz, Robert E

    Samuel A. Assefa, Danial Dervovic, Mahmoud Mahfouz, Robert E. Tillman, Prashant Reddy, and Manuela Veloso. Generating synthetic data in finance: opportunities, challenges and pitfalls. In Proceedings of the First ACM International Conference on AI in Finance, ICAIF ’20, New York, NY , USA, 2021. Association for Computing Machinery. ISBN 9781450375849. doi...

    arXiv 2021

  4. [4]

    Christopher M. Bishop. Pattern Recognition and Machine Learning (Information Science and Statistics). Springer-Verlag, Berlin, Heidelberg, 2006. ISBN 0387310738

    2006

  5. [5]

    Language models are realistic tabular data generators

    Vadim Borisov, Kathrin Seßler, Tobias Leemann, Martin Pawelczyk, and Gjergji Kasneci. Language models are realistic tabular data generators. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023

    2023

  6. [6]

    Breiman, Jerome H

    L. Breiman, Jerome H. Friedman, Richard A. Olshen, and C. J. Stone. Classification and regression trees. Biometrics, 40:874, 1984. URL https://api.semanticscholar.org/CorpusID:29458883

    1984

  7. [7]

    Random forests

    Leo Breiman. Random forests. Machine learning, 45(1):5–32, 2001. ISSN 1573-0565. doi: 10.1023/A: 1010933404324

    doi:10.1023/a: 2001

  8. [8]

    Rank consistent ordinal regression for neural networks with application to age estimation

    Wenzhi Cao, Vahid Mirjalili, and Sebastian Raschka. Rank consistent ordinal regression for neural networks with application to age estimation. Pattern Recognition Letters, 140:325–331, 2020. ISSN 0167-8655. doi: https://doi.org/10.1016/j.patrec.2020.11.008

    doi:10.1016/j.patrec.2020.11.008 2020

Show all 87 references
  1. [9]

    A simple log-based loss function for ordinal text classification

    François Castagnos, Martin Mihelich, and Charles Dognin. A simple log-based loss function for ordinal text classification. In Nicoletta Calzolari, Chu-Ren Huang, Hansaem Kim, James Pustejovsky, Leo Wanner, Key-Sun Choi, Pum-Mo Ryu, Hsin-Hsi Chen, Lucia Donatelli, Heng Ji, Sada...

    2022

  2. [10]

    XGBoost: A scalable tree boosting system

    Tianqi Chen and Carlos Guestrin. XGBoost: A scalable tree boosting system. In Balaji Krishnapuram, Mohak Shah, Alexander J. Smola, Charu C. Aggarwal, Dou Shen, and Rajeev Rastogi, editors,Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Da...

    2016

  3. [11]

    Carbonell, Quoc Viet Le, and Ruslan Salakhutdinov

    Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G. Carbonell, Quoc Viet Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-length context. In Anna Korhonen, David R. Traum, and Lluís Màrquez, editors, Proceedings of the 57th Conference of the As...

    2019 doi

  4. [12]

    Big data in healthcare: management, analysis and future prospects

    Sabyasachi Dash, Sushil Kumar Shakyawar, Mohit Sharma, and Sandeep Kaushik. Big data in healthcare: management, analysis and future prospects. Journal of Big Data, 6(1):54, 2019. ISSN 2196-1115. doi: 10.1186/s40537-019-0217-0

    2019 doi

  5. [13]

    DataCebo, Inc., 12 2023

    Synthetic Data Metrics. DataCebo, Inc., 12 2023. URL https://docs.sdv.dev/sdmetrics/. Version 0.13.0

    2023

  6. [14]

    Soft labels for ordinal regression

    Raul Diaz and Amit Marathe. Soft labels for ordinal regression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019

    2019

  7. [15]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at...

    2021

  8. [16]

    EU. Regulation (EU) 2016/679 of the European parliament and of the council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing directive 95/46/EC (General Data Protection Regu...

    2016

  9. [17]

    R. A. Fisher. The use of multiple measurements in taxonomic problems. Annals of Eugenics, 7(2):179–188, Sep 1936. doi: 10.1111/j.1469-1809.1936.tb02137.x

    1936

  10. [18]

    Generative adversarial nets

    Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger, editors, Advances in Neural Information Proc...

    2014

  11. [19]

    Revisiting deep learning models for tabular data

    Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning models for tabular data. In M. Ranzato, A. Beygelzimer, Y . Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 1...

    2021

  12. [20]

    On embeddings for numerical features in tabular deep learning

    Yury Gorishniy, Ivan Rubachev, and Artem Babenko. On embeddings for numerical features in tabular deep learning. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 24991–25004. Curr...

    2022

  13. [21]

    Why do tree-based models still outperform deep learning on typical tabular data? In S

    Léo Grinsztajn, Edouard Oyallon, and Gaël Varoquaux. Why do tree-based models still outperform deep learning on typical tabular data? In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pa...

    2022

  14. [22]

    TabMT: Generating tabular data with masked transformers

    Manbir Gulati and Paul Roysdon. TabMT: Generating tabular data with masked transformers. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 46245–46254, Red Hook, NY , USA, 2023. ...

    2023

  15. [23]

    Hedonic housing prices and the demand for clean air

    David Harrison and Daniel L Rubinfeld. Hedonic housing prices and the demand for clean air. JEEM, 5 (1):81–102, 1978. ISSN 0095-0696

    1978

  16. [24]

    Synthetic data generation for tabular health records: A systematic review

    Mikel Hernandez, Gorka Epelde, Ane Alberdi, Rodrigo Cilla, and Debbie Rankin. Synthetic data generation for tabular health records: A systematic review. Neurocomputing, 493:28–45, 2022. ISSN 0925-2312. doi: https://doi.org/10.1016/j.neucom.2022.04.053

    2022 doi

  17. [25]

    Denoising diffusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 6840–6851. Curran Associates, Inc., 2020

    2020

  18. [26]

    Statlog (German Credit Data)

    Hans Hofmann. Statlog (German Credit Data). UCI Machine Learning Repository, 1994. DOI: https://doi.org/10.24432/C5NC77

    1994 doi

  19. [27]

    TabPFN: A transformer that solves small tabular classification problems in a second

    Noah Hollmann, Samuel Müller, Katharina Eggensperger, and Frank Hutter. TabPFN: A transformer that solves small tabular classification problems in a second. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenRevi...

    2023

  20. [28]

    Squared earth mover’s distance-based loss for training deep neural networks

    Le Hou, Chen-Ping Yu, and Dimitris Samaras. Squared earth mover’s distance-based loss for training deep neural networks. CoRR, abs/1611.05916, 2016. URL http://arxiv.org/abs/1611.05916

    2016 arXiv

  21. [29]

    Xin Huang, Ashish Khetan, Milan Cvitkovic, and Zohar S. Karnin. TabTransformer: Tabular data modeling using contextual embeddings. CoRR, abs/2012.06678, 2020. URL https://arxiv.org/abs/2012. 06678

    2012 arXiv

  22. [30]

    Generating and imputing tabular data via diffusion and flow-based gradient-boosted trees

    Alexia Jolicoeur-Martineau, Kilian Fatras, and Tal Kachman. Generating and imputing tabular data via diffusion and flow-based gradient-boosted trees. In Sanjoy Dasgupta, Stephan Mandt, and Yingzhen Li, editors, Proceedings of The 27th International Conference on Artificial Int...

    2024

  23. [31]

    LightGBM: A highly efficient gradient boosting decision tree

    Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie- Yan Liu. LightGBM: A highly efficient gradient boosting decision tree. In I. Guyon, U. V on Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors,Advances in...

    2017

  24. [32]

    Kingma and Max Welling

    Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In Yoshua Bengio and Yann LeCun, editors, 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014. 12

    2014

  25. [33]

    Scaling up the accuracy of Naive-Bayes classifiers: a decision-tree hybrid

    Ron Kohavi. Scaling up the accuracy of Naive-Bayes classifiers: a decision-tree hybrid. In Proceedings of the Second International Conference on Knowledge Discovery and Data Mining, KDD’96, page 202–207. AAAI Press, 1996

    1996

  26. [34]

    Learning probabilistic ordinal embeddings for uncertainty-aware regression

    Wanhua Li, Xiaoke Huang, Jiwen Lu, Jianjiang Feng, and Jie Zhou. Learning probabilistic ordinal embeddings for uncertainty-aware regression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13896–13905, June 2021

    2021

  27. [35]

    MacQueen

    J. MacQueen. Some methods for classification and analysis of multivariate observations. In Lucien M. Le Cam and Jerzy Neyman, editors, Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability, volume 1, pages 281–297. University of California Pres...

    1967

  28. [36]

    H. B. Mann and D. R. Whitney. On a test of whether one of two random variables is stochastically larger than the other. The Annals of Mathematical Statistics, 18(1):50 – 60, 1947. doi: 10.1214/aoms/1177730491

    1947

  29. [37]

    Quantitative structure–activity relationship models for ready biodegradability of chemicals

    Kamel Mansouri, Tine Ringsted, Davide Ballabio, Roberto Todeschini, and Viviana Consonni. Quantitative structure–activity relationship models for ready biodegradability of chemicals. Journal of Chemical Information and Modeling, 53(4):867–878, 2013. doi: 10.1021/ci4000213. PMI...

    2013 doi

  30. [38]

    Unmasking trees for tabular data

    Calvin McCarter. Unmasking trees for tabular data. CoRR, abs/2407.05593, 2024. doi: 10.48550/ARXIV . 2407.05593. URL https://doi.org/10.48550/arXiv.2407.05593

    2024 doi arXiv

  31. [39]

    Using data mining for bank direct marketing: An application of the CRISP-DM methodology

    Sérgio Moro, Paulo Cortez, and Raul Laureano. Using data mining for bank direct marketing: An application of the CRISP-DM methodology. In Proceedings of the European Simulation and Modelling Conference, 10 2011

    2011

  32. [40]

    SLACE: A monotone and balance-sensitive loss function for ordinal regression

    Inbar Nachmani, Bar Genossar, Coral Scharf, Roee Shraga, and Avigdor Gal. SLACE: A monotone and balance-sensitive loss function for ordinal regression. Proceedings of the AAAI Conference on Artificial Intelligence, 39(18):19598–19606, Apr. 2025. doi: 10.1609/aaai.v39i18.34158

    2025 doi

  33. [41]

    Cat2Vec with position encoding: A new approach for handling ordinal features using learned embeddings with positional encoding

    Aditya Narvekar and Shubh Mehta. Cat2Vec with position encoding: A new approach for handling ordinal features using learned embeddings with positional encoding. International Journal of Computer Applications, 186(44):9–15, Oct 2024. ISSN 0975-8887. doi: 10.5120/ijca2024924052

    2024 doi

  34. [42]

    Andrew Y . Ng. Feature selection, L1 vs. L2 regularization, and rotational invariance. InProceedings of the Twenty-First International Conference on Machine Learning, ICML ’04, page 78, New York, NY , USA,

  35. [43]

    Ordinal regression with multiple output CNN for age estimation

    Zhenxing Niu, Mo Zhou, Le Wang, Xinbo Gao, and Gang Hua. Ordinal regression with multiple output CNN for age estimation. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4920–4928, 2016. doi: 10.1109/CVPR.2016.532

    2016 doi

  36. [44]

    Data synthesis based on generative adversarial networks

    Noseong Park, Mahmoud Mohammadi, Kshitij Gorde, Sushil Jajodia, Hongkyu Park, and Youngmin Kim. Data synthesis based on generative adversarial networks. Proc. VLDB Endow., 11(10):1071–1083, Jun

  37. [45]

    Proposed guide on synthetic data generation

    PDPC. Proposed guide on synthetic data generation. Personal Data Protection Com- mission, July 2024. URL https://www.pdpc.gov.sg/help-and-resources/2024/07/ proposed-guide-on-synthetic-data-generation . Accessed: 2024-12-26

    2024

  38. [46]

    CatBoost: unbiased boosting with categorical features

    Liudmila Prokhorenkova, Gleb Gusev, Aleksandr V orobev, Anna Veronika Dorogush, and Andrey Gulin. CatBoost: unbiased boosting with categorical features. In S. Bengio, H. Wallach, H. Larochelle, K. Grau- man, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Informat...

    2018

  39. [47]

    Synthcity: a benchmark framework for diverse use cases of tabular synthetic data

    Zhaozhi Qian, Rob Davis, and Mihaela van der Schaar. Synthcity: a benchmark framework for diverse use cases of tabular synthetic data. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36,...

    2023

  40. [48]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI Blog, 1(8), 2019. URL https://cdn.openai.com/ better-language-models/language_models_are_unsupervised_multitask_learners.pdf

    2019

  41. [49]

    Hamprecht, Yoshua Bengio, and Aaron C

    Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred A. Hamprecht, Yoshua Bengio, and Aaron C. Courville. On the spectral bias of neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference ...

  42. [50]

    DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter

    Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. In NeurIPS EMC2 Workshop, 2019

    2019

  43. [51]

    TabDiff: a mixed-type diffusion model for tabular data generation

    Juntong Shi, Minkai Xu, Harper Hua, Hengrui Zhang, Stefano Ermon, and Jure Leskovec. TabDiff: a mixed-type diffusion model for tabular data generation. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  44. [52]

    Deep neural networks for rank-consistent ordinal regression based on conditional probabilities

    Xintong Shi, Wenzhi Cao, and Sebastian Raschka. Deep neural networks for rank-consistent ordinal regression based on conditional probabilities. Pattern Anal. Appl., 26(3):941–955, June 2023. ISSN 1433-7541. doi: 10.1007/s10044-023-01181-9

    2023 doi

  45. [53]

    Tabular data: Deep learning is not all you need

    Ravid Shwartz-Ziv and Amitai Armon. Tabular data: Deep learning is not all you need. Inf. Fusion, 81(C): 84–90, May 2022. ISSN 1566-2535. doi: 10.1016/j.inffus.2021.11.011

    2022 doi

  46. [54]

    Everhart, W

    Jack Smith, J. Everhart, W. Dickson, W. Knowler, and Richard Johannes. Using the ADAP learning algorithm to forcast the onset of diabetes mellitus. Proc. Annu. Symp. Comput. Appl. Med. Care, 10, 11 1988

    1988

  47. [55]

    Solatorio and Olivier Dupriez

    Aivin V . Solatorio and Olivier Dupriez. REaLTabFormer: Generating realistic relational and tabular data using transformers. CoRR, abs/2302.02041, 2023. doi: 10.48550/ARXIV .2302.02041. URL https://doi.org/10.48550/arXiv.2302.02041

    2023 doi arXiv

  48. [56]

    Nick Street, W

    W. Nick Street, W. H. Wolberg, and O. L. Mangasarian. Nuclear feature extraction for breast tumor diagnosis. In Raj S. Acharya and Dmitry B. Goldgof, editors,Biomedical Image Processing and Biomedical Visualization, volume 1905, pages 861 – 870. International Society for Optic...

    1905

  49. [57]

    van Rijn, Bernd Bischl, and Luis Torgo

    Joaquin Vanschoren, Jan N. van Rijn, Bernd Bischl, and Luis Torgo. OpenML: Networked science in machine learning. SIGKDD Explorations, 15(2):49–60, 2013. doi: 10.1145/2641190.2641198

    2013

  50. [58]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. V on Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural...

    2017

  51. [59]

    Wang and Binh P

    Alex X. Wang and Binh P. Nguyen. TTV AE: Transformer-based generative modeling for tabular data generation. Artificial Intelligence, 340:104292, 2025. ISSN 0004-3702. doi: https://doi.org/10.1016/j. artint.2025.104292

    2025

  52. [60]

    HARMONIC: Harnessing llms for tabular data synthesis and privacy protection

    Yuxin Wang, Duanyu Feng, Yongfu Dai, Zhengyu Chen, Jimin Huang, Sophia Ananiadou, Qianqian Xie, and Hao Wang. HARMONIC: Harnessing llms for tabular data synthesis and privacy protection. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, edit...

    2024

  53. [61]

    Watson, Kristin Blesch, Jan Kapar, and Marvin N

    David S. Watson, Kristin Blesch, Jan Kapar, and Marvin N. Wright. Adversarial random forests for density estimation and generative modeling. In Francisco Ruiz, Jennifer Dy, and Jan-Willem van de Meent, editors, Proceedings of The 26th International Conference on Artificial Int...

    2023

  54. [62]

    Multisurface method of pattern separation for medical diagnosis applied to breast cytology

    W H Wolberg and O L Mangasarian. Multisurface method of pattern separation for medical diagnosis applied to breast cytology. PNAS, 87(23):9193–9196, 1990

    1990

  55. [63]

    CvT: Introducing convolutions to vision transformers

    Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. CvT: Introducing convolutions to vision transformers. In2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 22–31, 2021. doi: 10.1109/ICCV48922.2021.00009

    2021

  56. [64]

    Modeling tabular data using conditional GAN

    Lei Xu, Maria Skoularidou, Alfredo Cuesta-Infante, and Kalyan Veeramachaneni. Modeling tabular data using conditional GAN. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d 'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume ...

    2019

  57. [65]

    Mixed-type tabular data synthesis with score-based diffusion in latent space

    Hengrui Zhang, Jiani Zhang, Zhengyuan Shen, Balasubramaniam Srinivasan, Xiao Qin, Christos Faloutsos, Huzefa Rangwala, and George Karypis. Mixed-type tabular data synthesis with score-based diffusion in latent space. In The Twelfth International Conference on Learning Represen...

    2024

  58. [66]

    Improving deep regression with ordinal entropy

    Shihao Zhang, Linlin Yang, Michael Bi Mi, Xiaoxu Zheng, and Angela Yao. Improving deep regression with ordinal entropy. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. 14

    2023

  59. [67]

    Zilong Zhao, Aditya Kunar, Robert Birke, and Lydia Y . Chen. CTAB-GAN: Effective table data synthesiz- ing. In Vineeth N. Balasubramanian and Ivor Tsang, editors, Proceedings of The 13th Asian Conference on Machine Learning, volume 157 of Proceedings of Machine Learning Resear...

    2021

  60. [68]

    Zilong Zhao, Robert Birke, and Lydia Y . Chen. TabuLa: Harnessing language models for tabular data synthesis. CoRR, abs/2310.12746, 2023. doi: 10.48550/ARXIV .2310.12746. URL https://doi.org/ 10.48550/arXiv.2310.12746

    2023 doi arXiv

  61. [69]

    kmeans” and “quantile

    Zilong Zhao, Aditya Kunar, Robert Birke, Hiek Van der Scheer, and Lydia Y . Chen. CTAB-GAN+: enhancing tabular data synthesis. Frontiers in Big Data, 6, 2024. ISSN 2624-909X. doi: 10.3389/fdata. 2023.1296508. 15 Algorithm 1 Training of TabTreeFormer 1: Input: Tabular dataset X...

    2024

  62. [72]

    Detailed theoretical proofs are provided in the Appendix

    Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? Answer: [Yes] Justification: We provide our algorithm contribution in Section 3. Detailed theoretical proofs are provided in the Appendix. Experime...

  63. [73]

    Limitations

    Limitations Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: For our algorithm design, limitations and future work are discussed in Ap- pendix F. For broader challenges in generative model design for tabular da...

  64. [74]

    Except the Theorem 1 which is easy to prove, we provide detailed theoretical proofs for the rest of Theorems and Lemma in Appendix B.5, Appendix C.1 and Appendix C.2

    Theory assumptions and proofs Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes] 29 Justification: We provide 3 theorems and 1 lemma in our paper. Except the Theorem 1 which is easy to pro...

  65. [75]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experimental result reproducibility Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and da...

  66. [76]

    All datasets are sourced from OpenML; detailed loading instructions are provided in Appendix D.2

    Open access to data and code Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: The code is available via t...

  67. [77]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experimental setting/details Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? Answer: [Yes] Justification: This information is provided ...

  68. [78]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experiment statistical significance Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes] Justification: We repeated all our experiments 3 times, we report...

  69. [79]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experiments compute resources Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [Yes] Justification: In Appendix D.1. Guidel...

  70. [80]

    Guidelines: • The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics

    Code of ethics Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? Answer: [Yes] Justification: We reviewed it and comply with the code. Guidelines: • The answer NA means that...

  71. [81]

    Guidelines: • The answer NA means that there is no societal impact of the work performed

    Broader impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: In Appendix G. Guidelines: • The answer NA means that there is no societal impact of the work performed. ...

  72. [82]

    Safeguards Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] 32 Justification: Guidelines: ...

  73. [83]

    Guidelines: • The answer NA means that the paper does not use existing assets

    Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We acknowledged ...

  74. [84]

    • Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates

    New assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: https://anonymous.4open.science/r/tabtreeformer-9585 Guidelines: • The answer NA means that the paper does not relea...

  75. [85]

    Crowdsourcing and research with human subjects Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? A...

  76. [86]

    • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research

    Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals...

  77. [87]

    Declaration of LLM usage Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the ...

    2025

  78. [2004]

    ISBN 1581138385

    Association for Computing Machinery. ISBN 1581138385. doi: 10.1145/1015330.1015435

  79. [2018]

    doi: 10.14778/3231751.3231757

    ISSN 2150-8097. doi: 10.14778/3231751.3231757

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.