Pith. sign in

REVIEW 3 major objections 5 minor 3 cited by

Table Foundation Models: on knowledge pre-training for tabular learning

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read TARTE pre-trains a transformer on knowledge-base facts so that frozen table representations, used alone or boosted with a ridge or a strong base model, beat leading tree-based and pre-trained tabular predictors while improving the…

desk verdict Strong empirical paper; the reusable-knowledge result is real but narrower than claimed—transfer to out-of-distribution tables is untested. read the letter →

arxiv 2505.14415 v2 pith:TWP2EUVR submitted 2025-05-20 cs.LG

Myung Jun Kim , Félix Lefebvre , Gaëtan Brison , Alexandre Perez-Lebel , Gaël Varoquaux This is my paper
classification cs.LG
keywords tablefoundationmodelsknowledgepre-trainingtabularlearningcontrastiverepresentationtransferfew-shotdatasemantics
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

TARTE is a pre-trained transformer for tabular learning that tries to make knowledge pre-training actually reusable. It encodes each row as column-cell pairs, using a language model on strings and column names and a power transform on numbers, and pre-trains contrastively on 30 million facts drawn from two large knowledge bases. The paper's central claim is that these pre-trained representations work out of the box: frozen, they can be fed to a ridge, XGBoost, or TabPFNv2, and boosted variants improve prediction while staying computationally cheap. Across small samples (32 to 1,024 rows) and at 10,000 rows, TARTE variants rank best and sit on the prediction/computation Pareto frontier, beating strong tree-based and pre-trained baselines. Domain fine-tuning also gives representations that transfer to new tables without joint refitting.

What carries the argument

The load-bearing object is the column-cell pair representation: each entry is mapped to a common dimension as the sum of a language-model embedding of its column name and a language-model embedding of its string value (or a power-transformed, column-scaled numerical value, with datetimes converted to fractional years), and the row is a stack of these pairs plus a learnable readout token fed into a three-layer transformer. Pre-training uses contrastive learning with positives formed by replacing one or two facts (for example, Paris becomes London), so the readout embedding is pulled toward embeddings of similar rows; the training data merges a cleaned large knowledge base with richer numerical facts from a second one, giving 30 million facts across 687 relations. This mechanism is what carries the argument: the contrastive objective is meant to encode a smoothness prior, and the enriched data is meant to supply diverse world knowledge, so the readout token becomes a reusable table featurizer.

What would settle it

Take the TARTE architecture and pre-training data, replace the fact-swapping positives with random row pairs so no smoothness prior is taught, keep everything else fixed, and compare frozen-embedding performance on the 51 benchmark datasets: if the random-pair version matches TARTE's gains, the knowledge pre-training claim is falsified; alternatively, a benchmark of tables whose target is deliberately non-smooth in row space should shrink TARTE's advantage if the prior is real.

Watch

Extended reading notes

Core claim

The discovery the paper argues for is that a table foundation model can capture data semantics from knowledge bases and deliver reusable representations: TARTE's knowledge pre-training, unlike earlier string-and-number models whose gains come mainly from fine-tuning, produces embeddings that already help simple downstream learners. In the reported benchmarks, TARTE variants are the best-performing models regardless of sample size, and the frozen-or-boosted use gives most of the accuracy of fine-tuning at a fraction of the runtime. The representations also support domain specialization: fine-tuning on source tables once yields features that improve learning on new target tables in the same domain, without retraining on the source data.

Load-bearing premise

The load-bearing premise is that contrastive training on swapped knowledge-base facts teaches a smoothness prior that transfers to real downstream tables, rather than the gains coming mostly from FastText and the transformer architecture; the paper's own ablation shows random-weight FastText already performs well, so the added value of pre-training rests on that transfer.

Editorial extensions

If this is right

  • Used as a frozen featurizer, TARTE turns a plain ridge regression into a competitive tabular learner, outperforming the same ridge on standard table vectorization at small sample sizes.
  • Boosting a strong base model on TARTE residual features improves both XGBoost and TabPFNv2, and is the configuration that best combines accuracy with low runtime.
  • Fine-tuned TARTE remains the most accurate variant at n = 10,000, but costs about 14 times more than the boosted frozen representation, making the boosted version the better trade-off for deployment.
  • Once TARTE is fine-tuned on one or several source tables of a domain, it can be reused for new target tables in that domain without refitting the source data, a form of transfer that prior joint-learning baselines do not offer cheaply.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the fact-swapping contrastive prior suggests TARTE should help most on tasks where target values are locally smooth in row space; a direct test would be to benchmark on deliberately discontinuous label functions, where the pre-training gain should shrink.
  • Beyond the paper: because TARTE's domain specialization reuses fitted source models, it offers a path to cross-table transfer without sharing the source tables themselves, which could matter for privacy-sensitive settings.
  • Beyond the paper: the observed failure mode on long string entries points to a concrete extension, replacing FastText with a sentence-level encoder inside the same architecture should extend TARTE's benefits to tables with verbose text cells.
  • Beyond the paper: the Pareto results imply that TARTE-style knowledge pre-training could be combined with in-context learners other than TabPFNv2, such as newer prior-fitted networks, to compound gains, an experiment the paper does not run.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces TARTE, a transformer-based table encoder that maps column--cell pairs into a shared embedding space using FastText strings, a relation-wise power transform for numbers, and a datetime converter. TARTE is pre-trained with a contrastive objective on a large knowledge base built from YAGO4.5 enriched with numerical facts from Wikidata. The paper evaluates three reuse modes: fine-tuning, frozen featurization followed by Ridge/XGB/TabPFNv2, and a boosting scheme where TARTE embeddings fit the residuals of a base model. The empirical study covers 51 datasets from the CARTE benchmark at multiple sample sizes, additional numerical tables from TabLLM, and domain-specialization experiments on 12 domains. The main claims are that TARTE representations improve downstream prediction and improve the prediction/computation trade-off compared with XGBoost, CatBoost, TabPFNv2, and CARTE, and that the benefit comes from knowledge pre-training rather than architecture alone.

Significance. If the central claim holds, TARTE would be a meaningful step toward reusable table foundation models: it is the first model in this line, to my knowledge, whose pre-trained representations are shown to help without task-specific fine-tuning and at low downstream cost. The paper's strengths are substantial: 51 datasets, multiple training sizes from 32 to 10,000, several post-training protocols, ablations in Figure 6 that separate architecture from pre-training data, statistical comparisons via critical difference diagrams, Pareto analyses of prediction versus runtime, and domain-specialization experiments against the strong CARTE multi-table baseline. The study is more controlled than much of the tabular foundation-model literature. The main weakness is that the evidence for the general reusable-knowledge claim is restricted to tables whose strings resemble the pre-training corpus; the paper's own meta-analysis quantifies this dependence but does not test the representation on out-of-distribution tables.

major comments (3)
  1. [Section 4.3, Table 1; Section 4.1]
  2. [Figure 6, Section 4.3]
  3. [Section 4.4, Figures 7 and 8]
minor comments (5)
  1. [Abstract, Section 1]
  2. [Section 3.2, Appendix A.4]
  3. [Table 1]
  4. [Figure 6]
  5. [Appendix B.1]

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation; the central claim is empirical and rests on separate pre-training and evaluation, with minor self-citation that is not load-bearing.

full rationale

TARTE's central claim is that knowledge pre-training on relational data yields reusable representations for downstream tabular learning. This is an empirical claim, not a derived identity: the pre-training objective (contrastive InfoNCE on Wikidata/YAGO facts) is defined independently of the downstream benchmark, and the benchmark performance is measured on held-out tables rather than read off from the pre-training loss. The paper's own ablations (Figure 6: random weights, MinHash, YAGO3, enriched YAGO4.5) separate the contributions of architecture, string embeddings, and pre-training, so the improvement is not forced by construction. The factors-of-success regression (Table 1) is a post-hoc explanatory analysis; it does not fit the central result, and its finding that inlier probability and string similarity correlate with gains is a stated limitation rather than a definitional target. Self-citations to CARTE (Kim et al., 2024) are frequent, and the main benchmark originates from the same group, but that benchmark is a fixed, publicly released collection and the comparisons include external baselines (XGBoost, CatBoost, TabPFNv2) and external TabLLM datasets. At most this is a generalizability concern about lexical overlap between pre-training and benchmark strings; it is not circularity, because success is not defined in terms of the pre-training corpus or of CARTE's predictions.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim is empirical: TARTE's representations improve downstream learning. The model introduces no invented entities. Free parameters are limited to per-relation preprocessing and hyperparameters. The load-bearing assumptions are that knowledge-base triples proxy tabular rows, FastText captures semantics, contrastive learning transfers, and downstream tables resemble the pre-training data.

free parameters (2)
  • Yeo-Johnson power transform lambdas per relation = Not stated; estimated on pre-training data
    Numerical and datetime columns are transformed relation-wise before embedding; these parameters are learned from the pre-training data and affect all downstream representations (Section 3.2).
  • Number of facts per row after trimming = Fixed across batch, exact value not stated
    Rows are trimmed to a fixed number of facts to mimic table width; this is a hand-chosen hyperparameter (Section 3.2, Appendix A.4).
assumptions (4)
  • domain assumption Knowledge-base triples (h,r,t) are a faithful proxy for tabular rows, and pre-training on them transfers to downstream tables.
    The entire pre-training pipeline is built on this premise (Section 3.2).
  • domain assumption FastText embeddings capture enough semantics for short strings in tables.
    TARTE relies on FastText for all string tokens; the authors acknowledge limitations for long strings (Appendix A.2).
  • domain assumption Contrastive learning with in-batch negatives and substitution positives induces a smooth representation space useful for prediction.
    The positive-pair construction is justified by smoothness, not by an external benchmark (Section 3.2).
  • domain assumption Downstream tables share vocabulary and semantics with the pre-training knowledge base.
    The factors-of-success analysis in Table 1 shows TARTE helps most when strings are inliers to pre-training (Section 4.3).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Table Foundation Models: on knowledge pre-training for tabular learning." pith.science (2026). https://pith.science/paper/TWP2EUVR

@misc{pith2026250514415,
  author       = {Pith},
  title        = {Pith review of: Table Foundation Models: on knowledge pre-training for tabular learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TWP2EUVR}},
  note         = {Machine review of arXiv:2505.14415}
}
read the original abstract

Table foundation models bring high hopes to data science: pre-trained on tabular data to embark knowledge or priors, they should facilitate downstream tasks on tables. One specific challenge is that of data semantics: numerical entries take their meaning from context, e.g., column name. Pre-trained neural networks that jointly model column names and table entries have recently boosted prediction accuracy. While these models outline the promises of world knowledge to interpret table values, they lack the convenience of popular foundation models in text or vision. Indeed, they must be fine-tuned to bring benefits, come with sizeable computation costs, and cannot easily be reused or combined with other architectures. Here we introduce TARTE, a foundation model that transforms tables to knowledge-enhanced vector representations using the string to capture semantics. Pre-trained on large relational data, TARTE yields representations that facilitate subsequent learning with little additional cost. These representations can be fine-tuned or combined with other learners, giving models that push the state-of-the-art prediction performance and improve the prediction/computation performance trade-off. Specialized to a task or a domain, TARTE gives domain-specific representations that facilitate further learning. Our study demonstrates an effective approach to knowledge pre-training for tabular learning.

Figures

Figures reproduced from arXiv: 2505.14415 by the authors.

Figure 1
Figure 1. Transformer-based architecture of TARTE. TARTE models a row in a table as a set of column(E)– cell(X) pairs. Given a tabular data with multiple data types, TARTE maps the representation of column and cell values to the same dimension using a language model (LM). From the mapped input, the transformer takes in a combination of both column and cell information to contextualize the cell content. 3.1 A transformer-based… view at source ↗
Figure 2
Figure 2. TARTE Pre-training. We extract facts from knowledge bases and replicate the input structure in [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. TARTE performs best for learning on small tables – Top: Learning curve for normalized pre￾diction scores for regression and classification. In general, pre-trained models perform better, with variants of TARTE surpassing all baseline models. Middle: Critical difference diagram of average rank at n = 32 and 1 024. Bottom: Pareto diagrams normalized prediction scores with respect to total runtime (log-scale). Efficien… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Results for n = 10 000 – Left: Pareto diagram – Right: critical difference diagram of average rank Fine-tuned TARTE and TARTE boosting continues to surpass the baselines, but fine-tuning TARTE brings more benefits than on smaller data [PITH_FULL_IMAGE:figures/full_fig…
Figure 6
Figure 6. Figure 6: Ablating architecture, pre-training, and preprocessing components. A ridge is fitted with em￾beddings from different schemes: random weights (no pre-training); replacement of FastText with skrub’s “MinHash” encoder; TARTE and CARTE pre-trained with YAGO3 and Enriched Y…
Figure 7
Figure 7. Figure 7: Domain specialization from a single source – Left: Pareto diagram – Right: critical difference diagram of average rank. ‘DS’ and ‘MT’ denote Domain-Specialized and Multi-Tables schemes, respectively. Models that blends inference from the target table with representatio…
Figure 9
Figure 9. Figure 9: Results for small tables – Left: Pareto diagram – Right: critical difference diagram of average rank TARTE surpasses the baselines, and can act as an effective table preparator, especially for small number of training samples (n ≤ 256). 20 [PITH_FULL_IMAGE:figures/ful…
Figure 10
Figure 10. Figure 10: Results for domain specialization with single source table Regardless of the train-sizes, TARTE– B can provide benefits with domain specialization. Once domain specialized models are available, TARTE can readily benefit without requiring complex refitting with the sou…

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. No Data? No Problem: Robust Vision-Tabular Learning with Missing Values

    cs.CV 2025-12 conditional novelty 5.0 of 10

    Missingness-aware contrastive pretraining plus a 'tabular more vs fewer' ranking loss lets a vision-tabular model perform across the full 0%–100% range of tabular attribute availability.

  2. Universal Embeddings of Tabular Data

    cs.LG 2025-07 conditional novelty 5.0 of 10

    Rows of a table are embedded by training a graph auto-encoder on a table-derived weighted graph, giving smaller universal embeddings than EmbDI-style random-walk embeddings on two Kaggle datasets.

  3. TReB: A Comprehensive Benchmark for Evaluating Table Reasoning Capabilities of Large Language Models

    cs.CL 2025-06 conditional novelty 5.0 of 10

    TReB evaluates 26 large language models on 26 table reasoning subtasks using textual, programmatic, and interleaved reasoning modes, finding that the best model reaches only about 70 on a 0-100 judging scale.

Reference graph

Works this paper leans on

27 extracted references · 12 canonical work pages · cited by 3 Pith papers

  1. [1]

    Achiam, S

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774,

    arXiv

  2. [7]

    D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948 ,

    arXiv

  3. [10]

    Koshil, T

    M. Koshil, T. Nagler, M. Feurer, and K. Eggensperger. Towards localization via data embedding for tabpfn. In NeurIPS 2024 Third Table Representation Learning Workshop,

    work page 2024

  4. [11]

    Liu and H.-J

    S.-Y . Liu and H.-J. Ye. Tabpfn unleashed: A scalable and effective solution to tabular classification problems. arXiv preprint arXiv:2502.02527,

    arXiv

  5. [12]

    Mikolov, E

    T. Mikolov, E. Grave, P. Bojanowski, C. Puhrsch, and A. Joulin. Advances in pre-training distributed word representations. arXiv preprint arXiv:1712.09405,

    arXiv

  6. [13]

    A. v. d. Oord, Y . Li, and O. Vinyals. Representation learning with contrastive predictive coding.arXiv preprint arXiv:1807.03748,

    arXiv

  7. [15]

    Reimers and I

    N. Reimers and I. Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Pro- ceedings of the 2019 Conference on Empirical Methods in Natural Language Processing . Association for Computational Linguistics, 11

    work page 2019

  8. [16]

    URL https://arxiv.org/abs/1908.10084. D. Saul. Biggest market loss in history: Nvidia stock sheds nearly $600 billion as DeepSeek shakes AI darling, Jan

    arXiv 1908

Show all 27 references
  1. [17]

    14 KNOWLEDGE PRE -TRAINING OF TABLE FOUNDATION MODELS - SEPTEMBER 9, 2025 R

    URL https://www.forbes.com/sites/dereksaul/2025/01/27/biggest- market-loss-in-history-nvidia-stock-sheds-nearly-600-billion-as-deepseek-shakes- ai-darling/. 14 KNOWLEDGE PRE -TRAINING OF TABLE FOUNDATION MODELS - SEPTEMBER 9, 2025 R. Shwartz-Ziv and A. Armon. Tabular data: Dee...

    2025

  2. [18]

    Spinaci, M

    M. Spinaci, M. Polewczyk, J. Hoffart, M. C. Kohler, S. Thelin, and T. Klein. Portal: Scalable tabular foundation models via content-specific tokenization. In NeurIPS 2024 Third Table Representation Learning Workshop,

    2024

  3. [19]

    Thomas, J

    V . Thomas, J. Ma, R. Hosseinzadeh, K. Golestan, G. Yu, M. V olkovs, and A. L. Caterini. Retrieval & fine- tuning for in-context tabular models. In ICML 2024 Workshop on In-Context Learning,

    2024

  4. [20]

    Varoquaux, A

    G. Varoquaux, A. S. Luccioni, and M. Whittaker. Hype, sustainability, and the price of the bigger-is-better paradigm in ai. arXiv preprint arXiv:2409.14160,

    arXiv

  5. [22]

    Ye, S.-Y

    H.-J. Ye, S.-Y . Liu, H.-R. Cai, Q.-L. Zhou, and D.-C. Zhan. A closer look at deep learning on tabular data. arXiv preprint arXiv:2407.00956,

  6. [23]

    Zhang, X

    T. Zhang, X. Yue, Y . Li, and H. Sun. Tablellama: Towards open large generalist models for tables. InProceed- ings of the 2024 Conference of the North American Chapter of the Association for Computational Linguis- tics: Human Language Technologies (Volume 1: Long Papers), page...

    2024

  7. [24]

    The column information is crucial to supplement context for the transformers (Kim et al., 2024)

    15 KNOWLEDGE PRE -TRAINING OF TABLE FOUNDATION MODELS - SEPTEMBER 9, 2025 A Further details on backbone and pre-training A.1 Modeling with column–cell pairs: context-aware for transformers Given a table, TARTE models with a set of column – cell pairs and combine the embeddings...

    2025

  8. [26]

    For the runtime, it measures the total time for data preparation, hyperparameter optimization, and prediction

    The performance was measured with R2 score for regression and the Area Under Receiver Operating Curve (AUROC) for classification tasks. For the runtime, it measures the total time for data preparation, hyperparameter optimization, and prediction. Overall, the results were reco...

    2011

  9. [256]

    For each baseline, some additional details were considered

    The splits were set as same as that of singletables to enable comparable results. For each baseline, some additional details were considered. • TARTE: The runtime of TARTE models include the training time of source tables. For each source table, we fine-tuned the pre-trained T...

    2024

  10. [500]

    For models without native handling missing values, we imputed with the mean for numerical features, and treated as another category for categorical features

    For TableVectorizer from the skrub, categorical columns are differently encoded depending on the cardinality (number of categories): Columns with low cardinality are one-hot encoded while those with high cardinality are encoded using the Gamma-Poisson encoder (Cerda and Varoqu...

    2022

  11. [2014]

    R. Wang, Z. Wang, and J. Sun. Unipredict: Large language models are universal tabular predictors. arXiv preprint arXiv:2310.03266,

    arXiv

  12. [2018]

    J. Qu, D. Holzm ¨uller, G. Varoquaux, and M. L. Morvan. Tabicl: A tabular foundation model for in-context learning on large data. arXiv preprint arXiv:2502.05564,

    arXiv

  13. [2019]

    Bommasani, D

    R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258,

    arXiv

  14. [2020]

    D. Chen, Y . Lin, W. Li, P. Li, J. Zhou, and X. Sun. Measuring and relieving the over-smoothing problem for graph neural networks from the topological view. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 3438–3445, 2020a. J. Chen, Z. Lin, Q....

    arXiv

  15. [2021]

    12 KNOWLEDGE PRE -TRAINING OF TABLE FOUNDATION MODELS - SEPTEMBER 9, 2025 L

    URL https://openreview.net/forum?id= i80OPhOCVH2. 12 KNOWLEDGE PRE -TRAINING OF TABLE FOUNDATION MODELS - SEPTEMBER 9, 2025 L. Bernardi, T. Mavridis, and P. Estevez. 150 successful machine learning models: 6 lessons learned at booking. com. In Proceedings of the 25th ACM SIGKD...

    2025

  16. [2022]

    Feuer, R

    B. Feuer, R. T. Schirrmeister, V . Cherepanova, C. Hegde, F. Hutter, M. Goldblum, N. Cohen, and C. White. Tunetables: Context optimization for scalable prior-data fitted networks. arXiv:2402.11137,

    arXiv

  17. [2023]

    Herzig, T

    J. Herzig, T. Mueller, S. Krichene, and J. Eisenschlos. Open domain question answering over tables via dense retrieval. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 512–519,

    2021

  18. [2024]

    den Breejen, S

    F. den Breejen, S. Bae, S. Cha, and S.-Y . Yun. Fine-tuned in-context learning transformers are excellent tabular data classifiers. arXiv preprint arXiv:2405.13396,

    arXiv

  19. [2025]

    Holzm¨uller, L

    13 KNOWLEDGE PRE -TRAINING OF TABLE FOUNDATION MODELS - SEPTEMBER 9, 2025 D. Holzm¨uller, L. Grinsztajn, and I. Steinwart. Better by default: Strong pre-tuned mlps and boosted trees on tabular data. arXiv preprint arXiv:2407.04491,

    2025 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.