Pith. sign in

REVIEW 3 major objections 4 minor 38 references

Do Tabular Foundation Models Agree with Themselves?

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Every tabular foundation model tested fails basic probability checks

desk verdict Crisp negative result on TFM consistency; the classification half is exact and convincing, but the regression half overclaims the rigor of its discretization argument. read the letter →

arxiv 2608.06004 v1 pith:CLXHSH5P submitted 2026-08-06 cs.LG

Christian Klötergens , Vijaya Krishna Yalavarthi , Lars Schmidt-Thieme , Tom Hanika This is my paper · ORCID
classification cs.LG
keywords tabularfoundationmodelsmarginalizationconsistencyfactorizationautoregressivesamplingtotalvariationdistanceBayesianposteriorpredictivecoherentsystemsin-contextlearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether tabular foundation models (TFMs) — transformers that predict one column of a table given the others — behave as if their predictions come from any single joint distribution over the table's columns. Since the true posterior is unknown, the authors check two necessary properties that any coherent joint must obey: marginalizing a conditional over an intermediate target must recover the directly predicted marginal, and the two chain-rule orderings must build the same joint. They prove that the second property implies the first, and then measure violations with total variation distance using only each model's own predictions. Across six classifiers and four regressors, on every dataset tested, every model violates both conditions; the authors conclude that the univariate outputs of current TFMs do not arise from any well-defined joint, so autoregressive constructions from them are order-dependent and incoherent. This matters because practitioners use these models to generate multivariate records and to reason about dependence between targets.

What carries the argument

The load-bearing objects are two consistency conditions stated for a pair of targets $A$ and $B$. Marginalization consistency (C1) requires the directly predicted marginal of $A$ to equal the mixture of the conditional of $A$ given $B$ weighted by the marginal of $B$, i.e. the law of total probability. Factorization consistency (C2) requires the two chain-rule products, $A$-then-$B$ and $B$-then-$A$, to be equal, i.e. compatibility of conditionals and marginals. The argument runs on these identities: if a model satisfied (C2), integrating over $B$ would reduce the mixture to the direct marginal, yielding (C1) by normalization; the contrapositive makes the cheaper univariate check a certificate of order-dependence. Violations are quantified by total-variation distance between the two sides of each identity, computed exactly for classification and via deterministic quantile-based discretization for regression; because total-variation distance is non-increasing under coarsening, the regression gaps are lower bounds.

What would settle it

Run the paper's two consistency checks on a model whose predictive conditionals are computed exactly from a single joint distribution (e.g., a hand-built graphical model); if the total-variation distance is not exactly zero, the measurement procedure rather than the model is producing the violation. Alternatively, any current TFM that scores exactly zero on both checks for a dataset with two dependent targets would falsify the paper's universal claim.

Watch

Extended reading notes

Core claim

The paper's central discovery is a universal negative result: every tabular foundation model evaluated — TabPFNv2/v3, TabICLv1/v2, TabDPTv1.2, and TabFM — violates both marginalization consistency (C1) and factorization consistency (C2) on all ten classification and seven regression datasets. The violations are systematic and measurable from the models' own outputs: for classification, the checks are exact finite sums; for regression, they are evaluated on discretized grids with total-variation distance, giving lower bounds. As a corollary, the autoregressive joint constructed by sampling one target and conditioning on it is order-dependent: choosing which target to sample first changes the predicted distribution. The paper also proves that factorization consistency implies marginalization consistency, so each observed marginalization failure already certifies that the two factorization orders disagree.

Load-bearing premise

For regression, the conclusion assumes that the $K=1{,}000$ equal-mass atom quadrature and the 20-cell or $128\times128$ discretization faithfully represent the continuous predictive distributions; the paper argues that coarsening can only hide discrepancies, but no formal error bound for the quadrature is given.

Editorial extensions

If this is right

  • Autoregressive data generation with current TFMs produces order-dependent joints: the distribution over a generated row changes with the sampling order of the columns.
  • Marginalization consistency is a cheap, label-free diagnostic (it compares only univariate heads) and, by the paper's contrapositive, a detected violation certifies that factorization also fails.
  • Benchmarks that rank TFMs by accuracy or RMSE do not capture this axis; the paper finds no systematic correlation between predictive quality and consistency, so a state-of-the-art model can be among the most inconsistent.
  • Training a TFM with a self-supervised objective that swaps target and feature roles (as TabDPT does) does not in itself yield factorization-consistent conditionals.
  • The two consistency conditions extend from a pair of targets to arbitrary subsets, so future models with native multivariate heads would still need to align the directly predicted joint with the joint induced by one-target-at-a-time factorization.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the result holds across more models and datasets, consistency should become a standard evaluation axis alongside accuracy and calibration for any TFM used as a joint density estimator.
  • The two consistency conditions suggest a concrete training target: penalize the marginalization or factorization gap directly, or derive all univariate heads from a shared latent joint so coherence holds by construction.
  • The same label-free TV-measurement lens transfers to any conditionally specified predictor, including large language models, whose autoregressive token probabilities have been separately suspected of violating probability laws.
  • Once a TFM emits classification and regression from shared weights, the mixed categorical-continuous test the paper outlines would probe coherence across the two heads — a boundary current models cannot cross.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper asks whether the univariate predictive distributions issued by tabular foundation models (TFMs) can be the marginals and conditionals of any single joint distribution over the columns of a table. It formalizes two necessary conditions: marginalization consistency (C1), which requires the model's direct marginal for a target to equal the mixture of its conditionals over an intermediate target, and factorization consistency (C2), which requires the two autoregressive chain-rule orders to produce the same joint. The paper proves that (C2) implies (C1), gives a counterexample showing the converse fails, and then evaluates six TFMs on ten classification and seven regression datasets. Classification checks are exact finite sums; regression checks discretize continuous predictive distributions via quantile heads, using K=1,000 equal-mass atoms and a 20-cell or 128x128 grid. The reported result is that every evaluated TFM violates both conditions on every dataset, for both classification and regression, and that the two violation measures correlate strongly.

Significance. If the claims were fully established, the paper would make a valuable contribution: it provides label-free, ground-truth-free consistency diagnostics for TFMs, it draws a clean logical connection between the two forms of inconsistency, and the classification-side experiments are exact finite-sum checks that give mathematically certain violations. The proposed metrics could become a useful additional evaluation axis alongside accuracy and calibration. The paper is also careful in places: the proof of Proposition 1 is a direct calculation, the counterexample in Appendix A correctly separates the two properties, and the classification methodology leaves no room for sampling noise. The main weakness is that the regression-side evidence, which is needed for the abstract's universal claim, relies on discretization and quadrature steps whose error is not bounded and whose preservability of zero-TV for known-consistent models is not validated.

major comments (3)
  1. [§4.2 and Appendix C] The inference from a nonzero discretized TV to a violation of the continuous marginalization condition (C1) is not justified. The experiment computes TV between the S-cell coarsening of the direct marginal and the S-cell coarsening of the K-atom mixture. Coarsening is non-expansive, so TV(q_dir^S, q_mixK^S) is at most TV(q_dir, q_mixK), and the triangle inequality gives TV(q_dir, q_mixK) ≤ TV(q_dir, q_mix) + TV(q_mix, q_mixK). The observed quantity is therefore not a lower bound on TV(q_dir, q_mix); a large value can be produced entirely by quadrature error TV(q_mix, q_mixK). The paper asserts the contrary in §4.2 ('The resulting value for TV is therefore a lower bound ... in particular for the TV between the underlying continuous distributions'), but that assertion holds only for the K-atom approximation, not for the true mixture. A formal bound on the quadrature error, or a controlled experiment with a model known to be continuous-consistent, is needed before the regression violations can be claimed.
  2. [§4.3 and Appendix C] In the factorization check, the two discretized joints are computed by evaluating conditionals at cell representatives (midpoint quadrature in probability space, Appendix C). For a continuous model that is exactly factorization-consistent, the true continuous joints coincide, but the two discretized tables need not coincide, because each cell mass in order a→b uses p(a|b*_l,x) evaluated at a single representative rather than averaged over the cell. Hence a nonzero table TV does not establish a violation of (C2). This is not a minor technicality: the regression factorization violations in Table 2 are the main evidence for the regression part of the abstract's universal claim. The authors should either bound the discretization error, demonstrate that a known-consistent model yields zero table TV on these datasets, or present the numbers as violations of the discretized conditions only.
  3. [Abstract and §4.4] The central conclusion as stated in the abstract and §4.4 ('Every TFM that we evaluate violates both of these requirements for both classification and regression across all datasets') is too strong relative to the evidence. The classification side is exact, but the regression side is not yet rigorously connected to the continuous conditions on (C1) and (C2). Since this is the paper's headline claim, the manuscript needs a major revision either to supply the missing error analysis or validation, or to restrict the claim to the classification results and to the discretized regression checks.
minor comments (4)
  1. [Abstract and §4.1] The phrase 'for both classification and regression' is imprecise because TabICLv1 and TabFM lack probabilistic regression heads; the claim should specify that it applies to the models that provide probabilistic regression.
  2. [Appendix D] The decision to drop instances with NLL below 0.05 is a post hoc filtering step that can affect the reported per-row analysis; it should be described in the main text and accompanied by a sensitivity check over the threshold.
  3. [General] A code or data release link would aid reproducibility of the exact finite-sum classification checks and the deterministic regression pipeline described in Appendix C.
  4. [Table 3] The column header 'Feat. num./cat.' is ambiguous; splitting it into two separate columns for numerical and categorical feature counts would make the table easier to read.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the consistency checks compare a model's own predictions against other predictions of the same model via standard probability identities, with no fitted input being renamed as a prediction.

full rationale

The paper's derivation chain is self-contained. The central comparison evaluates necessary conditions for any joint distribution (law of total probability for (C1), compatibility of conditionals for (C2)) directly from TFM outputs. No parameter is fitted to produce the headline result; pretrained models are used as-is and the metrics are computed in closed form for classification and by a deterministic discretization for regression. Proposition 1 is an algebraic proof that (C2) implies (C1) using only the normalization of the conditional density, and Proposition 2 is an explicit counterexample; neither assumes the empirical conclusion. The only self-citation (Yalavarthi et al. 2026) appears in related work and is not load-bearing. The regression-side quadrature issue (the S=20 coarse-grid TV is a lower bound for the K-atom mixture versus the direct marginal, but not rigorously for the true continuous mixture) is a potential validity/approximation caveat, not a circularity: the observed nonzero TV is not forced by construction, and the paper does not fit a parameter and then call it a prediction. Accordingly, the circularity score is 0.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central claim rests only on standard probability axioms, a normalization assumption on TFM heads, and the interpretation that different context configurations of the same table should induce a coherent joint. No new entities or fitted constants are introduced.

free parameters (1)
  • Regression discretization resolutions = K=1000, S=20, grid=128x128, L=2000
    Chosen by hand as measurement hyperparameters. The lower-bound property of TV means the exact values do not determine whether a violation is detected, only its measured magnitude.
assumptions (4)
  • standard math Law of total probability and chain rule
    Definitions C1 and C2 are direct applications. Any well-defined joint distribution must satisfy both identities.
  • domain assumption Predicted densities are normalized
    The proof of Proposition 1 requires p(b|a) to integrate to 1. TFM heads are treated as normalized distributions, either softmax or quantile-based.
  • domain assumption Context sets differing in target/feature role represent the same underlying table
    C1 and C2 compare predictions from D^A, D^B_{-A}, and D^A_{-B}. The interpretation assumes these are the same data with different column roles, which is required for the consistency check to be meaningful.
  • standard math Total variation distance is non-increasing under coarsening
    Used to argue that the S=20 and 128x128 discretizations yield lower bounds for regression violations; cited to Csiszar and Shields 2004.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Do Tabular Foundation Models Agree with Themselves?." pith.science (2026). https://pith.science/paper/CLXHSH5P

@misc{pith2026260806004,
  author       = {Pith},
  title        = {Pith review of: Do Tabular Foundation Models Agree with Themselves?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CLXHSH5P}},
  note         = {Machine review of arXiv:2608.06004}
}
read the original abstract

Tabular Foundation Models (TFMs) are currently the best approach to tabular prediction problems. They are constructed as transformers that approximate the Bayesian posterior predictive distribution based on a pre-training prior. These univariate predictors can be converted into multivariate ones autoregressively by sampling one target and adding it to the features. However, the faithfulness of the resulting joint has not been investigated. Furthermore, TFMs cannot be evaluated against the posterior itself, at least not on real-world datasets, because the ground-truth distribution is unknown. We therefore propose asking a different question: could a model's predictions result from any joint distribution? To answer this question, we pose two requirements that any such model must satisfy. The first is marginalization consistency, which demands that marginalized conditionals are equal to directly predicted marginals. The second is factorization consistency, which demands that different factorization orders result in equal joint distributions. Every TFM that we evaluate violates both of these requirements for both classification and regression across all datasets.

Figures

Figures reproduced from arXiv: 2608.06004 by the authors.

Figure 1
Figure 1. A marginalization-consistency violation on a sin [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The prior-data fitted network blueprint. (a) A trans [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Factorization- versus marginalization-consistency [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Per-instance predictive loss versus marginalization [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]
Figure 5
Figure 5. Figure 5: Predictive accuracy versus factorization-consistency TV (Table 2), one panel per classification dataset and one marker [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: Predictive accuracy versus marginalization-consistency TV (Table 1), one panel per classification dataset and one [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Predictive RMSE versus factorization-consistency TV (Table 2), one panel per regression dataset and one marker per [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Predictive RMSE versus marginalization-consistency TV (Table 1), one panel per regression dataset and one marker [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 11
Figure 11. Figure 11: Factorization-consistency TV versus the number [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]
Figure 10
Figure 10. Figure 10: Marginalization-consistency TV versus the num [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

38 extracted references · 30 canonical work pages

  1. [1]

    doi:10.1007/0-387-28678-0 , isbn =

    An Introduction to Copulas , author =. doi:10.1007/0-387-28678-0 , isbn =

    doi:10.1007/0-387-28678-0

  2. [2]

    doi:10.48550/arXiv.2602.11139 , urldate =

    Qu, Jingang and Holzm. doi:10.48550/arXiv.2602.11139 , urldate =. arXiv , keywords =:2602.11139 , primaryclass =

    doi:10.48550/arxiv.2602.11139

  3. [3]

    doi:10.48550/arXiv.2605.13986 , urldate =

    Grinsztajn, L. doi:10.48550/arXiv.2605.13986 , urldate =. arXiv , keywords =:2605.13986 , primaryclass =

    arXiv doi:10.48550/arxiv.2605.13986

  4. [4]

    Transformers Can Do Bayesian Inference , booktitle =

    Samuel M. Transformers Can Do Bayesian Inference , booktitle =. 2022 , url =

    2022

  5. [5]

    Nature , volume =

    Accurate Predictions on Small Data with a Tabular Foundation Model , author =. Nature , volume =. doi:10.1038/s41586-024-08328-6 , urldate =

    doi:10.1038/s41586-024-08328-6

  6. [6]

    Effortless,

    Vetter, Julius and Gloeckler, Manuel and Gedon, Daniel and Macke, Jakob H , year = 2025, volume =. Effortless,. Advances in

    work page 2025

  7. [7]

    doi:10.1007/978-3-642-14394-6 , urldate =

    Stochastic. doi:10.1007/978-3-642-14394-6 , urldate =

    doi:10.1007/978-3-642-14394-6

  8. [8]

    Journal of Statistical Planning and Inference , volume =

    Energy Statistics: A Class of Statistics Based on Distances , author =. Journal of Statistical Planning and Inference , volume =

    work page

Show all 38 references
  1. [9]

    Journal of the Royal Statistical Society: Series B (Methodological) , volume =

    Spatial Interaction and the Statistical Analysis of Lattice Systems , author =. Journal of the Royal Statistical Society: Series B (Methodological) , volume =

  2. [10]

    Journal of the American Statistical Association , volume =

    Compatible Conditional Distributions , author =. Journal of the American Statistical Association , volume =

  3. [11]

    Statistical Science , volume =

    Conditionally Specified Distributions: An Introduction , author =. Statistical Science , volume =

  4. [12]

    Journal of Machine Learning Research , volume =

    Dependency Networks for Inference, Collaborative Filtering, and Data Visualization , author =. Journal of Machine Learning Research , volume =

  5. [13]

    The Annals of Probability , volume =

    Limit Theorems for a Class of Identically Distributed Random Variables , author =. The Annals of Probability , volume =. doi:10.1214/009117904000000676 , urldate =

    doi

  6. [14]

    Journal of the Royal Statistical Society Series B: Statistical Methodology , volume=

    Martingale posterior distributions , author=. Journal of the Royal Statistical Society Series B: Statistical Methodology , volume=. 2023 , publisher=

    2023

  7. [15]

    Reliable

    Yalavarthi, Vijaya Krishna and Scholz, Randolf and Kl. Reliable. International Conference on Learning Representations , volume =

  8. [16]

    Uria, Benigno and Murray, Iain and Larochelle, Hugo , year = 2014, month = jan, pages =. A. Proceedings of the 31st

    2014

  9. [17]

    and Bastings, Jasmijn and Poole, Ben and van den Berg, Rianne and Salimans, Tim , year = 2022, month = oct, urldate =

    Hoogeboom, Emiel and Gritsenko, Alexey A. and Bastings, Jasmijn and Poole, Ben and van den Berg, Rianne and Salimans, Tim , year = 2022, month = oct, urldate =. Autoregressive. International

    2022

  10. [18]

    Training and

    Shih, Andy and Sadigh, Dorsa and Ermon, Stefano , year = 2022, volume =. Training and. Advances in

    2022

  11. [19]

    Is In-Context Learning in Large Language Models

    Falck, Fabian and Wang, Ziyu and Holmes, Chris , year = 2024, booktitle =. Is In-Context Learning in Large Language Models

    2024

  12. [20]

    Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) , publisher =

    Deriving Language Models from Masked Language Models , author =. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) , publisher =

  13. [21]

    arXiv preprint arXiv:2301.00068 , year=

    Inconsistencies in masked language models , author=. arXiv preprint arXiv:2301.00068 , year=

    arXiv

  14. [22]

    doi:10.48550/arXiv.2505.08739 , archiveprefix =

    Probability Consistency in Large Language Models: Theoretical Foundations Meet Empirical Discrepancies , author =. doi:10.48550/arXiv.2505.08739 , archiveprefix =. 2505.08739 , primaryclass =

    doi arXiv

  15. [23]

    and Knoblauch, Jeremias and Wei, Susan , year = 2025, number =

    Ng, Kenyon and Fong, Edwin and Frazier, David T. and Knoblauch, Jeremias and Wei, Susan , year = 2025, number =. doi:10.48550/arXiv.2510.25154 , archiveprefix =. 2510.25154 , primaryclass =

    2025 doi arXiv

  16. [24]

    Young, Robin , year = 2026, month = feb, journal =. On the

    2026

  17. [25]

    Proceedings of the workshop on methods for optimizing and evaluating neural language generation , pages=

    BERT has a mouth, and it must speak: BERT as a Markov random field language model , author=. Proceedings of the workshop on methods for optimizing and evaluating neural language generation , pages=

  18. [26]

    Garnelo, Marta and Rosenbaum, Dan and Maddison, Christopher and Ramalho, Tiago and Saxton, David and Shanahan, Murray and Teh, Yee Whye and Rezende, Danilo and Eslami, S. M. Ali , year = 2018, month = jul, pages =. Conditional. Proceedings of the 35th

    2018

  19. [27]

    Incoherent Probability Judgments in Large Language Models , booktitle =

    Jian. Incoherent Probability Judgments in Large Language Models , booktitle =. 2024 , url =

    2024

  20. [28]

    Zhang, Xiyuan and Maddix Robinson, Danielle and Yin, Junming and Erickson, Nick and Ansari, Abdul Fatir and Han, Boran and Zhang, Shuai and Akoglu, Leman and Faloutsos, Christos and Mahoney, Michael and Hu, Tony and Rangwala, Huzefa and Karypis, George and Wang, Yuyang (Bernie...

    2025

  21. [29]

    Forty-Third

    Wang, Yuanrui and Zhang, Xingxuan and Yu, Han and Hao, Mingchao and Ren, Gang and Yuan, Hao and Mao, Li and Zhang, Yunjia and Yuan, Chun and Cui, Peng , year = 2026, month = jun, urldate =. Forty-Third

    2026

  22. [30]

    Management Science , volume =

    Is It Better to Average Probabilities or Quantiles? , author =. Management Science , volume =

  23. [31]

    Introducing

    Kong, Weihao and Das, Abhimanyu , year=2026, urldate =. Introducing

    2026

  24. [32]

    and Kadra, Arlind and Gijsbers, Pieter and Mallik, Neeratyoy and Ravi, Sahithya and M

    Feurer, Matthias and van Rijn, Jan N. and Kadra, Arlind and Gijsbers, Pieter and Mallik, Neeratyoy and Ravi, Sahithya and M. Journal of Machine Learning Research , volume =

  25. [33]

    Proceedings of the 42nd

    Qu, Jingang and Holzm. Proceedings of the 42nd

  26. [34]

    Advances in

    Ma, Junwei and Thomas, Valentin and Hosseinzadeh, Rasa and Labach, Alex and Cresswell, Jesse and Golestan, Keyvan and Yu, Guangwei and Caterini, Anthony L and Volkovs, Maks , year = 2025, volume =. Advances in

    2025

  27. [35]

    Information Theory and Statistics:

    Csisz. Information Theory and Statistics:. Foundations and Trends in Communications and Information Theory , volume =. 2004 , doi =

    2004

  28. [36]

    Guaranteed Deterministic Bounds on the

    Nielsen, Frank and Sun, Ke , year = 2018, month = sep, pages =. Guaranteed Deterministic Bounds on the. 2018. doi:10.1109/MLSP.2018.8517093 , urldate =

    2018

  29. [37]

    , year = 2026, month = may, urldate =

    Hosseinzadeh, Rasa and Labach, Alex and Xue, Zexin and Han, Shuyi and Thomas, Valentin and Caterini, Anthony L. , year = 2026, month = may, urldate =. 2nd

    2026

  30. [38]

    Advances in Neural Information Processing Systems , volume =

    Erickson, Nick and Purucker, Lennart and Tschalzev, Andrej and Holzm. Advances in Neural Information Processing Systems , volume =

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.