Pith. sign in

REVIEW 3 major objections 5 minor 34 references

Enhancing Tabular Learners with Context-Aware Semantic Embeddings

T0 review · 3 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read CASE claims that priming a table-tuned LLM's context memory with a sample of the dataset's own rows produces semantic embeddings which, appended to any tabular learner, substantially improve predictions on semantically rich data—especially

desk verdict CASE is a genuinely new featurization with real low-data gains; the memorization-vs-semantics question is the one open wound, and the paper overclaims 'SOTA' given the AutoGluon comparison is not significant. read the letter →

arxiv 2608.03565 v1 pith:B4PMBXBB submitted 2026-08-04 cs.AI cs.LG

Günther Schindler , Maximilian Schambach , Johannes Höhne This is my paper
classification cs.AIcs.LG
keywords tabularlearningsemanticembeddingslargelanguagemodelsKVcacheprimingin-contextfoundationfeatureenrichmentlow-dataregime
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to give tabular learners access to the meaning carried by column headers and text entries, which they usually treat as arbitrary symbols. It proposes CASE, a way to embed rows with a large language model that has been fine-tuned on tables and whose context memory is primed with a sample of rows from the very table being predicted. The resulting embeddings, compressed with PCA and appended to the original features, improve accuracy and R2 across semantic benchmarks for every downstream learner tested, with the largest gains in low-data regimes. The paper argues that row semantics are distributional: an isolated row is ambiguous, while a row placed next to representative rows of its own table is not.

What carries the argument

The load-bearing object is the context-primed key-value memory of a Tabular Language Model. The model serializes a table into header, row, and target tokens and is trained to predict only target tokens. At inference, a random sample of training rows, including both features and targets, is pre-filled into the model's context memory as a permanent prefix; each query row is then embedded by reading the final hidden state at its Beginning-of-Target token. This single mechanism converts a row embedding from an isolated string translation into a vector positioned relative to the table's semantics and predictive task, and it is what distinguishes CASE from cell-wise or row-wise embedding baselines

What would settle it

Run CASE on newly created, post-cutoff semantically rich tables whose rows provably never appeared in the T4 corpus or in the backbone's pretraining data; if the gains over non-semantic baselines disappear, the claim that CASE supplies genuine semantic reasoning fails. A cheaper probe is a string-matching audit of CARTE, TextTab, and TabArena rows and headers against the T4 corpus and the backbone's training data—a substantial hit rate would undermine the central claim.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that table-specific context, not the LLM alone, carries the semantic signal. A decoder-only model continued-pretrained on a large corpus of real tables with a target-imputation objective, then given a pre-filled context of 128 training rows, produces row embeddings that are both target-aware and anchored in the table's distribution. Appended as 32 PCA components to the features of CatBoost, RealMLP, TabPFN-2.6, TabICLv2, or ConTextTab, these embeddings yield consistent gains: 2.9–6.8 points accuracy and 3.8–19.1 points R2 on semantics-heavy benchmarks, with TabICLv2 plus CASE reported as a new state of the art. The paper also reports that th

Load-bearing premise

The evaluation assumes the benchmark tables and labels were not seen during continued pretraining or in the proprietary pretraining data of the backbone LLM; if they were, the reported gains could reflect memorization rather than semantic reasoning.

Editorial extensions

If this is right

  • Any tabular learner can be upgraded with world knowledge without changing its architecture, by appending PCA-reduced contextual embeddings to its feature matrix.
  • In low-data regimes, semantic priors can compensate for sparse statistical signal, making CASE most valuable exactly where tabular models usually struggle.
  • The observed 'contextual leap' from 0 to 1k context tokens implies that a small anchor sample suffices to ground semantics, keeping the computational overhead modest.
  • A standalone decoder-only TLM, given only 8 context rows, can outperform a strong tabular in-context learner given the full training split on CARTE classification, showing the pretrained backbone itself is competitive as a few-shot predictor.
  • Because semantic corruption erases the gains, the method's benefit is tied to genuine semantic content and degrades gracefully when that content is absent.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the contamination concern is eventually cleared, CASE-type enrichment could become a cheap, model-agnostic upgrade: feature extraction happens once at fit time, and prediction-time cost is a single forward pass through a context-primed model.
  • The regression gains at higher PCA dimensions suggest that a non-linear or task-adaptive compression of the embedding space might extract more signal than the default 32 components.
  • The same context-priming mechanism could plausibly transfer to other structured modalities, such as time series, graphs, or document tables, where a small labeled anchor set defines the local semantics.
  • Because the TLM is trained on real tables with a target-imputation objective, embedding quality is coupled to target distribution, so conditioning or fine-tuning the TLM on a user's own table semantics could yield further gains.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The manuscript proposes CASE, a method for semantic enrichment of tabular data. A Gemma 3 decoder-only model is continued-pretrained on the T4 corpus with a target-masked autoregressive imputation objective, producing a Tabular Language Model. At inference, a random subset of training rows is serialized and used to pre-fill the KV cache; each query row is then embedded at the BOT token, the vector is PCA-reduced and concatenated to the original feature matrix, and any downstream tabular learner can be trained on the enriched representation. Experiments on CARTE, TextTab, TabArena, and STRABLE report large gains for several learners, particularly TabICLv2, on semantically rich tasks and in low-data regimes. Ablations study context size, model scale, PCA dimension, semantic corruption, column ordering, and context sampling.

Significance. If the reported gains reflect semantic understanding rather than dataset memorization, CASE is a significant and practical contribution: it is model-agnostic, computationally efficient through KV-cache priming, and empirically strong in low-data regimes. The evaluation is extensive—multiple benchmarks, baselines, ablations, statistical tests, robustness checks, and runtime analyses—and the inference code is available. The main unresolved issue is contamination of the pretraining data by the evaluation benchmarks; the paper's own rebuttal is incomplete, so the central attribution of gains to 'genuine semantic reasoning' is not yet fully established. This is a correctness-risk concern that can be addressed with additional overlap analyses or a private benchmark.

major comments (3)
  1. [Appendix C and Appendix E] The contamination analysis only checks CARTE against T4, and does so by citing [27]; it does not check TextTab, TabArena, or STRABLE against T4, and Gemma 3's pretraining corpus is proprietary. The internal evidence (low zero-context accuracy, monotonic improvement with context size, corruption dropping CASE to baseline) is not dispositive: all three patterns are also compatible with a model that has memorized benchmark rows and requires matching n-grams or table-level triggers to retrieve them. Since the abstract and Section 4 attribute the gains to genuine semantic reasoning, please (i) run string-overlap/membership tests for every benchmark against T4 and any accessible public portions of the backbone corpus, (ii) evaluate on a newly constructed or private string-heavy benchmark, or (iii) explicitly restrict the SOTA claim to 'no detectable overlap' and discuss residual risk. This is
  2. [Section 3.2, Context Priming] The procedure as written does not state that the row whose embedding is being extracted is excluded from the context sample C. If a training row appears in the context prefix, the embedding at its [BOT] position is computed after the model has already seen that row's target value, making the training feature label-dependent while the test feature is not. Even if downstream test metrics are not directly inflated, this introduces a train/test feature distribution shift and affects the interpretation of the ablation 'CASE w/o X_org.' Please clarify whether overlapping rows are removed; if not, rerun the pipeline with exclusion and report whether the gains persist.
  3. [Section 4.2 and Appendix B.1] The paper states that TabICLv2 with CASE 'largely outperforms all existing baselines' and later 'setting a new state of the art for semantically rich datasets.' However, the critical-difference diagram in Figure 7 shows that the difference to AutoGluon is not statistically significant at α=0.05, even though rank, win ratio, and Elo favor CASE. Please either soften the SOTA statement to reflect that the advantage over AutoGluon is not significant, or provide a statistical test that supports the stronger claim.
minor comments (5)
  1. [Figure 1 and Table 1] Headline accuracy/R2 numbers are reported without error bars, confidence intervals, or per-task spread. Given the acknowledged sensitivity to context sampling and seeds, please add uncertainty intervals or explicitly refer readers to the appendix statistics in the main text.
  2. [Section 4.3, Model/Context Scaling] The statement that 'all CASE configurations regardless of base model scale or utilized context consistently outperform the vanilla TabICLv2 baseline' is stronger than the plotted min/max bands alone can verify. Please report the exact comparison, e.g., a table of mean and worst-case performance across runs, or qualify the claim.
  3. [Table 2] The 'TabICL [Default]' row reports a mean rank of 7.06, which is much worse than the TabICLv2 row in Table 1 (rank 4.5). This is apparently because Table 2 covers only CARTE and TextTab while Table 1 includes TabArena; please state this explicitly to avoid confusion.
  4. [Reproducibility] Only inference code is released. For reproducibility of the main results, please also release the trained TLM checkpoints or detailed training scripts, or state clearly why this is not possible.
  5. [Figure 1] The axis label 'ConT extT ab' appears to be a rendering artifact; please check the final PDF for correct spacing.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: CASE is an empirical feature-engineering pipeline; no benchmark-derived parameter is renamed as a prediction, and the only self-citation concern is a contamination caveat, not a load-bearing derivation step.

full rationale

CASE is an empirical pipeline rather than a derivation. The TLM is continued-pretrained on the T4 corpus with a target-masked next-token objective (Eq. 4); row embeddings are extracted from a KV-cache-primed forward pass using context rows sampled from the train split; PCA is fit only on training embeddings and frozen before application to test rows; and downstream tabular learners are trained on the enriched feature matrix. No benchmark label or test prediction is used to set TLM weights, context sampling, or PCA components, so no reported gain reduces to a fitted parameter by construction. The only self-citation of note is Appendix C's reliance on ConTextTab [27], an overlapping-author prior work, for the T4-CARTE overlap search; that search covers CARTE only and cannot audit the proprietary Gemma 3 pretraining corpus. This is a legitimate data-contamination/external-validity caveat, and the paper itself flags it as a theoretical risk, but it does not make the method's derivation circular. The semantic-corruption and context-scaling experiments are independent, falsifiable evidence for the semantic-reasoning interpretation, even if a memorization alternative cannot be fully retired. Accordingly, there is no circular step under the strict definition used here.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim rests on transfer from LLM pretraining and a specific table-tuning objective. The free parameters are hyperparameters chosen by hand; the axioms are domain assumptions about transfer, representativeness, contamination, and the embedding location. No new physical or theoretical entities are introduced.

free parameters (4)
  • context priming window k = 128 rows (up to 32k tokens)
    Chosen as default; ablations in Figure 5 show gains saturate after ~1k tokens, but the exact value is a hyperparameter, not derived from theory.
  • PCA target dimension d' = 32
    Default setting; Appendix H shows 128 and 256 components give better ranks, so 32 is a practical compromise, chosen by hand.
  • TLM training steps = 8500 steps, batch size 128
    Pretraining budget; the paper does not show that more steps would not change results.
  • learning rate = 1e-6, no dropout or weight decay
    Standard fine-tuning choice, not derived from data.
assumptions (5)
  • domain assumption Gemma 3's pretrained world knowledge transfers to tabular semantics through continued pretraining.
    The entire method rests on the LLM recognizing concepts from column headers and cell values; this is assumed and only empirically validated.
  • domain assumption The T4 table corpus is representative and disjoint from the evaluation benchmarks.
    T4 is used for continued pretraining; contamination evidence is partial and partly delegated to a paper with overlapping authors.
  • domain assumption The hidden state at the [BOT] token, after KV-cache priming, captures the row's target-predictive semantics.
    This is the architectural assumption that makes the embeddings useful; the paper validates it empirically but does not prove it.
  • domain assumption Uniform random context rows provide a stable semantic anchor.
    Appendix G tests 5 seeds and finds no significant difference, but this is empirical on specific benchmarks, not a general guarantee.
  • domain assumption PCA preserves the task-relevant embedding signal at 32 dimensions.
    Appendix H shows higher dimensions are better, so the default sacrifices some signal; this is a modeling choice.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Enhancing Tabular Learners with Context-Aware Semantic Embeddings." pith.science (2026). https://pith.science/paper/B4PMBXBB

@misc{pith2026260803565,
  author       = {Pith},
  title        = {Pith review of: Enhancing Tabular Learners with Context-Aware Semantic Embeddings},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/B4PMBXBB}},
  note         = {Machine review of arXiv:2608.03565}
}
read the original abstract

While modern tabular learners excel at capturing statistical patterns, they frequently operate in a semantic vacuum, treating textual features as discrete symbols, ignoring the rich semantics inherent in feature names or cell entries. We propose CASE (Context-Aware Semantic Embeddings), a novel framework that bridges the gap between the semantic understanding of Large Language Models (LLMs) and the statistical capabilities of tabular learners. Unlike existing methods that embed rows in isolation, CASE utilizes a contextualization strategy: we pre-fill the KV cache of a custom-trained Gemma 3-based Tabular Language Model with a representative sample of rows to establish a persistent anchor of the dataset's semantics. This ensures that generated row embeddings are dynamically contextualized, resolving semantic ambiguities and anchoring representations in domain-specific context. Our experiments across several benchmarks (CARTE, TextTab, and TabArena) demonstrate that CASE substantially improves the performance of tabular learners on semantically rich datasets, particularly in low-data regimes.

Figures

Figures reproduced from arXiv: 2608.03565 by the authors.

Figure 1
Figure 1. Our proposed Context-Aware Semantic Embeddings (CASE) boost prediction quality of tabular learners on semantically rich benchmarks (CARTE [18] and TextTab [22]). els [28] using a column-imputation objective. We apply the loss only to target-column tokens while conditioning on all feature tokens, aligning next-token prediction with tabular prediction. This yields task-specific representations that capture both the se… view at source ↗
Figure 2
Figure 2. Illustration of our proposed Tabular Language Model (TLM) architecture with table [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Overview of our proposed context-aware semantic embedding-based table enrichment [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: Impact of Context-Aware Semantic Embeddings (CASE) on predictive performance of [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Performance across CARTE and TextTab for TabICLv2 enriched with CASE embeddings [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Architectural Comparison on CARTE classi [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Critical difference diagrams, win ratios and ELO scores for the semantic-heavy CARTE [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8: , show that the gains in absolute performance are substantial in many cases. Overall, also these extended results underline the superior performance of our approach, setting a new state of the art for semantically rich datasets. CD 9 8 7 6 5 4 3 2 1 Random forest TabIC…
Figure 9
Figure 9. Figure 9: Critical difference diagrams, win ratios and ELO scores for the numerics-heavy TabArena [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: Per-task accuracy (left) and R2 scores (right) on TabArena comparing TabICLv2 with (y-axis) and without CASE (x-axis). On-diagonal scores indicate identical performance while above￾diagonal scores indicate that performance is better when using CASE instead of the defa…
Figure 11
Figure 11. Figure 11: Predictive Quality vs. Computational Cost Trade-offs on CARTE. Pareto frontiers demonstrate that CASE offers highly competitive inference latency while improving predictive performance over non-semantic baselines. CD 4 3 2 1 TabICL [CASE, corrupted] TabICL [corrupted]…
Figure 12
Figure 12. Figure 12: Critical Difference Diagram under Semantic Corruption. Across CARTE and TextTab benchmarks (α = 0.05), standard TabICL [CASE] achieves a significantly superior performance rank compared to all non-semantic or corrupted configurations. E Impact of Semantic Corruption a…
Figure 13
Figure 13. Figure 13: Critical difference diagrams for CARTE and TextTab benchmarks using different seeds [PITH_FULL_IMAGE:figures/full_fig_p018_13.png]
Figure 14
Figure 14. Figure 14: Critical difference diagrams for CARTE and TextTab benchmarks using different seeds [PITH_FULL_IMAGE:figures/full_fig_p019_14.png]
Figure 15
Figure 15. Figure 15: Critical Difference Diagram across PCA Dimensions. Higher PCA dimensions (k = 128, 256) yield statistically significant rank improvements across CARTE and TextTab benchmarks (α = 0.05). illustrated in the Critical Difference (CD) diagram ( [PITH_FULL_IMAGE:figures/fu…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

34 extracted references · 22 canonical work pages

  1. [27]

    ConTextTab: A semantics- aware tabular in-context learner

    Marco Spinaci, Marek Polewczyk, Maximilian Schambach, and Sam Thelin. ConTextTab: A semantics- aware tabular in-context learner. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025

    work page 2025

  2. [1]

    STRABLE: Benchmarking Tabular Machine Learning with Strings

    Gioia Blayer, Myung Jun Kim, Félix Lefebvre, Lennart Purucker, Alan Arazi, Eilam Shapira, Roi Reichart, Frank Hutter, Marine Le Morvan, David Holzmüller, and Gaël Varoquaux. Strable: Benchmarking tabular machine learning with strings, 2026. URLhttps://arxiv.org/abs/2605.12292

    work page Pith review arXiv 2026

  3. [2]

    XGBoost: A scalable tree boosting system

    Tianqi Chen and Carlos Guestrin. XGBoost: A scalable tree boosting system. InProceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’16, pages 785–794, 2016. ISBN 978-1-4503-4232-2

    work page 2016

  4. [3]

    Chatbot Arena: An open platform for evaluating llms by human preference.arXiv preprint arXiv:2403.04132, 2024

    Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E Gonzalez, et al. Chatbot Arena: An open platform for evaluating llms by human preference.arXiv preprint arXiv:2403.04132, 2024

    arXiv 2024

  5. [4]

    LIFT: Language-interfaced fine-tuning for non-language machine learning tasks.Advances in Neural Information Processing Systems, 35:11763–11784, 2022

    Tuan Dinh, Yuchen Zeng, Ruisu Zhang, Ziqian Lin, Michael Gira, Shashank Rajput, Jy-yong Sohn, Dimitris Papailiopoulos, and Kangwook Lee. LIFT: Language-interfaced fine-tuning for non-language machine learning tasks.Advances in Neural Information Processing Systems, 35:11763–11784, 2022

    work page 2022

  6. [5]

    Set-LLM: A permutation-invariant LLM

    Beni Egressy and Jan Stühmer. Set-LLM: A permutation-invariant LLM. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems

    work page

  7. [6]

    AutoGluon-Tabular: Robust and accurate AutoML for structured data.arXiv preprint arXiv:2003.06505, 2020

    Nick Erickson, Jonas Mueller, Alexander Shirkov, Hang Zhang, Pedro Larroy, Mu Li, and Alexander Smola. AutoGluon-Tabular: Robust and accurate AutoML for structured data.arXiv preprint arXiv:2003.06505, 2020

    arXiv 2003

  8. [7]

    Tabarena: A living benchmark for machine learning on tabular data

    Nick Erickson, Lennart Purucker, Andrej Tschalzev, David Holzmüller, Prateek Mutalik Desai, David Salinas, and Frank Hutter. Tabarena: A living benchmark for machine learning on tabular data. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2025

    2025

Show all 34 references
  1. [8]

    Large scale transfer learning for tabular data via language modeling

    Joshua P Gardner, Juan Carlos Perdomo, and Ludwig Schmidt. Large scale transfer learning for tabular data via language modeling. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  2. [9]

    Revisiting deep learning models for tabular data.Advances in Neural Information Processing Systems, 34:18932–18943, 2021

    Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning models for tabular data.Advances in Neural Information Processing Systems, 34:18932–18943, 2021

    2021

  3. [10]

    TabM: Advancing tabular deep learning with parameter-efficient ensembling

    Yury Gorishniy, Akim Kotelnikov, and Artem Babenko. TabM: Advancing tabular deep learning with parameter-efficient ensembling. InInternational Conference on Learning Representations, 2025. 10

    2025

  4. [11]

    Vectorizing string entries for data processing on tables: when are larger language models better?arXiv preprint arXiv:2312.09634, 2023

    Léo Grinsztajn, Edouard Oyallon, Myung Jun Kim, and Gaël Varoquaux. Vectorizing string entries for data processing on tables: when are larger language models better?arXiv preprint arXiv:2312.09634, 2023

    2023 arXiv

  5. [12]

    TabLLM: Few-shot classification of tabular data with large language models

    Stefan Hegselmann, Alejandro Buendia, Hunter Lang, Monica Agrawal, Xiaoyi Jiang, and David Sontag. TabLLM: Few-shot classification of tabular data with large language models. InInternational Conference on Artificial Intelligence and Statistics, pages 5549–5581. PMLR, 2023

    2023

  6. [13]

    Autorank: A Python package for automated ranking of classifiers.Journal of Open Source Software, 5(48):2173, 2020

    Steffen Herbold. Autorank: A Python package for automated ranking of classifiers.Journal of Open Source Software, 5(48):2173, 2020. doi: 10.21105/joss.02173. URL https://doi.org/10.21105/ joss.02173

    2020 doi

  7. [14]

    TabPFN: A transformer that solves small tabular classification problems in a second

    Noah Hollmann, Samuel Müller, Katharina Eggensperger, and Frank Hutter. TabPFN: A transformer that solves small tabular classification problems in a second. InThe Eleventh International Conference on Learning Representations, 2023

    2023

  8. [15]

    Accurate predictions on small data with a tabular foundation model.Nature, 637(8045):319–326, 2025

    Noah Hollmann, Samuel Müller, Lennart Purucker, Arjun Krishnakumar, Max Körfer, Shi Bin Hoo, Robin Tibor Schirrmeister, and Frank Hutter. Accurate predictions on small data with a tabular foundation model.Nature, 637(8045):319–326, 2025

    2025

  9. [16]

    Better by default: Strong pre-tuned MLPs and boosted trees on tabular data.Advances in Neural Information Processing Systems, 37:26577–26658, 2024

    David Holzmüller, Léo Grinsztajn, and Ingo Steinwart. Better by default: Strong pre-tuned MLPs and boosted trees on tabular data.Advances in Neural Information Processing Systems, 37:26577–26658, 2024

    2024

  10. [17]

    LightGBM: A highly efficient gradient boosting decision tree

    Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. LightGBM: A highly efficient gradient boosting decision tree. InAdvances in Neural Information Processing Systems, 2017

    2017

  11. [18]

    CARTE: Pretraining and transfer for tabular learning

    Myung Jun Kim, Leo Grinsztajn, and Gael Varoquaux. CARTE: Pretraining and transfer for tabular learning. InForty-first International Conference on Machine Learning, 2024

    2024

  12. [19]

    Knowledge-rich embeddings for tabular learning

    Félix Lefebvre, Myung Jun Kim, and Gaël Varoquaux. Knowledge-rich embeddings for tabular learning. InEurIPS 2025 Workshop: AI for Tabular Data, 2025

    2025

  13. [20]

    Generalization can emerge in tabular foundation models from a single table

    Junwei Ma, Nour Shaheen, Alex Labach, Amine Mhedhbi, Frank Hutter, Anthony L Caterini, and Valentin Thomas. Generalization can emerge in tabular foundation models from a single table. InEurIPS Workshop on AI for Tabular Data, 2025

    2025

  14. [21]

    TabDPT: Scaling tabular foundation models.Advances in Neural Information Processing Systems, 2025

    Junwei Ma, Valentin Thomas, Rasa Hosseinzadeh, Hamidreza Kamkari, Alex Labach, Jesse C Cresswell, Keyvan Golestan, Guangwei Yu, Maksims V olkovs, and Anthony L Caterini. TabDPT: Scaling tabular foundation models.Advances in Neural Information Processing Systems, 2025

    2025

  15. [22]

    Towards benchmarking foundation models for tabular data with text

    Martin Mráz, Breenda Das, Anshul Gupta, Lennart Purucker, and Frank Hutter. Towards benchmarking foundation models for tabular data with text. InICML 2025 Workshop on Foundation Models for Structured Data (FMSD), 2025

    2025

  16. [23]

    CatBoost: Unbiased boosting with categorical features.Advances in Neural Information Processing Systems, 31, 2018

    Liudmila Prokhorenkova, Gleb Gusev, Aleksandr V orobev, Anna Veronika Dorogush, and Andrey Gulin. CatBoost: Unbiased boosting with categorical features.Advances in Neural Information Processing Systems, 31, 2018

    2018

  17. [24]

    TabICL: A tabular foundation model for in-context learning on large data

    Jingang Qu, David Holzmüller, Gaël Varoquaux, and Marine Le Morvan. TabICL: A tabular foundation model for in-context learning on large data. InInternational Conference on Machine Learning, 2025

    2025

  18. [25]

    TabICLv2: A better, faster, scalable, and open tabular foundation model

    Jingang Qu, David Holzmüller, Gaël Varoquaux, and Marine Le Morvan. TabICLv2: A better, faster, scalable, and open tabular foundation model. InInternational Conference on Machine Learning, 2026

    2026

  19. [26]

    Tabgemma: Text-based tabular ICL via LLM using continued pretraining and retrieval

    Günther Schindler, Maximilian Schambach, Michael Medek, and Sam Thelin. Tabgemma: Text-based tabular ICL via LLM using continued pretraining and retrieval. InEurIPS 2025 Workshop: AI for Tabular Data, 2025

    2025

  20. [28]

    Gemma 3 technical report.arXiv preprint arXiv:2503.19786, 2025

    Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report.arXiv preprint arXiv:2503.19786, 2025. 11

    2025 arXiv

  21. [29]

    T5gemma 2: Seeing, reading, and understanding longer.arXiv preprint arXiv:2512.14856, 2025

    Biao Zhang, Paul Suganthan, Gaël Liu, Ilya Philippov, Sahil Dua, Ben Hora, Kat Black, Gus Martins, Omar Sanseviero, Shreya Pathak, Cassidy Hardin, Francesco Visin, Jiageng Zhang, Kathleen Kenealy, Qin Yin, Xiaodan Song, Olivier Lacombe, Armand Joulin, Tris Warkentin, and Adam ...

    2025

  22. [30]

    Mitra: Mixed synthetic priors for enhancing tabular foundation models

    Xiyuan Zhang, Danielle C Maddix, Junming Yin, Nick Erickson, Abdul Fatir Ansari, Boran Han, Shuai Zhang, Leman Akoglu, Christos Faloutsos, Michael W Mahoney, et al. Mitra: Mixed synthetic priors for enhancing tabular foundation models. InAdvances in Neural Information Processi...

    2025

  23. [31]

    XTab: Cross- table pretraining for tabular transformers

    Bingzhao Zhu, Xingjian Shi, Nick Erickson, Mu Li, George Karypis, and Mahsa Shoaran. XTab: Cross- table pretraining for tabular transformers. InProceedings of the 40th International Conference on Machine Learning, 2023. 12 A Baseline Details TabICLv2:We use the model from the ...

    2023

  24. [32]

    At prediction time, context rows do not need to be re-encoded, avoiding redundant forward passes

    KV-Cache Context Priming:Background row representations and table-level semantics are processed and cached duringfit(). At prediction time, context rows do not need to be re-encoded, avoiding redundant forward passes

  25. [33]

    Age" or cell values like

    Generation-Free Inference:Unlike standard LLM applications that rely on autoregressive token sampling—the dominant computational bottleneck in LLM inference—CASE extracts target representations from the final hidden states in a single, fast forward pass without invoking the la...

  26. [34]

    Statistical Impact and Analysis:As shown in the Critical Difference diagram (Fig

    TabICL [CASE]: Combined pipeline integrating CASE-12B semantic embeddings with TF-IDF. Statistical Impact and Analysis:As shown in the Critical Difference diagram (Fig. 12) and Table 3, three key insights emerge regarding when and why CASE provides performance gains: • Perform...

    2026

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.