Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Transfer Learning of Tabular Data by Finetuning Large Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read End-to-end finetuning of a small generative language model outperforms state-of-the-art machine and deep learning classifiers on tabular datasets with fewer than ten features, while using a fraction of the computational cost.

desk verdict The paper's own Table 2 contradicts its headline claim: on the two datasets with fewer than ten features, the proposed end-to-end finetuned LLM is beaten by MLP and GBT, and the 'transfer learning' mechanism is never tested. read the letter →

arxiv 2501.06863 v1 pith:NNXFXEYJ submitted 2025-01-12 cs.LG cs.AIcs.CL

Shourav B. Rabbani , Ibna Kowsar , Manar D. Samad This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords tabulardatatransferlearninglargelanguagemodelsDistilGPT2fine-tuningin-contextclassification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that transfer learning for tabular classification can be done by finetuning a small generative language model end to end, turning each data row into a short English sentence. The claim is that for the common case of tabular datasets with fewer than ten features, this finetuned model matches or beats gradient-boosted trees and other strong baselines, and outperforms a contrastive deep learning method on most of the benchmark sets. The reason this matters is that tabular data lacks the large pretrained foundation models that exist for text and images, so a text-to-tabular transfer route would give small scientific, medical, and business datasets access to deep learning. The paper also shows the finetuning route is far cheaper than API-based in-context prompting, because it runs locally on a small open-weight model.

What carries the argument

The load-bearing machinery is the combination of row serialization and full-weight finetuning. Each tabular row is converted to a prompt such as 'Age is 25. Sex is male.' with feature names and values, tokenized into sequences up to 1024 tokens, and passed through DistilGPT2, whose final 768-dimensional embedding is mapped to class logits through a classifier head trained with cross-entropy loss. With all transformer weights frozen the model underfits and ranks last; with all weights trainable, the pretrained English representations adapt to the tabular task, giving the reported gains. The same serialization format has no need for the elaborate prompt templates and token budgets that constrain API-based in-context learning.

What would settle it

Train the exact same end-to-end pipeline on the blood-transfusion and diabetes datasets using a randomly initialized DistilGPT2 of the same architecture; if its five-fold AUC matches or exceeds the reported values (0.738 and 0.820), the claim that transfer from English text drives the result is falsified.

Watch

Extended reading notes

Core claim

The paper's central discovery is that end-to-end finetuning of DistilGPT2—an 82-million-parameter distilled causal language model pretrained on English text—converts tabular classification into a text task and achieves competitive or superior accuracy. On the ten benchmark sets, the finetuned model with all weights trainable ranks ahead of the frozen-weight variant, the API-based in-context method FeatLLM, and the contrastive deep learning method SCARF, and its average ranking is close to gradient-boosted trees. The authors highlight the blood-transfusion and diabetes datasets, with five and nine features respectively, where the finetuned LLM matches or exceeds the best GBT baseline, and conclude that transfer learning via an LLM can be the best solution when the feature size is less than ten. They further report that the approach uses roughly 10 to 50 percent of the compute of the API or deep learning alternatives.

Load-bearing premise

The load-bearing assumption is that DistilGPT2's pretrained English knowledge, not the transformer architecture or the text serialization format, is what makes the finetuned model perform well; if a randomly initialized model of the same size did equally well, the transfer-learning explanation would collapse.

Editorial extensions

If this is right

  • If the central claim holds, tabular datasets with fewer than ten features—very common in practice—can be handled by deep learning through a finetuned text model, closing a gap that standard deep networks have not closed.
  • The finetuned LLM would replace API-based in-context prompting for small tabular tasks, since local training is faster and cheaper and does not depend on external token limits.
  • Frozen-weight finetuning is not a viable transfer strategy for this setup; the model weights must be trained end to end, which clarifies where the transfer gain comes from.
  • The approach outperforms the contrastive deep learning baseline SCARF on six of the ten datasets, so LLM finetuning becomes a stronger default than self-supervised tabular pretraining on these benchmarks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 'fewer than ten features' boundary is supported by only two datasets (blood-transfusion and diabetes); an inference worth testing is whether the advantage generalizes across a broader suite of low-feature tabular benchmarks or is specific to these medical and transfusion domains.
  • If the pretrained English representations are doing the work, the finetuned model's strong results on medical datasets may partly reflect domain knowledge absorbed during pretraining; comparing against a randomly initialized model of the same architecture would separate transfer from architecture.
  • A natural extension the paper does not explore is using the finetuned LLM's penultimate-layer embeddings as features for a gradient-boosted tree, which could combine the LLM's semantic representation with the classical model's strong tabular inductive bias.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes end-to-end finetuning of DistilGPT2 with a classification head for tabular data classification. It describes a feature-to-text serialization and tokenization pipeline, compares frozen-weight and fully trainable finetuning against GBT, MLP, SCARF, and FeatLLM on ten benchmark datasets, reports five-fold cross-validated AUC and training times, and claims that the proposed method outperforms state-of-the-art machine and deep learning methods on tabular data with fewer than ten features. The paper also argues that text-to-tabular transfer learning is feasible and computationally cheaper than API-based in-context learning.

Significance. If substantiated, the central claim would be practically useful: a lightweight generative LLM finetuned end-to-end could serve as a strong classifier for small tabular datasets, where deep learning is often ineffective. The paper has some strengths: a transparent serialization procedure, a public lightweight model (DistilGPT2), a five-fold cross-validation setup, and direct runtime comparisons in Table 3. However, the headline assertion is contradicted by the paper's own results, and the attribution of observed performance to pretrained English text representations is untested. As reported, the evidence does not support the claimed advantage.

major comments (4)
  1. [Abstract; §4.4; Table 2] The central claim that the proposed end-to-end finetuned LLM 'outperforms state-of-the-art machine and deep learning methods on tabular data with less than ten features' is contradicted by Table 2. On the only two datasets with fewer than ten features, blood-transfusion (5 features) has End-to-end AUC 0.738, below MLP's 0.762, and diabetes (9 features) has End-to-end AUC 0.820, below GBT's 0.828 and FeatLLM's 0.829. The unqualified abstract claim and the §4.4 statement that 'transfer learning via LLM can be the best solution when the feature size is less than ten' are therefore not supported by the paper's own numbers.
  2. [§4.4; §3.1.3] The 'transfer learning' attribution rests on the assumption that DistilGPT2's pretrained English representations, rather than the model architecture or the serialization setup, cause the observed performance. No ablation with a randomly initialized DistilGPT2 of identical architecture and training pipeline is reported, so the results do not establish text-to-tabular transfer. This is load-bearing because the novelty and the central framing of the paper depend on this attribution; the background section itself cites prior LLM finetuning approaches such as TabLLM and LIFT.
  3. [§4.4; Table 2] The 'fewer than ten features' regime is defined post hoc from only two datasets, and no statistical test, confidence-interval comparison, or multiple-comparison correction is provided. One of the two point estimates favors the proposed method over only some baselines, not over all of them, so the threshold cannot support a general claim about feature size. The paper should either narrow the conclusion to 'competitive on some small datasets' or provide a statistically grounded, preregistered analysis of a feature-size effect.
  4. [Table 2; §4.4; §5] The overall ranking results are also in tension with the conclusions. End-to-end finetuning has average rank 3.2 across all datasets, whereas GBT has rank 1.9; the paper itself notes in §4.4 that GBT 'appears to be the best performing method.' The Conclusion's statement that 'transfer learning using LLM is superior to in-context LLM learning and deep learning methods proposed for tabular data in terms of performance and computational costs' is not supported by the reported AUC comparisons.
minor comments (5)
  1. [§3.2] The paper says the authors 'updated the prompt generator of the FeatLLM method to include as many samples as possible under the constraint on token size,' but it does not report the exact prompt or how the update differs from the original FeatLLM protocol, which makes the in-context baseline difficult to reproduce.
  2. [§4.4] In the sentence 'text input prompts (0.733 (0.021)) are able to achieve a better performance than the best overall GBT model (0.711 (0.039))', the number 0.733 is not explicitly tied to FeatLLM in the text; the reader must infer it from Table 2.
  3. [Table 1; §4.1] The easy/hard difficulty split is taken from reference [26], an unpublished manuscript, without a description of the criterion. The authors should either define the criterion in this paper or make the basis for the split available.
  4. [§4.3; Figure 4] The discussion of overfitting after 30 epochs is based on training loss curves; since early stopping selects the model by validation loss, showing validation loss curves would be more informative.
  5. [References] Several references are incomplete or inaccessible, including [3] and [26]; in particular [26] is used for the baseline setup and difficulty labels and should be described or made available for reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: empirical benchmark with self-contained cross-validated comparisons; self-citations affect setup, not the central result.

full rationale

This paper does not contain a derivation chain whose conclusion is equivalent to its inputs. The central comparisons are produced by five-fold cross-validation on held-out folds (Sections 3.3-4.4), so the reported AUC values are measurements, not fitted parameters renamed as predictions. The only self-references are to the authors' prior benchmark [26] for dataset selection criteria, the easy/hard split, and the SCARF/MLP experimental setup; these choices affect baseline comparability but do not define or force the paper's central 'transfer learning via LLM' claim. The 'less than ten features' conclusion in the abstract and Section 4.4 is post hoc and is contradicted by the paper's own Table 2 on blood-transfusion (5 features: End-to-end 0.738 vs MLP 0.762) and diabetes (9 features: End-to-end 0.820 vs GBT 0.828 and FeatLLM 0.829); however, an internal empirical inconsistency is a correctness risk, not circularity. No equation-level reduction, imported uniqueness theorem, ansatz-smuggling citation, or renamed known result appears. The transfer-learning mechanism (benefit of DistilGPT2 pretrained weights) is asserted without a random-init ablation, but this is an untested assumption rather than a circular step.

Assumptions & free parameters 2 free parameters · 2 assumptions · 0 invented entities

No new entities are proposed. The paper's central claims rest on two untested assumptions: that pretrained text knowledge transfers to tabular data, and that the chosen serialization format is lossless enough. The hyperparameters are standard but not tuned systematically.

free parameters (2)
  • LLM finetuning hyperparameters = lr=5e-5, weight decay=0.01, batch size=16, max epochs=100, patience=10
    Chosen by hand without a reported tuning protocol; performance comparisons may shift with different settings, and no sensitivity analysis is provided.
  • Validation split fraction = 1/8 of training data
    Ad hoc choice for early stopping; not varied.
assumptions (2)
  • ad hoc to paper DistilGPT2's pretrained English text representations provide a useful inductive bias for tabular classification
    The transfer learning claim requires this, but no ablation against a randomly initialized model is reported.
  • domain assumption Serializing each tabular row as 'Feature is value' text preserves the information needed for classification
    The method relies on this serialization; alternative serializations are not tested.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Transfer Learning of Tabular Data by Finetuning Large Language Models." pith.science (2026). https://pith.science/paper/NNXFXEYJ

@misc{pith2026250106863,
  author       = {Pith},
  title        = {Pith review of: Transfer Learning of Tabular Data by Finetuning Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NNXFXEYJ}},
  note         = {Machine review of arXiv:2501.06863}
}
read the original abstract

Despite the artificial intelligence (AI) revolution, deep learning has yet to achieve much success with tabular data due to heterogeneous feature space and limited sample sizes without viable transfer learning. The new era of generative AI, powered by large language models (LLM), brings unprecedented learning opportunities to diverse data and domains. This paper investigates the effectiveness of an LLM application programming interface (API) and transfer learning of LLM in tabular data classification. LLM APIs respond to input text prompts with tokenized data and instructions, whereas transfer learning finetunes an LLM for a target classification task. This paper proposes an end-to-end finetuning of LLM to demonstrate cross-data transfer learning on ten benchmark data sets when large pre-trained tabular data models do not exist to facilitate transfer learning. The proposed LLM finetuning method outperforms state-of-the-art machine and deep learning methods on tabular data with less than ten features - a standard feature size for tabular data sets. The transfer learning approach uses a fraction of the computational cost of other deep learning or API-based solutions while ensuring competitive or superior classification performance.

Figures

Figures reproduced from arXiv: 2501.06863 by the authors.

Figure 1
Figure 1. Tabular data transfer learning framework using a large language model (LLM). In-context learning using [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Text prompt template for in-context learning using FeatLLM [8]. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Text prompt template for transfer learning using DistilGPT2 [22]. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Convergence plots of proposed transfer learning of tabular data using a large language model (LLM) [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Deep Learning with Pretrained 'Internal World' Layers: A Gemma 3-Based Modular Architecture for Wildfire Prediction

    cs.LG 2025-04 reject novelty 4.0 of 10

    A Gemma-3 middle-layer module for wildfire prediction is outperformed, on the paper's own metrics, by a lightweight MLP with positional encodings.

Reference graph

Works this paper leans on

30 extracted references · 25 canonical work pages · cited by 1 Pith paper

  1. [26]

    Rabbani, Ivan V

    Shourav B. Rabbani, Ivan V . Medri, and Manar D. Samad. Attention versus contrastive learning of tabular data – a data-centric benchmarking, 2024

    work page 2024

  2. [1]

    Survey on Deep Neural Networks in Speech and Vision Systems

    M Alam, M D Samad, L Vidyaratne, A Glandon, and K M Iftekharuddin. Survey on Deep Neural Networks in Speech and Vision Systems. Neurocomputing, 417:302–321, 2020

    work page 2020

  3. [2]

    Deep clustering of electronic health records tabular data for clinical interpretation

    Ibna Kowsar, Shourav B Rabbani, Kazi Fuad B Akhter, and Manar D Samad. Deep clustering of electronic health records tabular data for clinical interpretation. In 2023 IEEE International Conference on Telecommunications and Photonics (ICTP), pages 01–05. IEEE, 2023

    work page 2023

  4. [3]

    Rabbani, Ivan V

    Shourav B. Rabbani, Ivan V . Medri, and Manar D. Samad. Deep clustering of tabular data by weighted gaussian distribution learning, 2024

    work page 2024

  5. [4]

    Well-tuned simple nets excel on tabular datasets

    Arlind Kadra, Marius Lindauer, Frank Hutter, and Josif Grabocka. Well-tuned simple nets excel on tabular datasets. Advances in neural information processing systems, 34:23928–23941, 2021

    work page 2021

  6. [5]

    Tabular data: Deep learning is not all you need

    Ravid Shwartz-Ziv and Amitai Armon. Tabular data: Deep learning is not all you need. Information Fusion, 81:84–90, may 2022

    work page 2022

  7. [6]

    Kwok, and Lionel M

    Yaqing Wang, Quanming Yao, James T. Kwok, and Lionel M. Ni. Generalizing from a few examples: A survey on few-shot learning. ACM Comput. Surv., 53(3), jun 2020

    work page 2020

  8. [7]

    STUNT: Few-shot tabular learn- ing with self-generated tasks from unlabeled tables

    Jaehyun Nam, Jihoon Tack, Kyungmin Lee, Hankook Lee, and Jinwoo Shin. STUNT: Few-shot tabular learn- ing with self-generated tasks from unlabeled tables. In The Eleventh International Conference on Learning Representations, 2023

    work page 2023

Show all 30 references
  1. [8]

    Large Language Models Can Automatically Engineer Features for Few-Shot Tabular Learning

    Sungwon Han, Jinsung Yoon, Sercan O Arik, and Tomas Pfister. Large Language Models Can Automatically Engineer Features for Few-Shot Tabular Learning. apr 2024

    2024

  2. [9]

    Gradient boosting trees and large language models for tabular data few-shot learning

    Carlos Huertas. Gradient boosting trees and large language models for tabular data few-shot learning. In 19th Conference on Computer Science and Intelligence Systems, Annals of Computer Science and Information Systems. IEEE, 2024

    2024

  3. [10]

    Tabllm: Few-shot classification of tabular data with large language models

    Stefan Hegselmann, Alejandro Buendia, Hunter Lang, Monica Agrawal, Xiaoyi Jiang, and David Sontag. Tabllm: Few-shot classification of tabular data with large language models. In International Conference on Artificial Intelligence and Statistics, pages 5549–5581. PMLR, 2023

    2023

  4. [11]

    Ho, Carl Yang, and May Dongmei Wang

    Wenqi Shi, Ran Xu, Yuchen Zhuang, Yue Yu, Jieyu Zhang, Hang Wu, Yuanda Zhu, Joyce C. Ho, Carl Yang, and May Dongmei Wang. EHRAgent: Code empowers large language models for few-shot complex tabular reasoning on electronic health records. In ICLR 2024 Workshop on Large Language ...

    2024

  5. [12]

    Cancergpt for few shot drug pair synergy prediction using large pretrained language models

    Tianhao Li, Sandesh Shetty, Advaith Kamath, Ajay Jaiswal, Xiaoqian Jiang, Ying Ding, and Yejin Kim. Cancergpt for few shot drug pair synergy prediction using large pretrained language models. NPJ Digital Medicine, 7(1):40, 2024

    2024

  6. [13]

    Tablet: Learning from instructions for tabular data, 2023

    Dylan Slack and Sameer Singh. Tablet: Learning from instructions for tabular data, 2023

    2023

  7. [14]

    Deep learning does not outperform classical machine learning for cell-type annotation

    Niklas Kohler, Maren Buttner, and Fabian Theis. Deep learning does not outperform classical machine learning for cell-type annotation. bioRxiv, page 653907, 2019

    2019

  8. [15]

    Deep Neural Networks and Tabular Data: A Survey

    Vadim Borisov, Tobias Leemann, Kathrin Seßler, Johannes Haug, Martin Pawelczyk, and Gjergji Kasneci. Deep Neural Networks and Tabular Data: A Survey. arXiv preprint arXiv:2110.01889, oct 2021. 8 A PREPRINT - JANUARY 14, 2025

    2021 arXiv

  9. [16]

    Sakib Abrar, Ali Sekmen, and Manar D. Samad. Effectiveness of deep image embedding clustering methods on tabular data. In 2023 15th International Conference on Advanced Computational Intelligence (ICACI), pages 1–7, 2023

    2023

  10. [17]

    Revisiting deep learning models for tabular data

    Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning models for tabular data. Advances in Neural Information Processing Systems, 23:18932–18943, 2021

    2021

  11. [18]

    T2g-former: organizing tabular features into relation graphs promotes heterogeneous feature interaction

    Jiahuan Yan, Jintai Chen, Yixuan Wu, Danny Z Chen, and Jian Wu. T2g-former: organizing tabular features into relation graphs promotes heterogeneous feature interaction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 10720–10728, 2023

    2023

  12. [19]

    Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding – A Survey

    Xi Fang, Weijie Xu, Fiona Anting Tan, Jiani Zhang, Ziqing Hu, Yanjun Qi, Scott Nickleach, Diego Socolinsky, Srinivasan Sengamedu, and Christos Faloutsos. Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding – A Survey. pages 1–41, 2024

    2024

  13. [20]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Nee- lakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-V oss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Je...

    2020

  14. [21]

    LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Tasks

    Tuan Dinh, Yuchen Zeng, Ruisu Zhang, Ziqian Lin, Michael Gira, Shashank Rajput, Jy Yong Sohn, Dimitris Papailiopoulos, and Kangwook Lee. LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Tasks. Advances in Neural Information Processing Systems, 35(NeurIPS), 2022

    2022

  15. [22]

    Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter

    Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. In NeurIPS EMC Workshop, 2019

    2019

  16. [23]

    Distilgpt2

    Hugging Face. Distilgpt2. https://huggingface.co/distilgpt2, 2019. Accessed: 2024-09-20

    2019

  17. [24]

    Aligning books and movies: Towards story-like visual explanations by watching movies and reading books

    Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In 2015 IEEE International Conference on Computer Vision (ICCV),...

    2015

  18. [25]

    SCARF: Self-supervised contrastive learning using random feature corruption

    Dara Bahri, Heinrich Jiang, Yi Tay, and Donald Metzler. SCARF: Self-supervised contrastive learning using random feature corruption. In International Conference on Learning Representations, 2022

    2022

  19. [27]

    Pedregosa, G

    F. Pedregosa, G. Varoquaux, A. Gramfort, V . Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V . Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learni...

    2011

  20. [28]

    UCI machine learning repository, 2017

    Dheeru Dua and Casey Graff. UCI machine learning repository, 2017

    2017

  21. [29]

    van Rijn, Bernd Bischl, and Luis Torgo

    Joaquin Vanschoren, Jan N. van Rijn, Bernd Bischl, and Luis Torgo. Openml: Networked science in machine learning. SIGKDD Explorations, 15(2):49–60, 2013

    2013

  22. [30]

    The health gym: synthetic health-related datasets for the development of reinforcement learning algorithms

    Nicholas I-Hsien Kuo, Mark N Polizzotto, Simon Finfer, Federico Garcia, Anders S¨onnerborg, Maurizio Zazzi, Michael B¨ohm, Rolf Kaiser, Louisa Jorm, and Sebastiano Barbieri. The health gym: synthetic health-related datasets for the development of reinforcement learning algorit...

    2022

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.