Pith. sign in

REVIEW 4 major objections 5 minor 48 references

Basis Transformers for Multi-Task Tabular Regression

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A transformer that encodes numbers bit-by-bit beats fine-tuned LLMs on multi-task tabular regression.

desk verdict A genuinely new architecture for tabular regression, but the headline result against LLMs depends on a baseline protocol that probably flatters the proposed method; worth reviewing, not accepting as-is. read the letter →

arxiv 2506.06926 v1 pith:6RIAHO7T submitted 2025-06-07 cs.LG

Wei Min Loh , Jiaqi Shang , Pascal Poupart This is my paper
classification cs.LG
keywords tabularregressionmulti-tasklearningsign-magnituderepresentationbasistransformersmulti-labelclassificationcolumn-orderinvarianceOpenML-CTR23LLMbaselines
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a small, randomly initialized transformer built around tabular structure can outperform fine-tuned pretrained large language models at multi-task tabular regression. Its key moves are to represent every numeric value as a sign-magnitude bit vector, turning regression into multi-label classification, and to process rows with attention layers that respect column names, column-order invariance, mixed entry types, and missing values. On the 34-dataset OpenML-CTR23 benchmark, the basis transformer reports a median test $R^2$ of 0.241 versus $-0.097$ for the best LLM baseline, with the lowest standard deviation across datasets and about five times fewer parameters. If correct, this would mean that structural inductive biases plus a faithful numeric representation can beat much larger pretrained language models in this setting without any preprocessing.

What carries the argument

The central object is the sign-magnitude representation (SMR): a real number is written as a sign bit plus bits for powers of two from $2^{h-1}$ down to $2^{-\ell}$, and the model's output is the same bit vector, which converts regression into multi-label classification with summed binary cross-entropy. Around that representation, each basis transformer block performs basis compression, where learned basis queries (akin to inducing points) summarize each column into a fixed-size sequence via cross attention in linear time; a latent mixture merges column-name and entry-value information per column; latent compression projects the result down; latent contextualization applies self-attention across columns; and latent decompression produces fresh queries for the next block. Stacked blocks with residual connections behave like an ensemble of weak learners, and an adaptive loss-reweighting scheme upweights examples whose predicted magnitude is far from the target magnitude.

What would settle it

Re-run the 34-task comparison giving LLM baselines per-task tuned learning rates, full-precision numbers instead of five-significant-digit rounding, and counting unparsable outputs as worst-case predictions; if any LLM then reaches or exceeds the basis transformer's median $R^2$ of 0.241, the headline advantage is partly an artifact of baseline configuration. A complementary check is to hold out a fresh suite of heterogeneous tabular regression tasks and see whether the median $R^2$ gap persists.

Watch

Extended reading notes

Core claim

The central claim is that basis transformers — a stack of cross-attention blocks driven by learned basis queries — give the best reported multi-task tabular regression performance on OpenML-CTR23 while satisfying six desiderata: handling numeric, categorical, textual, and missing entries; variable column counts; use of column names; invariance to column order; preservation of the row–column–entry hierarchy; and preservation of numeric scale and precision. Numbers are encoded with a sign-magnitude bit vector and the target is predicted in the same representation, so training minimizes summed binary cross-entropy over bits rather than a scale-sensitive regression loss. The headline result is a median test $R^2$ of 0.241 across 34 datasets, a gain of 0.338 over the best LLM baseline (median $-0.097$), the lowest standard deviation (3.295), roughly five times fewer parameters, and no reliance on pretrained weights or dataset-specific normalization.

Load-bearing premise

The claim rests on the assumption that the OpenML-CTR23 multi-task setup, with LLM baselines configured exactly as in the paper (default learning rates, numbers rounded to five significant digits, equal gradient steps, and unparsed outputs excluded from the $R^2$ computation), is a fair and representative test of multi-task tabular regression.

Editorial extensions

If this is right

  • Tabular regression can be treated as bit-wise classification, eliminating per-dataset normalization and enabling cross-task transfer across targets with very different scales.
  • A randomly initialized, structure-aware architecture can beat fine-tuned pretrained LLMs up to five times its size on multi-task tabular regression.
  • One checkpoint can serve many heterogeneous tables in a single forward pass, ingesting free text, categories, missing values, and variable column counts together.
  • The low reported variance across seeds and datasets indicates that the design reduces the training instability that language-model baselines exhibit on tabular tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • My inference: the recipe should transfer beyond OpenML-CTR23 to any heterogeneous tabular domain — sensor fusion, time series, or medical records — since none of its components is benchmark-specific, though the paper does not test this.
  • My inference: part of the reported gap may come from LLM baseline configuration rather than architecture; per-task learning-rate tuning, full-precision numbers, or more fine-tuning steps could narrow it, and that comparison would settle the question.
  • My inference: because no normalization is required, the model is a natural candidate for zero-shot regression on unseen columns, a capability the paper motivates through its scale-preservation desideratum but does not directly benchmark.
  • My inference: the learned basis queries could serve as interpretability probes, because their attention weights reveal which value patterns each column summary mixes; the paper does not pursue this.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces basis transformers (BT), a transformer architecture for multi-task tabular regression that treats each row as a set of (column name, value) pairs, encodes numeric values with a sign-magnitude representation (SMR), converts regression into multi-label bit prediction, and uses cross-attention modules for basis compression and latent mixing. On 34 datasets from OpenML-CTR23, BT is reported to achieve a median test R2 of 0.241, the lowest variance among compared methods, with roughly five times fewer parameters than the best LLM baseline, and to outperform fine-tuned pretrained LLM baselines even when the BT stack is randomly initialized. The paper also provides ablations on numeric encoding, number of BT blocks, SMR versus scalar outputs, and the adaptive loss reweighting scheme.

Significance. The architecture is well motivated by six explicit desiderata and combines several plausible ideas: SMR as a scale-preserving numeric tokenization, set/permutation invariance, use of column-name metadata, and regression-as-multilabel-classification with binary cross entropy. The paper includes 5-seed experiments, per-dataset results in the appendix, and several ablation studies, which is useful for a system-style contribution. If the empirical claims survive a more rigorous baseline protocol, BT would be a meaningful step toward tabular foundation models that handle heterogeneous, multi-task, and partially textual data. The main result is not circular in the derivation sense, but its significance depends heavily on the fairness and competence of the LLM baselines and on the breadth of the claim that BT is the best multi-task tabular regression model.

major comments (4)
  1. [Appendix F.4, Tables 3 and 7] The LLM baseline protocol does not currently establish that the baselines are competent. Learning rates are taken as default or recommended values, the number of gradient steps is fixed at 40,000 while the batch size is smaller than BT, serialized numbers are rounded to five significant digits, and R2 is computed after discarding unparseable LLM outputs. Table 7 shows success rates as low as 0.13 (Pythia-1B on health_i) and 0.18 (Cerebras-1.3B on student), so some baseline R2 entries are computed on small, possibly nonrandom subsets. Because the central claim is superiority over fine-tuned pretrained LLMs, the authors should either provide a validation-guided hyperparameter sweep (at least per-family learning rates and step counts) or a convincing argument that the default protocol is adequate, together with a sensitivity analysis for the effects of rounding and of excluding parse failures. As written, the reported margin may partly reflect an under-tuned comparison rather than an architectural advantage.
  2. [Sections 4.1, 5.2, Appendix F.3] The statement that BT surpasses pretrained LLM baselines 'even when initialized from randomized weights' and 'without any pretraining' is undercut by the use of a pretrained distilled BERT text encoder. The manuscript does not state whether the BERT weights are frozen or fine-tuned. If they are frozen, the model still relies on a pretrained representation and the random-initialization slogan should be restricted to the BT stack; if they are fine-tuned, the parameter counts and the comparison with pretrained baselines should include them. The authors should state the training status explicitly and report both learnable and total parameter counts.
  3. [Table 2, Abstract, Section 5.2] The headline improvement of 0.338 in median R2 appears to be computed against Cerebras-1.3B, whose median R2 is -0.097 but whose mean R2 is -123.7, a model that the paper itself describes as having 'incredibly poor mean' performance. Because this baseline is highly unstable, the choice of median as the headline statistic is fragile, and the reported margin could change with small improvements in Cerebras-1.3B's calibration. The authors should report per-dataset win/loss counts, add paired nonparametric comparisons (e.g., Wilcoxon signed-rank tests) between BT and each baseline, and identify unambiguously which baseline is 'best-performing' when this term is used in the abstract.
  4. [Section 5.2, Appendix D] The contribution statement that BT has the 'best performance in multi-task tabular regression on the OpenML-CTR23 benchmark' is broader than what is evaluated: the only implemented baselines are LLMs, selected because they satisfy the desiderata. Strong tabular methods such as XGBoost, TabPFN, FT-Transformer, and CARTE are excluded for documented reasons, but the claim should still be narrowed to 'best among implemented LLM-based baselines under desiderata D1-D6,' or the evaluation should include at least one feasible non-LLM tabular baseline on a subset of numeric-only datasets to support the broader title and contribution claim.
minor comments (5)
  1. [Appendix F.3] The SMR bit counts (29 high bits, 14 low bits, 1 sign bit) are described as 'chosen to fit the range of values in the dataset,' which makes the representation benchmark-specific; a sensitivity analysis over bit counts or a test with targets outside the chosen range would strengthen the D6 scale-preservation claim.
  2. [Abstract, Section 5.2, Figure 2] No parameter count is given for the full BT model in the text, despite the 'five times fewer parameters' claim; a table with learnable and total parameter counts for BT and all baselines would make this claim verifiable.
  3. [Table 2] The table formatting is inconsistent, with negative values rendered as leading minus signs separated from digits (e.g., 'Pythia410M - 2.980') and decimal spacing artifacts such as '3 .295'; these should be cleaned up.
  4. [Section 5.1, Figure 3] The x-axis of Figure 3 is labeled [0,200] while the text says validation is performed every 200 gradient steps; the axis unit (strides versus gradient steps) should be clarified.
  5. [Overall] No code or configuration release is mentioned; for a system paper with this much experimental detail, a code link or a clear availability statement is needed.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the benchmark comparison is an independent held-out evaluation; tuned hyperparameters are not predictions derived from the targets.

full rationale

The paper's central claim is empirical: basis transformers achieve a higher median test R2 than LLM baselines on 34 OpenML-CTR23 datasets. I checked whether any 'prediction' or design-specific result reduces to its own input by construction. The SMR bit budget (29 high bits, 14 low bits, 1 sign bit) is a hyperparameter chosen to fit the range of values in the benchmark (Appendix F.3), and gamma = 0.2 is selected via ablations. These are tuned design choices; they do not make the held-out test R2 a restatement of the tuning targets. The adaptive loss reweighting g(y, yhat) is defined from prediction and target but is used only as a training weight; it does not define the test metric or the decoded predictions. The LLM baselines are external pretrained models, and no load-bearing result is justified by a self-citation or by a uniqueness claim from the authors. Concerns about baseline competence (default learning rates, equal gradient steps, 5-significant-digit rounding, and excluding parse failures from R2) bear on fairness and external validity, not on circularity, because the comparison is still an independent measurement of held-out performance. I therefore find no circular step.

Assumptions & free parameters 5 free parameters · 5 assumptions · 2 invented entities

The central result depends on a large set of design choices: SMR bit ranges, number of blocks, gamma, embedding dimensions, and the evaluation protocol itself. None are derived from theory, so they are free parameters in the ledger. The main domain assumption is that the benchmark and LLM baseline configuration constitute a fair test of the architecture's value.

free parameters (5)
  • SMR exponent ranges = 29 high bits, 14 low bits, 1 sign bit
    Chosen to fit the range of values in the dataset (Appendix F.3). This is a free design parameter affecting capacity and resolution.
  • Number of BT blocks = 4
    Selected via ablation on 3 datasets (Appendix G) and then fixed for the full 34-task run. It is a model architecture parameter tuned on data.
  • Adaptive loss reweighting gamma = 0.2
    Selected via ablation on 3 tasks (Appendix I); the paper states the full experiment uses 0.2 because there are more tasks, a heuristic extrapolation.
  • Global dimension hyperparameters = D=144, heads=8, queries=64, r=6, self-attn blocks=9
    Chosen by the authors for the full run without a systematic search; these jointly determine model capacity.
  • Evaluation split sampling = 20% of the smallest dataset for validation and test
    This defines the benchmark evaluation protocol and affects difficulty and variance of the reported R2.
assumptions (5)
  • domain assumption OpenML-CTR23 splits and dataset selection are representative for multi-task tabular regression.
    The paper evaluates on this benchmark and generalizes from it to claims about tabular multi-task learning.
  • domain assumption Default HuggingFace hyperparameters are a fair baseline configuration for the tabular LLM baselines.
    Appendix F.4 says learning rates and related hyperparameters are default or recommended ones, without evidence that the baselines are well-tuned for the tasks.
  • ad hoc to paper Rounding numbers to 5 significant digits in LLM serialization preserves enough numeric precision.
    Appendix F.4, introduced to fit LLM memory constraints; it only affects LLM baselines and may worsen their accuracy.
  • ad hoc to paper Excluding failed LLM parses from R2 is a fair comparison.
    Appendix E.2: LLM R2 is computed on parsed predictions only; Table 3 shows non-100% success rates for many baselines, which could inflate their R2.
  • domain assumption BERT semantic space is small enough for a distilled 8-layer 256-dim encoder.
    Appendix F.3 assumes the text space across the 34 datasets is small, so a tiny BERT suffices.
invented entities (2)
  • Basis queries
    purpose: Learned inducing points that summarize each column's text and numeric values into a fixed-size representation, playing a role similar to soft decision-tree splits.
    No falsifiable external handle; they are learned parameters of the architecture.
  • Latent mixture, compression, decompression modules
    purpose: Internal modules that merge column name and value, compress to a fixed vector, and generate basis queries for the next block.
    Internal architectural components with no observable consequences beyond the benchmark numbers.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Basis Transformers for Multi-Task Tabular Regression." pith.science (2026). https://pith.science/paper/6RIAHO7T

@misc{pith2026250606926,
  author       = {Pith},
  title        = {Pith review of: Basis Transformers for Multi-Task Tabular Regression},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6RIAHO7T}},
  note         = {Machine review of arXiv:2506.06926}
}
abstract

Dealing with tabular data is challenging due to partial information, noise, and heterogeneous structure. Existing techniques often struggle to simultaneously address key aspects of tabular data such as textual information, a variable number of columns, and unseen data without metadata besides column names. We propose a novel architecture, \textit{basis transformers}, specifically designed to tackle these challenges while respecting inherent invariances in tabular data, including hierarchical structure and the representation of numeric values. We evaluate our design on a multi-task tabular regression benchmark, achieving an improvement of 0.338 in the median $R^2$ score and the lowest standard deviation across 34 tasks from the OpenML-CTR23 benchmark. Furthermore, our model has five times fewer parameters than the best-performing baseline and surpasses pretrained large language model baselines -- even when initialized from randomized weights.

Figures

Figures reproduced from arXiv: 2506.06926 by the authors.

Figure 1
Figure 1. A basis transformer block, and a simplistic two-column diabetes example annotated with [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Median and standard deviation of R2 against the number of learnable parameters. High standard deviation (> 108 ) results are omitted as outliers. pretraining. Furthermore, the right side of the figure shows a trend in how larger models tend to have much higher variability in performance across datasets. The LLMs do not always abide by the serialization template. During evaluation, the generated outputs by the LLMs a… view at source ↗
Figure 3
Figure 3. Validation R2 scores over gradient steps per dataset. The range of y-axis is [-20, 1] in symmetric log scale and the range of x-axis is [0, 200]. The blue line is BT . Higher values are better. a higher dimension tensor leads to more wastage. This is especially the case where the variance in sequence length L is high as BT handles numeric values (L = 1), and textual values (L ≫ 1). Alignment to multi-task Bayesian r… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Relative performance (with a standard error of [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]
Figure 5
Figure 5. Figure 5: An enlarged view of validation R2 scores over gradient steps per dataset. The shaded area represents the 1 standard deviation across 5 random seeds from the mean. The range of y-axis is [-20, 1] (in symmetric log scale) and the range of x-axis is [0, 200]. The blue lin…
Figure 6
Figure 6. Figure 6: Test performance against the number of BT blocks over several choices of embedding dimension D. The shaded area represents the region of 1 standard error over 5 random seeds. Higher y-values are better. To show the impact of the number of blocks on a smaller scale expe…
Figure 7
Figure 7. Figure 7: Rescaled test R2 score against log mean target value. Each point represents a dataset, averaged over 5 random seeds. One dataset was excluded because the mean target value is negative valued. Higher y-values are better. We trained an identical model as outlined in Appe…
Figure 8
Figure 8. Figure 8: Test performance with and without adaptive loss reweighing scheme over different degrees [PITH_FULL_IMAGE:figures/full_fig_p023_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

48 extracted references · 24 canonical work pages

  1. [1]

    Tabnet: Attentive interpretable tabular learning

    Sercan Ö Arik and Tomas Pfister. Tabnet: Attentive interpretable tabular learning. In Proceed- ings of the AAAI conference on artificial intelligence, volume 35, pages 6679–6687, 2021

    work page 2021

  2. [2]

    Task clustering and gating for bayesian multitask learning

    Bart Bakker and Tom Heskes. Task clustering and gating for bayesian multitask learning. Journal of Machine Learning Research, 4(May):83–99, 2003

    work page 2003

  3. [3]

    Applying natural language processing and hierarchical machine learning approaches to text difficulty classification

    Renu Balyan, Kathryn S McCarthy, and Danielle S McNamara. Applying natural language processing and hierarchical machine learning approaches to text difficulty classification. Inter- national Journal of Artificial Intelligence in Education, 30(3):337–370, 2020

    work page 2020

  4. [4]

    Pythia: A suite for analyzing large language models across training and scaling

    Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pages 2397–2430. PMLR, 2023

    2023

  5. [5]

    Bayesian Hierarchical Mixtures of Experts

    Christopher M Bishop and Markus Svensén. Bayesian hierarchical mixtures of experts. arXiv preprint arXiv:1212.2447, 2012

    work page Pith review arXiv 2012

  6. [6]

    Revisiting multimodal transformers for tabular data with text fields

    Thomas Bonnier. Revisiting multimodal transformers for tabular data with text fields. In Findings of the Association for Computational Linguistics ACL 2024, pages 1481–1500, 2024

    work page 2024

  7. [7]

    Deep neural networks and tabular data: A survey

    Vadim Borisov, Tobias Leemann, Kathrin Seßler, Johannes Haug, Martin Pawelczyk, and Gjergji Kasneci. Deep neural networks and tabular data: A survey. IEEE transactions on neural networks and learning systems, 2022

    2022

  8. [8]

    A comprehensive review on data preprocessing techniques in data analysis

    V olkan Çetin and Oktay Yıldız. A comprehensive review on data preprocessing techniques in data analysis. Pamukkale Üniversitesi Mühendislik Bilimleri Dergisi, 28(2):299–312, 2022

    work page 2022

Show all 48 references
  1. [9]

    Xgboost: A scalable tree boosting system

    Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pages 785–794, 2016

    2016

  2. [10]

    Scaling instruction-finetuned language models

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25(70):1–53, 2024

    2024

  3. [11]

    Bayesian multitask learning with latent hierarchies

    Hal Daumé III. Bayesian multitask learning with latent hierarchies. arXiv preprint arXiv:0907.0783, 2009

    2009 arXiv

  4. [12]

    Bert: Pre-training of deep bidirectional transformers for language understanding, 2019

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019. URL https://arxiv.org/ abs/1810.04805

    2019 arXiv

  5. [13]

    Cerebras-gpt: Open compute-optimal language models trained on the cerebras wafer-scale cluster

    Nolan Dey, Gurpreet Gosal, Hemant Khachane, William Marshall, Ribhu Pathria, Marvin Tom, Joel Hestness, et al. Cerebras-gpt: Open compute-optimal language models trained on the cerebras wafer-scale cluster. arXiv preprint arXiv:2304.03208, 2023

    2023 arXiv

  6. [14]

    How numerical precision affects mathematical reasoning capabilities of llms

    Guhao Feng, Kai Yang, Yuntian Gu, Xinyue Ai, Shengjie Luo, Jiacheng Sun, Di He, Zhenguo Li, and Liwei Wang. How numerical precision affects mathematical reasoning capabilities of llms. arXiv preprint arXiv:2410.13857, 2024

    2024 arXiv

  7. [15]

    Openml-ctr23–a curated tabular regression benchmarking suite

    Sebastian Felix Fischer, Matthias Feurer, and Bernd Bischl. Openml-ctr23–a curated tabular regression benchmarking suite. In AutoML Conference 2023 (Workshop), 2023

    2023

  8. [16]

    Tabkanet: Tabular data modeling with kolmogorov-arnold network and transformer

    Weihao Gao, Zheng Gong, Zhuo Deng, Fuju Rong, Chucheng Chen, and Lan Ma. Tabkanet: Tabular data modeling with kolmogorov-arnold network and transformer. arXiv preprint arXiv:2409.08806, 2024

    2024 arXiv

  9. [17]

    Large scale transfer learning for tabular data via language modeling

    Josh Gardner, Juan C Perdomo, and Ludwig Schmidt. Large scale transfer learning for tabular data via language modeling. arXiv preprint arXiv:2406.12031, 2024. 10

    2024 arXiv

  10. [18]

    What every computer scientist should know about floating-point arithmetic

    David Goldberg. What every computer scientist should know about floating-point arithmetic. ACM computing surveys (CSUR), 23(1):5–48, 1991

    1991

  11. [19]

    Revisiting deep learning models for tabular data

    Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning models for tabular data. Advances in neural information processing systems , 34: 18932–18943, 2021

    2021

  12. [20]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  13. [21]

    On typical range, sensitivity, and normalization of mean squared error and nash-sutcliffe efficiency type metrics

    Hoshin Vijai Gupta and Harald Kling. On typical range, sensitivity, and normalization of mean squared error and nash-sutcliffe efficiency type metrics. Water Resources Research, 47(10), 2011

    2011

  14. [22]

    The coefficient of determination exposed

    Gerald J Hahn. The coefficient of determination exposed. Chemtech, 3(10):609–612, 1973

    1973

  15. [23]

    Tabllm: Few-shot classification of tabular data with large language models

    Stefan Hegselmann, Alejandro Buendia, Hunter Lang, Monica Agrawal, Xiaoyi Jiang, and David Sontag. Tabllm: Few-shot classification of tabular data with large language models. In International Conference on Artificial Intelligence and Statistics, pages 5549–5581. PMLR, 2023

    2023

  16. [24]

    Tabpfn: A transformer that solves small tabular classification problems in a second

    Noah Hollmann, Samuel Müller, Katharina Eggensperger, and Frank Hutter. Tabpfn: A transformer that solves small tabular classification problems in a second. In Proceedings of the Eleventh International Conference on Learning Representations (ICLR) , 2023. URL https://openrevie...

    2023

  17. [25]

    Tabtransformer: Tabular data modeling using contextual embeddings

    Xin Huang, Ashish Khetan, Milan Cvitkovic, and Zohar Karnin. Tabtransformer: Tabular data modeling using contextual embeddings. arXiv preprint arXiv:2012.06678, 2020

    2012 arXiv

  18. [26]

    Carte: pretraining and transfer for tabular learning

    Myung Jun Kim, Léo Grinsztajn, and Gaël Varoquaux. Carte: pretraining and transfer for tabular learning. In Proceedings of the 41st International Conference on Machine Learning, pages 23843–23866, 2024

    2024

  19. [27]

    The stack: 3 tb of permissively licensed source code

    Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jia Li, Chenghao Mou, Carlos Muñoz Ferrandis, Yacine Jernite, Margaret Mitchell, Sean Hughes, Thomas Wolf, et al. The stack: 3 tb of permissively licensed source code. arXiv preprint arXiv:2211.15533, 2022

    2022 arXiv

  20. [28]

    A comprehen- sive analysis of deep regression.IEEE transactions on pattern analysis and machine intelligence, 42(9):2065–2081, 2019

    Stéphane Lathuilière, Pablo Mesejo, Xavier Alameda-Pineda, and Radu Horaud. A comprehen- sive analysis of deep regression.IEEE transactions on pattern analysis and machine intelligence, 42(9):2065–2081, 2019

    2019

  21. [29]

    Set transformer: A framework for attention-based permutation-invariant neural networks

    Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set transformer: A framework for attention-based permutation-invariant neural networks. In International conference on machine learning, pages 3744–3753. PMLR, 2019

    2019

  22. [30]

    Bart: Denoising sequence-to-sequence pre- training for natural language generation, translation, and comprehension

    Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequence pre- training for natural language generation, translation, and comprehension. In Proceedings of the 58th Ann...

    2020

  23. [31]

    One transformer for all time series: Representing and training with time-dependent heterogeneous tabular data

    Simone Luetto, Fabrizio Garuti, Enver Sangineto, Lorenzo Forni, and Rita Cucchiara. One transformer for all time series: Representing and training with time-dependent heterogeneous tabular data. arXiv preprint arXiv:2302.06375, 2023

    2023 arXiv

  24. [32]

    A bounded version of the nash-sutcliffe criterion for better model assessment on large sets of basins

    THIBAULT Mathevet, CLAUDE Michel, Vazken Andréassian, CJIP Perrin, et al. A bounded version of the nash-sutcliffe criterion for better model assessment on large sets of basins. IAHS PUBLICATION, 307:211, 2006

    2006

  25. [33]

    A survey of sensor fusion methods in wearable robotics

    Domen Novak and Robert Riener. A survey of sensor fusion methods in wearable robotics. Robotics and Autonomous Systems, 73:155–170, 2015

    2015

  26. [34]

    Tabular transformers for modeling multivariate time series

    Inkit Padhi, Yair Schiff, Igor Melnyk, Mattia Rigotti, Youssef Mroueh, Pierre Dognin, Jerret Ross, Ravi Nair, and Erik Altman. Tabular transformers for modeling multivariate time series. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processi...

    2021

  27. [35]

    Neural oblivious decision ensembles for deep learning on tabular data

    Sergei Popov, Stanislav Morozov, and Artem Babenko. Neural oblivious decision ensembles for deep learning on tabular data. arXiv preprint arXiv:1909.06312, 2019. 11

    1909 arXiv

  28. [36]

    The statistical measurement of agreement

    William S Robinson. The statistical measurement of agreement. American sociological review, 22(1):17–25, 1957

    1957

  29. [37]

    Saint: Improved neural networks for tabular data via row attention and contrastive pre-training

    Gowthami Somepalli, Avi Schwarzschild, Micah Goldblum, C Bayan Bruss, and Tom Goldstein. Saint: Improved neural networks for tabular data via row attention and contrastive pre-training. In NeurIPS 2022 First Table Representation Workshop

    2022

  30. [38]

    Multi-task bayesian optimization

    Kevin Swersky, Jasper Snoek, and Ryan P Adams. Multi-task bayesian optimization. Advances in neural information processing systems, 26, 2013

    2013

  31. [39]

    Well-read students learn better: On the importance of pre-training compact models

    Iulia Turc, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Well-read students learn better: On the importance of pre-training compact models. arxiv 2019. arXiv preprint arXiv:1908.08962, 1908

    2019 arXiv

  32. [40]

    Why tabular foundation models should be a research priority

    Boris van Breugel and Mihaela van der Schaar. Why tabular foundation models should be a research priority. arXiv preprint arXiv:2405.01147, 2024

    2024 arXiv

  33. [41]

    Attention is all you need

    A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017

    2017

  34. [42]

    Unipredict: Large language models are universal tabular classifiers

    Ruiyu Wang, Zifeng Wang, and Jimeng Sun. Unipredict: Large language models are universal tabular classifiers. arXiv preprint arXiv:2310.03266, 2023

    2023 arXiv

  35. [43]

    Multi-task reinforcement learning: a hierarchical bayesian approach

    Aaron Wilson, Alan Fern, Soumya Ray, and Prasad Tadepalli. Multi-task reinforcement learning: a hierarchical bayesian approach. In Proceedings of the 24th international conference on Machine learning, pages 1015–1022, 2007

    2007

  36. [44]

    Huggingface’s transformers: State-of-the-art natural language processing

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R Louf, Morgan Funtowicz, et al. Huggingface’s transformers: State-of-the-art natural language processing. arxiv 2019. arXiv preprint arXiv:1910.03771, 10, 2020

    2019 arXiv

  37. [45]

    Lamini-lm: A diverse herd of distilled models from large-scale instructions

    Minghao Wu, Abdul Waheed, Chiyu Zhang, Muhammad Abdul-Mageed, and Alham Fikri Aji. Lamini-lm: A diverse herd of distilled models from large-scale instructions. CoRR, abs/2304.14402, 2023. URL https://arxiv.org/abs/2304.14402

    2023 arXiv

  38. [46]

    On layer normalization in the transformer architecture

    Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. On layer normalization in the transformer architecture. In International Conference on Machine Learning, pages 10524–10533. PMLR, 2020

    2020

  39. [47]

    Qwen2.5 technical report

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  40. [48]

    Tinyllama: An open-source small language model

    Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model. arXiv preprint arXiv:2401.02385, 2024. 12 A Discussion of Societal Impacts While the scope of the paper focuses narrowly on tabular modality, the broader impact could be ext...

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.