Pith. sign in

REVIEW 4 major objections 8 minor 80 references

TabFlex: Scaling Tabular Learning to Millions with Linear Attention

T0 review · 4 major / 8 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read TabFlex's central claim is that non-causal linear attention can replace softmax attention in TabPFN-style in-context tabular models, preserving accuracy while scaling to datasets with millions of rows.

desk verdict A clean architectural idea with a wide empirical sweep, but the headline 1M-sample claims rest on an unexamined 20x context-length extrapolation and a flawed HBM theorem; still worth a serious referee. read the letter →

arxiv 2506.05584 v1 pith:F3GFXPOA submitted 2025-06-05 cs.LG

Yuchen Zeng , Tuan Dinh , Wonjun Kang , Andreas C Mueller This is my paper
classification cs.LG
keywords tabularclassificationin-contextlearninglinearattentionscalabilityTabPFNstate-spacemodelslarge-scaledatasets
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

TabFlex claims that the main obstacle to scaling TabPFN-style in-context tabular classification is the quadratic cost of softmax attention, and that swapping in non-causal linear attention removes that obstacle without sacrificing accuracy. The paper supports this with two architectural findings: causal mechanisms such as state-space models hurt in-context learning on tabular data, while linear attention keeps accuracy and runs far faster. If true, the payoff is a single model family that classifies datasets with thousands of features and hundreds of classes, and processes more than a million rows in seconds. That matters because in-context tabular learning promised training-free adaptation to new datasets but was previously confined to small ones.

What carries the argument

The load-bearing mechanism is non-causal linear attention. Rather than computing $\exp(q_i^\top k_j)$ for every pair of positions, it applies an element-wise feature map $\phi$ and maintains two shared statistics, $\sum_j \phi(k_j) v_j$ and $\sum_j \phi(k_j)$; each position's output is then $\phi(q_i)$ multiplied by those statistics. This keeps the order-invariance of non-causal attention, which the paper argues is necessary for in-context learning, while lowering asymptotic cost from quadratic in sequence length to $O(ND^2)$ FLOPs and $O(ND)$ HBM accesses. The paper proves that the straightforward PyTorch implementation of this mechanism matches the HBM access count of causal FlashLinearAttention, so no specialized fused attention kernel is needed. A conditional rule selects among three trained variants, S100, L100, and H1K, based on dataset sample count and feature dimensionality.

What would settle it

Run TabFlex on a single large dataset at increasing context lengths, e.g. 50K, 200K, 500K, and 1M rows, holding everything else fixed. If AUC drops sharply as the context passes the 50K training length, or if runtime or memory stops scaling linearly, the million-row claim is falsified.

Watch

Extended reading notes

Core claim

The central discovery is that non-causal linear attention, using the element-wise kernel feature map $\phi(x)=\mathrm{elu}(x)+1$, is a drop-in replacement for softmax attention in TabPFN's in-context architecture. The model computes two shared statistics, $\sum_j \phi(k_j) v_j$ and $\sum_j \phi(k_j)$, so each token's output is $\phi(q_i)$ multiplied by those statistics; this preserves permutation-invariant, non-causal attention while reducing per-token cost to constant time with respect to sequence length. The paper trains three variants on synthetic priors at different prompt lengths and feature and class counts, and routes each dataset to the matching variant. On the 57 small benchmarks TabFlex reaches top-2 AUC while running about 2.8x faster than TabPFN per 1000 instances, and on the 1,025,009-row poker-hand dataset it reaches AUC 0.84 in 4.88 seconds.

Load-bearing premise

The central claim relies on the model's in-context behavior carrying over from 50K-row synthetic training prompts to datasets with over a million rows, and the paper does not describe how inference contexts longer than the training length are handled.

Editorial extensions

If this is right

  • On the 57 small benchmarks, TabFlex matches TabPFN's top-2 AUC while cutting per-1000-instance time about 2.8x; if this transfers, linear attention is a strict efficiency upgrade for small in-context tabular models.
  • On the 36-dataset TabZilla hard benchmark, TabFlex, TabPFN, and XGBoost are the only tested methods that run on every dataset, and TabFlex is the fastest of the three; this widens the reach of in-context tabular learning to datasets where most neural baselines fail.
  • On poker-hand with 1,025,009 rows, TabFlex reports AUC 0.84 in 4.88 seconds, while TabPFN takes 15.36 seconds for AUC 0.72 and the fifth-fastest baseline takes over 500 seconds; million-row in-context inference becomes practical.
  • Reducing feature dimensionality to 10% or sampling 20% of the training rows preserves reported accuracy while at least halving latency, so TabFlex's advantage compounds with standard data-reduction preprocessing.
  • Because the models are pretrained on synthetic priors and never update parameters per dataset, the speed gain is available immediately on unseen tasks, not after task-specific tuning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If TabFlex's accuracy holds at 1M rows despite being trained on 50K-row prompts, that would suggest linear-attention in-context models extrapolate in context length far beyond training; a direct sweep of context sizes between 50K and 1M would settle this.
  • The paper's ablation shows XGBoost overtakes TabFlex around 800 features, which suggests the headline speedup is most robust on wide-but-not-extreme feature ranges; projecting features above 1000 through random projection is a testable part of the method's high-dimensional claim.
  • Because the paper treats regression by discretizing targets into bins, the approach's applicability to regression is only as strong as the chosen binning; a more principled regression-specific prior is a natural next test.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 8 minor

Summary. This paper proposes TabFlex, an in-context-learning tabular classifier that replaces TabPFN's quadratic softmax attention with non-causal linear attention to target scalability to large datasets. TabFlex consists of three sub-models trained on synthetic data drawn from TabPFN's prior: TabFlex-S100 (1,152-token prompts, 100 features, 10 classes), TabFlex-L100 (50,000-token prompts, 100 features, 10 classes), and TabFlex-H1K (50,000-token prompts, 1,000 features, 100 classes), selected by the dataset-driven rule in Algorithm 1. The authors analyze HBM access counts for linear attention (Theorem 1 and Lemmas 2-4), report that causal/SSM architectures underperform non-causal attention in this ICL setting while linear attention matches softmax accuracy with lower runtime, and evaluate TabFlex on 115 OpenML datasets plus the TabZilla hard benchmark. Reported results include top-2 median AUC on 57 small datasets with a roughly 2.8x per-instance speedup over TabPFN, competitive accuracy with XGBoost on smaller data, and a poker-hand (1,025,009 samples) result of AUC 0.84 in 4.88 seconds, with additional experiments on regression, image data, and data-efficient techniques such as PCA/SVD/random projection and sample selection.

Significance. If the results hold, TabFlex is a practically valuable contribution: it is a simple modification of a widely used method with a large constant-factor speedup, preserved accuracy on small data, and a credible path to million-sample in-context prediction. The evaluation is broad (115 datasets, 25 baselines), the code is released, and the design findings (notably the importance of non-causality for linear attention in ICL) are directly useful to practitioners. However, the flagship 'scaling to millions' claim rests on a 20-40x context-length extrapolation that is neither described nor tested, the HBM-access proof in Section C.1 contains a genuine counting error, and several quantitative efficiency claims compare timings from different GPU generations. These issues are load-bearing for the central claims but appear fixable within the manuscript's scope through a corrected analysis, a length-generalization study, and re-qualified or re-run baseline comparisons.

major comments (4)
  1. [Sec. 5, 6.3, C.2; Tables 2, 13] The headline claim that TabFlex 'scales seamlessly to millions of samples' is currently an extrapolation rather than a demonstrated capability. TabFlex-L100 and TabFlex-H1K are trained on prompts of length 50K (Section C.2 and Table 6), yet Section 6.3 reports TabFlex on poker-hand (1,025,009 samples) while stating that TabFlex 'utilizes all available training data,' and Table 13 reports datasets up to 1,997,410 samples. Neither Section 5 nor Section C.2 describes how inference handles sequences 20-40x longer than any training prompt (e.g., chunking, recomputation of the linear-attention statistics S and z, or memory layout), and no length-generalization evaluation is provided. Since non-causal linear attention forms global sums over all keys, moving from N=50K to N=1M changes both the scale of the normalization denominator and the effective weight per training sample relative to the training regime, so the poker-hand AUC of 0.84 at N~1M is not covered by the training distribution. This is compounded by the evaluation protocol in Table 2: TabPFN is restricted to 3,000 training samples (and, per Section D.2, to 100 randomly selected features and the nine most frequent classes), so the reported superiority over TabPFN conflates the architectural change with the removal of these restrictions and with data quantity. Section 8's Limitations paragraph discusses feature dimensionality and regression but never mentions context-length generalization. I would ask the authors to (i) document the inference-time procedure for sequences beyond 50K, (ii) report a length-ablation on real large datasets (e.g., AUC versus training-subset size at 10K/50K/100K/250K/500K/1M), and (iii) add a controlled comparison where TabFlex is also evaluated with 3,000 training samples.
  2. [Sec. C.1; Lemma 2, Theorem 1, Table 5] The proof of Lemma 2 miscounts HBM accesses for Algorithm 3. The proof states that the operations in Algorithm 3 'are also repeated T times,' but the inner loop over j (lines 10-13) executes T times for each outer iteration i, that is, T^2 times in total. Loading Qj and Oj (2BD) and writing Oj (BD) therefore contribute 3BD * T^2 = 3N^2D/B, not 5ND; combined with the 2ND from loading Ki and Vi, Algorithm 3 as written requires 2ND + 3N^2D/B HBM accesses, which is quadratic in N. Consequently, Theorem 1's assertion that non-causal linear attention 'requires O(ND) HBM accesses' is not established by the cited lemma, and the '5ND' entry for Algorithm 3 in Table 5 is incorrect. The conclusion for Listing 1 appears recoverable by a direct count (roughly 4ND + 2D^2 reads/writes), so the practical design choice may survive, but as written the proof of Lemma 2, the statement of Theorem 1, and the pseudocode of Algorithm 3 (which should be restructured into two sequential passes: first accumulate S, then process Q) all need to be corrected.
  3. [Sec. D.2; Tables 1, 2; abstract] Several quantitative efficiency claims compare TabFlex timings on an A100 GPU against baseline timings taken from a V100-based benchmark. Section D.2 states that results for all baselines except TabPFN 'are taken directly from McElfresh et al. (2023), who conducted their experiments using a V100 GPU, while our experiments are run on an A100 GPU.' Since the A100 offers roughly twice the memory bandwidth and FLOPs of the V100, the abstract's '1.5x speedup over XGBoost,' the per-1000-instance runtimes against XGBoost and other baselines in Table 1, and the Table 2 statement that the 'fifth fastest method requires more than 500 seconds' on poker-hand are not like-for-like comparisons. The appendix disclosure is welcome, but the main-text claims are stated without this caveat. At minimum, XGBoost and the other fast baselines in Figure 4 should be re-run on the same A100 hardware used for TabFlex, or the claimed speedups should be re-qualified as hardware-confounded.
  4. [Secs. 6.3 and D.5; Tables 2 and 13] The large-scale evaluation is difficult to interpret as reported. TabFlex on poker-hand appears as AUC 0.84 in 4.88 seconds in Table 2, but Table 13 reports TabFlex accuracy 0.542 with a runtime of 15 seconds on the same dataset and reports TuneTables accuracy 1.000; the paper does not reconcile these discrepancies, which may be explainable by different metrics, splits, or hardware, but no explanation is given. In addition, Table 2 reports single runs without variance, and the '5th-best' summarization does not identify which specific baselines produced the comparison values, so the reader cannot judge how the comparison set varies across datasets. I would ask the authors to state the metric used in each table, identify the baselines underlying the 5th-best entries (or report the full baseline distribution), and provide variance or multiple-seed results for at least the TabFlex numbers in Table 2.
minor comments (8)
  1. [Table 10] Table 10 lists 'blood-transfusion-service-center' and 'colic' twice each and uses inconsistent identifiers for what appears to be the same 'visualizing_livestock' dataset; the rows should be deduplicated and the dataset names harmonized.
  2. [Table 3] Table 3 does not state the reported metric in its caption or header (the values are presumably R^2), and the all-zero row for yprop_4_1 indicates degenerate behavior for both TabFlex and XGBoost that deserves at least a footnote.
  3. [Abstract and Sec. 5] The abstract's claim that TabFlex handles 'hundreds of classes' overstates the trained configuration, since TabFlex-H1K is trained with at most 100 classes and TabFlex-L100 with 10 classes (Section 5).
  4. [Sec. 6.3, Fig. 4] The sentence in Section 6.3 stating that 'only TABFLEX, TABPFN, and XGBoost successfully run on all datasets' should be reconciled with the star markers in Figure 4, which appear to include additional methods.
  5. [Sec. C.4, Table 8] The sensitivity analysis for the Algorithm 1 decision thresholds uses only two datasets per regime; a summary over the full 115-dataset evaluation would more convincingly support the claim that performance is not sensitive to the chosen boundaries.
  6. [Sec. 8] Given that TabPFNv2 (Hollmann et al., 2025) is the current state of the art for prior-fitted tabular networks, the paper should either include a comparison against it or explicitly justify its exclusion from the evaluation.
  7. [Eq. (4)] Equation (4) and the surrounding notation render with garbled summation symbols and without an explicit fraction bar for the linear-attention normalization; the formulas should be typeset with clear indices and denominators.
  8. [Sec. 4.1, Fig. 2a] The causal-versus-non-causal finding in Figure 2a is averaged over only 20 synthetic datasets with no error bars or significance test; since Findings 1 and 2 motivate the entire architecture choice, a few more replications or confidence intervals would strengthen the claim.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: TabFlex's reported predictions are produced by a synthetic-prior-pretrained model evaluated on external benchmarks, with no target-derived fit.

full rationale

The paper's derivation chain is empirical and externally anchored. TabFlex is trained on synthetic datasets generated from TabPFN's prior (Section C.2), and its reported AUCs on the 57 small datasets, TabZilla hard datasets, and additional OpenML/TabZilla benchmarks are evaluated on held-out real-world data. No parameter is fitted to the test labels or to the reported benchmark scores. The architectural findings (non-causal vs. causal attention, Mamba vs. Transformer, softmax vs. linear attention) come from controlled experiments on synthetic or independently curated datasets, not from the target results. The only self-citation involving the authors' prior work (LIFT, Dinh et al. 2022) is a related-work mention and is not load-bearing. Theorem 1 and the HBM analysis are internal complexity calculations with explicit assumptions, not circular. The most notable gap is that the L100/H1K models are trained on 50K-token prompts (Section C.2, Table 6) while Table 2 reports inference on datasets up to 1,025,009 instances; this is a genuine context-length extrapolation and a robustness risk, but it is not circular because the 1M-sample predictions are not constructed from or fitted to the model's training targets. Similarly, the comparison where TabPFN uses 3,000 samples while TabFlex uses all data is an experimental-design asymmetry, not a circular reduction. Accordingly, no circular step is identified.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central contribution relies on the synthetic-data training paradigm of TabPFN and on a hardware-matched evaluation that the paper does not actually run. The free parameters are architectural and hand-set, not fit, so the circularity burden is low; the main risks are empirical overclaims and an unproven length-generalization assumption.

free parameters (4)
  • Feature kernel phi
    elu(x)+1 chosen from Katharopoulos et al., an architectural hyperparameter not fit to data.
  • Model selection thresholds (n>=3K, d<=100, d/n>=0.2, d<=1000)
    Hand-set boundaries in Alg. 1; sensitivity analysis only across three dataset types.
  • Random projection target dimension 1000
    Used to reduce high-dimensional datasets before TabFlex-H1K; chosen without tuning.
  • Training hyperparameters (lr=3e-5, batch sizes, epochs)
    Adopted from TabPFN; not fitted to target datasets.
assumptions (4)
  • domain assumption Synthetic data generated from TabPFN's prior is representative of real-world tabular classification tasks, including large and high-dimensional ones.
    All three TabFlex models are trained only on synthetic data; the paper does not validate the prior on the test distributions beyond the reported benchmark sets.
  • domain assumption A TabFlex model trained with 50K-token prompts can be applied to sequences up to 1M tokens without degradation or special handling.
    The paper reports poker-hand (1M samples) results but never describes how the context length limit of L100/H1K is exceeded; inferred from Alg. 1 and Table 2.
  • domain assumption Runtimes measured on A100 are directly comparable to baseline runtimes taken from McElfresh et al. on V100.
    Used for all efficiency claims against XGBoost and other baselines; stated in Appendix D.2.
  • domain assumption Non-causal linear attention with an elementwise kernel preserves the in-context learning behavior of softmax attention for tabular data.
    Supported empirically by Fig. 2c/8c, but no theoretical bound; the kernel is fixed as elu+1.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TabFlex: Scaling Tabular Learning to Millions with Linear Attention." pith.science (2026). https://pith.science/paper/F3GFXPOA

@misc{pith2026250605584,
  author       = {Pith},
  title        = {Pith review of: TabFlex: Scaling Tabular Learning to Millions with Linear Attention},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/F3GFXPOA}},
  note         = {Machine review of arXiv:2506.05584}
}
read the original abstract

Leveraging the in-context learning (ICL) capability of Large Language Models (LLMs) for tabular classification has gained significant attention for its training-free adaptability across diverse datasets. Recent advancements, like TabPFN, excel in small-scale tabular datasets but struggle to scale for large and complex datasets. Our work enhances the efficiency and scalability of TabPFN for larger datasets by incorporating linear attention mechanisms as a scalable alternative to complexity-quadratic self-attention. Our model, TabFlex, efficiently handles tabular datasets with thousands of features and hundreds of classes, scaling seamlessly to millions of samples. For instance, TabFlex processes the poker-hand dataset with over a million samples in just 5 seconds. Our extensive evaluations demonstrate that TabFlex can achieve over a 2x speedup compared to TabPFN and a 1.5x speedup over XGBoost, outperforming 25 tested baselines in terms of efficiency across a diverse range of datasets. Furthermore, TabFlex remains highly effective on large-scale datasets, delivering strong performance with significantly reduced computational costs, especially when combined with data-efficient techniques such as dimensionality reduction and data sampling.

Figures

Figures reproduced from arXiv: 2506.05584 by the authors.

Figure 1
Figure 1. Illustration of TABPFN’s for classifying the entire dataset in one forward pass. In each layer, attention outputs for training sample positions attend to all other training samples, ensuring that predictions are invariant to the order of training samples. Conversely, attention outputs for test sample positions attend only to training samples, ensuring independent predictions for each test instance, unaffected by oth… view at source ↗
Figure 2
Figure 2. Impact of model architecture on tabular classification performance. Please refer to [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Runtime and AUC comparison of TABPFN and TABFLEX on validation datasets. across a broader spectrum of datasets. Since models trained on large, high-dimensional datasets often struggle in small regions due to optimization issues, we introduce three spe￾cialized models to address this limitation. • TABFLEX-S100 is trained on prompts with 1152 length (same as TABPFN), 100 features, and 10 classes. This is optimized for… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Visualization of tested methods with processing times under 0.5 seconds per 1000 instances on the TabZilla hard benchmark. We report the median AUC across the completed datasets, as several methods completed only a subset of the datasets. Compared to other methods (XGB…
Figure 7
Figure 7. Figure 7 [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 6
Figure 6. Figure 6: Varying feature dimension with dimensionality reduc￾tion methods. Results are measured on a set of datasets whose number of features is greater than 100. Dimension of features can be reduced up to 90% to preserve the performance (left) with inference being 2× faster (r…
Figure 8
Figure 8. Figure 8: Impact of model architecture on tabular classification performance. Detailed version of [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: Time and CUDA memory usage comparison of FlashAttention-2 (Dao, 2024), causal FlashLinearAttention (Yang et al., 2024), and linear attention (Katharopoulos et al., 2020) (implemented as in Listing 1). Results for FlashLinearAttention in two configurations: (1) head dim…
Figure 10
Figure 10. Figure 10: Visualization of training loss for TABFLEX models as a function of epoch and wallclock time [PITH_FULL_IMAGE:figures/full_fig_p022_10.png]
Figure 11
Figure 11. Figure 11: Accuracy and runtime versus the number of samples. Two settings are considered: (a) 800 features and (b) 1000 features. Each curve is averaged over 20 synthetic datasets with varying data distributions, generated the same algorithm as employed in TabPFN (Hollmann et a…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

80 extracted references · 47 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  3. [3]

    Agarwal, R., Melnick, L., Frosst, N., Zhang, X., Lengerich, B., Caruana, R., and Hinton, G. E. Neural additive models: Interpretable machine learning with neural nets. Advances in neural information processing systems, 34: 0 4699--4711, 2021

    work page 2021

  4. [4]

    Ahamed, M. A. and Cheng, Q. Mambatab: A simple yet effective approach for handling tabular data. arXiv preprint arXiv:2401.08867, 2024

    work page Pith review arXiv 2024

  5. [5]

    Transformers learn to implement preconditioned gradient descent for in-context learning

    Ahn, K., Cheng, X., Daneshmand, H., and Sra, S. Transformers learn to implement preconditioned gradient descent for in-context learning. Advances in Neural Information Processing Systems, 36: 0 45614--45650, 2023

    work page 2023

  6. [6]

    Arik, S. \"O . and Pfister, T. Tabnet: Attentive interpretable tabular learning. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp.\ 6679--6687, 2021

    work page 2021

  7. [7]

    Loan approval prediction based on machine learning approach

    Arun, K., Ishan, G., and Sanmeet, K. Loan approval prediction based on machine learning approach. IOSR J. Comput. Eng, 18 0 (3): 0 18--21, 2016

    work page 2016

  8. [8]

    Qwen technical report

    Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023 a

    arXiv 2023

Show all 80 references
  1. [9]

    Qwen-vl: A frontier large vision-language model with versatile abilities

    Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., and Zhou, J. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966, 2023 b

    2023 arXiv

  2. [10]

    Transformers as statisticians: Provable in-context learning with in-context algorithm selection

    Bai, Y., Chen, F., Wang, H., Xiong, C., and Mei, S. Transformers as statisticians: Provable in-context learning with in-context algorithm selection. Advances in neural information processing systems, 36: 0 57125--57211, 2023 c

    2023

  3. [11]

    E., and Cohan, A

    Beltagy, I., Peters, M. E., and Cohan, A. LongFormer : The long-document transformer. arXiv preprint arXiv:2004.05150, 2020

    2004 arXiv

  4. [12]

    G., van Rijn, J

    Bischl, B., Casalicchio, G., Feurer, M., Hutter, F., Lang, M., Mantovani, R. G., van Rijn, J. N., and Vanschoren, J. Openml benchmarking suites. arXiv:1708.03731v2 [stat.ML], 2019

    2019 arXiv

  5. [13]

    M., Gir \'o -i Nieto, X., and Ioannidis, A

    Bonet, D., Montserrat, D. M., Gir \'o -i Nieto, X., and Ioannidis, A. G. Hyperfast: Instant classification for tabular data. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp.\ 11114--11123, 2024

    2024

  6. [14]

    D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pp.\ 1877--1901, 2020

    1901

  7. [15]

    Z., and Wu, J

    Chen, J., Liao, K., Wan, Y., Chen, D. Z., and Wu, J. Danets: Deep abstract networks for tabular data classification and regression. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp.\ 3930--3938, 2022

    2022

  8. [16]

    and Guestrin, C

    Chen, T. and Guestrin, C. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pp.\ 785--794, 2016

    2016

  9. [17]

    Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models

    Chu, Y., Xu, J., Zhou, X., Yang, Q., Zhang, S., Yan, Z., Zhou, C., and Zhou, J. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arXiv:2311.07919, 2023

    2023 arXiv

  10. [18]

    Qwen2-audio technical report

    Chu, Y., Xu, J., Yang, Q., Wei, H., Wei, X., Guo, Z., Leng, Y., Lv, Y., He, J., Lin, J., et al. Qwen2-audio technical report. arXiv preprint arXiv:2407.10759, 2024

    2024 arXiv

  11. [19]

    Support-vector networks

    Cortes, C. Support-vector networks. Machine Learning, 1995

    1995

  12. [20]

    and Hart, P

    Cover, T. and Hart, P. Nearest neighbor pattern classification. IEEE transactions on information theory, 13 0 (1): 0 21--27, 1967

    1967

  13. [21]

    Cox, D. R. The regression analysis of binary sequences. Journal of the Royal Statistical Society Series B: Statistical Methodology, 20 0 (2): 0 215--232, 1958

    1958

  14. [22]

    Flashattention-2: Faster attention with better parallelism and work partitioning

    Dao, T. Flashattention-2: Faster attention with better parallelism and work partitioning. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=mZn2Xyh9Ec

    2024

  15. [23]

    and Gu, A

    Dao, T. and Gu, A. Transformers are SSM s: Generalized models and efficient algorithms through structured state space duality. In Forty-first International Conference on Machine Learning, 2024. URL https://openreview.net/forum?id=ztn8FCR1td

    2024

  16. [24]

    Y., Ermon, S., Rudra, A., and Re, C

    Dao, T., Fu, D. Y., Ermon, S., Rudra, A., and Re, C. Flashattention: Fast and memory-efficient exact attention with IO -awareness. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/f...

    2022

  17. [25]

    CausalLM is not optimal for in-context learning

    Ding, N., Levinboim, T., Wu, J., Goodman, S., and Soricut, R. CausalLM is not optimal for in-context learning. In International Conference on Learning Representations (ICLR), 2024

    2024

  18. [26]

    LIFT : Language-interfaced fine-tuning for non-language machine learning tasks

    Dinh, T., Zeng, Y., Zhang, R., Lin, Z., Gira, M., Rajput, S., yong Sohn, J., Papailiopoulos, D., and Lee, K. LIFT : Language-interfaced fine-tuning for non-language machine learning tasks. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Informat...

    2022

  19. [27]

    The llama 3 herd of models

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  20. [28]

    T., Cherepanova, V., Hegde, C., Hutter, F., Goldblum, M., Cohen, N., and White, C

    Feuer, B., Schirrmeister, R. T., Cherepanova, V., Hegde, C., Hutter, F., Goldblum, M., Cohen, N., and White, C. Tunetables: Context optimization for scalable prior-data fitted networks. arXiv preprint arXiv:2402.11137, 2024

    2024 arXiv

  21. [29]

    Auto-sklearn 2.0: Hands-free automl via meta-learning

    Feurer, M., Eggensperger, K., Falkner, S., Lindauer, M., and Hutter, F. Auto-sklearn 2.0: Hands-free automl via meta-learning. arXiv:2007.04074 [cs.LG], 2021

    2007 arXiv

  22. [30]

    Friedman, J. H. Greedy function approximation: a gradient boosting machine. Annals of statistics, pp.\ 1189--1232, 2001

    2001

  23. [31]

    M., Hauth, A., et al

    Gemini Team, G., Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  24. [32]

    Improving input-label mapping with demonstration replay for in-context learning

    Gong, Z., Liu, J., Wang, Q., Wang, J., Cai, X., Zhao, D., and Yan, R. Improving input-label mapping with demonstration replay for in-context learning. arXiv preprint arXiv:2310.19572, 2023

    2023 arXiv

  25. [33]

    Revisiting deep learning models for tabular data

    Gorishniy, Y., Rubachev, I., Khrulkov, V., and Babenko, A. Revisiting deep learning models for tabular data. Advances in Neural Information Processing Systems, 34: 0 18932--18943, 2021

    2021

  26. [34]

    TabR : Tabular deep learning meets nearest neighbors

    Gorishniy, Y., Rubachev, I., Kartashev, N., Shlenskii, D., Kotelnikov, A., and Babenko, A. TabR : Tabular deep learning meets nearest neighbors. In International Conference on Learning Representations (ICLR), 2024

    2024

  27. [35]

    Why do tree-based models still outperform deep learning on typical tabular data? Advances in neural information processing systems, 35: 0 507--520, 2022

    Grinsztajn, L., Oyallon, E., and Varoquaux, G. Why do tree-based models still outperform deep learning on typical tabular data? Advances in neural information processing systems, 35: 0 507--520, 2022

    2022

  28. [36]

    and Dao, T

    Gu, A. and Dao, T. Mamba: Linear-time sequence modeling with selective state spaces. In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=tEYskw1VY2

    2024

  29. [37]

    K., Dao, T., Rudra, A., and Re, C

    Gu, A., Johnson, I., Goel, K., Saab, K. K., Dao, T., Rudra, A., and Re, C. Combining recurrent, convolutional, and continuous-time models with linear state space layers. In Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Proce...

    2021

  30. [38]

    Efficiently modeling long sequences with structured state spaces

    Gu, A., Goel, K., and Re, C. Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=uYLFoz1vlAC

    2022

  31. [39]

    Deepfm: a factorization-machine based neural network for ctr prediction

    Guo, H., Tang, R., Ye, Y., Li, Z., and He, X. Deepfm: a factorization-machine based neural network for ctr prediction. arXiv preprint arXiv:1703.04247, 2017

    2017 arXiv

  32. [40]

    J., Oktay, D., Lin, Z., Verkuil, R., Tran, V

    Hayes, T., Rao, R., Akin, H., Sofroniew, N. J., Oktay, D., Lin, Z., Verkuil, R., Tran, V. Q., Deaton, J., Wiggert, M., Badkundri, R., Shafkat, I., Gong, J., Derry, A., Molina, R. S., Thomas, N., Khan, Y. A., Mishra, C., Kim, C., Bartie, L. J., Nemeth, M., Hsu, P. D., Sercu, T....

    2024 doi

  33. [41]

    Deep residual learning for image recognition

    He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 770--778, 2016

    2016

  34. [42]

    Tab PFN : A transformer that solves small tabular classification problems in a second

    Hollmann, N., M \"u ller, S., Eggensperger, K., and Hutter, F. Tab PFN : A transformer that solves small tabular classification problems in a second. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=cp5PvcI6w8_

    2023

  35. [43]

    u ller, S., Purucker, L., Krishnakumar, A., K \

    Hollmann, N., M \"u ller, S., Purucker, L., Krishnakumar, A., K \"o rfer, M., Hoo, S. B., Schirrmeister, R. T., and Hutter, F. Accurate predictions on small data with a tabular foundation model. Nature, 637 0 (8045): 0 319--326, 2025

    2025

  36. [44]

    Tabtransformer: Tabular data modeling using contextual embeddings

    Huang, X., Khetan, A., Cvitkovic, M., and Karnin, Z. Tabtransformer: Tabular data modeling using contextual embeddings. arXiv preprint arXiv:2012.06678, 2020

    2012 arXiv

  37. [45]

    E., Pollard, T

    Johnson, A. E., Pollard, T. J., Shen, L., Lehman, L.-w. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., and Mark, R. G. Mimic-iii, a freely accessible critical care database. Scientific data, 3 0 (1): 0 1--9, 2016

    2016

  38. [46]

    Transformers are rnns: Fast autoregressive transformers with linear attention

    Katharopoulos, A., Vyas, A., Pappas, N., and Fleuret, F. Transformers are rnns: Fast autoregressive transformers with linear attention. In International conference on machine learning, pp.\ 5156--5165. PMLR, 2020

    2020

  39. [47]

    Lightgbm: A highly efficient gradient boosting decision tree

    Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q., and Liu, T.-Y. Lightgbm: A highly efficient gradient boosting decision tree. Advances in neural information processing systems, 30, 2017

    2017

  40. [48]

    Learning multiple layers of features from tiny images

    Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009

    2009

  41. [49]

    MNIST handwritten digit database

    LeCun, Y., Cortes, C., and Burges, C. MNIST handwritten digit database. 2, 2010

    2010

  42. [50]

    Classification and regression by randomforest

    Liaw, A., Wiener, M., et al. Classification and regression by randomforest. R news, 2 0 (3): 0 18--22, 2002

    2002

  43. [51]

    In-context data distillation with TabPFN

    Ma, J., Thomas, V., Yu, G., and Caterini, A. In-context data distillation with TabPFN . arXiv preprint arXiv:2402.06971, 2024

    2024 arXiv

  44. [52]

    and Ratajczak, W

    Ma \'c kiewicz, A. and Ratajczak, W. Principal components analysis (pca). Computers & Geosciences, 19 0 (3): 0 303--342, 1993

    1993

  45. [53]

    C., Khandagale, S., Valverde, J., C, V

    McElfresh, D. C., Khandagale, S., Valverde, J., C, V. P., Ramakrishnan, G., Goldblum, M., and White, C. When do neural nets outperform boosted trees on tabular data? In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL ...

    2023

  46. [54]

    L., Gu, A., Fernando, A., Gulcehre, C., Pascanu, R., and De, S

    Orvieto, A., Smith, S. L., Gu, A., Fernando, A., Gulcehre, C., Pascanu, R., and De, S. Resurrecting recurrent neural networks for long sequences. In International Conference on Machine Learning, pp.\ 26670--26698. PMLR, 2023

    2023

  47. [55]

    G., Albalak, A., Arcadinho, S., Biderman, S., Cao, H., Cheng, X., Chung, M

    Peng, B., Alcaide, E., Anthony, Q. G., Albalak, A., Arcadinho, S., Biderman, S., Cao, H., Cheng, X., Chung, M. N., Derczynski, L., et al. Rwkv: Reinventing rnns for the transformer era. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023

    2023

  48. [56]

    Neural oblivious decision ensembles for deep learning on tabular data

    Popov, S., Morozov, S., and Babenko, A. Neural oblivious decision ensembles for deep learning on tabular data. arXiv preprint arXiv:1909.06312, 2019

    1909 arXiv

  49. [57]

    V., and Gulin, A

    Prokhorenkova, L., Gusev, G., Vorobev, A., Dorogush, A. V., and Gulin, A. Catboost: unbiased boosting with categorical features. Advances in neural information processing systems, 31, 2018

    2018

  50. [58]

    The devil in linear transformer

    Qin, Z., Han, X., Sun, W., Li, D., Kong, L., Barnes, N., and Zhong, Y. The devil in linear transformer. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp.\ 7025--7041, 2022

    2022

  51. [59]

    Quinlan, J. R. Induction of decision trees. Machine learning, 1: 0 81--106, 1986

    1986

  52. [60]

    L., Ma, J., and Fergus, R

    Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., and Fergus, R. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. PNAS, 2019. doi:10.1101/622803. URL https://www....

    2019 doi

  53. [61]

    E., Hinton, G

    Rumelhart, D. E., Hinton, G. E., and Williams, R. J. Learning representations by back-propagating errors. nature, 323 0 (6088): 0 533--536, 1986

    1986

  54. [62]

    Flashattention-3: Fast and accurate attention with asynchrony and low-precision

    Shah, J., Bikshandi, G., Zhang, Y., Thakkar, V., Ramani, P., and Dao, T. Flashattention-3: Fast and accurate attention with asynchrony and low-precision. arXiv preprint arXiv:2407.08608, 2024

    2024 arXiv

  55. [63]

    B., and Goldstein, T

    Somepalli, G., Goldblum, M., Schwarzschild, A., Bruss, C. B., and Goldstein, T. Saint: Improved neural networks for tabular data via row attention and contrastive pre-training. arXiv preprint arXiv:2106.01342, 2021

    2021 arXiv

  56. [64]

    Retentive network: A successor to transformer for large language models

    Sun, Y., Dong, L., Huang, S., Ma, S., Xia, Y., Xue, J., Wang, J., and Wei, F. Retentive network: A successor to transformer for large language models. arXiv preprint arXiv:2307.08621, 2023

    2023 arXiv

  57. [65]

    F., Kumar, M., Weisser, C., Reuter, A., S \"a fken, B., and Samiee, S

    Thielmann, A. F., Kumar, M., Weisser, C., Reuter, A., S \"a fken, B., and Samiee, S. Mambular: A sequential model for tabular deep learning. arXiv preprint arXiv:2408.06291, 2024

    2024 arXiv

  58. [66]

    Thomas, V., Ma, J., Hosseinzadeh, R., Golestaneh, K., Yu, G., Volkovs, M., and Caterini, A. L. Retrieval & fine-tuning for in-context tabular models. In NeurIPS, 2024

    2024

  59. [67]

    N., Bischl, B., and Torgo, L

    Vanschoren, J., van Rijn, J. N., Bischl, B., and Torgo, L. OpenML : Networked science in machine learning. SIGKDD Explorations, 15 0 (2): 0 49--60, 2013

    2013

  60. [68]

    N., Kaiser, ., and Polosukhin, I

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, ., and Polosukhin, I. Attention is all you need. Advances in neural information processing systems, 30, 2017

    2017

  61. [69]

    Vempala, S. S. The random projection method, volume 65. American Mathematical Soc., 2005

    2005

  62. [70]

    Eegformer: A transformer--based brain activity classification method using eeg signal

    Wan, Z., Li, M., Liu, S., Huang, J., Tan, H., and Duan, W. Eegformer: A transformer--based brain activity classification method using eeg signal. Frontiers in Neuroscience, 17: 0 1148855, 2023

    2023

  63. [71]

    Fashion-MNIST : a novel image dataset for benchmarking machine learning algorithms

    Xiao, H., Rasul, K., and Vollgraf, R. Fashion-MNIST : a novel image dataset for benchmarking machine learning algorithms. 2017

    2017

  64. [72]

    Y.-C., Li, W., Gilani, A., Goan, H.-S., and Liu, H

    Xu, C., Huang, Y.-C., Hu, J. Y.-C., Li, W., Gilani, A., Goan, H.-S., and Liu, H. BiSHop : Bi-directional cellular learning for tabular data with generalized sparse modern hopfield model. In International Conference on Machine Learning (ICML), 2024

    2024

  65. [73]

    Q., Cirik, F

    Xu, D. Q., Cirik, F. O., Asadi, R., Sun, Y., and Wang, W. Mixture of in-context prompters for tabular PFN s. In International Conference on Learning Representations (ICLR), 2025

    2025

  66. [74]

    Feature selection using stochastic gates

    Yamada, Y., Lindenbaum, O., Negahban, S., and Kluger, Y. Feature selection using stochastic gates. In International conference on machine learning, pp.\ 10648--10659. PMLR, 2020

    2020

  67. [75]

    Gated linear attention transformers with hardware-efficient training

    Yang, S., Wang, B., Shen, Y., Panda, R., and Kim, Y. Gated linear attention transformers with hardware-efficient training. In Forty-first International Conference on Machine Learning, 2024

    2024

  68. [76]

    Vime: Extending the success of self-and semi-supervised learning to tabular domain

    Yoon, J., Zhang, Y., Jordon, J., and Van der Schaar, M. Vime: Extending the success of self-and semi-supervised learning to tabular domain. Advances in Neural Information Processing Systems, 33: 0 11033--11043, 2020

    2020

  69. [77]

    You, H., Fu, Y., Wang, Z., Yazdanbakhsh, A., and Lin, Y. C. When linear attention meets autoregressive decoding: Towards more effective and efficient linearized large language models. In Forty-first International Conference on Machine Learning, 2024

    2024

  70. [78]

    The hedgehog & the porcupine: Expressive linear attentions with softmax mimicry

    Zhang, M., Bhatia, K., Kumbong, H., and Re, C. The hedgehog & the porcupine: Expressive linear attentions with softmax mimicry. In The Twelfth International Conference on Learning Representations, 2024

    2024

  71. [79]

    Deep learning based recommender system: A survey and new perspectives

    Zhang, S., Yao, L., Sun, A., and Tay, Y. Deep learning based recommender system: A survey and new perspectives. ACM computing surveys (CSUR), 52 0 (1): 0 1--38, 2019

    2019

  72. [80]

    XTab : Cross-table pretraining for tabular transformers

    Zhu, B., Shi, X., Erickson, N., Li, M., Karypis, G., and Shoaran, M. XTab : Cross-table pretraining for tabular transformers. In International Conference on Machine Learning (ICML), 2023

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.