pith. sign in

arxiv: 2606.31126 · v1 · pith:AC5FP4BXnew · submitted 2026-06-30 · 💻 cs.LG · q-bio.QM· stat.ML

Can Tabular In-Context Learners Generalize to Biomolecular Property Prediction?

Davy Guan , Lu Zhang , Asiri Wijesinghe , Allen Zhu , He Zhao , Helen Power , F. Hafna Ahmed , Andrew Warden
show 2 more authors
Cheng Soon Ong Daniel M. Steinberg
This is my paper

Pith reviewed 2026-07-01 06:59 UTC · model grok-4.3

classification 💻 cs.LG q-bio.QMstat.ML
keywords tabular in-context learningbiomolecular property predictionfew-shot learningprotein fitness predictionmolecular descriptorsfoundation modelsin-context learning
0
0 comments X

The pith

Tabular in-context learners transfer to biomolecular property prediction when paired with fixed representations.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper examines whether tabular foundation models pretrained on synthetic causal graphs can serve as data-efficient predictors for protein and small-molecule properties in the few-shot regime. It treats each model as a predictor-representation pair and tests this across protein fitness regression using an ESMC encoder on ProteinGym and an esterase dataset, plus small-molecule classification with ECFP or RDKit descriptors on TDC ADMET, MoleculeNet, FS-Mol, and DrugOOD. The central result is that these models remain competitive, especially on proteins, even though their pretraining prior has no direct link to biological sequence or graph generation. This matters because it shows that the difficulty in biomolecular prediction has largely shifted from representation learning to the choice of predictor once strong fixed encoders are available.

Core claim

Tabular in-context learners such as TabPFN3 and TabICL achieve competitive performance on biomolecular property prediction tasks despite their synthetic causal pretraining, with strong results on protein fitness regression over a fixed ESMC representation and results on small-molecule tasks that depend primarily on the choice of molecular descriptor.

What carries the argument

The predictor-representation pair, in which a tabular in-context learner acts as the predictor over a fixed pretrained encoder output.

If this is right

  • Over a fixed ESMC representation, tabular in-context learning is consistently competitive for protein fitness regression on ProteinGym and a diverse esterase dataset.
  • For small-molecule classification with ECFP or RDKit descriptors, no single predictor-representation pairing dominates across the tested datasets.
  • Representation choice becomes the primary determinant of performance when the predictor's prior is indifferent to molecular structure.
  • Tabular foundation models function as strong performers on biomolecular prediction tasks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The causal inductive bias learned from random graphs may capture transferable statistical regularities that appear in both protein sequences and molecular graphs.
  • Future experiments could test whether jointly optimizing the representation and the tabular predictor yields further gains beyond the fixed-encoder setting.
  • The result suggests that domain-specific predictor pretraining may not be required once a sufficiently rich fixed representation is supplied.

Load-bearing premise

The selected benchmarks and fixed representations are representative enough to show that the tabular causal prior generalizes to biomolecular processes.

What would settle it

A new biomolecular benchmark where tabular in-context learning with the same fixed representations falls substantially below strong baselines on both protein and molecular tasks.

Figures

Figures reproduced from arXiv: 2606.31126 by Allen Zhu, Andrew Warden, Asiri Wijesinghe, Cheng Soon Ong, Daniel M. Steinberg, Davy Guan, F. Hafna Ahmed, Helen Power, He Zhao, Lu Zhang.

Figure 1
Figure 1. Figure 1: ProteinGym random 5-fold method comparison across 217 assays. Bars report mean Spearman [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Assay-level paired difference in validation performance between TabPFN3 and TabICL on Prote [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: ProteinGym performance across official random, modulo, and contiguous holdout schemes. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: ProteinGym few-shot performance as support-set size increases. Curves show best-so-far task [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: ProteinGym assay-size diagnostic. Mean Spearman is plotted against assay variant count for [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Few-shot PpEST performance measured by Spearman correlation. Bold curves are monotone best [PITH_FULL_IMAGE:figures/full_fig_p010_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Few-shot PpEST performance measured by MSE. Bold curves are monotone best-so-far envelopes [PITH_FULL_IMAGE:figures/full_fig_p011_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Few-shot small-molecule learning curves. The x-axis is the number of labeled support molecules. The [PITH_FULL_IMAGE:figures/full_fig_p013_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Full-train small-molecule benchmark summary. Bars show mean ROC-AUC across tasks; DrugOOD [PITH_FULL_IMAGE:figures/full_fig_p013_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: DrugOOD full-train ID/OOD generalization gap. Bars report ID-test ROC-AUC minus OOD-test ROC [PITH_FULL_IMAGE:figures/full_fig_p014_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Paired few-shot comparison of TabPFN3 and TabICL on molecules. Positive values indicate higher [PITH_FULL_IMAGE:figures/full_fig_p014_11.png] view at source ↗
read the original abstract

Predicting biomolecular properties from limited labeled data is a central bottleneck in protein engineering and small-molecule design. As strong pretrained encoders now supply rich fixed-length representations, the difficulty has shifted from representation learning to building a data-efficient predictor for the few-shot regime. Tabular foundation models such as TabPFN3 and TabICL are unlikely candidates for this role: they are in-context learners pretrained on synthetic tables drawn from random causal graphs, a generative prior with no obvious correspondence to the processes that produce protein sequences or molecular graphs. That this tabular, causal inductive bias should transfer to biomolecular data at all is unintuitive, yet we find it does. Treating each method as a predictor-representation pair, we evaluate across two domains. Over a fixed ESMC representation, tabular in-context learning is consistently competitive for protein fitness regression on ProteinGym and a diverse esterase dataset. For small-molecule classification with ECFP/RDKit descriptors, no single pairing dominates across TDC ADMET, MoleculeNet, FS-Mol, and DrugOOD; representation choice becomes a primary determinant, as expected when the predictor's own prior is indifferent to molecular structure. We conclude that tabular foundation models are strong performers on biomolecular prediction tasks, but that their performance depends strongly on the sequence or molecular representation used.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper evaluates whether tabular in-context learners (TabPFN3, TabICL) pretrained on synthetic tables from random causal graphs can generalize to biomolecular property prediction. Treating each method as a predictor-representation pair, it reports that over a fixed ESMC representation these models are consistently competitive for protein fitness regression on ProteinGym and a diverse esterase dataset; for small-molecule classification with ECFP/RDKit descriptors, no single pairing dominates across TDC ADMET, MoleculeNet, FS-Mol, and DrugOOD, with representation choice emerging as the primary determinant. The central conclusion is that tabular foundation models are strong performers on these tasks but that performance depends strongly on the sequence or molecular representation used.

Significance. If the empirical findings hold, the work provides evidence that a causal inductive bias learned from synthetic tabular data can transfer to biomolecular domains despite the lack of obvious correspondence to protein sequences or molecular graphs. This is noteworthy for few-shot regimes in protein engineering and small-molecule design, where strong pretrained encoders already supply fixed representations. The explicit demonstration that representation choice dominates for molecular tasks (where the predictor prior is structure-indifferent) is a useful scoping result. The study ships an empirical benchmarking protocol on public datasets with no hidden fitted parameters or circular derivations.

major comments (2)
  1. [§4] §4 (protein results): the claim of 'consistent competitiveness' on ProteinGym and the esterase dataset is load-bearing for the protein-domain conclusion, yet the abstract and summary provide no quantitative metrics, baseline comparisons, or statistical tests; without these, the strength of evidence for transfer of the tabular causal prior cannot be assessed.
  2. [Methods / §3] Methods and benchmark selection (implicit in §3 and §5): the weakest assumption is that ProteinGym, the esterase set, TDC ADMET, MoleculeNet, FS-Mol, and DrugOOD with the chosen fixed representations (ESMC, ECFP/RDKit) are representative enough to support statements about generalization to biomolecular processes; a sensitivity analysis or explicit discussion of dataset selection criteria would be needed to make the scoping claim robust.
minor comments (2)
  1. [Results tables] Ensure all result tables include error bars, number of seeds, and exact baseline definitions so that 'competitive' can be interpreted quantitatively.
  2. [§2] Clarify the precise versions of TabPFN3 and TabICL and any hyper-parameter choices that were held fixed across domains.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback and the recommendation for minor revision. We address the two major comments point by point below.

read point-by-point responses

  1. Referee: [§4] §4 (protein results): the claim of 'consistent competitiveness' on ProteinGym and the esterase dataset is load-bearing for the protein-domain conclusion, yet the abstract and summary provide no quantitative metrics, baseline comparisons, or statistical tests; without these, the strength of evidence for transfer of the tabular causal prior cannot be assessed.

    Authors: We agree that the abstract would benefit from explicit quantitative support for the 'consistent competitiveness' claim. Section 4 of the manuscript already contains the full results, including average ranks across ProteinGym tasks, comparisons against standard baselines (e.g., random forests, XGBoost), and statistical tests. We will revise the abstract to include a concise statement of these key metrics (e.g., average rank of TabPFN3+ESMC) to make the evidence for transfer immediately assessable. revision: yes

  2. Referee: [Methods / §3] Methods and benchmark selection (implicit in §3 and §5): the weakest assumption is that ProteinGym, the esterase set, TDC ADMET, MoleculeNet, FS-Mol, and DrugOOD with the chosen fixed representations (ESMC, ECFP/RDKit) are representative enough to support statements about generalization to biomolecular processes; a sensitivity analysis or explicit discussion of dataset selection criteria would be needed to make the scoping claim robust.

    Authors: We accept that an explicit justification of benchmark selection strengthens the scoping claims. These datasets were selected because they are widely adopted community standards that together span protein fitness regression and small-molecule classification/regression across multiple domains and distribution shifts. We will add a dedicated paragraph in §3 (and a brief note in §5) that states the selection criteria and acknowledges the absence of a full sensitivity analysis across every possible biomolecular dataset as a limitation of the current study. revision: yes

Circularity Check

0 steps flagged

Empirical benchmarking study; no derivation chain present

full rationale

The paper is an empirical benchmarking study that evaluates tabular in-context learners (TabPFN3, TabICL) paired with fixed representations (ESMC, ECFP/RDKit) on external public datasets (ProteinGym, TDC ADMET, MoleculeNet, FS-Mol, DrugOOD). No equations, fitted parameters, or predictions are defined within the paper; all performance claims are direct experimental outcomes on held-out benchmarks. No self-citations are load-bearing for any central premise, and the abstract explicitly scopes conclusions to the evaluated representation-predictor pairs without asserting representation-free generalization. The derivation chain is therefore empty and self-contained against external data.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

No free parameters, invented entities, or ad-hoc axioms are introduced in the abstract; the work relies on standard external benchmarks and pretrained representations as domain assumptions.

axioms (1)
  • domain assumption The selected benchmarks and representations are representative for testing generalization to biomolecular property prediction.
    Invoked to support the conclusion that tabular models are strong performers.

pith-pipeline@v0.9.1-grok · 5797 in / 1232 out tokens · 32842 ms · 2026-07-01T06:59:25.884453+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

29 extracted references · 29 canonical work pages

  1. [1]

    International Conference on Learning Representations , year=

    Hollmann, Noah and M. International Conference on Learning Representations , year=

    work page

  2. [2]

    Nature , volume=

    Accurate predictions on small data with a tabular foundation model , author=. Nature , volume=. 2025 , doi=

    work page 2025

  3. [3]

    Proceedings of the 42nd International Conference on Machine Learning , pages=

    Qu, Jingang and Holzm. Proceedings of the 42nd International Conference on Machine Learning , pages=. 2025 , volume=

    work page 2025

  4. [4]

    2026 , note=

    TabICLv2: A better, faster, scalable, and open tabular foundation model , author=. 2026 , note=

    work page 2026

  5. [5]

    2025 , volume=

    Liu, Siyang and Ye, Han-Jia , booktitle=. 2025 , volume=

    work page 2025

  6. [6]

    2025 , note=

    Spinaci, Marco and Polewczyk, Marek and Schambach, Maximilian and Thelin, Sam , booktitle=. 2025 , note=

    work page 2025

  7. [7]

    2025 , note=

    Arbel, Michael and Salinas, David and Hutter, Frank , booktitle=. 2025 , note=

    work page 2025

  8. [8]

    Advances in Neural Information Processing Systems , volume=

    Proteingym: Large-scale benchmarks for protein fitness prediction and design , author=. Advances in Neural Information Processing Systems , volume=

    work page

  9. [9]

    Science , volume=

    Simulating 500 million years of evolution with a language model , author=. Science , volume=. 2025 , publisher=

    work page 2025

  10. [10]

    2026 , doi=

    Wu, Dan-Ni and Jen, Joey and Fajiculay, Erickson and Hsu, Min-Fen and Chang, Ming-Chu and Yeh, Jen-Chen and Sargsyan, Karen and Kupcinskas, Juozas and Skieceviciene, Jurgita and Steponaitiene, Ruta and Morkunas, Egidijus and Gedgaudiene, Greta and Hsu, Chao-Ping and Chang, Yu-Ting and Hu, Chun-Mei and others , journal=. 2026 , doi=

    work page 2026

  11. [11]

    and Betta, Emanuela and Khomenko, Iuliia and Pedrotti, Michele and Romano, Andrea and Biasioli, Franco and others , journal=

    Granitto, Pablo M. and Betta, Emanuela and Khomenko, Iuliia and Pedrotti, Michele and Romano, Andrea and Biasioli, Franco and others , journal=. On the use of. 2026 , doi=

    work page 2026

  12. [12]

    npj Computational Materials , year=

    In context learning foundation models for materials property prediction with small datasets , author=. npj Computational Materials , year=

    work page

  13. [13]

    Journal of Chemical Information and Modeling , volume =

    Analyzing Learned Molecular Representations for Property Prediction , author =. Journal of Chemical Information and Modeling , volume =. 2019 , publisher =

    work page 2019

  14. [14]

    and Maziarz, Krzysztof and Misztela, Hubert and Lanini, Jessica and Segler, Marwin and Schneider, Nadine and Brockschmidt, Marc , booktitle =

    Stanley, Megan and Bronskill, John F. and Maziarz, Krzysztof and Misztela, Hubert and Lanini, Jessica and Segler, Marwin and Schneider, Nadine and Brockschmidt, Marc , booktitle =

    work page

  15. [15]

    Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks , volume =

    Therapeutics Data Commons: Machine Learning Datasets and Tasks for Drug Discovery and Development , author =. Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks , volume =

    work page

  16. [16]

    and Gomes, Joseph and Geniesse, Caleb and Pappu, Aneesh S

    Wu, Zhenqin and Ramsundar, Bharath and Feinberg, Evan N. and Gomes, Joseph and Geniesse, Caleb and Pappu, Aneesh S. and Leswing, Karl and Pande, Vijay , journal =. 2018 , publisher =

    work page 2018

  17. [17]

    2023 , doi =

    Ji, Yuanfeng and Zhang, Lu and Wu, Jiaxiang and Wu, Bingzhe and Li, Lanqing and Huang, Long-Kai and Xu, Tingyang and Rong, Yu and Ren, Jie and Xue, Ding and Lai, Houtim and Xu, Shaoyong and Feng, Jing and Liu, Wei and Luo, Ping and Zhou, Shuigeng and Huang, Junzhou and Zhao, Peilin and Bian, Yatao , booktitle =. 2023 , doi =

    work page 2023

  18. [18]

    Journal of Chemical Information and Modeling , volume =

    Extended-Connectivity Fingerprints , author =. Journal of Chemical Information and Modeling , volume =. 2010 , publisher =

    work page 2010

  19. [19]

    2025 , note =

    Descriptor-based Foundation Models for Molecular Property Prediction , author =. 2025 , note =

    work page 2025

  20. [20]

    Kolberg, Christopher and Eggensperger, Katharina and Pfeifer, Nico , year =

    work page

  21. [21]

    2026 , note =

    Tabular foundation models for in-context prediction of molecular properties , author =. 2026 , note =

    work page 2026

  22. [22]

    Science , volume =

    Evolutionary-Scale Prediction of Atomic-Level Protein Structure with a Language Model , author =. Science , volume =. 2023 , publisher =

    work page 2023

  23. [23]

    International Conference on Learning Representations , year =

    Transformers Can Do Bayesian Inference , author=. International Conference on Learning Representations , year =

    work page

  24. [24]

    and Lee, Jin Sub and Bruguera, Elise S

    Candido, Salvatore and Hayes, Thomas and Derry, Alexander and Rao, Roshan and Lin, Zeming and Verkuil, Robert and Wu, Bryan Z. and Lee, Jin Sub and Bruguera, Elise S. and Keval, Jehan A. and Kopylov, Mykhailo and Pak, John E. and Wu, Wesley and Thomas, Neil and Mataraso, Samson and Hsu, Alvin and Trotman-Grant, Ashton C. and Fatras, Kilian and dos Santos ...

    work page 2026

  25. [25]

    Ahmed, F. Hafna and Bender, Asher and Wijesinghe, Asiri and Zhu, Allen and Zhang, Lu and Gebbie, Leigh and Marsh, Adrian and Ishitate, Chie and Holdsworth, William and Jones, Candice and Warden, Andrew C. and Power, Helen and Ong, Cheng Soon and Steinberg, Daniel and Speight, Robert E. , title =. 2026 , doi =

    work page 2026

  26. [26]

    2026 , eprint=

    TabPFN-3: Technical Report , author=. 2026 , eprint=

    work page 2026

  27. [27]

    and Varoquaux, G

    Pedregosa, F. and Varoquaux, G. and Gramfort, A. and Michel, V. and Thirion, B. and Grisel, O. and Blondel, M. and Prettenhofer, P. and Weiss, R. and Dubourg, V. and Vanderplas, J. and Passos, A. and Cournapeau, D. and Brucher, M. and Perrot, M. and Duchesnay, E. , journal=. Scikit-learn: Machine Learning in

    work page

  28. [28]

    A scalable tree boosting system

    Chen, Tianqi and Guestrin, Carlos , title =. 2016 , isbn =. doi:10.1145/2939672.2939785 , booktitle =

    work page doi:10.1145/2939672.2939785 2016

  29. [29]

    IEEE Transactions on Pattern Analysis and Machine Intelligence , year =

    Elnaggar, Ahmed and Heinzinger, Michael and Dallago, Christian and Rehawi, Ghalia and Wang, Yu and Jones, Llion and Gibbs, Tom and Feher, Tamas and Angerer, Christoph and Steinegger, Martin and Bhowmik, Debsindhu and Rost, Burkhard , title=. IEEE Transactions on Pattern Analysis and Machine Intelligence , year =

    work page