Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

Position: The Future of Bayesian Prediction Is Prior-Fitted

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Bayesian prediction may become a neural forward pass: prior-fitted networks trained on synthetic data replace per-dataset MCMC and variational inference with a single amortized forward pass.

desk verdict A clear, honest position paper from the PFN inventors, but the dominance claim rests on an assumption about limitations that the paper's own counting experiment undercuts. read the letter →

arxiv 2505.23947 v1 pith:YJTEI7L7 submitted 2025-05-29 cs.LG cs.AI

Samuel Müller , Arik Reuter , Noah Hollmann , David Rügamer , Frank Hutter This is my paper
classification cs.LGcs.AI MSC 68T0762F15
keywords Prior-DataFittedNetworksBayesianpredictionamortizedinferencein-contextlearningsyntheticdataposteriorpredictivedistributiontabularoptimization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This position paper argues that Bayesian prediction will increasingly be performed by Prior-Data Fitted Networks (PFNs): neural networks pre-trained on millions of datasets sampled from a user-defined prior, which then approximate the posterior predictive distribution in a single forward pass. The rationale is that pre-training compute is scaling faster than the supply of real-world data, so the efficient division of labor is to spend compute once on a prior and amortize it across every future dataset drawn from that prior. If the authors are right, the standard toolkit of Bayesian prediction—MCMC sampling, variational inference, and Gaussian-process marginalization—would be superseded for most supervised-prediction tasks by a standardized neural-network forward pass. The paper also maps the open problems—interpretability, scaling to large datasets, inference speed, architectural counting limits—and argues that most are engineering obstacles rather than fundamental barriers.

What carries the argument

The load-bearing object is the prior-fitting objective: sampling datasets $D \sim p(D)$ from a prior over datasets and optimizing the network parameters $\theta$ to minimize $\ell = \mathbb{E}_{D \sim p(D)}[-\log q_\theta(y_{\text{test}} | x_{\text{test}}, D_{\text{train}})]$, which equals the expected KL divergence between the true posterior predictive $p(\cdot | x, D)$ and the network approximation $q_\theta(\cdot | x, D)$ up to a constant. The prior is supplied as a sampling program—for example, weights of a Bayesian neural network, Gaussian-process hyperparameters, or a structural causal model as in TabPFN—which means the user declares the inductive bias without needing tractable densities. The forward pass with masked attention, where training points attend to each other and the test point attends only to training points, then performs amortized Bayesian inference by in-context learning.

What would settle it

Train a PFN on a prior with a known posterior predictive (e.g., a Gaussian process or linear regression with a conjugate prior), then measure the KL divergence between the PFN's predictive distribution and the exact posterior predictive as the in-context dataset size grows within the prior's support; if increasing model scale and pre-training compute by fixed factors leaves a non-shrinking KL gap, or if the martingale violation observed in the paper's Figure 2 grows with scale instead of stabilizing, the dominance claim is falsified. A sharper version: find any prior from which the PFN can sample such that no scaling of the current architecture class achieves near-Bayesian predictions on all datasets in the prior's support.

Watch

Extended reading notes

Core claim

On the authors' own terms, the central claim is that PFNs will dominate most applications of Bayesian prediction and create new ones. A PFN is trained by minimizing the cross-entropy of held-out examples across datasets sampled from a prior, which is exactly the KL divergence from the network's predictive distribution to the true posterior predictive distribution; the prior itself is defined declaratively by a synthetic data-generating process rather than by a tractable likelihood. Because the network is conditioned only on the observed dataset and query at inference time, prediction becomes in-context learning: no latent posterior is instantiated, no per-dataset sampling is run, and the same amortized model serves any dataset from the prior. The authors point to TabPFN as the poster child, reporting that it outperforms tuned classical baselines on small tabular datasets, and to applications in Bayesian optimization, learning-curve extrapolation, time series, and biology as evidence that the amortization pattern transfers across domains.

Load-bearing premise

The claim stands on the assumption that the current PFN weaknesses—small-support effectiveness only, slow inference, and failures like not counting repeated identical examples—are efficiency and engineering problems that scale away with compute and architectural tweaks, rather than fundamental limits of amortized in-context prediction.

Editorial extensions

If this is right

  • Bayesian prediction in data-scarce domains becomes a forward pass on a pre-trained network, removing the per-dataset cost of MCMC or variational inference.
  • Priors that are only defined by a sampling process—complex graphs, architectures, simulators—become usable for Bayesian prediction, since no density evaluation is needed.
  • The same pre-trained PFN can serve many datasets from one prior, shifting compute from inference time to pre-training time, which suits the trend of falling pre-training cost and stagnant real-world data growth.
  • Application areas such as tabular learning, Bayesian optimization, learning-curve extrapolation, and time-series forecasting already exhibit this pattern, and more domains should follow as the efficiency gaps are closed.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the amortization thesis holds, the economic structure of Bayesian practice flips: the scarce resource becomes prior engineering (writing good sampling programs) rather than inference engineering (converging samplers or tuning variational families).
  • A testable extension is that PFN performance on a given prior should track a scaling law in pre-training compute and training datasets, with deviations from the true posterior predictive shrinking monotonically—something the paper does not yet demonstrate.
  • The counting failure on identical examples (Section 6.5) suggests a boundary condition: architectures without positional or counting mechanisms cannot represent sufficient statistics that depend on multiplicities, so the dominance claim implicitly requires architectural progress of exactly the kind the paper lists as open.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This position paper argues that Prior-Data Fitted Networks (PFNs), trained on synthetic datasets sampled from a prior, will dominate most applications of Bayesian prediction and create new applications. It introduces the PFN formalism, reviews the prior-fitting objective, gives examples of priors, compares PFNs with MCMC/VI/GPs and amortized methods, discusses extensions (extra inputs, in-context interpreters, RL for BO, latent prediction), presents two small experiments (martingale property and duplicate counting), and proposes research directions for scaling, fast inference, and architectural improvements. It closes with alternative views and responses.

Significance. If the position is correct, Bayesian prediction would become a one-forward-pass operation, amortizing compute across datasets and enabling priors that are specified only through a sampler. The paper is strongest as a roadmap: it names concrete open problems (counting, heterogeneous features, interpretability, scaling) and points to published successes such as TabPFN v2, which has independent peer review. The paper is appropriately cautious in some places, but the strength of the dominance claim is not matched by the empirical evidence: the two new experiments are small-scale, and one of them (duplicate counting) exposes a representational limitation rather than an efficiency issue. The position is plausible and worth publishing, but the argument needs to be tightened before the strong claims can be endorsed.

major comments (3)
  1. [§6.5, Figure 3] The duplicate-counting experiment is a load-bearing counterexample to the claim in Section 3 (limitation 3) that current PFN shortcomings are mostly efficiency- and compute-related. On the coin-flip prior, the encoder-only transformer without positional embeddings cannot represent the count of identical observations, which is the sufficient statistic for the posterior; Figure 3 shows the predicted head probability remaining flat as the number of observed heads increases. Counting duplicated examples is a representational requirement, not a speed or compute issue, and duplicate feature vectors occur in real tabular data. The proposed remedies (zero attention, inference-time noise features) are mentioned only as suggestions and are not demonstrated to restore the correct posterior. The paper should either provide evidence that these architectural fixes recover Bayesian behavior on this prior, or substantially weaken the 'will dominate' claim.
  2. [§5.1, Figure 2] The martingale experiment shows a systematic decrease in predictive standard deviation over the first few rollout steps, which the authors themselves state 'should not be the case for a Bayesian predictor.' Since the central premise of the paper is that PFNs approximate the posterior predictive distribution, this deviation is directly relevant; calling it small-scale does not remove the need to explain it. The paper should report a more systematic evaluation (multiple priors, longer rollouts, statistical significance) and either reconcile the deviation with the Bayesian claim or qualify the scope of the claim in Section 1 and the Position box.
  3. [§6.1, §7 View 2] The claim that PFNs will dominate most Bayesian prediction applications assumes that scaling limitations are engineering challenges. Section 6.1 offers architectural ideas (Perceiver, linear attention, state-space models) but no results showing that these preserve the prior-fitting guarantee at larger sample sizes. Since Section 2.1's approximation guarantee is stated for a fixed architecture trained on a fixed prior, it is not automatic that changing the architecture preserves posterior predictive approximation; this transfer should be demonstrated or explicitly discussed as an open assumption. Without this, the response to View 2 remains speculative, so the dominance claim is not fully supported.
minor comments (5)
  1. [§2.2] The sentence on the BNN prior contains a grammatical error ('it's weight') and the notation 'N(0, σ2)' is not tied to the σ introduced in the text.
  2. [§6.1] 'knowledege' is a typo; it should read 'knowledge'.
  3. [§6.5] 'add zero attnin PyTorch' has a typo; it should read 'add zero attention in PyTorch'.
  4. [Appendix B, Figure 4 caption] The caption describes training on 'sines and sloped lines' and 'slightly sloped sines', but the surrounding text and the rest of the appendix concern Martingale rollouts; the caption appears to be from a different experiment and should be corrected.
  5. [§3] The bullet 'No Possibility for Data Leakage' is too absolute; leakage can occur through the choice of prior, preprocessing, or evaluation protocol, even if pre-training data itself is synthetic.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper's PFN-to-PPD derivation is a standard cross-entropy argument, its empirical support is externally published, and its own negative experiments are candid.

full rationale

This is a position paper whose central claim is an argument about the future trajectory of PFNs, not a derivation whose conclusion is smuggled in through its premises. The one technical derivation, in Section 2.1, shows that minimizing the held-out cross-entropy loss is equivalent to minimizing the KL divergence to the true posterior predictive distribution; this is a standard identity and is stated explicitly in the paper with the relevant expectation. It does not depend on defining a PFN in terms of the claim being made, and it is not a fitted parameter renamed as a prediction. The empirical backbone (TabPFN, PFNs4BO, learning-curve extrapolation) is drawn from prior work by the authors and others, but it is externally published, including TabPFN v2 in Nature, and the paper does not invoke a uniqueness theorem from the authors' own work to foreclose alternatives. The paper also candidly reports limitations that cut against its own thesis: the Martingale-property experiment in Section 5.1 shows a deviation from Bayesian behavior, and Section 6.5 explicitly demonstrates that the original PFN architecture cannot count identical examples, a representational rather than purely computational failure. These admissions are the opposite of circularity: they are falsifiable evidence offered against the paper's own optimism. No equation in the paper reduces by construction to its inputs, and no load-bearing argument rests on an unverified self-citation. The finding is therefore no significant circularity.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new physical or mathematical entities. Its assumptions are the standard PFN correctness claim, a compute-vs-data scaling projection, and a generalization-beyond-prior assumption. The free parameters listed are experimental hyperparameters, not fitted to force the central claim.

free parameters (2)
  • GP prior hyperparameters (length scale, output scale, noise) = 0.1, 1.0, 1e-4
    Fixed values from Müller et al. (2022); not fitted to the paper's claims but used for the Martingale experiment.
  • Transformer architecture hyperparameters (layers, batch size, learning rate, embedding size) = Grid search over 4/8 layers, 32/64 batch, 1e-4/3e-4/1e-3 LR, 128/256/512 embedding
    Grid search on training loss in Appendix A. These are standard training choices, not fitted to outcome.
assumptions (3)
  • domain assumption Minimizing the PFN cross-entropy loss approximates the true posterior predictive distribution.
    Used in Section 2.1, formula for loss and KL-divergence. Cited to Müller et al. (2022) and Goodfellow et al. (2016); this is the foundational correctness claim of PFNs.
  • domain assumption Pre-training compute will continue to scale exponentially and faster than the rate of new real-world data generation.
    Used in Sections 1 and 3, based on Epoch AI projections. The central 'compute-rich, data-scarce' premise rests on this.
  • domain assumption PFNs can generalize beyond their training prior's support and thus work on real-world datasets that are not exactly drawn from the generated prior.
    Invoked in Section 5.3 and in the practical claims about TabPFN and forecasting applications.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Position: The Future of Bayesian Prediction Is Prior-Fitted." pith.science (2026). https://pith.science/paper/YJTEI7L7

@misc{pith2026250523947,
  author       = {Pith},
  title        = {Pith review of: Position: The Future of Bayesian Prediction Is Prior-Fitted},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YJTEI7L7}},
  note         = {Machine review of arXiv:2505.23947}
}
read the original abstract

Training neural networks on randomly generated artificial datasets yields Bayesian models that capture the prior defined by the dataset-generating distribution. Prior-data Fitted Networks (PFNs) are a class of methods designed to leverage this insight. In an era of rapidly increasing computational resources for pre-training and a near stagnation in the generation of new real-world data in many applications, PFNs are poised to play a more important role across a wide range of applications. They enable the efficient allocation of pre-training compute to low-data scenarios. Originally applied to small Bayesian modeling tasks, the field of PFNs has significantly expanded to address more complex domains and larger datasets. This position paper argues that PFNs and other amortized inference approaches represent the future of Bayesian inference, leveraging amortized learning to tackle data-scarce problems. We thus believe they are a fruitful area of research. In this position paper, we explore their potential and directions to address their current limitations.

Figures

Figures reproduced from arXiv: 2505.23947 by the authors.

Figure 1
Figure 1. (a) The PFN learns to approximate the Bayesian predic￾tion offline by training on datasets sampled from the prior and transfers to real-world data. (b) In a typical PFN architecture train￾ing samples (xi, yi) can attend only to each other; test positions (x4 and x5) attend only to the training positions. of available computational resources for neural network pre-training in these areas. Initially, the most prominen… view at source ↗
Figure 2
Figure 2. We show the average standard deviation of 100 datasets sampled from our prior, each normalized by its final standard deviation. We can see that the standard deviations tend to shrink in the first few steps of each roll-out. This should not be the case for a Bayesian predictor, but it stabilizes after some steps, as expected. Further, the deviations in standard deviation are small in absolute terms, compared to an av… view at source ↗
Figure 3
Figure 3. On the left (top), we outline our prior, sampling a probability p for heads (green) and generating samples by coin flips. At test time (left, bottom), we condition on varying counts of coins displaying heads. On the right, we can see that the transformer-based PFN model is not able to approximate the posterior, as it would need to count the number of examples in the context, which are all identical. 6. Improving PFN… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: We train a model on two distinct classes of functions, sines and sloped lines, only (left). It not only learns fit both function classes well (center), but also learns to model slightly sloped sines, when prompted with a data from a sloped sine. 15 [PITH_FULL_IMAGE:fi…

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. TabPFN beyond Tabular Data: Calibration and Accuracy on Multimodal Embeddings

    cs.LG 2026-07 conditional novelty 6.0 of 10

    TabPFN as a zero-gradient head on frozen multimodal embeddings ranks best on NLL and ECE across 22 820 episodes while matching accuracy in mid-shot, mid-dimension regimes and also fixes miscalibration after fine-tuning.

  2. In-Context Learning of Temporal Point Processes with Foundation Inference Models

    cs.LG 2025-09 conditional novelty 6.0 of 10

    A pretrained in-context transformer infers Hawkes-style conditional intensities from event histories and transfers zero-shot to real-world event data, roughly matching specialized models after finetuning.

Reference graph

Works this paper leans on

97 extracted references · 53 canonical work pages · cited by 2 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Efficient bayesian learning curve extrapolation using prior-data fitted networks

    Adriaensen, S., Rakotoarison, H., M \"u ller, S., and Hutter, F. Efficient bayesian learning curve extrapolation using prior-data fitted networks. In neurips23

  3. [3]

    Andrieu, C., De Freitas, N., Doucet, A., and Jordan, M. I. An introduction to mcmc for machine learning. Machine learning, 50 0 (1): 0 5--43, 2003

    2003

  4. [4]

    Transformers as statisticians: Provable in-context learning with in-context algorithm selection

    Bai, Y., Chen, F., Wang, H., Xiong, C., and Mei, S. Transformers as statisticians: Provable in-context learning with in-context algorithm selection. Advances in neural information processing systems, 36, 2024

    2024

  5. [5]

    G., Vitvitskyi, A., Pascanu, R., and Veli c kovi \'c , P

    Barbero, F., Banino, A., Kapturowski, S., Kumaran, D., Ara \'u jo, J. G., Vitvitskyi, A., Pascanu, R., and Veli c kovi \'c , P. Transformers need glasses! information over-squashing in language tasks. arXiv preprint arXiv:2406.04267, 2024

    arXiv 2024

  6. [6]

    K., Swelam, O., Siems, J., Salinas, D., and Hutter, F

    Bhethanabhotla, S. K., Swelam, O., Siems, J., Salinas, D., and Hutter, F. Mamba4cast: Efficient zero-shot time series forecasting with state space models. In NeurIPS Workshop on Time Series in the Age of Large Models, 2024. URL https://openreview.net/forum?id=YBOQ5HnzI6

    2024

  7. [7]

    Weight uncertainty in neural network

    Blundell, C., Cornebise, J., Kavukcuoglu, K., and Wierstra, D. Weight uncertainty in neural network. In Bach, F. and Blei, D. (eds.), Proceedings of the 32nd International Conference on Machine Learning ( ICML '15) , volume 37, pp.\ 1613--1622. Omnipress, 2015

    2015

  8. [8]

    Fine-Tuning the Retrieval Mechanism for Tabular Deep Learning

    Breejen, F. d., Bae, S., Cha, S., Kim, T.-Y., Koh, S. H., and Yun, S.-Y. Fine-tuning the retrieval mechanism for tabular deep learning. arXiv preprint arXiv:2311.07343, 2023

    work page Pith review arXiv 2023

Show all 97 references
  1. [9]

    Random forests

    Breiman, L. Random forests. Machine Learning, 45: 0 5--32, 2001

    2001

  2. [10]

    R., Ober, S

    Burt, D. R., Ober, S. W., Garriga-Alonso, A., and van der Wilk, M. Understanding variational inference in function-space. In Third Symposium on Advances in Approximate Bayesian Inference, 2020

    2020

  3. [11]

    In-context learning for latency estimation

    Carstensen, T., Elsken, T., and Rapp, M. In-context learning for latency estimation. In AutoML Conference 2024 (Workshop Track), 2024

    2024

  4. [12]

    and Guestrin, C

    Chen, T. and Guestrin, C. XGBoost : A scalable tree boosting system. In Krishnapuram, B., Shah, M., Smola, A., Aggarwal, C., Shen, D., and Rastogi, R. (eds.), Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining ( KDD '16) , pp.\ 7...

    2016

  5. [13]

    The frontier of simulation-based inference

    Cranmer, K., Brehmer, J., and Louppe, G. The frontier of simulation-based inference. Proceedings of the National Academy of Sciences, 117 0 (48): 0 30055--30062, 2020

    2020

  6. [14]

    Autoaugment : Learning augmentation strategies from data

    Cubuk, E., Zoph, B., Mane, D., Vasudevan, V., and Le, Q. Autoaugment : Learning augmentation strategies from data. In Proceedings of the International Conference on Computer Vision and Pattern Recognition ( CVPR '19) , pp.\ 113--123. Computer Vision Foundation and IEEE Compute...

    2019

  7. [15]

    and Dalca, A

    Czolbe, S. and Dalca, A. V. Neuralizer: General neuroimage analysis without re-training. In Proceedings of the International Conference on Computer Vision and Pattern Recognition ( CVPR '23) , pp.\ 6217--6230. Computer Vision Foundation and IEEE Computer Society, IEEE, 2023

    2023

  8. [16]

    Deep symbolic regression for recurrent sequences

    d'Ascoli, S., Kamienny, P.-A., Lample, G., and Charton, F. Deep symbolic regression for recurrent sequences. arXiv preprint arXiv:2201.04600, 2022

    2022 arXiv

  9. [17]

    DeepSeek-AI, Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., Zhang, X., Yu, X., Wu, Y., Wu, Z. F., Gou, Z., Shao, Z., Li, Z., Gao, Z., Liu, A., Xue, B., Wang, B., Wu, B., Feng, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., D...

    2025 arXiv

  10. [18]

    ImageNet: A Large-Scale Hierarchical Image Database

    Deng, J., Dong, W., Socher, R., Li, L., Li, K., and Fei-Fei, L. ImageNet: A Large-Scale Hierarchical Image Database . In Proceedings of the International Conference on Computer Vision and Pattern Recognition ( CVPR '09) , pp.\ 248--255. ieeecs, IEEE, 2009

    2009

  11. [19]

    developers, O. R. Onnx runtime. https://onnxruntime.ai/, 2021

    2021

  12. [20]

    S., Mohapatra, C., Naidu, S

    Dooley, S., Khurana, G. S., Mohapatra, C., Naidu, S. V., and White, C. Forecastpfn: Synthetically-trained zero-shot forecasting. In neurips23

  13. [21]

    Data on machine learning hardware, 2024 a

    Epoch AI . Data on machine learning hardware, 2024 a . URL https://epoch.ai/data/machine-learning-hardware. Accessed: 2025-01-27

    2024

  14. [22]

    Data on notable ai models, 2024 b

    Epoch AI . Data on notable ai models, 2024 b . URL https://epoch.ai/data/notable-ai-models. Accessed: 2025-01-27

    2024

  15. [23]

    Falck, F., Wang, Z., and Holmes, C. C. Is in-context learning in large language models bayesian? a martingale perspective. In Forty-first International Conference on Machine Learning, 2024. URL https://openreview.net/forum?id=b1YQ5WKY3w

    2024

  16. [24]

    Reducing transformer depth on demand with structured dropout

    Fan, A., Grave, E., and Joulin, A. Reducing transformer depth on demand with structured dropout. arXiv preprint arXiv:1909.11556, 2019

    1909 arXiv

  17. [25]

    White, C

    Feuer, B., Schirrmeister, R., Cherepanova, V., Hegde, C., Hutter, F., Goldblum, M., Cohen, N., and C. White, C. Tunetables: Context optimization for scalable prior-data fitted networks. In Globerson, A., Mackey, L., Belgrave, D., Fan, A., Paquet, U., Tomczak, J., and Zhang, C....

    2024

  18. [26]

    T., Cherepanova, V., Hegde, C., Hutter, F., Goldblum, M., Cohen, N., and White, C

    Feuer, B., Schirrmeister, R. T., Cherepanova, V., Hegde, C., Hutter, F., Goldblum, M., Cohen, N., and White, C. Tunetables: Context optimization for scalable prior-data fitted networks. arXiv preprint arXiv:2402.11137, 2024 b

    2024 arXiv

  19. [27]

    Model-agnostic meta-learning for fast adaptation of deep networks

    Finn, C., Abbeel, P., and Levine, S. Model-agnostic meta-learning for fast adaptation of deep networks. In Precup, D. and Teh, Y. (eds.), Proceedings of the 34th International Conference on Machine Learning ( ICML '17) , volume 70, pp.\ 1126--1135. Proceedings of Machine Learn...

    2017

  20. [28]

    Conditional neural processes

    Garnelo, M., Rosenbaum, D., Maddison, C., Ramalho, T., Saxton, D., Shanahan, M., Teh, Y., Rezende, D., and Eslami, S. Conditional neural processes. In Dy, J. and Krause, A. (eds.), Proceedings of the 35th International Conference on Machine Learning ( ICML '18) , volume 80, pp...

    2018

  21. [29]

    J., Eslami, S., and Teh, Y

    Garnelo, M., Schwarz, J., Rosenbaum, D., Viola, F., Rezende, D. J., Eslami, S., and Teh, Y. W. Neural processes. arXiv preprint arXiv:1807.01622, 2018 b

    2018 arXiv

  22. [30]

    Bayesian Optimization

    Garnett, R. Bayesian Optimization . Cambridge University Press, 2023

    2023

  23. [31]

    D., Wildberger, J., Dax, M., Kofler, A., Angerhausen, D., Quanz, S

    Gebhard, T. D., Wildberger, J., Dax, M., Kofler, A., Angerhausen, D., Quanz, S. P., and Sch \"o lkopf, B. Flow matching for atmospheric retrieval of exoplanets: Where reliability meets adaptive noise levels. Astronomy & Astrophysics, 693: 0 A42, 2025

    2025

  24. [32]

    Gloeckler, M., Deistler, M., Weilbach, C., Wood, F., and Macke, J. H. All-in-one simulation-based inference. arXiv preprint arXiv:2404.09636, 2024

    2024 arXiv

  25. [33]

    Deep Learning

    Goodfellow, I., Bengio, Y., and Courville, A. Deep Learning. MIT Press, 2016

    2016

  26. [34]

    Green, P. J. Reversible jump markov chain monte carlo computation and bayesian model determination. Biometrika, 82 0 (4): 0 711--732, 1995

    1995

  27. [35]

    Automatic posterior transformation for likelihood-free inference

    Greenberg, D., Nonnenmacher, M., and Macke, J. Automatic posterior transformation for likelihood-free inference. In International Conference on Machine Learning, pp.\ 2404--2414. PMLR, 2019

    2019

  28. [36]

    and Dao, T

    Gu, A. and Dao, T. Mamba: Linear time sequence modeling with selective state spaces. arXiv:2312.00752 [cs.LG], 2023

    2023 arXiv

  29. [37]

    Drift-resilient tab PFN : In-context learning temporal distribution shifts on tabular data

    Helli, K., Schnurr, D., Hollmann, N., M \"u ller, S., and Hutter, F. Drift-resilient tab PFN : In-context learning temporal distribution shifts on tabular data. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/fo...

    2024

  30. [38]

    C., Atkinson, D., Thompson, N., and Sevilla, J

    Ho, A., Besiroglu, T., Erdil, E., Owen, D., Rahman, R., Guo, Z. C., Atkinson, D., Thompson, N., and Sevilla, J. Algorithmic progress in language models. CoRR, 2024

    2024

  31. [39]

    D., Blei, D

    Hoffman, M. D., Blei, D. M., Wang, C., and Paisley, J. Stochastic variational inference. Journal of Machine Learning Research, 14 0 (4): 0 1303--1347, 2013. URL http://jmlr.org/papers/v14/hoffman13a.html

    2013

  32. [40]

    D., Gelman, A., et al

    Hoffman, M. D., Gelman, A., et al. The no-u-turn sampler: adaptively setting path lengths in hamiltonian monte carlo. J. Mach. Learn. Res., 15 0 (1): 0 1593--1623, 2014

    2014

  33. [41]

    Tab PFN : A transformer that solves small tabular classification problems in a second

    Hollmann, N., M \"u ller, S., Eggensperger, K., and Hutter, F. Tab PFN : A transformer that solves small tabular classification problems in a second. In The Eleventh International Conference on Learning Representations ( ICLR '23) . ICLR, 2023. Published online: iclr.cc

    2023

  34. [42]

    u ller, S., Purucker, L., Krishnakumar, A., K \

    Hollmann, N., M \"u ller, S., Purucker, L., Krishnakumar, A., K \"o rfer, M., Hoo, S. B., Schirrmeister, R. T., and Hutter, F. Accurate predictions on small data with a tabular foundation model. Nature, 637 0 (8045): 0 319--326, 2025

    2025

  35. [43]

    B., M \"u ller, S., Salinas, D., and Hutter, F

    Hoo, S. B., M \"u ller, S., Salinas, D., and Hutter, F. The tabular foundation model tabpfn outperforms specialized time series forecasting models based on simple features. arXiv preprint arXiv:2501.02945, 2025

    2025

  36. [44]

    Perceiver: General perception with iterative attention

    Jaegle, A., Gimeno, F., Brock, A., Vinyals, O., Zisserman, A., and Carreira, J. Perceiver: General perception with iterative attention. In International conference on machine learning, pp.\ 4651--4664. PMLR, 2021

    2021

  37. [45]

    Billion-scale similarity search with gpus

    Johnson, J., Douze, M., and J \'e gou, H. Billion-scale similarity search with gpus. IEEE Transactions on Big Data, 7 0 (3): 0 535--547, 2019

    2019

  38. [46]

    I., Ghahramani, Z., Jaakkola, T

    Jordan, M. I., Ghahramani, Z., Jaakkola, T. S., and Saul, L. K. An introduction to variational methods for graphical models. Machine learning, 37 0 (2): 0 183--233, 1999

    1999

  39. [47]

    End-to-end symbolic regression with transformers

    Kamienny, P.-A., d'Ascoli, S., Lample, G., and Charton, F. End-to-end symbolic regression with transformers. In Proceedings of the 36th International Conference on Neural Information Processing Systems, pp.\ 10269--10281, 2022

    2022

  40. [48]

    Kimi, T., Du, A., Gao, B., Xing, B., Jiang, C., Chen, C., Li, C., Xiao, C., Du, C., Liao, C., et al. Kimi k1. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599, 2025

    2025 arXiv

  41. [49]

    When ensembling smaller models is more efficient than single large models

    Kondratyuk, D., Tan, M., Brown, M., and Gong, B. When ensembling smaller models is more efficient than single large models. arXiv preprint arXiv:2005.00570, 2020

    2005 arXiv

  42. [50]

    Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.). Proceedings of the 40th International Conference on Machine Learning ( ICML '23) , volume 202 of Proceedings of Machine Learning Research, 2023. PMLR

    2023

  43. [51]

    J., Loftus, J., Russell, C., and Silva, R

    Kusner, M. J., Loftus, J., Russell, C., and Silva, R. Counterfactual fairness. Advances in neural information processing systems, 30, 2017

    2017

  44. [52]

    Lake, B. M. and Baroni, M. Human-like systematic generalization through a meta-learning neural network. Nature, 623 0 (7985): 0 115--121, 2023

    2023

  45. [53]

    Deepseek-v3 technical report

    Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  46. [54]

    Amortized inference for causal structure learning

    Lorch, L., Sussex, S., Rothfuss, J., Krause, A., and Sch \"o lkopf, B. Amortized inference for causal structure learning. Advances in Neural Information Processing Systems, 35, 2022

    2022

  47. [55]

    Lueckmann, J.-M., Bassetto, G., Karaletsos, T., and Macke, J. H. Likelihood-free inference with emulator networks. In Symposium on Advances in Approximate Bayesian Inference, pp.\ 32--53. PMLR, 2019

    2019

  48. [56]

    J., Macke, J

    Manzano-Patr \'o n, J., Deistler, M., Schr \"o der, C., Kypraios, T., Gon c alves, P. J., Macke, J. H., and Sotiropoulos, S. S. Uncertainty mapping and probabilistic tractography using simulation-based inference in diffusion mri: A comparison with classical bayes. bioRxiv, pp....

    2024

  49. [57]

    When do neural nets outperform boosted trees on tabular data? In neurips23 , pp.\ 76336--76369

    McElfresh, D., Khandagale, S., Valverde, J., Prasad C , V., Ramakrishnan, G., Goldblum, M., and White, C. When do neural nets outperform boosted trees on tabular data? In neurips23 , pp.\ 76336--76369

  50. [59]

    Mothernet: A foundational hypernetwork for tabular classification

    M \"u ller, A., Curino, C., and Ramakrishnan, R. Mothernet: A foundational hypernetwork for tabular classification. arXiv preprint arXiv:2312.08598, 2023 b

    2023 arXiv

  51. [60]

    and Hutter, F

    M \"u ller, S. and Hutter, F. Trivialaugment: Tuning-free yet state-of-the-art data augmentation. In Proceedings of the 24nd IEEE/CVF International Conference on Computer Vision ( ICCV '21) , pp.\ 774--782. Computer Vision Foundation and IEEE Computer Society, IEEE, 2021

    2021

  52. [61]

    Transformers can do B ayesian inference

    M \"u ller, S., Hollmann, N., Arango, S., Grabocka, J., and Hutter, F. Transformers can do B ayesian inference. In The Tenth International Conference on Learning Representations ( ICLR '22) . ICLR, 2022. Published online: iclr.cc

    2022

  53. [62]

    PFNs4BO: In-Context Learning for Bayesian Optimization

    M \"u ller, S., Feurer, M., Hollmann, N., and Hutter, F. PFNs4BO: In-Context Learning for Bayesian Optimization . In icml23

  54. [63]

    Bayes' power for explaining in-context learning generalizations

    M \"u ller, S., Hollmann, N., and Hutter, F. Bayes' power for explaining in-context learning generalizations. arXiv preprint arXiv:2410.01565, 2024

    2024 arXiv

  55. [64]

    shapiq: Shapley interactions for machine learning

    Muschalik, M., Baniecki, H., Fumagalli, F., Kolpaczki, P., Hammer, B., and H\" u llermeier, E. shapiq: Shapley interactions for machine learning. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL https://openreview...

    2024

  56. [65]

    Statistical foundations of prior-data fitted networks

    Nagler, T. Statistical foundations of prior-data fitted networks. In icml23 , pp.\ 25660--25676

  57. [66]

    Bayesian Learning for Neural Networks

    Neal, R. Bayesian Learning for Neural Networks. Lecture Notes in Statistics. Springer, 1996

    1996

  58. [67]

    and Grover, A

    Nguyen, T. and Grover, A. Transformer neural processes: Uncertainty-aware meta learning via sequence modeling. arXiv preprint arXiv:2207.04179, 2022

    2022 arXiv

  59. [68]

    Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.). Proceedings of the 36th International Conference on Advances in Neural Information Processing Systems ( N eur IPS '23) , 2023. Curran Associates

    2023

  60. [69]

    Neural density estimation and likelihood-free inference

    Papamakarios, G. Neural density estimation and likelihood-free inference. arXiv preprint arXiv:1910.13233, 2019

    1910 arXiv

  61. [70]

    and Murray, I

    Papamakarios, G. and Murray, I. Fast -free inference of simulation models with bayesian conditional density estimation. Advances in neural information processing systems, 29, 2016

    2016

  62. [71]

    PyTorch : An imperative style, high-performance deep learning library

    Paszke, A., Gross, S., Massa, F., Lerer, A., et al. PyTorch : An imperative style, high-performance deep learning library. In Wallach, H., Larochelle, H., Beygelzimer, A., d'Alche Buc, F., Fox, E., and Garnett, R. (eds.), Proceedings of the 32nd International Conference on Adv...

    2019

  63. [72]

    Adapting tabpfn for zero-inflated metagenomic data

    Perciballi, G., Granese, F., Fall, A., Zehraoui, F., Prifti, E., and Zucker, J.-D. Adapting tabpfn for zero-inflated metagenomic data. In NeurIPS 2024 Third Table Representation Learning Workshop, 2024

    2024

  64. [73]

    The devil in linear transformer

    Qin, Z., Han, X., Sun, W., Li, D., Kong, L., Barnes, N., and Zhong, Y. The devil in linear transformer. arXiv preprint arXiv:2210.10340, 2022

    2022 arXiv

  65. [74]

    In-context freeze-thaw bayesian optimization for hyperparameter optimization

    Rakotoarison, H., Adriaensen, S., Mallik, N., Garibov, S., Bergman, E., and Hutter, F. In-context freeze-thaw bayesian optimization for hyperparameter optimization. In Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp, F. (eds.), P...

    2024

  66. [75]

    and Williams, C

    Rasmussen, C. and Williams, C. Gaussian Processes for Machine Learning. The MIT Press, 2006

    2006

  67. [76]

    Pretraining task diversity and the emergence of non-bayesian in-context learning for regression

    Raventos, A., Paul, M., Chen, F., and Ganguli, S. Pretraining task diversity and the emergence of non-bayesian in-context learning for regression. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=BtAz4a5xDg

    2023

  68. [77]

    Reuter, A., Rudner, T. G. J., Fortuin, V., and R \"u gamer, D. Can transformers learn full bayesian inference in context? arXiv preprint arXiv:2501.16825, 2025. doi:10.48550/arXiv.2501.16825. URL https://arxiv.org/abs/2501.16825

    2025 doi arXiv

  69. [78]

    Fairpfn: Transformers can do counterfactual fairness

    Robertson, J., Hollmann, N., Awad, N., and Hutter, F. Fairpfn: Transformers can do counterfactual fairness. arXiv preprint arXiv:2407.05732, 2024

    2024 arXiv

  70. [79]

    G., Chen, Z., Teh, Y

    Rudner, T. G., Chen, Z., Teh, Y. W., and Gal, Y. Tractable function-space variational inference in bayesian neural networks. Advances in Neural Information Processing Systems, 35: 0 22686--22698, 2022

    2022

  71. [80]

    Interpretable machine learning for tabpfn

    Rundel, D., Kobialka, J., von Crailsheim, C., Feurer, M., Nagler, T., and R \"u gamer, D. Interpretable machine learning for tabpfn. In World Conference on Explainable Artificial Intelligence, pp.\ 465--476. Springer, 2024

    2024

  72. [81]

    Meta-learning with memory-augmented neural networks

    Santoro, A., Bartunov, S., Botvinick, M., Wierstra, D., and Lillicrap, T. Meta-learning with memory-augmented neural networks. In Balcan, M. and Weinberger, K. (eds.), Proceedings of the 33rd International Conference on Machine Learning ( ICML '17) , volume 48. Proceedings of ...

    2016

  73. [82]

    K., Wolfinger, M

    Scheuer, D., Runge, F., Franke, J. K., Wolfinger, M. T., Flamm, C., and Hutter, F. Kinpfn: Bayesian approximation of rna folding kinetics using prior-data fitted networks. bioRxiv, pp.\ 2024--10, 2024

    2024

  74. [83]

    Fast transformer decoding: One write-head is all you need

    Shazeer, N. Fast transformer decoding: One write-head is all you need. arXiv preprint arXiv:1911.02150, 2019

    1911 arXiv

  75. [84]

    Zero-shot outlier detection via prior-data fitted networks: Model selection bygone! CoRR, abs/2409.05672, 2024

    Shen, Y., Wen, H., and Akoglu, L. Zero-shot outlier detection via prior-data fitted networks: Model selection bygone! CoRR, abs/2409.05672, 2024. doi:10.48550/ARXIV.2409.05672. URL https://doi.org/10.48550/arXiv.2409.05672

    2024 doi

  76. [85]

    Sutton, R. S. and Barto, A. G. Reinforcement learning: An introduction. Adaptive computation and machine learning. MIT Press, 2 edition, 2018

    2018

  77. [86]

    Thomas, V., Ma, J., Hosseinzadeh, R., Golestan, K., Yu, G., Volkovs, M., and Caterini, A. L. Retrieval & fine-tuning for in-context tabular models. In ICML 2024 Workshop on In-Context Learning, 2024

    2024

  78. [87]

    T., Le, Q., He, H., and Luong, T

    Trinh, T., Wu, Y. T., Le, Q., He, H., and Luong, T. Solving olympiad geometry without human demonstrations. Nature, 625: 0 476--482, 2024. URL https://www.nature.com/articles/s41586-023-06747-5

    2024

  79. [88]

    Ubbens, J., Stavness, I., and Sharpe, A. G. Gpfn: Prior-data fitted networks for genomic prediction. bioRxiv, pp.\ 2023--09, 2023

    2023

  80. [89]

    Verdenius, S., Zerio, A., and Wang, R. L. Lat-pfn: A joint embedding predictive architecture for in-context time-series forecasting. arXiv preprint arXiv:2405.10093, 2024

    2024 arXiv

  81. [90]

    Wainwright, M. J. and Jordan, M. I. Graphical models, exponential families, and variational inference. Now Publishers Inc, 2008

    2008

  82. [91]

    Z., Khabsa, M., Fang, H., and Ma, H

    Wang, S., Li, B. Z., Khabsa, M., Fang, H., and Ma, H. Linformer: Self-attention with linear complexity, 2020 a

    2020

  83. [92]

    M., Alon, Y., and Eban, E

    Wang, X., Kondratyuk, D., Christiansen, E., Kitani, K. M., Alon, Y., and Eban, E. Wisdom of committees: An overlooked approach to faster and more accurate models. arXiv preprint arXiv:2012.01988, 2020 b

    2012 arXiv

  84. [93]

    V., Zhou, D., et al

    Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35: 0 24824--24837, 2022

    2022

  85. [94]

    and Teh, Y

    Welling, M. and Teh, Y. W. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-11), pp.\ 681--688. Citeseer, 2011

    2011

  86. [95]

    H., and Sch \"o lkopf, B

    Wildberger, J., Dax, M., Buchholz, S., Green, S., Macke, J. H., and Sch \"o lkopf, B. Flow matching for scalable simulation-based inference. Advances in Neural Information Processing Systems, 36, 2024

    2024

  87. [96]

    J., Pritzel, A., and Blundell, C

    Wirnsberger, P., Papamakarios, G., Ibarz, B., Racaniere, S., Ballard, A. J., Pritzel, A., and Blundell, C. Normalizing flows for atomic solids. Machine Learning: Science and Technology, 3 0 (2): 0 025009, 2022

    2022

  88. [97]

    Mixture of in-context prompters for tabular pfns

    Xu, D., Cirit, O., Asadi, R., Sun, Y., and Wang, W. Mixture of in-context prompters for tabular pfns. arXiv:2405.16156 [cs.LG], 2024

    2024 arXiv

  89. [98]

    When can transformers count to n? arXiv preprint arXiv:2407.15160, 2024

    Yehudai, G., Kaplan, H., Ghandeharioun, A., Geva, M., and Globerson, A. When can transformers count to n? arXiv preprint arXiv:2407.15160, 2024

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.