Pith. sign in

REVIEW 4 major objections 5 minor 3 cited by

Can Transformers Learn Full Bayesian Inference in Context?

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that transformers can learn to output full posterior distributions in context, matching Hamiltonian Monte Carlo performance on generalized linear models, factor analysis, and low-dimensional Gaussian mixture models…

desk verdict The per-scenario experiments are the real content and mostly support the method, but the headline summary tables don't reproduce from the appendix, and the abstract's comparative claim leans on those tables. read the letter →

arxiv 2501.16825 v2 pith:EDV37HRP submitted 2025-01-28 cs.LG

Arik Reuter , Tim G. J. Rudner , Vincent Fortuin , David Rügamer This is my paper · ORCID
classification cs.LG MSC 62F1568T0762J1262H30
keywords in-contextlearningBayesianinferenceflowmatchingposteriorsamplinggeneralizedlinearmodelsfactoranalysisGaussianmixtureamortized
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a transformer can learn to do full Bayesian inference in context: given a dataset as its input context, the network directly outputs posterior samples for the latent variables, with no per-dataset parameter updates or posterior-family assumptions. The authors train one model per model class on synthetic samples from the joint distribution of data and latents, and evaluate the resulting posterior samples against Hamiltonian Monte Carlo and several variational baselines. Across 50 synthetic and 17 real-world tabular datasets, the in-context learner matches or beats the variational baselines and approaches HMC quality for generalized linear models, factor analysis, and low-dimensional Gaussian mixture models. If correct, this means a single trained forward pass plus a short ODE solve could replace per-dataset MCMC or variational inference for these model classes.

What carries the argument

The load-bearing machinery is conditional flow matching on the joint distribution. A transformer encoder (of the prior-data fitted network style) encodes the context dataset $x$; a diffusion-transformer decoder with adaptive layer normalization takes an interpolated point $\gamma_t(z^{(0)}|z^{(1)}) = (1-\omega t) z^{(0)} + t z^{(1)}$ and outputs a vector field $v_{t,x}$, trained by the flow-matching loss in Equation (7). Because the expected divergence between the model posterior and the true posterior can be rewritten as an expectation over samples from the joint $P(x,z)$ (Proposition 1), the training objective is tractable: sample a latent $z$ from the prior, a dataset $x$ from the likelihood, a base sample $z^{(0)}$, and a time $t$, then regress the vector field toward the optimal-transport velocity $z^{(1)} - \omega z^{(0)}$. At deployment, posterior sampling is simply ODE integration of the learned field from Gaussian noise, conditioned on $x$.

What would settle it

Run the trained GLM model on a real-world dataset without the power transform, standardization, and feature selection used in preprocessing, and measure C2ST, MMD, and Wasserstein-2 against HMC; if scores degrade substantially relative to the preprocessed version, the real-world result is carried by preprocessing rather than by the in-context posterior mechanism.

Watch

Extended reading notes

Core claim

The central discovery is that posterior inference can be amortized at the dataset level: the mapping $x \mapsto P(z \mid x)$ is learned end-to-end by training a transformer encoder-decoder on samples $(x, z) \sim P(x, z)$, without ever evaluating a likelihood or a divergence between posterior distributions directly. The learned object is a conditional continuous normalizing flow: the decoder outputs a time-dependent vector field $v_{t,x}$, and solving the ODE $d\psi/dt = v_{t,x}(\psi)$ from a standard Gaussian base distribution yields posterior samples. On generalized linear models the ICL posterior agrees with the analytical posterior or with HMC more closely than Gaussian variational inference does, especially when the posterior is skewed; on factor analysis the synthetic C2ST reaches 0.568, close to the chance level of 0.5; and on low-dimensional Gaussian mixture models the ICL samples reproduce the bimodal posterior structure that Gaussian variational approximations miss. The authors report these results as evidence that transformers can perform full Bayesian inference in context.

Load-bearing premise

The method's real-world performance rests on the assumption that, after preprocessing, real datasets look like datasets drawn from the synthetic training prior; if that mismatch is large, the learned posterior map has to extrapolate out-of-distribution and can fail.

Editorial extensions

If this is right

  • For generalized linear models, the approach yields the best or tied-best posterior-similarity scores against HMC in most scenarios, and clearly beats all tested variational methods when the prior induces a skewed posterior.
  • For factor analysis, synthetic C2ST approaches chance level, indicating samples nearly indistinguishable from HMC; on real data it is best or tied on C2ST and MMD, though the Wasserstein-2 result is mixed.
  • For low-dimensional Gaussian mixture models, ICL captures the two modes of a bimodal posterior, while even flow-based variational inference collapses to one mode.
  • Ablations show that replacing flow matching with a diffusion objective or score matching, or with a multivariate Gaussian posterior, substantially degrades sample quality; the flow-matching objective and transformer encoder are essential.
  • At inference time the method replaces per-dataset MCMC: reported runtimes are consistently lower than the HMC reference, with the largest speedups in the factor analysis and mixture model scenarios.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the only requirement is the ability to sample from $P(x,z)$, the same recipe could be applied to probabilistic programs and simulator-based models with non-standard likelihoods, where HMC and variational inference are harder to set up.
  • The real-world results depend on preprocessing that makes datasets roughly match the synthetic covariate prior; a direct test is whether the method stays reliable when covariates are categorical, missing, or otherwise outside that prior.
  • The reported performance drops at 20 or 50 latent dimensions suggest that the method's advantage is clearest in low dimensions; evaluating with metrics less sensitive to the curse of dimensionality would clarify whether the drop is in sampling or in measurement.
  • The full-posterior sampler could be used as a drop-in component in downstream tasks like active learning, calibration, or sequential decision-making, where having the whole distribution rather than a point estimate is what matters.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes an in-context learning framework for full Bayesian inference: a TabPFN-style transformer encoder ingests a dataset x, and a diffusion-transformer decoder with continuous normalizing flows outputs samples from the posterior P(z|x). The model is trained on synthetic datasets drawn from the joint distribution P(x,z) for generalized linear models, factor analysis, and Gaussian mixture models. The authors evaluate the method against Hamiltonian Monte Carlo and several variational inference baselines on 50 synthetic and 17 real-world datasets using C2ST, MMD, and Wasserstein-2 metrics, and report that the ICL approach produces posterior samples similar in quality to HMC and often better than VI.

Significance. If the empirical claims hold, this is a useful step toward fast amortized Bayesian inference for moderate-dimensional models: a single forward pass replaces per-dataset MCMC or variational inference. The paper ships source code, trains models on a large scale, and includes a broad set of ablations (flow matching vs. diffusion, MLP vs. transformer encoders, out-of-distribution robustness, dimensionality scaling, and alternative C2ST classifiers). The authors are also unusually explicit about limitations. However, the central comparative claim rests on summary tables whose numbers do not reproduce from the per-scenario tables in the appendix, so the quantitative contribution is currently not verifiable from the manuscript itself.

major comments (4)
  1. [Section 4, Tables 1/3/4 vs. Appendix I, Tables 12/13/14] The headline summary tables are not consistent with the per-scenario results that the paper itself provides. For example, using the seven real-world GLM rows of Table 12, the unweighted mean MMD for ICL is about (0.175+0.143+0.037+0.105+0.046+0.035+0.294)/7 ≈ 0.119 and for VI-Full is about 0.112, so VI-Full is better; Table 1 prints 0.090 for ICL and 0.116 for VI-Full and marks ICL as best. Similarly, the synthetic GLM C2ST rows of Table 12 imply an ICL mean of about 0.689, not the printed 0.657. The FA and GMM summaries in Tables 3 and 4 likewise differ from straightforward means of their appendix tables. Because the abstract and Section 4 make explicitly comparative claims, the aggregation rule must be documented precisely and the tables corrected, or the claims must be qualified to the per-scenario level. This is a load-bearing issue, not a cosmetic one.
  2. [Section 3.1.2, Eq. (7)] Equation (7) appears to have a sign error: as printed it reads ||v_theta + z(1) - omega z(0)||^2, whereas the discrepancy in Eq. (5) is expressed with v_theta - (z(1) - omega z(0)). If the implementation follows Eq. (5), the printed objective is wrong and must be corrected; if the implementation uses the printed sign, the relationship to the stated flow-matching objective needs to be explained. Since this equation defines the training loss of the central method, the correct sign should be confirmed and stated explicitly.
  3. [Appendix F.2 and Section 4 (Evaluation Process)] HMC is used as the gold standard for most comparisons, but no convergence diagnostics are reported. The paper states that 500 burn-in samples are used and that unimodal problems use a single chain, but it does not report R-hat, effective sample size, or trace diagnostics for representative datasets. Without such diagnostics, 'samples similar to HMC' could mean samples similar to an unconverged or poorly mixed HMC run. I request at least R-hat and ESS summaries for representative GLM, FA, and GMM scenarios, and preferably chain diagnostics for the multimodal GMM cases.
  4. [Appendix C and Section 4 (Real-World Datasets)] The real-world evaluation depends on strong preprocessing: Yeo-Johnson transformation, scaling and shifting the target to match the generative prior, feature selection by number of distinct values, and subsampling to 50 observations. Since the ICL model is trained on covariates drawn from the TabPFN prior, the validity of the real-world comparisons depends on whether the preprocessed real datasets are approximately in-distribution. The paper does not quantify this, and the per-scenario real-world results are mixed: for example, in Table 12 the VI-Full or VI-Structured baselines have lower C2ST than ICL in scenarios 1, 3, 6, and 7. The claims about real-world superiority should be weakened or supplemented with an in-distribution check, such as comparing the covariate and target distributions of preprocessed real data against the training prior.
minor comments (5)
  1. [Section 4, first paragraph of Experiments] The text says the model is trained on samples from the joint 'P x,x'; this should presumably be P(x,z).
  2. [Appendix I.3, Table 14] Table 14 is captioned as covering six GMM scenarios but contains only four scenario rows, and the text says 'six different scenarios' while Section 4.3 correctly describes four GMM scenarios.
  3. [Section 3.2 and Appendix A] The main text refers to Appendix B for details on data-generating processes, but the relevant details are in Appendix A.
  4. [Table 12] Scenario 1 includes an HMC row even though the summary table and method description present HMC as the reference; clarify whether HMC here is compared against the analytical solution and how this row is treated in aggregations.
  5. [Appendix K, Table 18 caption] The caption contains a duplicated comma and an extra space ('KL-divergence,,'); this is a minor typo that should be cleaned up.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the posterior target is defined by the generative model, and the ICL model is trained on samples from that same joint distribution; evaluation against independently obtained HMC/analytical reference samples is an external benchmark.

full rationale

The paper's load-bearing claim is empirical: a transformer trained on synthetic samples (x, z) ~ P(x, z) via the flow-matching objective in Eq. (7) produces samples close to HMC/analytical posteriors. This is standard amortized inference, not circular: the target P(z|x) is the conditional distribution of the specified generative model, and the training data are draws from that model's joint distribution; no fitted parameter is subsequently renamed as a prediction of the same quantity. Proposition 1 is a mathematical identity (law of total expectation) used to make the objective tractable, not a self-referential definition. The GLM covariate prior borrowed from TabPFN and the Yeo-Johnson/scaling preprocessing (Appendix C) are explicitly discussed as model-misspecification choices (Appendix B) and tested with out-of-distribution ablations; they may weaken external validity but do not make the reported posterior comparisons definitional. Self-citations (Rudner et al. 2018; Sommer et al. 2024, 2025) are background and are not load-bearing. The apparent mismatch between the summary Tables 1/3/4 and the per-scenario Appendix tables is a reproducibility/aggregation concern, not an instance of circularity, because the per-scenario numbers are independent measurements and the discrepancy does not show that any claimed prediction is equal to a fitted input by construction.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The method's empirical claims rest on hand-specified generative scenarios, the representational capacity of flow matching, and the validity of HMC and discrepancy metrics as references. These are standard domain assumptions rather than ad hoc inventions, but they bound the generality of the results.

free parameters (1)
  • Scenario prior hyperparameters (normal, Laplace, gamma, inverse-gamma, Dirichlet, K, P, zdim) = Tables 5-7 (e.g., IG(5,2), N(0,9), Dir(1), lambda=3)
    Chosen by hand to define the seven GLM, six FA, and four GMM scenarios; all empirical conclusions are conditional on these modeling choices, though they are not fitted to evaluation data.
assumptions (5)
  • domain assumption Ability to sample from the joint distribution p(x,z)=p(x|z)p(z) for the target model (Sec 3.2)
    The whole training procedure requires a generative process for each model; if such sampling is impossible, the method cannot be applied.
  • domain assumption The TabPFN prior on covariates is a faithful enough proxy for real-world covariate distributions (Appendix A.1, B)
    The claimed real-world transfer depends on this; the paper explicitly acknowledges this as model misspecification risk.
  • domain assumption Target posteriors admit a density and can be represented by continuous normalizing flows with Gaussian conditional paths (Sec 3.1.2)
    Flow matching only learns distributions that can be pushed forward from a Gaussian base with the chosen path; multimodal posteriors are only partially captured, as seen in GMM scenarios.
  • domain assumption HMC and NUTS samples are the gold-standard posterior (Sec 4)
    All comparisons assume HMC samples are exact; no convergence diagnostics such as R-hat or ESS are reported.
  • domain assumption The discrepancy metrics, C2ST with a random forest, MMD, and Wasserstein-2, faithfully rank posterior approximations (Sec 4)
    C2ST saturates at 1.0 for many VI baselines and for ICL in high-dimensional GMM scenarios, limiting its discriminative power; the authors validate with a neural network in Appendix Q, but the metric choice still shapes all conclusions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Can Transformers Learn Full Bayesian Inference in Context?." pith.science (2026). https://pith.science/paper/EDV37HRP

@misc{pith2026250116825,
  author       = {Pith},
  title        = {Pith review of: Can Transformers Learn Full Bayesian Inference in Context?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EDV37HRP}},
  note         = {Machine review of arXiv:2501.16825}
}
read the original abstract

Transformers have emerged as the dominant architecture in the field of deep learning, with a broad range of applications and remarkable in-context learning (ICL) capabilities. While not yet fully understood, ICL has already proved to be an intriguing phenomenon, allowing transformers to learn in context -- without requiring further training. In this paper, we further advance the understanding of ICL by demonstrating that transformers can perform full Bayesian inference for commonly used statistical models in context. More specifically, we introduce a general framework that builds on ideas from prior fitted networks and continuous normalizing flows and enables us to infer complex posterior distributions for models such as generalized linear models and latent factor models. Extensive experiments on real-world datasets demonstrate that our ICL approach yields posterior samples that are similar in quality to state-of-the-art MCMC or variational inference methods that do not operate in context. The source code for this paper is available at https://github.com/ArikReuter/ICL_for_Full_Bayesian_Inference.

Figures

Figures reproduced from arXiv: 2501.16825 by the authors.

Figure 1
Figure 1. (a) An LLM generates a summary s1, s2, . . . of a text t1, t2, . . . , tK through autoregressive sampling while referring to the context using masked self-attention. (b) A dataset x is processed with a transformer encoder. Subsequently, cross attention allows generating samples from the posterior conditioned on x in context using a diffusion transformer (decoder). The samples are generated by solving a neural differ… view at source ↗
Figure 2
Figure 2. Architecture to perform ICL for full Bayesian [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Density plots for the marginals of the posterior [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (4 more)
Figure 5
Figure 5. Figure 5: Learning curves for GMM scenario 1 with M = 5 components, K = 50 datapoints and L = 1 dimen￾sions [PITH_FULL_IMAGE:figures/full_fig_p020_5.png]
Figure 7
Figure 7. Figure 7: Density plots for first three the marginals of the posterior in a GLM with a gamma prior on the coefficients [PITH_FULL_IMAGE:figures/full_fig_p024_7.png]
Figure 8
Figure 8. Figure 8: Marginal distribution for GLM scenario 2 (left) and GMM scenario 1 (right). The in-context learner is trained [PITH_FULL_IMAGE:figures/full_fig_p033_8.png]
Figure 9
Figure 9. Figure 9: Out-of-distribution (OOD) performance of the ICL method in GLM Scenario 2. The [PITH_FULL_IMAGE:figures/full_fig_p037_9.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Bayesian Wind Tunnels for Model Selection

    cs.LG 2026-07 conditional novelty 6.0 of 10

    A 2.8M-parameter transformer matches Bayesian model-selection posteriors to 0.01-bit entropy error on relational function classes, while failing completely on arithmetic classes with opaque tokens.

  2. Early Stopping Tabular In-Context Learning

    cs.LG 2025-06 conditional novelty 4.0 of 10

    Early-exit decoders let TabPFN stop at earlier transformer layers when confidence is high, giving up to 2.2x faster tabular classification with small accuracy losses.

  3. Position: The Future of Bayesian Prediction Is Prior-Fitted

    cs.LG 2025-05 conditional novelty 4.0 of 10

    PFNs, which amortize Bayesian inference by training on datasets sampled from a prior, are likely to supersede MCMC and variational inference for most prediction tasks, the authors argue.

Reference graph

Works this paper leans on

107 extracted references · 34 canonical work pages · cited by 3 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  3. [3]

    A., Hassan, M

    Abdullah, A. A., Hassan, M. M., and Mustafa, Y. T. A review on bayesian deep learning in healthcare: Applications and challenges. IEEE Access, 10: 0 36538--36562, 2022

    2022

  4. [4]

    In-context learning through the bayesian prism

    Ahuja, K., Panwar, M., and Goyal, N. In-context learning through the bayesian prism. arXiv preprint arXiv:2306.04891, 2023

    arXiv 2023

  5. [5]

    M., Hauth, A., Millican, K., et al

    Anil, R., Borgeaud, S., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023

    arXiv 2023

  6. [6]

    A conceptual introduction to hamiltonian monte carlo

    Betancourt, M. A conceptual introduction to hamiltonian monte carlo. arXiv preprint arXiv:1701.02434, 2017

    arXiv 2017

  7. [7]

    P., Jankowiak, M., Obermeyer, F., Pradhan, N., Karaletsos, T., Singh, R., Szerlip, P., Horsfall, P., and Goodman, N

    Bingham, E., Chen, J. P., Jankowiak, M., Obermeyer, F., Pradhan, N., Karaletsos, T., Singh, R., Szerlip, P., Horsfall, P., and Goodman, N. D. Pyro: Deep universal probabilistic programming. Journal of machine learning research, 20 0 (28): 0 1--6, 2019

    2019

  8. [8]

    Vibes: A variational inference engine for bayesian networks

    Bishop, C., Spiegelhalter, D., and Winn, J. Vibes: A variational inference engine for bayesian networks. Advances in neural information processing systems, 15, 2002

    2002

Show all 107 references
  1. [9]

    Blei, D. M. Probabilistic topic models. Communications of the ACM, 55 0 (4): 0 77--84, 2012

    2012

  2. [10]

    M., Kucukelbir, A., and McAuliffe, J

    Blei, D. M., Kucukelbir, A., and McAuliffe, J. D. Variational inference: A review for statisticians. Journal of the American statistical Association, 112 0 (518): 0 859--877, 2017

    2017

  3. [11]

    and Cranmer, K

    Brehmer, J. and Cranmer, K. Simulation-based inference methods for particle physics. In Artificial Intelligence for High Energy Physics, pp.\ 579--611. World Scientific, 2022

    2022

  4. [12]

    The promises and pitfalls of stochastic gradient langevin dynamics

    Brosse, N., Durmus, A., and Moulines, E. The promises and pitfalls of stochastic gradient langevin dynamics. Advances in Neural Information Processing Systems, 31, 2018

    2018

  5. [13]

    D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 0 1877--1901, 2020

    1901

  6. [14]

    C., Dasgupta, I., Kim, J., Kumaran, D., Lampinen, A

    Chan, S. C., Dasgupta, I., Kim, J., Kumaran, D., Lampinen, A. K., and Hill, F. Transformers generalize differently from information stored in context vs in weights. arXiv preprint arXiv:2210.05675, 2022

    2022 arXiv

  7. [15]

    Chen, R. T. Q. torchdiffeq, 2018. URL https://github.com/rtqichen/torchdiffeq

    2018

  8. [16]

    Stochastic gradient hamiltonian monte carlo

    Chen, T., Fox, E., and Guestrin, C. Stochastic gradient hamiltonian monte carlo. In International conference on machine learning, pp.\ 1683--1691. PMLR, 2014

    2014

  9. [17]

    The frontier of simulation-based inference

    Cranmer, K., Brehmer, J., and Louppe, G. The frontier of simulation-based inference. Proceedings of the National Academy of Sciences, 117 0 (48): 0 30055--30062, 2020

    2020

  10. [18]

    Inference suboptimality in variational autoencoders

    Cremer, C., Li, X., and Duvenaud, D. Inference suboptimality in variational autoencoders. In International conference on machine learning, pp.\ 1078--1086. PMLR, 2018

    2018

  11. [19]

    Flow matching in latent space

    Dao, Q., Phung, H., Nguyen, B., and Tran, A. Flow matching in latent space. arXiv preprint arXiv:2307.08698, 2023

    2023 arXiv

  12. [20]

    R., Gair, J., Macke, J

    Dax, M., Green, S. R., Gair, J., Macke, J. H., Buonanno, A., and Sch \"o lkopf, B. Real-time gravitational wave science with neural posterior estimation. Physical review letters, 127 0 (24): 0 241103, 2021

    2021

  13. [21]

    u rrer, M., Raymond, V., Wildberger, J., Macke, J. H., Buonanno, A., and Sch \

    Dax, M., Green, S. R., Gair, J., Gupte, N., P \"u rrer, M., Raymond, V., Wildberger, J., Macke, J. H., Buonanno, A., and Sch \"o lkopf, B. Real-time gravitational-wave inference for binary neutron stars using machine learning. arXiv preprint arXiv:2407.09602, 2024

    2024 arXiv

  14. [22]

    Laplace redux-effortless bayesian deep learning

    Daxberger, E., Kristiadi, A., Immer, A., Eschenhagen, R., Bauer, M., and Hennig, P. Laplace redux-effortless bayesian deep learning. Advances in Neural Information Processing Systems, 34: 0 20089--20103, 2021

    2021

  15. [23]

    A survey on in-context learning

    Dong, Q., Li, L., Dai, D., Zheng, C., Wu, Z., Chang, B., Sun, X., Xu, J., and Sui, Z. A survey on in-context learning. arXiv preprint arXiv:2301.00234, 2022

    2022 arXiv

  16. [24]

    Dormand, J. R. and Prince, P. J. A family of embedded runge-kutta formulae. Journal of computational and applied mathematics, 6 0 (1): 0 19--26, 1980

    1980

  17. [25]

    Gpts are gpts: An early look at the labor market impact potential of large language models

    Eloundou, T., Manning, S., Mishkin, P., and Rock, D. Gpts are gpts: An early look at the labor market impact potential of large language models. arXiv preprint arXiv:2303.10130, 2023

    2023 arXiv

  18. [26]

    Scaling rectified flow transformers for high-resolution image synthesis

    Esser, P., Kulal, S., Blattmann, A., Entezari, R., M \"u ller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F., et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning, 2024

    2024

  19. [27]

    Etzioni, R. D. and Kadane, J. B. Bayesian statistical methods in public health and medicine. Annual review of public health, 16 0 (1): 0 23--41, 1995

    1995

  20. [28]

    Regression models

    Fahrmeir, L., Kneib, T., Lang, S., Marx, B., Fahrmeir, L., Kneib, T., Lang, S., and Marx, B. Regression models. Springer, 2013

    2013

  21. [29]

    and Markram, H

    Fan, X. and Markram, H. A brief history of simulation neuroscience. Frontiers in neuroinformatics, 13: 0 32, 2019

    2019

  22. [30]

    Z., Boisbunon, A., Chambon, S., Chapel, L., Corenflos, A., Fatras, K., Fournier, N., et al

    Flamary, R., Courty, N., Gramfort, A., Alaya, M. Z., Boisbunon, A., Chambon, S., Chapel, L., Corenflos, A., Fatras, K., Fournier, N., et al. Pot: Python optimal transport. Journal of Machine Learning Research, 22 0 (78): 0 1--8, 2021

    2021

  23. [31]

    S., and Valiant, G

    Garg, S., Tsipras, D., Liang, P. S., and Valiant, G. What can transformers learn in-context? a case study of simple function classes. Advances in Neural Information Processing Systems, 35: 0 30583--30598, 2022

    2022

  24. [32]

    W., Rezende, D., and Eslami, S

    Garnelo, M., Rosenbaum, D., Maddison, C., Ramalho, T., Saxton, D., Shanahan, M., Teh, Y. W., Rezende, D., and Eslami, S. A. Conditional neural processes. In International conference on machine learning, pp.\ 1704--1713. PMLR, 2018 a

    2018

  25. [33]

    J., Eslami, S., and Teh, Y

    Garnelo, M., Schwarz, J., Rosenbaum, D., Viola, F., Rezende, D. J., Eslami, S., and Teh, Y. W. Neural processes. arXiv preprint arXiv:1807.01622, 2018 b

    2018 arXiv

  26. [34]

    D., Wildberger, J., Dax, M., Kofler, A., Angerhausen, D., Quanz, S

    Gebhard, T. D., Wildberger, J., Dax, M., Kofler, A., Angerhausen, D., Quanz, S. P., and Sch \"o lkopf, B. Flow matching for atmospheric retrieval of exoplanets: Where reliability meets adaptive noise levels. Astronomy & Astrophysics, 693: 0 A42, 2025

    2025

  27. [35]

    Givens, C. R. and Shortt, R. M. A class of wasserstein metrics for probability distributions. Michigan Mathematical Journal, 31 0 (2): 0 231--240, 1984

    1984

  28. [36]

    Gloeckler, M., Deistler, M., Weilbach, C., Wood, F., and Macke, J. H. All-in-one simulation-based inference. arXiv preprint arXiv:2404.09636, 2024

    2024 arXiv

  29. [37]

    M., Rasch, M

    Gretton, A., Borgwardt, K. M., Rasch, M. J., Sch \"o lkopf, B., and Smola, A. A kernel two-sample test. The Journal of Machine Learning Research, 13 0 (1): 0 723--773, 2012

    2012

  30. [38]

    Why do tree-based models still outperform deep learning on typical tabular data? Advances in neural information processing systems, 35: 0 507--520, 2022

    Grinsztajn, L., Oyallon, E., and Varoquaux, G. Why do tree-based models still outperform deep learning on typical tabular data? Advances in neural information processing systems, 35: 0 507--520, 2022

    2022

  31. [39]

    and van Ommen, T

    Gr \"u nwald, P. and van Ommen, T. Inconsistency of bayesian inference for misspecified linear models, and a proposal for repairing it. Bayesian Analysis, 12 0 (4): 0 1069--1103, 2017

    2017

  32. [40]

    Monte carlo sampling methods using markov chains and their applications

    Hastings, W. Monte carlo sampling methods using markov chains and their applications. Biometrika, 57 0 (1): 0 97--109, 1970

    1970

  33. [41]

    D., Gelman, A., et al

    Hoffman, M. D., Gelman, A., et al. The no-u-turn sampler: adaptively setting path lengths in hamiltonian monte carlo. J. Mach. Learn. Res., 15 0 (1): 0 1593--1623, 2014

    2014

  34. [42]

    Tabpfn: A transformer that solves small tabular classification problems in a second

    Hollmann, N., M \"u ller, S., Eggensperger, K., and Hutter, F. Tabpfn: A transformer that solves small tabular classification problems in a second. arXiv preprint arXiv:2207.01848, 2022

    2022 arXiv

  35. [43]

    u ller, S., Purucker, L., Krishnakumar, A., K \

    Hollmann, N., M \"u ller, S., Purucker, L., Krishnakumar, A., K \"o rfer, M., Hoo, S. B., Schirrmeister, R. T., and Hutter, F. Accurate predictions on small data with a tabular foundation model. Nature, 637 0 (8045): 0 319--326, 2025

    2025

  36. [44]

    B., M \"u ller, S., Salinas, D., and Hutter, F

    Hoo, S. B., M \"u ller, S., Salinas, D., and Hutter, F. The tabular foundation model tabpfn outperforms specialized time series forecasting models based on simple features. In NeurIPS Workshop on Time Series in the Age of Large Models, 2024

    2024

  37. [45]

    Meta-learning in neural networks: A survey

    Hospedales, T., Antoniou, A., Micaelli, P., and Storkey, A. Meta-learning in neural networks: A survey. IEEE transactions on pattern analysis and machine intelligence, 44 0 (9): 0 5149--5169, 2021

    2021

  38. [46]

    Batch normalization: Accelerating deep network training by reducing internal covariate shift

    Ioffe, S. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015

    2015 arXiv

  39. [47]

    D., and Wilson, A

    Izmailov, P., Vikram, S., Hoffman, M. D., and Wilson, A. G. G. What are bayesian neural network posteriors really like? In International conference on machine learning, pp.\ 4629--4640. PMLR, 2021

    2021

  40. [48]

    Semi-amortized variational autoencoders

    Kim, Y., Wiseman, S., Miller, A., Sontag, D., and Rush, A. Semi-amortized variational autoencoders. In International Conference on Machine Learning, pp.\ 2678--2687. PMLR, 2018

    2018

  41. [49]

    Kingma, D. P. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013

    2013 arXiv

  42. [50]

    Kingma, D. P. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  43. [51]

    P., Salimans, T., Jozefowicz, R., Chen, X., Sutskever, I., and Welling, M

    Kingma, D. P., Salimans, T., Jozefowicz, R., Chen, X., Sutskever, I., and Welling, M. Improved variational inference with inverse autoregressive flow. Advances in neural information processing systems, 29, 2016

    2016

  44. [52]

    General-purpose in-context learning by meta-learning transformers

    Kirsch, L., Harrison, J., Sohl-Dickstein, J., and Metz, L. General-purpose in-context learning by meta-learning transformers. arXiv preprint arXiv:2212.04458, 2022

    2022 arXiv

  45. [53]

    Kucukelbir, A., Tran, D., Ranganath, R., Gelman, A., and Blei, D. M. Automatic differentiation variational inference. Journal of machine learning research, 18 0 (14): 0 1--45, 2017

    2017

  46. [54]

    R., Fahmi, A., and Fenton, N

    Kyrimi, E., McLachlan, S., Dube, K., Neves, M. R., Fahmi, A., and Fenton, N. A comprehensive scoping review of bayesian networks in healthcare: Past, present and future. Artificial Intelligence in Medicine, 117: 0 102108, 2021

    2021

  47. [55]

    Lawley, D. N. and Maxwell, A. E. Factor analysis as a statistical method. Journal of the Royal Statistical Society. Series D (The Statistician), 12 0 (3): 0 209--229, 1962

    1962

  48. [56]

    A., Baydin, A

    Le, T. A., Baydin, A. G., and Wood, F. Inference compilation and universal probabilistic programming. In Artificial Intelligence and Statistics, pp.\ 1338--1348. PMLR, 2017

    2017

  49. [57]

    Preconditioned stochastic gradient langevin dynamics for deep neural networks

    Li, C., Chen, C., Carlson, D., and Carin, L. Preconditioned stochastic gradient langevin dynamics for deep neural networks. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, pp.\ 1788--1794, 2016

    2016

  50. [58]

    Generative modeling with bayesian sample inference

    Lienen, M., Kollovieh, M., and G \"u nnemann, S. Generative modeling with bayesian sample inference. arXiv preprint arXiv:2502.07580, 2025

    2025

  51. [59]

    Task-adaptive neural process for user cold-start recommendation

    Lin, X., Wu, J., Zhou, C., Pan, S., Cao, Y., and Wang, B. Task-adaptive neural process for user cold-start recommendation. In Proceedings of the Web Conference 2021, pp.\ 1306--1316, 2021

    2021

  52. [60]

    T., Ben-Hamu, H., Nickel, M., and Le, M

    Lipman, Y., Chen, R. T., Ben-Hamu, H., Nickel, M., and Le, M. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022

    2022 arXiv

  53. [61]

    Lopes, H. F. and West, M. Bayesian model assessment in factor analysis. Statistica Sinica, pp.\ 41--67, 2004

    2004

  54. [62]

    and Oquab, M

    Lopez-Paz, D. and Oquab, M. Revisiting classifier two-sample tests. arXiv preprint arXiv:1610.06545, 2016

    2016 arXiv

  55. [63]

    and Hutter, F

    Loshchilov, I. and Hutter, F. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983, 2016

    2016 arXiv

  56. [64]

    J., Bassetto, G., \"O cal, K., Nonnenmacher, M., and Macke, J

    Lueckmann, J.-M., Goncalves, P. J., Bassetto, G., \"O cal, K., Nonnenmacher, M., and Macke, J. H. Flexible statistical inference for mechanistic models of neural dynamics. Advances in neural information processing systems, 30, 2017

    2017

  57. [65]

    Benchmarking simulation-based inference

    Lueckmann, J.-M., Boelts, J., Greenberg, D., Goncalves, P., and Macke, J. Benchmarking simulation-based inference. In International conference on artificial intelligence and statistics, pp.\ 343--351. PMLR, 2021

    2021

  58. [66]

    and Vishnoi, N

    Mangoubi, O. and Vishnoi, N. K. Nonconvex sampling with the metropolis-adjusted langevin algorithm. In Conference on learning theory, pp.\ 2259--2293. PMLR, 2019

    2019

  59. [67]

    Margossian, C. C. and Blei, D. M. Amortized variational inference: When and why? arXiv preprint arXiv:2307.11018, 2023

    2023 arXiv

  60. [68]

    In-context parametric inference: Point or distribution estimators? arXiv preprint arXiv:2502.11617, 2025 a

    Mittal, S., Bengio, Y., Malkin, N., and Lajoie, G. In-context parametric inference: Point or distribution estimators? arXiv preprint arXiv:2502.11617, 2025 a

    2025 arXiv

  61. [69]

    L., Lajoie, G., Jaini, P., and Brubaker, M

    Mittal, S., Bracher, N. L., Lajoie, G., Jaini, P., and Brubaker, M. Amortized in-context bayesian posterior estimation. arXiv preprint arXiv:2502.06601, 2025 b

    2025 arXiv

  62. [70]

    P., Grabocka, J., and Hutter, F

    M \"u ller, S., Hollmann, N., Arango, S. P., Grabocka, J., and Hutter, F. Transformers can do bayesian-inference by meta-learning on prior-data. In Fifth Workshop on Meta-Learning at the Conference on Neural Information Processing Systems, 2021

    2021

  63. [71]

    Murphy, K. P. Probabilistic machine learning: Advanced topics. MIT press, 2023

    2023

  64. [72]

    Nelder, J. A. and Wedderburn, R. W. Generalized linear models. Journal of the Royal Statistical Society Series A: Statistics in Society, 135 0 (3): 0 370--384, 1972

    1972

  65. [73]

    Gpt-4 technical report

    OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    2023 arXiv

  66. [74]

    J., Mohamed, S., and Lakshminarayanan, B

    Papamakarios, G., Nalisnick, E., Rezende, D. J., Mohamed, S., and Lakshminarayanan, B. Normalizing flows for probabilistic modeling and inference. Journal of Machine Learning Research, 22 0 (57): 0 1--64, 2021 a

    2021

  67. [75]

    J., Mohamed, S., and Lakshminarayanan, B

    Papamakarios, G., Nalisnick, E., Rezende, D. J., Mohamed, S., and Lakshminarayanan, B. Normalizing flows for probabilistic modeling and inference. Journal of Machine Learning Research, 22 0 (57): 0 1--64, 2021 b

    2021

  68. [76]

    Scikit-learn: Machine learning in python

    Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., et al. Scikit-learn: Machine learning in python. the Journal of machine Learning research, 12: 0 2825--2830, 2011

    2011

  69. [77]

    and Xie, S

    Peebles, W. and Xie, S. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 4195--4205, 2023

    2023

  70. [78]

    Composable effects for flexible and accelerated probabilistic programming in numpyro

    Phan, D., Pradhan, N., and Jankowiak, M. Composable effects for flexible and accelerated probabilistic programming in numpyro. arXiv preprint arXiv:1912.11554, 2019

    1912 arXiv

  71. [79]

    J., Mohamed, S., and Wierstra, D

    Rezende, D. J., Mohamed, S., and Wierstra, D. Stochastic backpropagation and approximate inference in deep generative models. In International conference on machine learning, pp.\ 1278--1286. PMLR, 2014

    2014

  72. [80]

    Fairpfn: Transformers can do counterfactual fairness

    Robertson, J., Hollmann, N., Awad, N., and Hutter, F. Fairpfn: Transformers can do counterfactual fairness. In ICML 2024 Next Generation of AI Safety Workshop, 2024

    2024

  73. [81]

    G., Fortuin, V., Teh, Y

    Rudner, T. G., Fortuin, V., Teh, Y. W., and Gal, Y. On the connection between neural processes and gaussian processes with deep kernels. In Workshop on Bayesian Deep Learning, NeurIPS, pp.\ 14, 2018

    2018

  74. [82]

    Rummel, R. J. Applied factor analysis. Northwestern University Press, 1988

    1988

  75. [83]

    M., and Kuleshov, V

    Sahoo, S., Gokaslan, A., De Sa, C. M., and Kuleshov, V. Diffusion models with learned adaptive noise. Advances in Neural Information Processing Systems, 37: 0 105730--105779, 2024

    2024

  76. [84]

    Vart: variational regression trees

    Salazar, S. Vart: variational regression trees. Advances in Neural Information Processing Systems, 36: 0 45681--45693, 2023

    2023

  77. [85]

    Schmit, C. J. and Pritchard, J. R. Emulation of reionization simulations for bayesian inference of astrophysics parameters using neural networks. Monthly Notices of the Royal Astronomical Society, 475 0 (1): 0 1213--1223, 2018

    2018

  78. [86]

    and Narain, D

    Sohn, H. and Narain, D. Neural implementations of bayesian inference. Current Opinion in Neurobiology, 70: 0 121--129, 2021

    2021

  79. [87]

    Sommer, E., Wimmer, L., Papamarkou, T., Bothmann, L., Bischl, B., and R \"u gamer, D. Connecting the dots: Is mode-connectedness the key to feasible sample-based inference in bayesian neural networks? In Proceedings of the 41st International Conference on Machine Learning. PMLR, 2024

    2024

  80. [88]

    Microcanonical Langevin Ensembles: Advancing the Sampling of Bayesian Neural Networks

    Sommer, E., Robnik, J., Nozadze, G., Seljak, U., and R\"ugamer, D. Microcanonical Langevin Ensembles: Advancing the Sampling of Bayesian Neural Networks . In The Thirteenth International Conference on Learning Representations, 2025

    2025

  81. [89]

    and Ermon, S

    Song, Y. and Ermon, S. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019

    2019

  82. [90]

    P., Kumar, A., Ermon, S., and Poole, B

    Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020

    2011 arXiv

  83. [91]

    and Sutton, C

    Srivastava, A. and Sutton, C. Autoencoding variational inference for topic models. arXiv preprint arXiv:1703.01488, 2017

    2017 arXiv

  84. [92]

    Llama 2: Open foundation and fine-tuned chat models

    Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  85. [93]

    Vetter, J., Gloeckler, M., Gedon, D., and Macke, J. H. Effortless, simulation-efficient bayesian inference using tabular foundation models. arXiv preprint arXiv:2504.17660, 2025

    2025

  86. [94]

    Walker, S. G. Bayesian inference with misspecified models. Journal of statistical planning and inference, 143 0 (10): 0 1621--1633, 2013

    2013

  87. [95]

    Wang, X., Zhu, W., Saxon, M., Steyvers, M., and Wang, W. Y. Large language models are latent variable models: Explaining and finding good demonstrations for in-context learning. Advances in Neural Information Processing Systems, 36, 2024

    2024

  88. [96]

    and Blei, D

    Wang, Y. and Blei, D. Variational bayes under model misspecification. Advances in Neural Information Processing Systems, 32, 2019

    2019

  89. [97]

    and Teh, Y

    Welling, M. and Teh, Y. W. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-11), pp.\ 681--688. Citeseer, 2011

    2011

  90. [98]

    H., and Sch \"o lkopf, B

    Wildberger, J., Dax, M., Buchholz, S., Green, S., Macke, J. H., and Sch \"o lkopf, B. Flow matching for scalable simulation-based inference. Advances in Neural Information Processing Systems, 36, 2024

    2024

  91. [99]

    and Johnson, R

    Yeo, I.-K. and Johnson, R. A. A new family of power transformations to improve normality or symmetry. Biometrika, 87 0 (4): 0 954--959, 2000

    2000

  92. [100]

    Y., Gastegger, M., Jim \'e nez-Luna, J., Lewis, S., Satorras, V

    Yim, J., Campbell, A., Foong, A. Y., Gastegger, M., Jim \'e nez-Luna, J., Lewis, S., Satorras, V. G., Veeling, B. S., Barzilay, R., Jaakkola, T., et al. Fast protein backbone generation with se (3) flow matching. arXiv preprint arXiv:2310.05297, 2023

    arXiv

  93. [101]

    Y., Gastegger, M., Jim \'e nez-Luna, J., Lewis, S., Satorras, V

    Yim, J., Campbell, A., Mathieu, E., Foong, A. Y., Gastegger, M., Jim \'e nez-Luna, J., Lewis, S., Satorras, V. G., Veeling, B. S., No \'e , F., et al. Improved motif-scaffolding with se (3) flow matching. arXiv preprint arXiv:2401.04082, 2024

    2024 arXiv

  94. [102]

    Autoencoder and its various variants

    Zhai, J., Zhang, S., Chen, J., and He, Q. Autoencoder and its various variants. In 2018 IEEE international conference on systems, man, and cybernetics (SMC), pp.\ 415--419. IEEE, 2018

    2018

  95. [103]

    Flowturbo: Towards real-time flow-based image generation with velocity refiner

    Zhao, W., Shi, M., Yu, X., Zhou, J., and Lu, J. Flowturbo: Towards real-time flow-based image generation with velocity refiner. arXiv preprint arXiv:2409.18128, 2024

    2024 arXiv

  96. [104]

    Improved techniques for maximum likelihood estimation for diffusion odes

    Zheng, K., Lu, C., Chen, J., and Zhu, J. Improved techniques for maximum likelihood estimation for diffusion odes. In International Conference on Machine Learning, pp.\ 42363--42389. PMLR, 2023

    2023

  97. [105]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  98. [106]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  99. [107]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.