Pith. sign in

REVIEW 2 major objections 2 minor 87 references

Neural inference accumulates positive evidence over a graph to monotonically increase Dirichlet precision and recover Bayes-optimal predictions under ideal conditions.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-06-30 15:47 UTC pith:DFE3UJYD

load-bearing objection NBSR puts exact Dirichlet conjugate updates inside a Gumbel-routed DAG and gets monotonic precision growth for free, but the Bayes-optimal terminal claim still rests on unshown idealized capacity assumptions. the 2 major comments →

arxiv 2605.26147 v1 pith:DFE3UJYD submitted 2026-05-22 cs.LG

Neural Bayesian Sequential Routing

Yongchao Huang This is my paper
classification cs.LG
keywords neural networksBayesian inferencesequential routingDirichlet distributionevidence accumulationuncertainty quantificationearly exitingconjugate updates
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents Neural Bayesian Sequential Routing as a way to turn static neural forward passes into sequential evidence gathering on a directed acyclic graph. Neural experts pull positive evidence vectors from a knowledge oracle and add them as pseudo-counts to a Dirichlet belief state via exact conjugate updates. This setup yields proofs that total precision grows steadily along any path and that predictive variance stays bounded, which formalizes the idea of hypothesis sharpening over time. A reader would care because the same mechanism supplies built-in uncertainty measures for early stopping, out-of-distribution rejection, and cost-sensitive computation while still allowing end-to-end training through a Gumbel-Softmax estimator. Under strong capacity and optimization assumptions the final Dirichlet mean matches the Bayes-optimal conditional distribution.

Core claim

Within a Dirichlet-Categorical conjugate model, neural experts query a persistent global oracle to obtain positive evidence vectors that update a shared belief state by conjugate addition. Hard path-dependent routing is realized with a Gumbel-Softmax Straight-Through estimator. The resulting Dirichlet precision and entropy support uncertainty-aware controls such as entropy-based early exit and abstention. The central theorems establish that, whenever evidence extraction remains strictly positive, total Dirichlet precision increases monotonically along every valid trajectory and marginal predictive variance remains bounded; under idealized capacity and optimization the terminal Dirichlet expe

What carries the argument

Dirichlet belief state updated by exact conjugate addition of positive evidence vectors extracted by neural experts, with Gumbel-Softmax Straight-Through routing preserving surrogate gradients.

Load-bearing premise

Evidence extraction must remain strictly positive on every step and the network must satisfy idealized capacity plus perfect optimization so that the terminal Dirichlet expectation equals the Bayes-optimal distribution.

What would settle it

Run a controlled synthetic task with known Bayes-optimal posterior; observe whether any valid trajectory exhibits non-monotonic Dirichlet precision or whether the final Dirichlet mean deviates from the known optimum once capacity and optimization assumptions are met.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Predictive accuracy stays competitive with standard networks while exposing explicit routing traces and path-dependent evidence attribution.
  • Dirichlet precision and entropy supply direct mechanisms for entropy-based early exiting, out-of-distribution abstention, and cost-aware evidence acquisition.
  • The framework applies across visual categorization, medical diagnosis, language modeling, partially observable control, and Bayesian experimental design.
  • Resource-rational inference emerges naturally from the monotonic sharpening property without separate regularization terms.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The conjugate update structure could be swapped for other exponential-family priors if the evidence vectors remain positive.
  • The same routing-plus-precision mechanism might serve as an internal clock for deciding computation budgets in larger agent loops.
  • If the idealized recovery result holds only under perfect optimization, practical training schedules that approach but do not reach the optimum would produce a quantifiable gap to Bayes optimality.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The paper introduces Neural Bayesian Sequential Routing (NBSR), which models neural inference as sequential evidence accumulation over a hierarchical DAG within a Dirichlet-Categorical conjugate framework. Neural experts extract positive evidence vectors from a global knowledge oracle to update a Dirichlet belief state exactly, with Gumbel-Softmax Straight-Through enabling hard routing and end-to-end training. The central claims are a proof that total Dirichlet precision increases monotonically (and marginal predictive variance is bounded) under strictly positive evidence extraction, plus recovery of the Bayes-optimal conditional distribution at termination under idealized capacity and optimization assumptions. Empirical results across five domains are asserted to show competitive performance with added interpretability, uncertainty control, and resource rationality.

Significance. If the monotonicity result and Bayes-optimal recovery can be rigorously established without unattainable assumptions, the framework would provide a mathematically grounded approach to uncertainty-aware, path-dependent neural computation that integrates exact conjugate updates with learned routing. This could advance interpretable and resource-rational agentic systems, particularly where evidence attribution and early exiting matter.

major comments (2)
  1. [Abstract] Abstract (final paragraph): The claim that 'under idealized capacity and optimization assumptions, the terminal Dirichlet expectation recovers the Bayes-optimal conditional distribution' is central to the theoretical contribution, yet these assumptions are never formalized, nor is it demonstrated that they are attainable by the finite-capacity neural experts, Gumbel-Softmax routing, and gradient-based training described in the framework.
  2. [Abstract] Abstract: The stated proof that 'under strictly positive evidence extraction, total Dirichlet precision increases monotonically along any valid trajectory' is load-bearing for the 'hypothesis sharpening' formalization, but the manuscript provides no derivation steps, lemmas, or verification that the oracle and routing mechanism automatically enforce strictly positive extraction.
minor comments (2)
  1. The empirical evaluations are described only qualitatively ('competitive predictive performance') without quantitative metrics, error bars, baseline comparisons, or domain-specific results, which limits assessment of the practical claims.
  2. The 'persistent global knowledge oracle' is introduced as an invented entity without a precise interface definition or discussion of how its capacity interacts with the idealized assumptions.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments highlighting the need for greater rigor in the theoretical claims. We address each point below and will revise the manuscript accordingly to formalize assumptions and include derivations.

read point-by-point responses

  1. Referee: [Abstract] Abstract (final paragraph): The claim that 'under idealized capacity and optimization assumptions, the terminal Dirichlet expectation recovers the Bayes-optimal conditional distribution' is central to the theoretical contribution, yet these assumptions are never formalized, nor is it demonstrated that they are attainable by the finite-capacity neural experts, Gumbel-Softmax routing, and gradient-based training described in the framework.

    Authors: We agree the assumptions require explicit formalization. The revised manuscript will add a dedicated theoretical subsection defining the idealized assumptions (infinite expert capacity as universal approximators of the evidence function, and convergence to the global optimum of the routing objective). We will also include a discussion of attainability, noting that finite networks provide approximations whose error can be bounded under standard neural network approximation theory, with the Gumbel-Softmax and conjugate updates preserving the recovery property in the limit. This will clarify the scope without overstating practical attainment. revision: yes

  2. Referee: [Abstract] Abstract: The stated proof that 'under strictly positive evidence extraction, total Dirichlet precision increases monotonically along any valid trajectory' is load-bearing for the 'hypothesis sharpening' formalization, but the manuscript provides no derivation steps, lemmas, or verification that the oracle and routing mechanism automatically enforce strictly positive extraction.

    Authors: The monotonicity follows directly from the Dirichlet conjugate update rule when evidence vectors are strictly positive. We will insert the full derivation as a lemma in the main text (with steps showing precision = sum of Dirichlet parameters strictly increases by the L1 norm of the evidence vector). For enforcement, the oracle is defined to output positive vectors by construction (as pseudo-counts), and routing selects only such experts; we will add an explicit proposition and verification in the methods. This supplies the missing steps and confirmation. revision: yes

Circularity Check

0 steps flagged

No significant circularity; claims rest on conjugate properties and explicit assumptions

full rationale

The derivation chain centers on Dirichlet-Categorical conjugate updates for evidence accumulation, with the monotonic precision increase and bounded variance proved under the stated condition of strictly positive evidence extraction. The Bayes-optimal recovery is separately conditioned on idealized capacity and optimization assumptions that are not derived from or equivalent to the model's fitted parameters or routing mechanism. No self-definitional reductions, fitted inputs renamed as predictions, or load-bearing self-citations appear in the provided abstract or claims; the framework introduces new components (Gumbel-Softmax routing, oracle queries) whose properties are analyzed via standard conjugate mathematics rather than reducing to the inputs by construction.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 1 invented entities

The framework rests on standard conjugate-update properties of the Dirichlet distribution and on the existence of a persistent global knowledge oracle that supplies positive evidence vectors; no numerical free parameters are named in the abstract, but the idealized capacity assumption functions as an ad-hoc modeling premise.

axioms (2)
  • standard math Dirichlet-Categorical conjugacy permits exact pseudo-count addition that preserves the Dirichlet form
    Invoked in the description of evidence-vector updates (abstract, paragraph 2)
  • standard math Gumbel-Softmax Straight-Through estimator supplies unbiased surrogate gradients for hard categorical routing
    Used to enable end-to-end training with discrete path selection (abstract, paragraph 2)
invented entities (1)
  • persistent global knowledge oracle no independent evidence
    purpose: Supplies positive evidence vectors that act as pseudo-counts for Dirichlet updates
    New component introduced to enable sequential evidence accumulation across the DAG

pith-pipeline@v0.9.1-grok · 5789 in / 1441 out tokens · 31922 ms · 2026-06-30T15:47:38.607657+00:00 · methodology

0 comments
read the original abstract

Human decision-making is sequential and uncertainty-aware, yet standard neural networks often rely on static, dense forward computation with limited visibility into evidence acquisition, uncertainty evolution, or when computation should stop. We introduce \textbf{Neural Bayesian Sequential Routing (NBSR)}, a framework that models neural inference as active evidence accumulation over a hierarchical Directed Acyclic Graph (DAG). Within a Dirichlet--Categorical conjugate framework, neural experts query a persistent global knowledge oracle to extract positive evidence vectors, which act as pseudo-counts and update a Dirichlet belief state by exact conjugate addition. Coupled with a Gumbel-Softmax Straight-Through estimator, this update enables hard, path-dependent routing while preserving surrogate gradients for end-to-end training. The resulting Dirichlet precision and entropy provide mechanisms for uncertainty quantification, entropy-based early exiting, OOD abstention, and cost-aware evidence acquisition. We prove that, under strictly positive evidence extraction, total Dirichlet precision increases monotonically along any valid trajectory and marginal predictive variance is bounded, formalizing sequential ``hypothesis sharpening''; under idealized capacity and optimization assumptions, the terminal Dirichlet expectation recovers the Bayes-optimal conditional distribution. Empirical evaluations across visual categorization, structured medical diagnosis, language modeling, partially observable control, and cost-aware Bayesian experimental design show that NBSR achieves competitive predictive performance while providing transparent routing traces, path-dependent evidence attribution, uncertainty-aware decision control, and resource-rational inference. Overall, NBSR offers a mathematically grounded framework for interpretable, modular, and resource-rational agentic AI.

Figures

Figures reproduced from arXiv: 2605.26147 by Yongchao Huang.

Figure 1
Figure 1. Figure 1: Computational graph of the NBSR process. The total number of sequential layers [PITH_FULL_IMAGE:figures/full_fig_p011_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Evolution of the 2D decision boundary and epistemic uncertainty on the Iris dataset. [PITH_FULL_IMAGE:figures/full_fig_p023_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Evolution of the two loss components over 150 epochs: the semantic NLL strictly [PITH_FULL_IMAGE:figures/full_fig_p026_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Evolution of the belief state αt , averaged across the CIFAR-10 test set. (Left) The monotonic increase of the mean total concentration α (t) 0 across tree depth t ∈ {0, 1, 2} as proven in Theorem 1. (Right) The corresponding sequential collapse in mean differential entropy H(αt). Accuracy-Efficiency Pareto Frontier. To evaluate the practical efficacy of our dynamic early-exiting strategy on the test set, … view at source ↗
Figure 5
Figure 5. Figure 5: Accuracy-Efficiency Pareto Frontier demonstrating dynamic early-exiting capabilities [PITH_FULL_IMAGE:figures/full_fig_p028_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Terminal Entropy Distribution demonstrating robust Out-Of-Distribution (OOD) [PITH_FULL_IMAGE:figures/full_fig_p028_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: A sample diagnostic audit trail for Patient 0. The graph explicitly tracks the sequential [PITH_FULL_IMAGE:figures/full_fig_p029_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Comparison of Feature Attribution. (Left) XGBoost provides a global, population [PITH_FULL_IMAGE:figures/full_fig_p030_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Illustration of the synthetic POMDP keyframe sequence. Rather than a contiguous [PITH_FULL_IMAGE:figures/full_fig_p035_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: The Imitation Learning (Behavioral Cloning) paradigm under partial observability. [PITH_FULL_IMAGE:figures/full_fig_p036_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Illustration of the Autoregressive NBSR (AR-NBSR) framework. Unlike a static feed [PITH_FULL_IMAGE:figures/full_fig_p042_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: AR-NBSR Training Convergence. The simultaneous decline in NLL (Blue) and [PITH_FULL_IMAGE:figures/full_fig_p043_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Cost-Accuracy Pareto Frontier. AR-NBSR (Blue) demonstrates superior resource [PITH_FULL_IMAGE:figures/full_fig_p044_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Dynamic Entropy Reduction for a single patient trajectory. The agent sequentially [PITH_FULL_IMAGE:figures/full_fig_p044_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Training loss and test accuracy dynamics for the standard Flat ResNet-18 baseline [PITH_FULL_IMAGE:figures/full_fig_p063_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Training loss and test accuracy dynamics for the Sparse MoE (Soft Routing) baseline [PITH_FULL_IMAGE:figures/full_fig_p064_16.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

87 extracted references · 87 canonical work pages

  1. [1]

    Mart´ ın Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Man´ e, Rajat Monga, Sherry Moore, Derek M...

    work page

  2. [2]

    Software available from tensorflow.org

    work page

  3. [3]

    Bengio, P

    Y. Bengio, P. Simard, and P. Frasconi. Learning long-term dependencies with gradient descent is difficult.Trans. Neur. Netw., 5(2):157–166, March 1994

    work page 1994

  4. [4]

    Estimating or propagating gradi- ents through stochastic neurons for conditional computation, 2013

    Yoshua Bengio, Nicholas L´ eonard, and Aaron Courville. Estimating or propagating gradi- ents through stochastic neurons for conditional computation, 2013

    work page 2013

  5. [5]

    Bishop.Pattern Recognition and Machine Learning

    Christopher M. Bishop.Pattern Recognition and Machine Learning. Information Science and Statistics. Springer, New York, NY, 1 edition, 2006

    work page 2006

  6. [6]

    Weight uncer- tainty in neural networks

    Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncer- tainty in neural networks. InProceedings of the 32nd International Conference on Machine Learning - Volume 37, ICML’15, page 1613–1622. JMLR.org, 2015

    work page 2015

  7. [7]

    Meltzoff, James Woodward, and Laura E

    Elizabeth Baraff Bonawitz, Darlene Ferranti, Rebecca Saxe, Alison Gopnik, Andrew N. Meltzoff, James Woodward, and Laura E. Schulz. Just do it? investigating the gap between prediction and action in toddlers’ causal inferences.Cognition, 115(1):104–117, 2010

    work page 2010

  8. [8]

    Friedman, Richard A

    Leo Breiman, Jerome H. Friedman, Richard A. Olshen, and Charles J. Stone.Classification and Regression Trees. Chapman and Hall/CRC, 1 edition, 1984

    work page 1984

  9. [9]

    Xgboost: A scalable tree boosting system

    Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. InProceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’16, page 785–794, New York, NY, USA, 2016. Association for Computing Machinery

    work page 2016

  10. [10]

    Collins, Ilia Sucholutsky, Umang Bhatt, et al

    Katherine M. Collins, Ilia Sucholutsky, Umang Bhatt, et al. Building machines that learn and think with people.Nature Human Behaviour, 8:1851–1863, 2024

    work page 2024

  11. [11]

    Approximation by superpositions of a sigmoidal function.Mathematics of Control, Signals, and Systems, 2:303–314, 1989

    George Cybenko. Approximation by superpositions of a sigmoidal function.Mathematics of Control, Signals, and Systems, 2:303–314, 1989

    work page 1989

  12. [12]

    ImageNet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database . In2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPR Workshops), pages 248–255, Los Alamitos, CA, USA, June 2009. IEEE Computer Society

    work page 2009

  13. [13]

    Calibration of pre-trained transformers, 2020

    Shrey Desai and Greg Durrett. Calibration of pre-trained transformers, 2020

    work page 2020

  14. [14]

    An image is worth 16x16 words: Transformers for image recognition at scale, 2021

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale, 2021

    work page 2021

  15. [15]

    von mises conditions revisited.The Annals of Probability, 21(3):1310–1328, 1993

    Michael Falk and Frank Marohn. von mises conditions revisited.The Annals of Probability, 21(3):1310–1328, 1993. Intro|Lite|Pre|Method|Theory|Tests|Disc|Con|App

    work page 1993

  16. [16]

    Hierarchical neural story generation

    Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. In Iryna Gurevych and Yusuke Miyao, editors,Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 889–898, Melbourne, Australia, July 2018. Association for Computational Linguistics

    work page 2018

  17. [17]

    Switch transformers: scaling to trillion parameter models with simple and efficient sparsity.J

    William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: scaling to trillion parameter models with simple and efficient sparsity.J. Mach. Learn. Res., 23(1), January 2022

    work page 2022

  18. [18]

    R. A. Fisher and L. H. C. Tippett. Limiting forms of the frequency distribution of the largest and smallest member of a sample.Mathematical Proceedings of the Cambridge Philosophical Society, 24(2):180–190, 1928

    work page 1928

  19. [19]

    Variational bayesian optimal experimental design

    Adam Foster, Martin Jankowiak, Elias Bingham, Paul Horsfall, Yee Whye Teh, Thomas Rainforth, and Noah Goodman. Variational bayesian optimal experimental design. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alch´ e-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019

    work page 2019

  20. [20]

    Sur la loi de probabilit´ e de l’´ ecart maximum.Annales de la Soci´ et´ e Polonaise de Math´ ematique, 6(1):93–116, 1927

    Maurice Fr´ echet. Sur la loi de probabilit´ e de l’´ ecart maximum.Annales de la Soci´ et´ e Polonaise de Math´ ematique, 6(1):93–116, 1927

    work page 1927

  21. [21]

    Dropout as a bayesian approximation: representing model uncertainty in deep learning

    Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: representing model uncertainty in deep learning. InProceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48, ICML’16, page 1050–1059. JMLR.org, 2016

    work page 2016

  22. [22]

    Carlin, Hal S

    Andrew Gelman, John B. Carlin, Hal S. Stern, David B. Dunson, Aki Vehtari, and Don- ald B. Rubin.Bayesian Data Analysis. Chapman and Hall/CRC, New York, 3 edition, 2013

    work page 2013

  23. [23]

    Gil-Leyva, Antonio Lijoi, Rams´ es H

    Mar´ ıa F. Gil-Leyva, Antonio Lijoi, Rams´ es H. Mena, and Igor Pr¨ unster. Markov stick- breaking processes, 2026

    work page 2026

  24. [24]

    B. V. Gnedenko. Sur la distribution limite du terme maximum d’une s´ erie al´ eatoire.Annals of Mathematics, 44(3):423–453, 1943

    work page 1943

  25. [25]

    Gemini 3 developer guide, 2026

    Google. Gemini 3 developer guide, 2026. Google AI for Developers

    work page 2026

  26. [26]

    Adaptive computation time for recurrent neural networks, 2017

    Alex Graves. Adaptive computation time for recurrent neural networks, 2017

    work page 2017

  27. [27]

    Weinberger

    Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. InProceedings of the 34th International Conference on Machine Learning - Volume 70, ICML’17, page 1321–1330. JMLR.org, 2017

    work page 2017

  28. [28]

    Dy- namic Neural Networks: A Survey .IEEE Transactions on Pattern Analysis & Machine Intelligence, 44(11):7436–7456, November 2022

    Yizeng Han, Gao Huang, Shiji Song, Le Yang, Honghui Wang, and Yulin Wang. Dy- namic Neural Networks: A Survey .IEEE Transactions on Pattern Analysis & Machine Intelligence, 44(11):7436–7456, November 2022

    work page 2022

  29. [29]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016

    work page 2016

  30. [30]

    A baseline for detecting misclassified and out-of- distribution examples in neural networks

    Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of- distribution examples in neural networks. InInternational Conference on Learning Repre- sentations, 2017. Intro|Lite|Pre|Method|Theory|Tests|Disc|Con|App

    work page 2017

  31. [31]

    Geoffrey E. Hinton. Training products of experts by minimizing contrastive divergence. Neural Comput., 14(8):1771–1800, August 2002

    work page 2002

  32. [32]

    Long short-term memory.Neural Comput., 9(8):1735–1780, November 1997

    Sepp Hochreiter and J¨ urgen Schmidhuber. Long short-term memory.Neural Comput., 9(8):1735–1780, November 1997

    work page 1997

  33. [33]

    The curious case of neural text degeneration, 2020

    Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration, 2020

    work page 2020

  34. [34]

    Approximation capabilities of multilayer feedforward networks.Neural Net- works, 4(2):251–257, 1991

    Kurt Hornik. Approximation capabilities of multilayer feedforward networks.Neural Net- works, 4(2):251–257, 1991

    work page 1991

  35. [35]

    Sampling via gaussian mixture approximations, 2025

    Yongchao Huang. Sampling via gaussian mixture approximations, 2025

    work page 2025

  36. [36]

    Vjepa: Variational joint embedding predictive architectures as proba- bilistic world models, 2026

    Yongchao Huang. Vjepa: Variational joint embedding predictive architectures as proba- bilistic world models, 2026

    work page 2026

  37. [37]

    On the information bottleneck of VJEPA.OpenReview, May 2026

    Yongchao Huang. On the information bottleneck of VJEPA.OpenReview, May 2026

    work page 2026

  38. [38]

    Imitation learning: A survey of learning methods.ACM Comput

    Ahmed Hussein, Mohamed Medhat Gaber, Eyad Elyan, and Chrisina Jayne. Imitation learning: A survey of learning methods.ACM Comput. Surv., 50(2), April 2017

    work page 2017

  39. [39]

    Gibbs sampling methods for stick-breaking priors

    Hemant Ishwaran and Lancelot F James. Gibbs sampling methods for stick-breaking priors. Journal of the American Statistical Association, 96(453):161–173, 2001

    work page 2001

  40. [40]

    Jacobs, Michael I

    Robert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. Adaptive mixtures of local experts.Neural Computation, 3(1):79–87, 1991

    work page 1991

  41. [41]

    Categorical reparameterization with gumbel- softmax

    Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel- softmax. InInternational Conference on Learning Representations, 2017

    work page 2017

  42. [42]

    Survey of hallucination in natural language generation.ACM Comput

    Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation.ACM Comput. Surv., 55(12), March 2023

    work page 2023

  43. [43]

    Jordan and R.A

    M.I. Jordan and R.A. Jacobs. Hierarchical mixtures of experts and the em algorithm. InProceedings of 1993 International Conference on Neural Networks (IJCNN-93-Nagoya, Japan), volume 2, pages 1339–1344 vol.2, 1993

    work page 1993

  44. [44]

    Artificial Intelligence: Foundations, Theory, and Algorithms

    Audun Jøsang.Subjective Logic: A Formalism for Reasoning Under Uncertainty. Artificial Intelligence: Foundations, Theory, and Algorithms. Springer Cham, 1 edition, 2016

    work page 2016

  45. [45]

    Prentice Hall, 1st edition, 2000

    Daniel Jurafsky and James H Martin.Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition. Prentice Hall, 1st edition, 2000

    work page 2000

  46. [46]

    Littman, and Anthony R

    Leslie Pack Kaelbling, Michael L. Littman, and Anthony R. Cassandra. Planning and acting in partially observable stochastic domains.Artificial Intelligence, 101(1):99–134, 1998

    work page 1998

  47. [47]

    Penguin Books, London, 2011

    Daniel Kahneman.Thinking, Fast and Slow. Penguin Books, London, 2011

    work page 2011

  48. [48]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017

    work page 2017

  49. [49]

    SAGE Publications Ltd, apr 2018

    Ben Lambert.A Student’s Guide to Bayesian Statistics. SAGE Publications Ltd, apr 2018

    work page 2018

  50. [50]

    A path towards autonomous machine intelligence version 0.9.2, 2022-06-27

    Yann LeCun. A path towards autonomous machine intelligence version 0.9.2, 2022-06-27. Open Review, 62(1):1–62, 2022. Intro|Lite|Pre|Method|Theory|Tests|Disc|Con|App

    work page 2022

  51. [51]

    Deep learning.Nature, 521:436–444, 2015

    Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning.Nature, 521:436–444, 2015

    work page 2015

  52. [52]

    Zachary C. Lipton. The mythos of model interpretability.Commun. ACM, 61(10):36–43, September 2018

    work page 2018

  53. [53]

    Lundberg and Su-In Lee

    Scott M. Lundberg and Su-In Lee. A unified approach to interpreting model predictions. InProceedings of the 31st International Conference on Neural Information Processing Sys- tems, NIPS’17, page 4768–4777, Red Hook, NY, USA, 2017. Curran Associates Inc

    work page 2017

  54. [54]

    Maddison, Andriy Mnih, and Yee Whye Teh

    Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A con- tinuous relaxation of discrete random variables. InInternational Conference on Learning Representations, 2017

    work page 2017

  55. [55]

    Maddison, Daniel Tarlow, and Tom Minka

    Chris J. Maddison, Daniel Tarlow, and Tom Minka. A* sampling. InProceedings of the 28th International Conference on Neural Information Processing Systems - Volume 2, NIPS’14, page 3086–3094, Cambridge, MA, USA, 2014. MIT Press

    work page 2014

  56. [56]

    Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz

    Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a large annotated corpus of English: The Penn Treebank.Computational Linguistics, 19(2):313– 330, 1993

    work page 1993

  57. [57]

    Bayesian neural networks in mortality modelling

    Patrick Moehrke and Yongchao Huang. Bayesian neural networks in mortality modelling. The Actuarial, 2024

    work page 2024

  58. [58]

    Bissacco, Bo Wu, and A

    Yuval Netzer, Tao Wang, Adam Coates, A. Bissacco, Bo Wu, and A. Ng. Reading digits in natural images with unsupervised feature learning, 2011

    work page 2011

  59. [59]

    Maxime Oquab, Timoth´ ee Darcet, Th´ eo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Jegou, Julien Mairal, Patrick...

    work page 2024

  60. [60]

    Sculley, Sebastian Nowozin, Joshua V

    Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D. Sculley, Sebastian Nowozin, Joshua V. Dillon, Balaji Lakshminarayanan, and Jasper Snoek.Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift. Curran Associates Inc., Red Hook, NY, USA, 2019

    work page 2019

  61. [61]

    On the difficulty of training recurrent neural networks

    Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. InProceedings of the 30th International Conference on International Con- ference on Machine Learning - Volume 28, ICML’13, page III–1310–III–1318. JMLR.org, 2013

    work page 2013

  62. [62]

    Curran Associates Inc., Red Hook, NY, USA, 2019

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmai- son, Andreas K¨ opf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.PyTorch: an imperative style, high-perfo...

    work page 2019

  63. [63]

    Pomerleau

    Dean A. Pomerleau. Alvinn: an autonomous land vehicle in a neural network. InPro- ceedings of the 2nd International Conference on Neural Information Processing Systems, NIPS’88, page 305–313, Cambridge, MA, USA, 1988. MIT Press. Intro|Lite|Pre|Method|Theory|Tests|Disc|Con|App

    work page 1988

  64. [64]

    Puterman.Markov Decision Processes: Discrete Stochastic Dynamic Program- ming

    Martin L. Puterman.Markov Decision Processes: Discrete Stochastic Dynamic Program- ming. Wiley Series in Probability and Statistics. John Wiley & Sons, 1994

    work page 1994

  65. [65]

    Learning transferable visual models from natural language supervision, 2021

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021

    work page 2021

  66. [66]

    Harvard Univer- sity Press, Boston, 1961

    Howard Raiffa and Robert Schlaifer.Applied Statistical Decision Theory. Harvard Univer- sity Press, Boston, 1961

    work page 1961

  67. [67]

    ”why should i trust you?”: Ex- plaining the predictions of any classifier

    Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. ”why should i trust you?”: Ex- plaining the predictions of any classifier. InProceedings of the 22nd ACM SIGKDD Inter- national Conference on Knowledge Discovery and Data Mining, KDD ’16, page 1135–1144, New York, NY, USA, 2016. Association for Computing Machinery

    work page 2016

  68. [68]

    Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead.Nature Machine Intelligence, 1:206–215, 2019

    Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead.Nature Machine Intelligence, 1:206–215, 2019

    work page 2019

  69. [69]

    Paola Sebastiani and Henry P. Wynn. Maximum entropy sampling and optimal bayesian experimental design.Journal of the Royal Statistical Society. Series B (Statistical Method- ology), 62(1):145–157, 2000

    work page 2000

  70. [70]

    Evidential deep learning to quantify classification uncertainty

    Murat Sensoy, Lance Kaplan, and Melih Kandemir. Evidential deep learning to quantify classification uncertainty. InProceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS’18, page 3183–3193, Red Hook, NY, USA, 2018. Curran Associates Inc

    work page 2018

  71. [71]

    Outrageously large neural networks: The sparsely-gated mixture- of-experts layer

    Noam Shazeer, *Azalia Mirhoseini, *Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture- of-experts layer. InInternational Conference on Learning Representations, 2017

    work page 2017

  72. [72]

    Opening the black box of deep neural networks via information, 2017

    Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via information, 2017

    work page 2017

  73. [73]

    La distribution de la plus grande de n valeurs.Revue de Math´ ematiques de l’Union Interbalkanique, 1:141–160, 1936

    Richard von Mises. La distribution de la plus grande de n valeurs.Revue de Math´ ematiques de l’Union Interbalkanique, 1:141–160, 1936

    work page 1936

  74. [74]

    A. Wald. Sequential tests of statistical hypotheses.The Annals of Mathematical Statistics, 16(2):117–186, 1945

    work page 1945

  75. [75]

    S. R. Waterhouse and A. J. Robinson. Constructive algorithms for hierarchical mixtures of experts. InProceedings of the 9th International Conference on Neural Information Processing Systems, NIPS’95, page 584–590, Cambridge, MA, USA, 1995. MIT Press

    work page 1995

  76. [76]

    Sharpening

    George Kingsley Zipf.Human behavior and the principle of least effort: an introduction to human ecology. Addison-Wesley, 1949. A The Dirichlet Distribution This section provides a formal overview of the Dirichlet distribution, detailing the mathematical properties that make it the foundational engine for the uncertainty-tracking and evidence- accumulating...

    work page 1949

  77. [77]

    The Irreducible Noise:the expectation of the first term depends entirely on the true distribution and representsthe inherent uncertainty in the data generation process(the Shannon entropy): Ey∼P ∗ −logP ∗(y|x) =H(P ∗) (36)

    work page

  78. [78]

    This is precisely the Kullback-Leibler (KL) divergence: Ey∼P ∗ logP ∗(y|x)−log ¯PG(y|x) =D KL P ∗∥ ¯PG (37)

    The Bias:the expectation of the second term measures the distance between the true distribution and the model’s average prediction. This is precisely the Kullback-Leibler (KL) divergence: Ey∼P ∗ logP ∗(y|x)−log ¯PG(y|x) =D KL P ∗∥ ¯PG (37)

    work page

  79. [79]

    The Variance:The expectation of the third term captures how much the predictions from models trained on specific datasets fluctuate around the average model prediction: Ey∼P ∗ log ¯PG(y|x)−logP G(y|x;D) = X y P ∗(y|x) log ¯PG(y|x) PG(y|x;D) (38) In standard generalized bias-variance decompositions for likelihood estimators, it is common practice to approx...

    work page

  80. [80]

    The Role of Intermediate Nodes: Routing Gates vs. Sequential Evidence Extrac- torsIn classical differentiable decision trees and hierarchical MoEs, the graph is composed of two strictly distinct types of nodes: •Internal Nodes (Routers):these nodes do not produce any classification output. They act purely as routing gates that calculate transitional proba...

    work page

Showing first 80 references.