Pith. sign in

REVIEW 3 major objections 5 minor 75 references

The paper claims that the Bellman equation is a theorem about compatible structures, not a primitive axiom: it emerges from three conditions, and the three dualities they generate organize a large body of sequential-decision methods.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Bellman's equation is derived from three compatibility conditions (D, R, A), and three known RL dualities are presented as faces of one unifying triangle.

T0 review reviewed 2026-08-01 challenge →

load-bearing objection A useful formalization of when Bellman recursion holds, with two solid theorems and one overclaimed duality restoration that needs a fix before I'd fully trust the unification. the 3 major comments →

arxiv 2607.18077 v1 pith:EF6QVJ4X submitted 2026-07-20 cs.LG cs.AI

Generalised Bellman recurrence and three dualities in sequential decision-making

Fernando E. Rosas , David Hyland , Daniel Polani This is my paper
classification cs.LG cs.AI
keywords Bellman equationprinciple of optimalitysequential decision-makingdualitydistortion aggregatorCVaRrisk-sensitive controlreinforcement learning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks what gives the Bellman equation its form, and answers that it is the mutual consistency of three structural conditions: dynamics that compress through a sufficient statistic, returns that decompose recursively, and an aggregation functional that composes with both. Under those conditions the value function satisfies a generalized Bellman recursion, and with monotonicity and attainment the optimal value function is its fixed point. The same three conditions generate three dualities — probability/return, return/aggregation, and aggregation/probability — which the paper presents as a single construction with many previously separate methods as special cases. A sympathetic reader would care because this turns the question of whether a sequential decision problem is tractable by dynamic programming into a checkable structural question, and explains why state augmentation and deformation of returns or dynamics restore tractability when one condition fails.

Core claim

On the paper's own terms, the central claim is that Bellman recursion is not assumed but derived. If the environment admits a unifilar sufficient statistic (Condition D), the return composes through a local reward and an associative operation (Condition R), and the aggregation functional satisfies kernel-composition and return-composition compatibility (Condition A), then the value of any policy that factors through the joint state is a fixed point of a generalized Bellman operator (Theorem 1), and with monotonicity and attainment the optimal value is the fixed point of the corresponding optimality operator (Theorem 2). The paper further claims that the three conditions are pairwise linked b

What carries the argument

The carrying mechanism is the joint Bellman state z = (z_D, z_R), where z_D is a dynamics-sufficient statistic and z_R a return-sufficient statistic, together with the backup operator B and the policy-evaluation operator T^π and optimality operator T^*. Conditions D, R, and A are defined so that the trajectory law and the tail return both factor step-by-step and the aggregation commutes with those factorizations; the proof of Theorem 1 is exactly the sequence of applying A1 to the two-stage law, A2 to the local return, and the induced factorized value. The dualities arise from holding one vertex of the triple (probability, return, aggregation) fixed and exchanging the other two.

Load-bearing premise

The load-bearing premise is that a nonlinear aggregation rewritten as a trajectory reweighting can be applied step by step from local data; the paper concedes the reweighting for rank-dependent distortions depends on the global rank of the whole trajectory and does not prove it factors into local transitions.

What would settle it

Take a standard MDP, a distortion aggregator that is not a fixed deformation (e.g., CVaR at level alpha), and implement the deformed expectation from the paper's Eq. (80) as the policy objective; if value iteration on the reweighted Bellman recursion either fails to converge, converges to a value different from the true CVaR optimum, or requires knowing the full-return quantile before the recursion can start, then the claimed restoration of Bellman recursion by representation duality is false.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the paper is right, the Bellman expectation and optimality equations are corollaries of Conditions D, R, A, not separate assumptions; verifying these conditions for a new decision problem is the route to knowing whether a recursion exists.
  • When one condition fails, the account predicts that tractability can often be restored by augmenting the state (e.g., adding a clock or belief state) or by deforming returns or dynamics, which organizes previously ad-hoc recovery strategies.
  • The assessment duality implies that risk-sensitive, robust, and soft/entropy-regularized Bellman formulas are all the same recursion with different conjugated local rewards, so algorithms developed for one formulation transfer directly to the others.
  • The representation duality implies that a family of distortion-type aggregators (including CVaR) can in principle be replaced by expectation under a deformed trajectory distribution, locating nonlinear aggregation entirely in the measure.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • An extension the paper leaves implicit: the three conditions could be used as a constructive checklist for designing new tractable decision problems; one would deliberately choose dynamics, returns, and aggregation so that D, R, and A hold on a common state.
  • A testable extension: for a given distortion function g, check whether the deformed measure dQ/dP = g'(S_J(J)) from the paper's Eq. (80) factors into a product of local Markov kernels; for rank-dependent distortions such as CVaR the paper itself concedes this is not a fixed deformation, so a natural test is whether value iteration under the deformed measure converges to the correct fixed point on
  • If the paper's structural account is correct, similar dualities should appear in other aggregation-based decision settings, such as multiobjective or lexicographic preferences, where an analogous triangle could be derived by replacing the return lattice and the aggregation accordingly.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes three structural conditions — Condition D (history sufficiency of the dynamics), Condition R (recursive decomposability of the return), and Condition A (compatibility of the aggregation functional with kernel composition and with return composition) — and shows that when all three hold on a common state, value functions satisfy a generalised Bellman recursion (Theorem 1) and, under additional monotonicity and attainment assumptions, the optimal value function satisfies a generalised Bellman optimality equation (Theorem 2). The paper then presents three dualities — object duality (probability–return), assessment duality (return–aggregation), and representation duality (dynamics–aggregation) — and claims that these arise from a single construction, with representation duality restoring Bellman recursion for distortion-type aggregators such as CVaR.

Significance. The conditional theorems in Section 2 are carefully stated and proved in Appendix A, and the derivations in Appendix D that recover soft, risk-sensitive, and robust Bellman equations from a common Gibbs identity are a useful and mostly correct synthesis. If the full unification claim were established, the paper would provide a valuable organising framework for several RL and decision-theoretic literatures. However, the representation-duality restoration in Section 3.3 is not established: the deformed measure constructed in Appendix E is rank-dependent and generally does not factor through local kernels, so Theorems 1–2 do not apply to it. This weakens the paper's central claim that the three dualities are unified through a single construction.

major comments (3)
  1. [§3.3 and App. E, Eq. (80)] The claim that representation duality 'can recover Bellman recursion for a deformed distribution P'_π' is not proven. The constructed density is dQ/dP^π = g'(S_J(J)), where S_J(J) is the upper-tail rank of the entire trajectory return J. Condition D requires the trajectory law to factor through a local unifilar statistic φ(h_t) as in Eq. (2); a rank-dependent reweighting of the global return generally destroys this factorization. For example, when J is an additive sum of per-step returns, Q's conditional distribution of later states depends on the cumulative rank of the full sum, not just on φ(h_t). The paper itself concedes in §3.3 that 'this is not a fixed deformation that could be applied without knowledge of J,' which contradicts the preceding sentence claiming recovery of Bellman recursion. The authors need either to prove that Q satisfies Condition D (or a suitable local-factorizat
  2. [§3.2, assessment duality and A2] The statement that when A2 fails, assessment duality 'can be used to recover Bellman recursion' for non-log-sum-exp utilities is not demonstrated. Transforming the objective from Shat F_u[J] to E[u(J)] changes the aggregation but does not preserve Condition R unless the return composition is changed as well: for additive returns J = ⊕ ℓ_t, the transformed return u(J) generally does not decompose as ⊕ of local returns. The paper mentions multiplicative dynamic programming, but does not specify the transformed return composition or prove that the transformed problem satisfies Conditions D, R, and A. This is a load-bearing part of the 'deforming returns' recovery claim and needs a precise statement.
  3. [Theorem 2 / App. A.2, Eq. (40)] The attainment assumption is stated only as 'If Sbar V^* is attained,' which is vague. The proof of the lower bound requires, for every successor history (h_t,a,s), the existence of a policy attaining V^* at that successor. This is stronger than the existence of a supremum and should be stated explicitly as an assumption. If the authors intend a weaker condition, the proof needs to be replaced with an ε-optimality argument.
minor comments (5)
  1. [§2.1] The sentence 'Condition D is always met (even in non-stationary settings)' is potentially confusing because the trivial full-history statistic satisfies the letter of the condition but gives no compression; Appendix B clarifies this, but the main text should point the reader there earlier.
  2. [§3.3] The definition of Stilde F_g in Eq. (35) does not state the regularity conditions used in Appendix E. The explicit deformed-measure formula in Proposition 1 assumes g ∈ C^1 and continuous F_J. These hypotheses should be stated in the main text, with the general case deferred to the Lebesgue–Stieltjes construction.
  3. [App. A.2, Eq. (40)] The notation V^*((h_t,A,S)) is used without definition; it should be written as V^*(h_{t+1}) or explicitly defined as the value at the concatenated history (h_t,A,S).
  4. [§3.3] The two consecutive sentences — 'representation duality can recover Bellman recursion' and 'this is not a fixed deformation...' — are in direct tension and should be reconciled or rewritten, not simply juxtaposed.
  5. [§3.2, Eq. (24)] The claim that Shat F_u satisfies A1 for all increasing u should be justified, since it is not immediate from the definition; a short argument or reference would help.

Circularity Check

0 steps flagged

No significant circularity: Theorems 1–2 are direct consequences of explicit assumptions, and the broader dualities are presented as known equivalences rather than as predicted outputs.

full rationale

I walked the paper's derivation chain. Condition D, Condition R, and Condition A are stated as definitions/assumptions, and Theorems 1 and 2 are proved directly by substituting those conditions into the value-function definition: the proof of Theorem 1 uses only the two-stage factorization in Eq. (39), the return recursion in Eq. (38), Condition A1 to interchange aggregations, and Condition A2 to pull the local return out of the inner aggregation. No fitted parameter is introduced, and no conclusion is assumed within the hypotheses. The "Bellman equation arises from their mutual consistency" language is a summary of Theorem 1, not an independent prediction. The three dualities in Section 3 are explicitly said to synthesize previously known work ('while each of these dualities was previously known'), and the equivalences such as Eq. (25) and the Gibbs variational identity are algebraic identities with cited external proofs, not predictions forced by the present framework. The one self-citation, Morales et al. (2023) for the log-sum-exp aggregator satisfying A2, is a standard, independently verifiable mathematical fact used only to illustrate Condition A; it is not load-bearing for Theorems 1–2 because those theorems take Condition A as an assumption. The skeptical concern that the rank-dependent deformation in Eq. (80) does not generally factor into local kernels is a correctness/scope concern about a heuristic claim in Section 3.3, not a circularity: the paper itself concedes that the deformation is 'not a fixed deformation that could be applied without knowledge of J,' and no theorem in the paper depends on that deformation. Overall, the central derivation is self-contained under the stated assumptions, so the appropriate circularity score is 0.

Axiom & Free-Parameter Ledger

0 free parameters · 6 axioms · 0 invented entities

The central claim depends on domain assumptions about sufficient statistics, associative return composition, and a strong algebraic compatibility of the aggregation functional. Theorem 2 additionally assumes attainment. No free parameters are fitted to data, and no new empirical entities are posited; the paper's contribution is in naming and framing, not in new postulates.

axioms (6)
  • domain assumption Trajectory uncertainty is a probability measure and the aggregation F maps Δ(U) to R (Definition 1).
    The entire framework is built on Δ-distributions and real-valued aggregators; non-probabilistic uncertainty is only sketched in App B.2.
  • domain assumption Return compositionality: there exists an associative operator ⊕ and a local ℓ such that J decomposes as in Condition R (Eq. 3).
    This is the key structural restriction; it excludes many non-additive or time-inconsistent return criteria.
  • domain assumption Aggregation compatibility (A1 and A2) holds (Condition A, Eqs. 4–5).
    This is the load-bearing algebra; it fails for most certainty-equivalent and distortion aggregators, which the paper tries to handle separately.
  • domain assumption The optimal value \bar V^* is attained (Theorem 2, App A.2).
    The proof of Theorem 2 selects attaining policies for every successor; this is not automatic for general state/action spaces.
  • standard math Donsker–Varadhan / Gibbs variational identity (App D, Eq. 55).
    Used to rewrite soft, risk-sensitive, and robust Bellman equations as regularized Bellman equations.
  • domain assumption Deformed-measure representation assumes g ∈ C^1, J bounded with continuous distribution (App E, Prop. 1).
    The main-text Eq. (36) for CVaR uses the a.e. derivative; the Bellman-recovery claim for these deformed measures is not proven.

reviewed 2026-08-01 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Generalised Bellman recurrence and three dualities in sequential decision-making." pith.science (2026). https://pith.science/paper/EF6QVJ4X

@misc{pith2026260718077,
  author       = {Pith},
  title        = {Pith review of: Generalised Bellman recurrence and three dualities in sequential decision-making},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EF6QVJ4X}},
  note         = {Machine review of arXiv:2607.18077}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

What gives the Bellman equation its form? We show that the recursive properties of optimal value functions follow from three conditions: that the dynamics decomposes through sufficient statistics, that the return decomposes recursively, and that the aggregation of uncertainty is compatible with both. When all three conditions hold on a common state, the Bellman equation arises from their mutual consistency; when one fails, tractability can often be recovered by augmenting the state or by deforming return or dynamics. The same conditions are shown to give rise to three dualities: one between probability and return, one between return and aggregation, and one between aggregation and probability. Our framework reveals these dualities as arising from a single construction, unifying methods developed separately across reinforcement learning, control, and decision theory.

Figures

Figures reproduced from arXiv: 2607.18077 by Daniel Polani, David Hyland, Fernando E. Rosas.

Figure 1
Figure 1. Figure 1: Three dualities in reinforcement learning. These can be arranged as a triangle, where each edge is a duality holding the opposite vertex fixed and exchanging the other two. Object duality relates the dynamics and return using the bilinearity of the expectation; assessment duality exchanges non-linearity of aggregation and return curvature or regularisation; and representation duality exchanges deformations… view at source ↗
Figure 2
Figure 2. Figure 2: Conditions giving rise to generalised Bellman recursion and optimality. The three conditions D, R, A yield a generalised Bellman recursion (Theorem 1). Adding an additional monotonicity assumption on F and ⊙ yields the Bellman optimality equation (Theorem 2). which equals (T∗V¯ ∗ )(z) after factorisation. For notational convenience, the proof will use Keπ(a, s|ht) := π(a | ht)K(s | ϕD(ht), a). (41) Upper b… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

75 extracted references · 8 linked inside Pith

  1. [1]

    More risk-sensitive Markov decision processes

    Nicole B \"a uerle and Ulrich Rieder. More risk-sensitive Markov decision processes. Mathematics of Operations Research, 39 0 (1): 0 105--120, 2014

    2014

  2. [2]

    Dynamic Programming

    Richard Bellman. Dynamic Programming. Princeton University Press, 1957

    1957

  3. [3]

    On aggregators and dynamic programming

    Philippe Bich, Jean-Pierre Drugeon, and Lisa Morhaim. On aggregators and dynamic programming. Technical report, Université Panthéon-Sorbonne, Centre d'Economie de la Sorbonne, 2015

    2015

  4. [4]

    A technical critique of some parts of the free energy principle

    Martin Biehl, Felix Pollock, and Ryota Kanai. A technical critique of some parts of the free energy principle. Entropy, 23 0 (3): 0 293, 2021

    2021

  5. [5]

    Functions resembling quotients of measures

    Ethan Bolker. Functions resembling quotients of measures. Transactions of the American Mathematical Society, 124 0 (2): 0 292--312, 1966

    1966

  6. [6]

    Your policy regularizer is secretly an adversary

    Rob Brekelmans, Tim Genewein, Jordi Grau-Moya, Gr \'e goire Del \'e tang, Markus Kunesch, Shane Legg, and Pedro Ortega. Your policy regularizer is secretly an adversary. arXiv preprint arXiv:2203.12592, 2022

    Pith/arXiv arXiv 2022

  7. [7]

    Profunctor optics, a categorical update

    Bryce Clarke, Derek Elkins, Jeremy Gibbons, Fosco Loregian, Bartosz Milewski, Emily Pillmore, and Mario Rom \'a n. Profunctor optics, a categorical update. Compositionality, 6 0 (1), 2024

    2024

  8. [8]

    Active inference on discrete state-spaces: A synthesis

    Lancelot Da Costa, Thomas Parr, Noor Sajid, Sebastijan Veselic, Victorita Neacsu, and Karl Friston. Active inference on discrete state-spaces: A synthesis. Journal of Mathematical Psychology, 99: 0 102447, 2020

    2020

  9. [9]

    Reward maximization through discrete active inference

    Lancelot Da Costa, Noor Sajid, Thomas Parr, Karl Friston, and Ryan Smith. Reward maximization through discrete active inference. Neural Computation, 35 0 (5): 0 807--852, 04 2023

    2023

  10. [10]

    Probability is real, and value is complex, 2018

    Abram Demski. Probability is real, and value is complex, 2018. URL https://www.lesswrong.com/posts/oheKfWA7SsvpK7SGp/probability-is-real-and-value-is-complex

    2018

  11. [11]

    Twice regularized MDP s and the equivalence between robustness and regularization

    Esther Derman, Matthieu Geist, and Shie Mannor. Twice regularized MDP s and the equivalence between robustness and regularization. Advances in Neural Information Processing Systems, 34: 0 22274--22287, 2021

    2021

  12. [12]

    Risk measures and comonotonicity: a review

    Jan Dhaene, Steven Vanduffel, Marc Goovaerts, Rob Kaas, Qihe Tang, and David Vyncke. Risk measures and comonotonicity: a review. Stochastic models, 22 0 (4): 0 573--606, 2006

    2006

  13. [13]

    Optimal Learning: Computational Procedures for Bayes -Adaptive Markov Decision Processes

    Michael Duff. Optimal Learning: Computational Procedures for Bayes -Adaptive Markov Decision Processes . PhD thesis, University of Massachusetts Amherst, 2002

    2002

  14. [14]

    Robust properties of risk-sensitive control

    Paul Dupuis, Matthew James, and Ian Petersen. Robust properties of risk-sensitive control. Mathematics of Control, Signals and Systems, 13 0 (4): 0 318--332, 2000

    2000

  15. [15]

    Long-term values in M arkov decision processes, (co)algebraically

    Frank Feys, Helle Hansen, and Lawrence Moss. Long-term values in M arkov decision processes, (co)algebraically. In Coalgebraic Methods in Computer Science: 14th IFIP WG 1.3 International Workshop (CMCS 2018), pp.\ 78–99, Berlin, Heidelberg, 2018. Springer-Verlag

    2018

  16. [16]

    Stochastic finance: an introduction in discrete time

    Hans F \"o llmer and Alexander Schied. Stochastic finance: an introduction in discrete time. Walter de Gruyter GmbH & Co KG, 2025

    2025

  17. [17]

    The free-energy principle: A unified brain theory? Nature Reviews Neuroscience, 11 0 (2): 0 127--138, 2010

    Karl Friston. The free-energy principle: A unified brain theory? Nature Reviews Neuroscience, 11 0 (2): 0 127--138, 2010

    2010

  18. [18]

    Active inference and epistemic value

    Karl Friston, Francesco Rigoli, Dimitri Ognibene, Christoph Mathys, Thomas Fitzgerald, and Giovanni Pezzulo. Active inference and epistemic value. Cognitive Neuroscience, 6 0 (4): 0 187--214, 2015

    2015

  19. [19]

    Sophisticated inference

    Karl Friston, Lancelot Da Costa, Danijar Hafner, Casper Hesp, and Thomas Parr. Sophisticated inference. Neural Computation, 33 0 (3): 0 713--763, 03 2021

    2021

  20. [20]

    A synthetic approach to Markov kernels, conditional independence and theorems on sufficient statistics

    Tobias Fritz. A synthetic approach to Markov kernels, conditional independence and theorems on sufficient statistics. Advances in Mathematics, 370: 0 107239, 2020

    2020

  21. [21]

    A theory of regularized M arkov decision processes

    Matthieu Geist, Bruno Scherrer, and Olivier Pietquin. A theory of regularized M arkov decision processes. In International conference on machine learning, pp.\ 2160--2169. PMLR, 2019

    2019

  22. [22]

    Bayesian reinforcement learning: A survey

    Mohammad Ghavamzadeh, Shie Mannor, Joelle Pineau, and Aviv Tamar. Bayesian reinforcement learning: A survey. Foundations and Trends in Machine Learning, 8 0 (5--6): 0 359--483, 2015

    2015

  23. [23]

    Maxmin expected utility with non-unique prior

    Itzhak Gilboa and David Schmeidler. Maxmin expected utility with non-unique prior. Journal of mathematical economics, 18 0 (2): 0 141--153, 1989

    1989

  24. [24]

    The minimum description length principle

    Peter Gr \"u nwald. The minimum description length principle. MIT press, 2007

    2007

  25. [25]

    Reinforcement learning with deep energy-based policies

    Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In Proceedings of the 34th International Conference on Machine Learning, pp.\ 1352--1361, 2017

    2017

  26. [26]

    Robustness

    Lars Hansen and Thomas Sargent. Robustness. Princeton university press, 2011

    2011

  27. [27]

    Value iteration is optic composition

    Jules Hedges and Riu Rodr \'i guez Sakamoto. Value iteration is optic composition. In Proceedings of Applied Category Theory 2022, volume 380 of Electronic Proceedings in Theoretical Computer Science, pp.\ 417--432, 2023

    2022

  28. [28]

    Reinforcement learning in categorical cybernetics

    Jules Hedges and Riu Rodr \'i guez Sakamoto. Reinforcement learning in categorical cybernetics. In Proceedings of Applied Category Theory 2024, volume 429 of Electronic Proceedings in Theoretical Computer Science, pp.\ 270--286, 2025

    2024

  29. [29]

    Risk-sensitive Markov decision processes

    Ronald Howard and James Matheson. Risk-sensitive Markov decision processes. Management Science, 18 0 (7): 0 356--369, 1972

    1972

  30. [30]

    Regularized policies are reward robust

    Hisham Husain, Kamil Ciosek, and Ryota Tomioka. Regularized policies are reward robust. In International Conference on Artificial Intelligence and Statistics, pp.\ 64--72. PMLR, 2021

    2021

  31. [31]

    Using reward machines for high-level task specification and decomposition in reinforcement learning

    Rodrigo Icarte, Toryn Klassen, Richard Valenzano, and Sheila McIlraith. Using reward machines for high-level task specification and decomposition in reinforcement learning. In International Conference on Machine Learning, pp.\ 2107--2116. PMLR, 2018

    2018

  32. [32]

    Robust dynamic programming

    Garud Iyengar. Robust dynamic programming. Mathematics of Operations Research, 30 0 (2): 0 257--280, 2005

    2005

  33. [33]

    The Logic of Decision

    Richard Jeffrey. The Logic of Decision. McGraw-Hill, 1965

    1965

  34. [34]

    Planning and acting in partially observable stochastic domains

    Leslie Kaelbling, Michael Littman, and Anthony Cassandra. Planning and acting in partially observable stochastic domains. Artificial Intelligence, 101 0 (1--2): 0 99--134, 1998

    1998

  35. [35]

    Optimal control as a graphical model inference problem

    Hilbert Kappen, Vicen c G \'o mez, and Manfred Opper. Optimal control as a graphical model inference problem. Machine Learning, 87 0 (2): 0 159--182, 2012

    2012

  36. [36]

    Empowerment: A universal agent-centric measure of control

    Alexander Klyubin, Daniel Polani, and Chrystopher Nehaniv. Empowerment: A universal agent-centric measure of control. In 2005 IEEE Congress on Evolutionary Computation, pp.\ 128--135, 2005

    2005

  37. [37]

    Reinforcement learning and control as probabilistic inference: Tutorial and review

    Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review. arXiv preprint arXiv:1805.00909, 2018

    Pith/arXiv arXiv 2018

  38. [38]

    The Kelly capital growth investment criterion: Theory and practice, volume 3

    Leonard MacLean, Edward Thorp, and William Ziemba. The Kelly capital growth investment criterion: Theory and practice, volume 3. World Scientific, 2011

    2011

  39. [39]

    A retrospective on active inference

    Beren Millidge. A retrospective on active inference. Blog post, 2024. URL https://www.beren.io/2024-07-27-A-Retrospective-on-Active-Inference/

    2024

  40. [40]

    On the relationship between active inference and control as inference

    Beren Millidge, Alexander Tschantz, Anil Seth, and Christopher Buckley. On the relationship between active inference and control as inference. arXiv preprint arXiv:2006.12964, 2020

    Pith/arXiv arXiv 2006

  41. [41]

    Whence the expected free energy? Neural Computation, 33 0 (2): 0 447--482, 2021

    Beren Millidge, Alexander Tschantz, and Christopher Buckley. Whence the expected free energy? Neural Computation, 33 0 (2): 0 447--482, 2021

    2021

  42. [42]

    Thermodynamics of exponential Kolmogorov--Nagumo averages

    Pablo Morales, Jan Korbel, and Fernando Rosas. Thermodynamics of exponential Kolmogorov--Nagumo averages. New Journal of Physics, 25 0 (7): 0 073011, 2023

    2023

  43. [43]

    Bridging the gap between value and policy based reinforcement learning

    Ofir Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans. Bridging the gap between value and policy based reinforcement learning. Advances in neural information processing systems, 30, 2017

    2017

  44. [44]

    Uncertainty in all its flavours, 2024

    Cleo Nardo. Uncertainty in all its flavours, 2024. URL https://www.lesswrong.com/posts/JhhFokDRkAfg5Tnu9/uncertainty-in-all-its-flavours

    2024

  45. [45]

    Robust control of Markov decision processes with uncertain transition matrices

    Arnab Nilim and Laurent El Ghaoui. Robust control of Markov decision processes with uncertain transition matrices. Operations Research, 53 0 (5): 0 780--798, 2005

    2005

  46. [46]

    Discovering reinforcement learning algorithms

    Junhyuk Oh, Matteo Hessel, Wojciech Czarnecki, Zhongwen Xu, Hado van Hasselt, Satinder Singh, and David Silver. Discovering reinforcement learning algorithms. Advances in Neural Information Processing Systems, 33: 0 1060--1070, 2020

    2020

  47. [47]

    Thermodynamics as a theory of decision-making with information-processing costs

    Pedro Ortega and Daniel Braun. Thermodynamics as a theory of decision-making with information-processing costs. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 469 0 (2153): 0 20120683, 2013

    2013

  48. [48]

    Memory shapes time perception and intertemporal choices

    Pedro Ortega and Naftali Tishby. Memory shapes time perception and intertemporal choices. arXiv preprint arXiv:1604.05129, 2016

    Pith/arXiv arXiv 2016

  49. [49]

    Information-theoretic bounded rationality

    Pedro Ortega, Daniel Braun, Justin Dyer, Kee-Eung Kim, and Naftali Tishby. Information-theoretic bounded rationality. arXiv preprint arXiv:1512.06789, 2015

    Pith/arXiv arXiv 2015

  50. [50]

    Robustness and risk-sensitivity in M arkov decision processes

    Takayuki Osogami. Robustness and risk-sensitivity in M arkov decision processes. Advances in neural information processing systems, 25, 2012

    2012

  51. [51]

    What is intrinsic motivation? a typology of computational approaches

    Pierre-Yves Oudeyer and Frederic Kaplan. What is intrinsic motivation? a typology of computational approaches. Frontiers in Neurorobotics, 1: 0 6, 2007

    2007

  52. [52]

    Active Inference: The Free Energy Principle in Mind, Brain, and Behavior

    Thomas Parr, Giovanni Pezzulo, and Karl Friston. Active Inference: The Free Energy Principle in Mind, Brain, and Behavior. The MIT Press, 03 2022

    2022

  53. [53]

    Curiosity-driven exploration by self-supervised prediction

    Deepak Pathak, Pulkit Agrawal, Alexei Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In Proceedings of the 34th International Conference on Machine Learning, pp.\ 2778--2787, 2017

    2017

  54. [54]

    Minimax optimal control of stochastic uncertain systems with relative entropy constraints

    Ian Petersen, Matthew James, and Paul Dupuis. Minimax optimal control of stochastic uncertain systems with relative entropy constraints. IEEE Transactions on Automatic Control, 45 0 (3): 0 398--412, 2000

    2000

  55. [55]

    Markov Decision Processes: Discrete Stochastic Dynamic Programming

    Martin Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley, 1994

    1994

  56. [56]

    A theoretical connection between statistical physics and reinforcement learning

    Jad Rahme and Ryan Adams. A theoretical connection between statistical physics and reinforcement learning. arXiv preprint arXiv:1906.10228, 2019

    Pith/arXiv arXiv 1906

  57. [57]

    Categories of optics

    Mitchell Riley. Categories of optics. arXiv preprint arXiv:1809.00738, 2018

    Pith/arXiv arXiv 2018

  58. [58]

    Optimization of conditional value-at-risk

    R Rockafellar and Stanislav Uryasev. Optimization of conditional value-at-risk. Journal of risk, 2: 0 21--42, 2000

    2000

  59. [59]

    Learning to optimize via information-directed sampling

    Daniel Russo and Benjamin Van Roy. Learning to optimize via information-directed sampling. In Advances in Neural Information Processing Systems 27, pp.\ 1583--1591, 2014

    2014

  60. [60]

    Learning to optimize via information-directed sampling

    Daniel Russo and Benjamin Van Roy. Learning to optimize via information-directed sampling. Operations Research, 66 0 (1): 0 230--252, 2018

    2018

  61. [61]

    Active inference: Demystified and compared

    Noor Sajid, Philip Ball, Thomas Parr, and Karl Friston. Active inference: Demystified and compared. Neural Computation, 33 0 (3): 0 674--712, 2021

    2021

  62. [62]

    Empowerment: An introduction

    Christoph Salge, Cornelius Glackin, and Daniel Polani. Empowerment: An introduction. In Guided Self-Organization: Inception, pp.\ 67--114. Springer, 2014

    2014

  63. [63]

    The foundations of statistics

    Leonard Savage. The foundations of statistics. John Wiley & Sons, 1954

    1954

  64. [64]

    Subjective probability and expected utility without additivity

    David Schmeidler. Subjective probability and expected utility without additivity. Econometrica: Journal of the Econometric Society, pp.\ 571--587, 1989

    1989

  65. [65]

    Equivalence between policy gradients and soft q-learning

    John Schulman, Xi Chen, and Pieter Abbeel. Equivalence between policy gradients and soft q-learning. arXiv preprint arXiv:1704.06440, 2017

    Pith/arXiv arXiv 2017

  66. [66]

    Horde: A scalable real-time architecture for learning knowledge from unsupervised sensorimotor interaction

    Richard Sutton, Joseph Modayil, Michael Delp, Thomas Degris, Patrick Pilarski, Adam White, and Doina Precup. Horde: A scalable real-time architecture for learning knowledge from unsupervised sensorimotor interaction. In The 10th international conference on autonomous agents and multiagent systems-volume 2, pp.\ 761--768, 2011

    2011

  67. [67]

    Recursive reward aggregation

    Yuting Tang, Yivan Zhang, Johannes Ackermann, Yu-Jie Zhang, Soichiro Nishimori, and Masashi Sugiyama. Recursive reward aggregation. In Reinforcement Learning Conference, 2025

    2025

  68. [68]

    Linearly-solvable Markov decision problems

    Emanuel Todorov. Linearly-solvable Markov decision problems. In Advances in Neural Information Processing Systems 19, pp.\ 1369--1376, 2007

    2007

  69. [69]

    Theory of games and economic behavior

    John Von Neumann and Oskar Morgenstern. Theory of games and economic behavior. Princeton University Press, 1944

    1944

  70. [70]

    Utility maximization = description length minimization

    John Wentworth. Utility maximization = description length minimization. LessWrong, 2021. URL https://www.lesswrong.com/posts/voLHQgNncnjjgAPH7/utility-maximization-description-length-minimization

    2021

  71. [71]

    Risk-sensitive linear/quadratic/ Gaussian control

    Peter Whittle. Risk-sensitive linear/quadratic/ Gaussian control. Advances in Applied Probability, 13 0 (4): 0 764--777, 1981

    1981

  72. [72]

    Robust Markov decision processes

    Wolfram Wiesemann, Daniel Kuhn, and Ber c Rustem. Robust Markov decision processes. Mathematics of Operations Research, 38 0 (1): 0 153--183, 2013

    2013

  73. [73]

    The dual theory of choice under risk

    Menahem Yaari. The dual theory of choice under risk. Econometrica: Journal of the Econometric Society, pp.\ 95--115, 1987

    1987

  74. [74]

    Modeling Purposeful Adaptive Behavior with the Principle of Maximum Causal Entropy

    Brian Ziebart. Modeling Purposeful Adaptive Behavior with the Principle of Maximum Causal Entropy. PhD thesis, Carnegie Mellon University, 2010

    2010

  75. [75]

    VariBAD : A very good method for Bayes -adaptive deep RL via meta-learning

    Luisa Zintgraf, Kyriacos Shiarlis, Maximilian Igl, Sebastian Schulze, Yarin Gal, Katja Hofmann, and Shimon Whiteson. VariBAD : A very good method for Bayes -adaptive deep RL via meta-learning. In International Conference on Learning Representations (ICLR), 2020

    2020

This paper was first reviewed by deepseek-v4-flash on August 1, 2026.