REVIEW 3 major objections 4 minor 44 references
Intrinsic motivation as constrained entropy maximization
T0 review · 3 major / 4 minor · reviewed 2026-08-09 · deepseek-v4-flash
Pith's one-line read The paper argues that empowerment, active inference, and maximum occupancy are all constrained entropy maximization in disguise.
desk verdict Conceptual synthesis with a genuine insight about maximum occupancy, but the formal empowerment–active inference bridge is undermined by a misstated mutual information identity. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is constrained entropy maximization as a shared objective. In empowerment it appears as channel capacity, the maximum over action distributions of $I(A_{t:T};O_T)$, which balances action entropy against controllability of observations. In active inference it appears in expected free energy, decomposed into risk plus ambiguity; the paper rewrites the empowerment objective in that vocabulary as ambiguity minus state information gain. In maximum occupancy, the value of a state is the discounted expected sum of action entropy and next-state entropy, with no explicit preference term. The load-bearing mechanism that turns pure entropy maximization into self-preserving behavior is the action–state partition of the agent-environment system, which acts as an implicit model-evidence constraint.
What would settle it
Compute both sides of $I_t(\pi;o_T)=H(Q(\pi))-H(Q(o_T|\pi))$ for a simple asymmetric binary channel with a nonuniform marginal over policies; any nonzero difference between the two sides shows the identity is false.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that empowerment, active inference, and maximum occupancy form a spectrum rather than rival theories. The paper derives a free energy of empowerment, $$F_t(\pi,o_T)=\mathbb{E}_{Q(s_T|\pi)}[H(P(o_T|s_T))]-I_t(\pi;s_T),$$ and says that maximizing empowerment $I_t(\pi;o_T)$ is equivalent to minimizing this quantity; the first term matches the ambiguity term of the expected free energy, while the two entropy terms combine into a state information gain. Active inference agents are therefore 'empowered' in the absence of preferences: they maximize the entropy of future state distributions under a controllability constraint. Maximum occupancy is the same theme run almost unconstrained, but the paper argues its apparent goal-directedness is underwritten by an implicit model-evidence constraint: the action–state partition that defines the agent is a precondition for future occupancy and thereby functions as a will to live. The unifying principle is constrained entropy maximization.
Load-bearing premise
The formal bridge is the displayed identity $I_t(\pi;o_T)=H(Q(\pi))-H(Q(o_T|\pi))$, but mutual information is $H(X)-H(X|Y)$ or $H(Y)-H(Y|X)$, not $H(X)-H(Y|X)$, so the equivalence to a free energy stands only if this identity can be justified.
Editorial extensions
If this is right
- If the equivalence holds, maximizing empowerment and minimizing preference-free expected free energy are the same computation, so algorithms developed for either framework should transfer directly.
- Active inference, viewed broadly to include policy entropy, retains an explicit drive to keep options open rather than deterministically selecting a single policy.
- Maximum occupancy agents depend on an implicit self-model; removing the action–state partition removes the formal reason to avoid absorbing states.
- Exploratory behavior is not merely an evolved means to reward but a fundamental aspect of agency, so intrinsic motivation can be studied without ad hoc reward functions.
- Comparisons among the three frameworks reduce to which constraints are imposed on entropy maximization, giving a common axis for empirical tests.
Reading between the lines
- A testable extension is to interpolate among the frameworks by tuning constraints—pure action-entropy maximization, empowerment's controllability, and active inference's controllability plus model evidence—on the same environment.
- If the formal bridge is repaired, empowerment could be implemented with standard variational inference machinery, making channel-capacity maximization tractable in high-dimensional partially observed settings.
- The argument implies that 'death is bad' cannot be derived from entropy maximization alone; a maximizing agent also needs an a priori boundary between itself and the world, connecting intrinsic motivation to accounts of selfhood.
- The displayed information-theoretic identity deserves scrutiny before the equivalence is taken literally; the looser conceptual framing of constraints may hold even if that derivation needs repair.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper argues that three formal accounts of intrinsic motivation—empowerment, active inference, and the maximum occupancy principle—can be viewed as variations on constrained entropy maximization. It develops this thesis through a conceptual comparison in Sections 2 and 3, with the central formal step in Section 3.1 attempting to show that the empowerment objective can be cast as a variational free energy, thereby bridging empowerment and active inference. Section 3.2 argues that maximum occupancy implicitly contains a model-evidence constraint via the action-state partition, and Section 3.3 draws broader philosophical conclusions about agency and the 'will to live'.
Significance. If the formal unification were correct, the paper would provide a useful synthetic perspective on influential frameworks, potentially clarifying their relationships and supporting the view that entropy maximization under constraints is a core principle of agency. The paper is clearly written, cites relevant literature, and makes no parameter-fitting claims, so its conceptual contribution is distinguishable from its formal one. However, the mathematical derivation in Section 3.1 contains load-bearing errors in the manipulation of mutual information and entropy, and these errors invalidate the claimed equivalence between empowerment and active inference as presented. The broader conceptual framing may survive a correction, but the technical centerpiece needs substantial revision.
major comments (3)
- [3.1, equation It(π; oT) = H(Q(π)) − H(Q(oT|π))] The displayed identity is not mutual information. Standard information theory gives I(X;Y) = H(X) − H(X|Y) = H(Y) − H(Y|X), so the correct expressions would be H(Q(π)) − H(Q(π|oT)) or H(Q(oT)) − H(Q(oT|π)). The paper uses H(Q(oT|π)), which is the conditional entropy of observations given policies, and omits the marginal entropy of observations. Consequently, the subsequent rearrangement into the 'free energy of empowerment' does not follow, and the claim that maximizing It(π; oT) is equivalent to minimizing this energy is unsupported as written.
- [3.1, final line of derivation] The statement that the two entropy terms H(Q(sT|π)) and −H(Q(π)) 'can be combined into a state information gain term' −It(π; sT) is incorrect. Mutual information I(π; sT) equals H(π) − H(π|sT) or H(sT) − H(sT|π); the expression H(sT|π) − H(π) is not generally equal to −I(π; sT). This is a second algebraic error in the same derivation, further undermining the claimed equivalence between the free-energy-of-empowerment expression and the ambiguity-minus-information-gain form.
- [3.1 (overall)] Because the two algebraic identities above are load-bearing for the paper's central formal claim—that empowerment can be derived as a variational free energy and is therefore equivalent to active inference's information-seeking objective—the technical bridge is not established. The conceptual discussion in Sections 2 and 3.2 may remain plausible, but the formal equivalence requires a corrected derivation, possibly using the correct entropy decomposition and an appropriate variational bound.
minor comments (4)
- [2.2] There is a typo: 'mosty saliently' should be 'most saliently'.
- [2.3] The quantity R(τ) is defined with a negative sign and then maximized, so calling it a 'reward' is potentially confusing; since it is a negative log-probability, it is more naturally a cost or surprise.
- [3.1] The notation It(π; sT) is introduced without a definition; please define it explicitly as mutual information between policies and final states, and distinguish it from the earlier It(π; oT).
- [3.2] The sentence 'It is argued in [7] on both conceptual and experimental grounds...' is followed by a discussion noting that the default β=0 setting limits the relevance of those experiments; consider clarifying which specific claims are being endorsed given those limitations.
Circularity Check
No circular reduction; the unificatory claim rests on in-text algebra, with only non-load-bearing self-citations; the Section 3.1 formal bridge is undermined by an incorrect mutual-information identity, which is a correctness issue rather than circularity.
full rationale
The paper does not fit parameters or rename fitted values as predictions, and the central claim that active inference, empowerment, and maximum occupancy are constrained entropy maximization is supported by the displayed equations rather than by importing the conclusion. The 'free energy of empowerment' F_t is introduced explicitly as an algebraic rearrangement of the empowerment objective I_t, so the equivalence between maximizing I_t and minimizing F_t is true by construction; this is a definitional identity, not a hidden circular load. Self-citations [32], [35], and [44] overlap with the author, but the substantive steps they support also appear as derivations in the text (e.g., the EFE-information gain relation in Section 3.1), so they are background support rather than the sole basis for the argument. A separate, non-circular correctness problem exists: Section 3.1 writes I_t(pi;o_T) = H(Q(pi)) - H(Q(o_T|pi)), which is not the standard mutual-information decomposition, and the subsequent 'ambiguity minus state information gain' rearrangement inherits this error. That undermines the formal bridge as written but is a mathematical defect, not a self-referential reduction. Therefore the circularity score is 2, reflecting only minor non-load-bearing self-citation.
Assumptions & free parameters
assumptions (4)
- domain assumption Empowerment is the channel capacity of P(O_T|A_t:T) with fixed conditional and variable action distribution.
- domain assumption Expected free energy decomposes into risk and ambiguity as G_t^pi = D_KL(Q(o_t|pi) || P(o_t)) + E_{Q(s_t|pi)}[H(P(o_t|s_t))].
- domain assumption In variational inference, the entropy of latent variables such as policies should be maximized ceteris paribus.
- standard math Standard identities for mutual information and entropy: I(X;Y) = H(Y) - H(Y|X) = H(X) - H(X|Y).
Cite this review
Pith. "Pith review of Intrinsic motivation as constrained entropy maximization." pith.science (2026). https://pith.science/paper/HRUVV3ZV
@misc{pith2026250202962,
author = {Pith},
title = {Pith review of: Intrinsic motivation as constrained entropy maximization},
year = {2026},
howpublished = {\url{https://pith.science/paper/HRUVV3ZV}},
note = {Machine review of arXiv:2502.02962}
}
read the original abstract
"Intrinsic motivation" refers to the capacity for intelligent systems to be motivated endogenously, i.e. by features of agential architecture itself rather than by learned associations between action and reward. This paper views active inference, empowerment, and other formal accounts of intrinsic motivation as variations on the theme of constrained maximum entropy inference, providing a general perspective on intrinsic motivation complementary to existing frameworks. The connection between free energy and empowerment noted in previous literature is further explored, and it is argued that the maximum-occupancy approach in practice incorporates an implicit model-evidence constraint.
Reference graph
Works this paper leans on
-
[15]
Friston, K.; Da Costa, L.; Hafner, D.; Hesp, C.; Parr, T. Sophisticated Inference. Neural Computation 2021, 33, 713–763
work page 2021
-
[1]
The Emerging Neuroscience of Intrinsic Motivation: A New Frontier in Self-Determination Research
Domenico, S.I.D.; Ryan, R.M. The Emerging Neuroscience of Intrinsic Motivation: A New Frontier in Self-Determination Research. Frontiers in Human Neuroscience 2017, 11
work page 2017
-
[2]
Empowerment: a universal agent-centric measure of control
Klyubin, A.S.; Polani, D.; Nehaniv, C.L. Empowerment: a universal agent-centric measure of control. 2005 IEEE Congress on Evolutionary Computation 2005, 1, 128–135 Vol.1
work page 2005
-
[3]
Empowerment -- an Introduction
Salge, C.; Glackin, C.; Polani, D. Empowerment - an Introduction. ArXiv 2013, abs/1310.1863
work page Pith review arXiv 2013
-
[4]
Active Inference: A Process Theory
Friston, K.J.; FitzGerald, T.H.B.; Rigoli, F.; Schwartenbeck, P .; Pezzulo, G. Active Inference: A Process Theory. Neural Computation 2017, 29, 1–49
work page 2017
-
[5]
Active inference and epistemic value
Friston, K.J.; Rigoli, F.; Ognibene, D.; Mathys, C.D.; FitzGerald, T.H.B.; Pezzulo, G. Active inference and epistemic value. Cognitive Neuroscience 2015, 6, 187 – 214
work page 2015
-
[6]
Active Inference as a Model of Agency, 2024, [arXiv:cs.AI/2401.12917]
Da Costa , L.; Tenka, S.; Zhao, D.; Sajid, N. Active Inference as a Model of Agency, 2024, [arXiv:cs.AI/2401.12917]
arXiv 2024
-
[7]
Complex behavior from intrinsic motivation to occupy future action-state path space
Ramirez-Ruiz, J.; Grytskyy, D.; Mastrogiuseppe, C.; Habib, Y.; Moreno-Bote, R. Complex behavior from intrinsic motivation to occupy future action-state path space. Nature Communications 2022, 15
work page 2022
Show all 44 references
-
[8]
Adaptive confidence and adaptive curiosity
Schmidhuber, J. Adaptive confidence and adaptive curiosity. Forschungsberichte, TU Munich 1991, FKI 149 91, 1–9. 13 The distinction between thermodynamic and merely information-theoretic or variational free energy [32,43] needn’t concern us here, as the entropy of observations...
1991
-
[9]
Formal Theory of Creativity, Fun, and Intrinsic Motivation (1990–2010)
Schmidhuber, J. Formal Theory of Creativity, Fun, and Intrinsic Motivation (1990–2010). IEEE T ransactions on Autonomous Mental Development 2010, 2, 230–247
1990
-
[10]
Bayesian surprise attracts human attention
Itti, L.; Baldi, P . Bayesian surprise attracts human attention. Vision Research 2009, 49, 1295–1306. Visual Attention: Psychophysics, electrophysiology and neuroimaging, https://doi.org/https://doi.org/10.1016/j. visres.2008.09.007
2009 doi
-
[11]
Curiosity-Driven Exploration via Latent Bayesian Surprise
Mazzaglia, P .; Çatal, O.; Verbelen, T.; Dhoedt, B. Curiosity-Driven Exploration via Latent Bayesian Surprise. In Proceedings of the AAAI Conference on Artificial Intelligence, 2021
2021
-
[12]
Empowerment, Free Energy Principle and Maximum Occupancy Principle Compared
Moreno-Bote, R.; Ramírez-Ruiz, J. Empowerment, Free Energy Principle and Maximum Occupancy Principle Compared. In Proceedings of the NeurIPS 2023 workshop: Information-Theoretic Principles in Cognitive Systems, 2023
2023
-
[13]
Expanding the Active Inference Landscape: More Intrinsic Motivations in the Perception-Action Loop
Biehl, M.; Guckelsberger, C.; Salge, C.; Smith, S.C.; Polani, D. Expanding the Active Inference Landscape: More Intrinsic Motivations in the Perception-Action Loop. Frontiers in Neurorobotics 2018, 12
2018
-
[14]
Towards a Geometry and Analysis for Bayesian Mechanics, 2022, [arXiv:math- ph/2204.11900]
Sakthivadivel, D.A.R. Towards a Geometry and Analysis for Bayesian Mechanics, 2022, [arXiv:math- ph/2204.11900]
2022 arXiv
-
[16]
Intrinsic Motivation in Dynamical Control Systems
Tiomkin, S.; Nemenman, I.; Polani, D.; Tishby, N. Intrinsic Motivation in Dynamical Control Systems. PRX Life 2024, 2, 033009
2024
-
[17]
Causal Entropic Forces
Wissner-Gross, A.D.; Freer, C.E. Causal Entropic Forces. Phys. Rev. Lett. 2013, 110, 168702. https: //doi.org/10.1103/PhysRevLett.110.168702
2013 doi
-
[18]
In Facets of Systems Science; Springer US: Boston, MA, 1991; pp
Ashby, W.R., Requisite Variety and Its Implications for the Control of Complex Systems. In Facets of Systems Science; Springer US: Boston, MA, 1991; pp. 405–417. https://doi.org/10.1007/978-1-4899-0718-9_28
1991 doi
-
[19]
The Predictive Mind; Oxford University Press UK: Oxford, GB, 2013
Hohwy, J. The Predictive Mind; Oxford University Press UK: Oxford, GB, 2013
2013
-
[20]
Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects
Rao, R.P .N.; Ballard, D.H. Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects. Nature Neuroscience 1999, 2, 79–87
1999
-
[21]
A Stable, Fast, and Fully Automatic Learning Algorithm for Predictive Coding Networks
Salvatori, T.; Song, Y.; Yordanov, Y.; Millidge, B.; Sha, L.; Emde, C.; Xu, Z.; Bogacz, R.; Lukasiewicz, T. A Stable, Fast, and Fully Automatic Learning Algorithm for Predictive Coding Networks. In Proceedings of the The Twelfth International Conference on Learning Representat...
2024
-
[22]
Helmholtz, H.v.; Southall, J.P .C.J.P .C.Helmholtz’s T reatise on physiological optics; Dover Publications: New York, 1962
1962
-
[23]
A free energy principle for a particular physics
Friston, K.J. A free energy principle for a particular physics. arXiv: Neurons and Cognition 2019
2019
-
[24]
Path integrals, particular kinds, and strange things
Friston, K.; Da Costa, L.; Sakthivadivel, D.A.; Heins, C.; Pavliotis, G.A.; Ramstead, M.; Parr, T. Path integrals, particular kinds, and strange things. Physics of Life Reviews 2023, 47, 35–62
2023
-
[25]
A step-by-step tutorial on active inference and its application to empirical data
Smith, R.; Friston, K.J.; Whyte, C.J. A step-by-step tutorial on active inference and its application to empirical data. Journal of Mathematical Psychology 2022, 107, 102632. https://doi.org/https://doi.org/10.1016/j.jmp. 2021.102632
2022
-
[26]
Active Inference, Attention, and Motor Preparation
Brown, H.R.; Friston, K.J.; Bestmann, S. Active Inference, Attention, and Motor Preparation. Frontiers in Psychology 2011, 2
2011
-
[27]
Planning as inference
Botvinick, M.; Toussaint, M. Planning as inference. T rends in Cognitive Sciences 2012, 16, 485–488. https: //doi.org/https://doi.org/10.1016/j.tics.2012.08.006
2012 doi
-
[28]
The Self-Evidencing Brain
Hohwy, J. The Self-Evidencing Brain. Noûs 2014, 50, 259–285. https://doi.org/10.1111/nous.12062
2014 doi
-
[29]
pymdp: A Python library for active inference in discrete state spaces
Heins, C.; Millidge, B.; Demekas, D.; Klein, B.; Friston, K.; Couzin, I.D.; Tschantz, A. pymdp: A Python library for active inference in discrete state spaces. Journal of Open Source Software 2022, 7, 4098. https: //doi.org/10.21105/joss.04098
2022 doi
-
[30]
Whence the Expected Free Energy? Neural Computation 2021, 33, 447–482
Millidge, B.; Tschantz, A.; Buckley, C.L. Whence the Expected Free Energy? Neural Computation 2021, 33, 447–482. https://doi.org/10.1162/neco_a_01354
2021 doi
-
[31]
Information Theory and Statistical Mechanics
Jaynes, E.T. Information Theory and Statistical Mechanics. Phys. Rev. 1957, 106, 620–630
1957
-
[32]
Psychophysical Identity and Free Energy
Kiefer, A. Psychophysical Identity and Free Energy. Journal of the Royal Society Interface 2020, 17
2020
-
[33]
Reward is not Necessary: How to Create a Modular & Compositional Self-Preserving Agent for Life-Long Learning, 2023, [arXiv:cs.AI/2211.10851]
Ringstrom, T.J. Reward is not Necessary: How to Create a Modular & Compositional Self-Preserving Agent for Life-Long Learning, 2023, [arXiv:cs.AI/2211.10851]
2023 arXiv
-
[34]
On the thermodynamics of prediction under dissipative adaptation
Ueltzhöffer, K. On the thermodynamics of prediction under dissipative adaptation. arXiv: Neurons and Cognition 2020
2020
-
[35]
Supervised structure learning
Friston, K.J.; Da Costa, L.; Tschantz, A.; Kiefer, A.; Salvatori, T.; Neacsu, V .; Koudahl, M.; Heins, C.; Sajid, N.; Markovic, D.; et al. Supervised structure learning. Biological Psychology 2024, 193, 108891. https://doi.org/https://doi.org/10.1016/j.biopsycho.2024.108891. 13 of 13
2024
-
[36]
Generalised free energy and active inference
Parr, T.; Friston, K.J. Generalised free energy and active inference. Biological Cybernetics 2018, 113, 495 – 513
2018
-
[37]
Understanding the origin of information-seeking exploration in probabilistic objectives for control
Millidge, B.; Tschantz, A.; Seth, A.K.; Buckley, C.L. Understanding the origin of information-seeking exploration in probabilistic objectives for control. ArXiv 2021, abs/2103.06859
2021 arXiv
-
[38]
Statistical physics of self-replication
England, J.L. Statistical physics of self-replication. The Journal of Chemical Physics 2013, 139, 121923
2013
-
[39]
Probabilistic Principles for Biophysics and Neuroscience: Entropy Production, Bayesian Mechan- ics & the Free-Energy Principle, 2024, [arXiv:math-ph/2410.11735]
Costa, L.D. Probabilistic Principles for Biophysics and Neuroscience: Entropy Production, Bayesian Mechan- ics & the Free-Energy Principle, 2024, [arXiv:math-ph/2410.11735]
2024 arXiv
-
[40]
The World as Will and Representation ; Dover Publications: New York„ 1958
Schopenhauer, A.; Payne, E.F.J. The World as Will and Representation ; Dover Publications: New York„ 1958
1958
-
[41]
A Multi-scale View of the Emergent Complexity of Life: A Free-Energy Proposal
Hesp, C.; Ramstead, M.; Constant, A.; Badcock, P .; Kirchhoff, M.; Friston, K. A Multi-scale View of the Emergent Complexity of Life: A Free-Energy Proposal. In Proceedings of the Evolution, Development and Complexity; Georgiev, G.Y.; Smart, J.M.; Flores Martinez, C.L.; Price,...
2019
-
[42]
From pixels to planning: scale-free active inference, 2024, [arXiv:cs.LG/2407.20292]
Friston, K.; Heins, C.; Verbelen, T.; Costa, L.D.; Salvatori, T.; Markovic, D.; Tschantz, A.; Koudahl, M.; Buckley, C.; Parr, T. From pixels to planning: scale-free active inference, 2024, [arXiv:cs.LG/2407.20292]
2024 arXiv
-
[43]
Making the Thermodynamic Cost of Active Inference Explicit
Fields, C.; Goldstein, A.; Sandved-Smith, L. Making the Thermodynamic Cost of Active Inference Explicit. Entropy 2024, 26. https://doi.org/10.3390/e26080622
2024 doi
-
[44]
Active Inference Models Do Not Contradict Folk Psychology.Synthese 2022, 200, 1–37
Smith, R.; Ramstead, M.J.D.; Kiefer, A. Active Inference Models Do Not Contradict Folk Psychology.Synthese 2022, 200, 1–37. https://doi.org/10.1007/s11229-022-03480-w
2022 doi
Reviewed August 9, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.