Pith. sign in

REVIEW 4 major objections 5 minor 68 references

Free Energy Projective Simulation (FEPS): Active inference with interpretability

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Free Energy Projective Simulation claims that a memory-graph agent with no external rewards and no neural network can learn a complete world model of a partially observable environment from prediction accuracy alone, and can then…

desk verdict Solid paper with a real derivation and useful agent architecture, but the abstract overstates what the experiments show; the clone count is doing more work than the 'prediction accuracy only' claim admits. read the letter →

arxiv 2411.14991 v1 pith:XJZBJORZ submitted 2024-11-22 cs.AI cs.LGq-bio.NCstat.ML

Joséphine Pazem , Marius Krumm , Alexander Q. Vining , Lukas J. Fiderer , Hans J. Briegel This is my paper · ORCID
classification cs.AIcs.LGq-bio.NCstat.ML
keywords activeinferencefreeenergyprincipleprojectivesimulationclone-structuredHMMpartiallyobservableMarkovdecisionprocessinternalrewardsexpectedinterpretablereinforcementlearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces Free Energy Projectile Simulation (FEPS), an active-inference agent whose memory is a graph of clone clips and whose only reinforcement signal is whether the next observation was predicted correctly. It claims that this reward-free learning rule is enough for an agent to build an accurate world model of a partially observable environment, including resolving cases where one observation is emitted by several hidden states. The claim is tested in a timed lever-press task and a 3x3 grid navigation task with symmetric smell observations, both inspired by behavioral biology. In both environments the paper reports that trained agents fully disambiguate the ambiguous observations using context alone, and that a single change to the preference distribution re-targets the policy to a different goal without additional environment interaction. If correct, this offers an interpretable, non-neural route to active inference in which goals are specified as preferences rather than learned as rewards.

What carries the argument

The load-bearing object is the clone-structured Episodic and Compositional Memory (ECM) with confidence-weighted trajectory rewards. Each clone clip is bound to exactly one observation, so the emission likelihood is a delta function; ambiguity therefore lives entirely in the transition edges, which are updated only when predictions are confirmed. The confidence attribute $f$ counts the number of consecutive correct predictions a transition edge enabled, and the $h$-value update distributes a reward proportional to $f$, creating a built-in learning schedule. Planning uses look-ahead preference propagation: an absolute preference $p^*$ on a target observation is propagated backwards through the world model for $T_h$ steps via $v_n(b_{t+1}) = \max( v_{n-1}(b_{t+1}), \max_{b^+} \beta^{n-1} r(b^+|b_{t+1}) v_{n-1}(b^+) )$, and the resulting preference distribution over belief states is fed into the expected free energy $G_{b_t}[a_t] = -H[B_{t+1}|b_t,a_t] + \mathbb{E}[\mathcal{S}^{pref}(S_{t+1},B_{t+1}|b_t,a_t)]$ with a softmax policy $\pi(a_j|b_i) = \operatorname{softmax}(\zeta G_{b_i}[a_j])$. During the wandering phase, the preference distribution is taken as the action-marginal world model, which reduces $G$ to the information gain $D_{KL}[p(B_{t+1}|b_t,a_t)\|p(B_{t+1}|b_t)]$, so exploration is driven by expected model improvement.

What would settle it

Run the timed-response task with $N_{\mathrm{clones}} = 1$ for the ambiguous observation 'light on, hungry' and otherwise identical parameters; the paper's account predicts the agent cannot distinguish the two hidden states, so the variational free energy should not reach the low plateau and the optimal wait-before-pressing policy should not emerge.

Watch

Extended reading notes

Core claim

FEPS is a Projective Simulation agent with a clone-structured world model: each of the $N_S$ observations is represented by $N_{\mathrm{clones}}$ clone clips, the emission function $p(s_t|b_t)$ is deterministic, and the transition edges between clone clips are the only trainable part. Learning is driven by prediction accuracy: when the sampled next belief state $b_{t+1}$ leads to a predicted observation $\hat{s}_{t+1}$ that matches the environment's signal $s_{t+1}^{\mathrm{env}}$, the confidence $f$ of every edge in the trajectory is incremented; when a prediction fails, the trajectory ends and each edge receives the update $h_{ij} \leftarrow h_{ij} - \gamma(h_{ij} - h_{ij}^0) + f_{ij} R$, after which confidences reset. The paper argues that this signal is sufficient for the clone clips to acquire distinct contextual meanings, so that in the timed-response environment the two clones of 'light on, hungry' come to represent the two hidden states $E_1$ and $E_2$, and in the grid the clones form a consistent topological map of the cells. From this world model the policy is set by the expected free energy $G_{b_t}[a_t]$, using an absolute preference for the target observation and a look-ahead preference over belief states computed by the recursive value update in Eq. (18). The paper's central claim is that this yields optimal policies in both environments, and that retargeting to another observation—for example, from smell 3 to smell 0 in the grid—requires only replacing the absolute preference, with no new interaction with the environment.

Load-bearing premise

The load-bearing premise is that the fixed number of clone clips per observation is large enough to represent every hidden state that emits that observation; choose too few and the agent cannot form the necessary distinct belief states, so the central claim fails.

Editorial extensions

If this is right

  • Without any external reward, an agent can learn a world model whose prediction accuracy reaches high levels, as shown by the plateau in correct-prediction trajectory lengths in both environments.
  • Clone clips acquire contextual meaning from the recent past, so observations that are emitted by several hidden states become distinguishable using only the structure of past predictions.
  • After learning, a new goal costs no retraining: changing the absolute preference distribution and recomputing the look-ahead preferences re-derives a near-optimal policy in one step, even for goals that require opposite policies.
  • A wandering phase with $\zeta>0$, where preferences equal the action-marginal world model, turns the expected free energy into an information-gain objective and avoids the dark-room problem while producing a model that predicts better when the policy is uniform.
  • Estimating belief states in superposition, by carrying all clone clips compatible with the current observation and eliminating those that fail to predict the next observation, doubles the length of correct-prediction trajectories in the grid.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One consequence the paper leaves implicit: if the required clone count grows with the number of hidden states behind an observation, a self-extending memory that adds clones when prediction errors persist would be the natural next step; the paper fixes $N_{\mathrm{clones}}$ by hand and notes that more clones may be needed for longer waiting times.
  • The wandering-phase identity $G = $ information gain suggests a bridge to model-based exploration in standard reinforcement learning: the same clone graph could serve as an intrinsic curiosity signal without a separate exploration bonus, though the paper does not test this.
  • The tested environments are deterministic; a stochastic environment would probe whether confidence-weighted rewards can still separate true context from noise, which is a direct test of the prediction-accuracy learning rule.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces Free Energy Projective Simulation (FEPS), an active-inference agent that replaces neural-network world models with a clone-structured projective-simulation graph. The world model is trained by internal reinforcement based on whether predicted observations match perceived observations (Eqs. 7-8), and actions are selected by minimizing the expected free energy (EFE) computed from this model (Eqs. 3-4, 9). The paper proposes a two-phase scheme: a wandering phase in which the EFE with model-marginal preferences reduces to information gain (Eqs. 10-14), and a task phase in which a goal observation is injected into the preference distribution and a look-ahead value heuristic propagates value over belief states (Eqs. 15-19). It also introduces a superposition method for belief-state estimation that maintains multiple candidate clone clips and prunes them on prediction mismatches (Sec. 5.2). The model is evaluated on a timed-response (Skinner-box) task and a 3x3 grid navigation task. The authors report that FEPS disambiguates aliased observations from prediction accuracy alone and recomputes near-optimal policies for new target observations without further environment interaction.

Significance. If the central claims hold, FEPS is a useful addition to interpretable active inference: it demonstrates that a graph-based, non-neural model can learn a POMDP world model from an internal prediction-accuracy signal and re-target policies by changing only the preference distribution. The paper's strongest pieces are concrete and checkable: the derivation that the wandering-phase EFE equals the information gain about belief states (Eqs. 10-14 and Appendix A) is genuine and not circular, and the numerical studies are reasonably extensive, including comparisons of task-oriented versus wandering training, several values of the scaling parameter ζ, and both single and superposition belief-state estimation. The clone-structured representation makes the learned models directly interpretable: the grid-world experiment shows a one-to-one map from clone clips to grid cells (Fig. 6b), which is a valuable demonstration.

major comments (4)
  1. [§4.1, Eq. (8), §6.1.1] The claim in the abstract that FEPS agents 'fully resolve the ambiguity ... based on prediction accuracy only' is conditional on the experimenter-supplied number of clone clips N_clones. The update rule in Eq. (8) can only strengthen or weaken existing edges; it cannot create new clone clips, so the world model cannot represent more distinct contexts per observation than the fixed N_clones. The experiments fix N_clones=2 (Sec. 6.1.1) and N_clones=3 (Sec. 6.2.1), matching the known latent multiplicity, and Sec. 6.1.1 concedes that larger waiting times 'might' require more clones. This means the agent is given prior information about the degree of aliasing, which is inconsistent with the Discussion's statement that the agent adapts 'without any prior knowledge about it' (Sec. 7). The revision should either add a data-driven mechanism for growing or splitting clone clips, or explicitly restate the result as valid for a fixed, experimenter-chosen clone count.
  2. [§5.1.2, Eq. (18), Table C] The discount factor β in Eq. (18) is not reported anywhere in Table C or the text; only its allowed range 0 ≤ β ≤ 1 is given. The look-ahead preference values v_n, and therefore the policies presented as optimal, depend quantitatively on β and on the horizon Th, so the planning experiments are not reproducible without this value. Please report the values used for both environments and, ideally, a small sensitivity analysis.
  3. [§5.1.2, Eqs. (16)-(19)] The planning method is heuristic and the paper provides no optimality guarantee for it. Eq. (18) is a monotone value-propagation rule that uses reachability under the current policy r^(k) rather than a Bellman optimality equation, and the final policy comes from a single EFE evaluation rather than an optimization over action sequences. The paper therefore does not establish the abstract's 'optimal policies for any target observation' as a theorem; the grid results show near-optimal behavior with an overhead of up to one step (Sec. 6.2.2, Fig. 6c). Please either present a formal optimality statement under explicit conditions or qualify the claim to 'empirically near-optimal' policies for the tested targets.
  4. [Abstract, §6.2.2] The phrase 'optimal policies flexibly for any target observation' is stronger than the evidence. In the grid environment the authors test two target observations (3 and 0), and in Fig. 6c the median times to target are not always equal to the shortest-path distances (an average overhead of 1.5 steps when starting directly under the target is reported). Moreover, when a target observation is emitted by more than one hidden state, a preference over that observation does not specify a unique goal; the paper itself acknowledges this limitation in Sec. 7. The abstract should be tempered to the tested target observations or the conditions under which 'any target' holds should be stated precisely.
minor comments (5)
  1. [Sec. 7] The sentence 'The behavior of FEPS agents does not depend on any reinforcement' is inaccurate as written, since Eq. (8) uses the internal reward R; it should read 'does not depend on external reinforcement.'
  2. [Sec. 5.2] The text refers to 'Figure 1c' when introducing the superposition method, but Figure 1 has only panels (a) and (b) in the caption; either add the missing panel or correct the reference.
  3. [Appendix A] There is a typo in the sentence 'In the fourth fourth line, we separated the expectation values'; it should read 'In the fourth line.'
  4. [Eq. (19)] The children set is defined using the inequality r^(k)(b^+|bt) > r^(k)(Bt+1|bt), where Bt+1 is a random variable on the right-hand side; the definition should use a threshold such as the mean of r^(k) over Bt+1, as described in the following sentence, or state this mean explicitly in the equation.
  5. [Table C] The table reports ζtask=-1 for the Skinner box and -3 for navigation, but the text in Sec. 6.1.1 says the timed task also tested a wandering condition with ζ=0; the table should clarify which rows correspond to which training phase.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the world model is trained on an external prediction-accuracy signal, and the EFE-to-information-gain reduction is a genuine algebraic derivation.

full rationale

The paper's derivations are self-contained and not circular. The world model update (Eqs. 7-8) is driven by comparing the agent's sampled prediction with the environment observation; the confidence attribute counts external prediction successes, and the h-values are reinforced proportionally to that count. This is a genuine external error signal, not a model output fed back into itself. The policy (Eq. 9) is a softmax over the expected free energy computed from the learned transition model and a user-specified preference distribution, so no fitted parameter is relabeled as a prediction. The wandering-phase identity in Eqs. (10)-(14) and Appendix A is an algebraic reduction: when the preference is set to the action-marginal p(B_{t+1},S_{t+1}|B_t), the EFE becomes D_KL[p(B_{t+1}|b_t,a_t)||p(B_{t+1}|b_t)], the information gain about the next belief state. The preference is defined before and independently of the EFE, so this is a derivation rather than a tautology. The look-ahead preference propagation (Eqs. 16-19) is a value-propagation heuristic explicitly compared to successor representation, and the final policy is tested against held-out goal observations (Fig. 6c), an external check. The main caveat, that N_clones is supplied by the experimenter and must be at least as large as the ambient ambiguity, is a hyperparameter limitation on the strength of the 'prediction accuracy only' claim; it is not an equation-level circularity because the clone count is not fit to the target performance and the learned edge weights do the disambiguation. Self-citations to Projective Simulation and clone-structured HMMs supply background building blocks rather than a load-bearing uniqueness argument.

Assumptions & free parameters 8 free parameters · 5 assumptions · 0 invented entities

The central claim rests primarily on the clone-structured world model with a hand-set number of clones per observation, on the assumption that prediction accuracy is a sufficient learning signal, and on the user-supplied ability to encode any goal as an observation. The free parameters are simulation hyperparameters; the most consequential is N_clones, which must be large enough to represent all hidden states sharing an observation. No new physical or metaphysical entities are introduced; the 'confidence' attribute and look-ahead preferences are algorithmic constructs, not entities.

free parameters (8)
  • N_clones = 2 (timed), 3 (grid)
    Fixed number of clone clips per observation; must be large enough to distinguish hidden states emitting the same observation. Chosen by the experimenter, not learned.
  • gamma = 0.001
    Forgetting rate in h-value update (Eq. 6 and Eq. 8), set in Table 1.
  • R = 3
    Reward scale in the h-value update (Eq. 8), set in Table 1.
  • zeta = -1 (timed task), -3 (grid task), +1 (grid wander); scanned over -3..3
    Softmax scaling parameter in the policy (Eq. 9). The paper reports that performance depends sensitively on its sign and magnitude, and the best value differs by training regime.
  • p_star = 0.99
    Probability assigned to the target observation in the absolute preference distribution (Sec 5.1.2).
  • N_pref = 1
    Number of policy update iterations in the look-ahead preference algorithm (Sec 5.1.2).
  • Th = 2 (timed), 3 (grid)
    Prediction horizon for the look-ahead value iteration (Eq. 18).
  • beta = not specified in paper
    Discount factor in Eq. (18), stated only as 0 <= beta <= 1; the actual value used in simulations is not reported in Table 1 or the text.
assumptions (5)
  • domain assumption The environment is a deterministic POMDP with finite hidden states, observations, and actions.
    Invoked throughout the simulations and explicitly used in Appendix B to derive the EFE limits. The results may not transfer to stochastic environments.
  • domain assumption The emission function is deterministic given the clone structure: p(s|b) = delta_{s,s(b)}.
    Core architectural assumption in Sec 4.1; this is what makes the EFE reduce to information gain in the wandering phase (Appendix A).
  • domain assumption The target observation can always be specified in the absolute preference distribution; hidden states that are not observable cannot be preferred.
    Acknowledged as a limitation in Sec 7; an externally given goal must be a sensory state.
  • standard math The expected free energy definition (Eq. 4) is a valid decision objective.
    Taken from active inference literature; the paper uses the most common formulation (citing Millidge et al. 2020).
  • ad hoc to paper Prediction accuracy alone is a sufficient reinforcement signal to learn a correct world model.
    This is the central hypothesis of FEPS (Sec 4.2). It is supported by simulations but not proven.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Free Energy Projective Simulation (FEPS): Active inference with interpretability." pith.science (2026). https://pith.science/paper/XJZBJORZ

@misc{pith2026241114991,
  author       = {Pith},
  title        = {Pith review of: Free Energy Projective Simulation (FEPS): Active inference with interpretability},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XJZBJORZ}},
  note         = {Machine review of arXiv:2411.14991}
}
read the original abstract

In the last decade, the free energy principle (FEP) and active inference (AIF) have achieved many successes connecting conceptual models of learning and cognition to mathematical models of perception and action. This effort is driven by a multidisciplinary interest in understanding aspects of self-organizing complex adaptive systems, including elements of agency. Various reinforcement learning (RL) models performing active inference have been proposed and trained on standard RL tasks using deep neural networks. Recent work has focused on improving such agents' performance in complex environments by incorporating the latest machine learning techniques. In this paper, we take an alternative approach. Within the constraints imposed by the FEP and AIF, we attempt to model agents in an interpretable way without deep neural networks by introducing Free Energy Projective Simulation (FEPS). Using internal rewards only, FEPS agents build a representation of their partially observable environments with which they interact. Following AIF, the policy to achieve a given task is derived from this world model by minimizing the expected free energy. Leveraging the interpretability of the model, techniques are introduced to deal with long-term goals and reduce prediction errors caused by erroneous hidden state estimation. We test the FEPS model on two RL environments inspired from behavioral biology: a timed response task and a navigation task in a partially observable grid. Our results show that FEPS agents fully resolve the ambiguity of both environments by appropriately contextualizing their observations based on prediction accuracy only. In addition, they infer optimal policies flexibly for any target observation in the environment.

Figures

Figures reproduced from arXiv: 2411.14991 by the authors.

Figure 1
Figure 1. Architecture and training of an FEPS agent a) Architecture of a FEPS agent, with four sensory states (squares) and two possible actions (diamonds). The agent has two main components: the world model and the policy. The world model is composed of vertices representing observations (squares) while clone clips represent all values a belief state can take (circles). As in a clone-structured graph, each clone clip b rela… view at source ↗
Figure 2
Figure 2. Estimation of belief states in superposition, after the world model has been trained. To minimize its prediction error due to faulty belief state estimation, an agent considers multiple clone clips as candidate belief states simultaneously. For the initial observation, s env 1 (on the left), the agent includes all corresponding clone clips {b i t} 3 i=1 to its hypothesis, as depicted on the right. Conditioned on the… view at source ↗
Figure 3
Figure 3. MDP for the timed response environment This environment has four hidden states. The obser￾vations are compositional and contain information that are both external (light on or off) and internal (hungry or satiated) to the agent. Arrows correspond to the transitions the actions the agent can result in. In this environment, the agent can either wait or press a lever. In order to complete the task, the agent must reach… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Training FEPS agents for the timed response task a) Evolution of the variational free energy (top) defined in Eq.(2) and expected free energy as in Eq. 4 (bottom) during the training, averaged over 100 agents and a time window of 100 episodes. At each step, the VFE dep…
Figure 5
Figure 5. Figure 5: Training results for the grid world environment. a) Evolution of the length of the trajectories during the training, for different scaling parameters ranging from -3 to 3, and different preference distributions: the agent can either learn to complete the task from the …
Figure 6
Figure 6. Figure 6: Model and robustness to reward reevaluation for the grid world environment. a) Example of a world model learned by an agent directly trained on the task, with the target positioned in the top right corner of the grid. The circles represent the belief states as in fig.1…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

68 extracted references · 58 canonical work pages

  1. [1]

    Sutton and A

    R. Sutton and A. Barto, Reinforcement Learning: An Introduction. A Bradford book, MIT Press, 1998

    work page 1998

  2. [2]

    The markov blankets of life: Autonomy, active inference and the free energy principle,

    M. Kirchhoff, T. Parr, E. Palacios, K. Friston, and J. Kiverstein, “The markov blankets of life: Autonomy, active inference and the free energy principle,” Journal of the Royal Society Interface , vol. 15, 1 2018

    work page 2018

  3. [3]

    The Active Inference Approach to Ecological Percep- tion: General Information Dynamics for Natural and Artificial Embodied Cognition,

    A. Linson, A. Clark, S. Ramamoorthy, and K. Friston, “The Active Inference Approach to Ecological Percep- tion: General Information Dynamics for Natural and Artificial Embodied Cognition,” Frontiers in Robotics and AI , vol. 5, p. 21, Mar. 2018

    work page 2018

  4. [4]

    Active Inference, homeostatic regulation and adaptive behavioural control,

    G. Pezzulo, F. Rigoli, and K. Friston, “Active Inference, homeostatic regulation and adaptive behavioural control,” Progress in Neurobiology, vol. 134, pp. 17–35, Nov. 2015

    work page 2015

  5. [5]

    The Markov blanket trick: On the scope of the free energy principle and active inference,

    V. Raja, D. Valluri, E. Baggs, A. Chemero, and M. L. Anderson, “The Markov blanket trick: On the scope of the free energy principle and active inference,” Physics of Life Reviews , vol. 39, pp. 49–72, Dec. 2021

    work page 2021

  6. [6]

    The free energy principle for perception and action: A deep learning perspective,

    P. Mazzaglia, T. Verbelen, O. C ¸ atal, and B. Dhoedt, “The free energy principle for perception and action: A deep learning perspective,” Entropy, vol. 24, 2022

    work page 2022

  7. [7]

    Reinforcement learning through active inference,

    A. Tschantz, B. Millidge, A. K. Seth, and C. L. Buckley, “Reinforcement learning through active inference,”

    work page

  8. [8]

    Active inference and learning,

    K. Friston, T. FitzGerald, F. Rigoli, P. Schwartenbeck, J. O’Doherty, and G. Pezzulo, “Active inference and learning,” Neuroscience & Biobehavioral Reviews, vol. 68, pp. 862–879, Sept. 2016

    work page 2016

Show all 68 references
  1. [9]

    Prediction, cognition and the brain,

    A. Bubic, D. Y. Von Cramon, and R. I. Schubotz, “Prediction, cognition and the brain,” Frontiers in Human Neuroscience, vol. 4, Mar. 2010. Publisher: Frontiers

    2010

  2. [10]

    Bayesian models: the structure of the world, uncertainty, behavior, and the brain,

    I. Vilares and K. Kording, “Bayesian models: the structure of the world, uncertainty, behavior, and the brain,” Annals of the New York Academy of Sciences , vol. 1224, no. 1, pp. 22–39, 2011. eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/j.1749-6632.2011.05965.x

    2011

  3. [11]

    The free-energy principle: a unified brain theory?,

    K. Friston, “The free-energy principle: a unified brain theory?,” Nature Reviews Neuroscience, vol. 11, pp. 127– 138, Feb. 2010

    2010

  4. [12]

    Predictive coding under the free-energy principle,

    K. Friston and S. Kiebel, “Predictive coding under the free-energy principle,” Philosophical Transactions of the Royal Society B: Biological Sciences , vol. 364, pp. 1211–1221, May 2009. Publisher: Royal Society

    2009

  5. [13]

    Neural representation in active inference: Using generative models to interact with—and understand—the lived world,

    G. Pezzulo, L. D’Amato, F. Mannella, M. Priorelli, T. Van De Maele, I. P. Stoianov, and K. Friston, “Neural representation in active inference: Using generative models to interact with—and understand—the lived world,” Annals of the New York Academy of Sciences , vol. 1534, pp....

    2024

  6. [14]

    Active inference in openai gym: A paradigm for computa- tional investigations into psychiatric illness,

    M. Cullen, B. Davey, K. J. Friston, and R. J. Moran, “Active inference in openai gym: A paradigm for computa- tional investigations into psychiatric illness,” Biological Psychiatry: Cognitive Neuroscience and Neuroimaging, vol. 3, pp. 809–818, 9 2018

    2018

  7. [15]

    Learned uncertainty: The free energy principle in anxiety,

    H. T. McGovern, A. De Foe, H. Biddell, P. Leptourgos, P. Corlett, K. Bandara, and B. T. Hutchinson, “Learned uncertainty: The free energy principle in anxiety,” Frontiers in Psychology , vol. 13, Sept. 2022. Publisher: Frontiers

    2022

  8. [16]

    Answering Schr¨ odinger’s question: A free-energy formulation,

    M. J. D. Ramstead, P. B. Badcock, and K. J. Friston, “Answering Schr¨ odinger’s question: A free-energy formulation,” Physics of Life Reviews , vol. 24, pp. 1–16, Mar. 2018

    2018

  9. [17]

    Collective behavior from sur- prise minimization,

    C. Heins, B. Millidge, L. da Costa, R. Mann, K. Friston, and I. Couzin, “Collective behavior from sur- prise minimization,” Proceedings of the National Academy of Sciences , vol. 121, p. e2320239121, Apr. 2024. arXiv:2307.14804 [nlin, q-bio]

    2024 arXiv

  10. [18]

    Contrastive Active Inference,

    P. Mazzaglia, T. Verbelen, and B. Dhoedt, “Contrastive Active Inference,” in Advances in Neural Information Processing Systems, vol. 34, pp. 13870–13882, Curran Associates, Inc., 2021

    2021

  11. [19]

    Deep active inference agents using Monte-Carlo methods,

    Z. Fountas, N. Sajid, P. Mediano, and K. Friston, “Deep active inference agents using Monte-Carlo methods,” in Advances in Neural Information Processing Systems, vol. 33, pp. 11662–11675, Curran Associates, Inc., 2020. 19

    2020

  12. [20]

    R-AIF: Solving Sparse-Reward Robotic Tasks from Pixels with Active Inference and World Models,

    V. D. Nguyen, Z. Yang, C. L. Buckley, and A. Ororbia, “R-AIF: Solving Sparse-Reward Robotic Tasks from Pixels with Active Inference and World Models,” Sept. 2024. arXiv:2409.14216

    2024 arXiv

  13. [21]

    Reinforcement Learning through Active Inference,

    A. Tschantz, B. Millidge, A. K. Seth, and C. L. Buckley, “Reinforcement Learning through Active Inference,” 2020

    2020

  14. [22]

    Spatial and temporal hierarchy for autonomous navigation using active inference in minigrid environment,

    D. de Tinguy, T. V. de Maele, T. Verbelen, and B. Dhoedt, “Spatial and temporal hierarchy for autonomous navigation using active inference in minigrid environment,” Entropy, vol. 26, p. 83, 1 2024

    2024

  15. [23]

    Whatever next? Predictive brains, situated agents, and the future of cognitive science,

    A. Clark, “Whatever next? Predictive brains, situated agents, and the future of cognitive science,” Behavioral and Brain Sciences , vol. 36, pp. 181–204, June 2013

    2013

  16. [24]

    How to Knit Your Own Markov Blanket:: Resisting the Second Law with Metamorphic Minds,

    A. Clark, “How to Knit Your Own Markov Blanket:: Resisting the Second Law with Metamorphic Minds,” Philosophy and Predictive Processing, 2017. Publisher: Theoretical Philosophy/MIND Group – JGU Mainz

    2017

  17. [25]

    In vitro neurons learn and exhibit sentience when embodied in a simulated game-world,

    B. J. Kagan, A. C. Kitchen, N. T. Tran, F. Habibollahi, M. Khajehnejad, B. J. Parker, A. Bhat, B. Rollo, A. Razi, and K. J. Friston, “In vitro neurons learn and exhibit sentience when embodied in a simulated game-world,” Neuron, vol. 110, pp. 3952–3969.e8, Dec. 2022

    2022

  18. [26]

    Recent advances in the application of predictive coding and active inference models within clinical neuroscience,

    R. Smith, P. Badcock, and K. J. Friston, “Recent advances in the application of predictive coding and active inference models within clinical neuroscience,” Psychiatry and Clinical Neurosciences , vol. 75, no. 1, pp. 3–13,

  19. [27]

    Active inference in robotics and artificial agents: Survey and challenges,

    P. Lanillos, C. Meo, C. Pezzato, A. A. Meera, M. Baioumy, W. Ohata, A. Tschantz, B. Millidge, M. Wisse, C. L. Buckley, and J. Tani, “Active inference in robotics and artificial agents: Survey and challenges,” 12 2021. arXiv:2112.01871

    2021 arXiv

  20. [28]

    Active preference inference using language models and probabilistic resasoning,

    T. Piriyakulkij, V. Kuleshov, and K. Ellis, “Active preference inference using language models and probabilistic resasoning,” Dec. 2023. arXiv:2312.12009

    2023 arXiv

  21. [29]

    A curiosity algorithm for robots based on the free energy principle,

    D. Kawahara, S. Ozeki, and I. Mizuuchi, “A curiosity algorithm for robots based on the free energy principle,” in 2022 IEEE/SICE International Symposium on System Integration, SII 2022 , pp. 53–59, Institute of Electrical and Electronics Engineers Inc., 2022

    2022

  22. [30]

    Intrinsic rewards for exploration without harm from observational noise: A simulation study based on the free energy principle,

    T. J. Tinker, K. Doya, and J. Tani, “Intrinsic rewards for exploration without harm from observational noise: A simulation study based on the free energy principle,” 5 2024. arXiv:2405.07473

    2024 arXiv

  23. [31]

    Projective simulation for artificial intelligence,

    H. J. Briegel and G. D. L. Cuevas, “Projective simulation for artificial intelligence,” Scientific Reports, vol. 2, 2012

    2012

  24. [33]

    Towards interpretable quantum machine learning via single-photon quantum walks,

    F. Flamini, M. Krumm, L. J. Fiderer, T. M¨ uller, and H. J. Briegel, “Towards interpretable quantum machine learning via single-photon quantum walks,” Oct. 2023. arXiv:2301.13669

    2023 arXiv

  25. [34]

    World models,

    D. Ha and J. Schmidhuber, “World models,” CoRR, vol. abs/1803.10122, 2018

    2018 arXiv

  26. [35]

    Discovering and Achieving Goals via World Models,

    R. Mendonca, O. Rybkin, K. Daniilidis, D. Hafner, and D. Pathak, “Discovering and Achieving Goals via World Models,” in Advances in Neural Information Processing Systems, vol. 34, pp. 24379–24391, Curran Associates, Inc., 2021

    2021

  27. [36]

    What is Intrinsic Motivation? A Typology of Computational Approaches,

    P.-Y. Oudeyer and F. Kaplan, “What is Intrinsic Motivation? A Typology of Computational Approaches,” Frontiers in Neurorobotics, vol. 1, p. 6, Nov. 2007

    2007

  28. [37]

    Whence the Expected Free Energy?,

    B. Millidge, A. Tschantz, and C. L. Buckley, “Whence the Expected Free Energy?,” Sept. 2020. arXiv:2004.08128

    2020 arXiv

  29. [38]

    Curiosity-Driven Exploration by Self-Supervised Predic- tion,

    D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell, “Curiosity-Driven Exploration by Self-Supervised Predic- tion,” in 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPR W) , (Hon- olulu, HI, USA), pp. 488–489, IEEE, July 2017

    2017

  30. [39]

    Active inference on discrete state-spaces: A synthesis,

    L. Da Costa, T. Parr, N. Sajid, S. Veselic, V. Neacsu, and K. Friston, “Active inference on discrete state-spaces: A synthesis,” Journal of Mathematical Psychology , vol. 99, p. 102447, Dec. 2020. 20

    2020

  31. [40]

    On efficient computation in active inference,

    A. Paul, N. Sajid, L. Da Costa, and A. Razi, “On efficient computation in active inference,” Expert Systems with Applications, vol. 253, p. 124315, Nov. 2024

    2024

  32. [41]

    Sophisticated Inference,

    K. Friston, L. Da Costa, D. Hafner, C. Hesp, and T. Parr, “Sophisticated Inference,” Neural Computation , vol. 33, pp. 713–763, Mar. 2021

    2021

  33. [42]

    Projective simulation applied to the grid-world and the mountain-car problem,

    A. A. Melnikov, A. Makmal, and H. J. Briegel, “Projective simulation applied to the grid-world and the mountain-car problem,” May 2014. arXiv:1405.5459

    2014 arXiv

  34. [43]

    Projective Simulation for Classical Learning Agents: A Comprehensive Investigation,

    J. Mautner, A. Makmal, D. Manzano, M. Tiersch, and H. J. Briegel, “Projective Simulation for Classical Learning Agents: A Comprehensive Investigation,” New Generation Computing , vol. 33, pp. 69–114, Jan. 2015

    2015

  35. [44]

    How a Minimal Learning Agent can Infer the Existence of Unobserved Variables in a Complex Environment,

    B. Eva, K. Ried, T. M¨ uller, and H. J. Briegel, “How a Minimal Learning Agent can Infer the Existence of Unobserved Variables in a Complex Environment,” Minds and Machines , vol. 33, pp. 185–219, Mar. 2023

    2023

  36. [45]

    Projective simulation with generalization,

    A. A. Melnikov, A. Makmal, V. Dunjko, and H. J. Briegel, “Projective simulation with generalization,” Scien- tific Reports, vol. 7, p. 14430, Oct. 2017

    2017

  37. [46]

    Multi-Excitation Projective Simulation with a Many-Body Physics Inspired Inductive Bias,

    P. A. LeMaitre, M. Krumm, and H. J. Briegel, “Multi-Excitation Projective Simulation with a Many-Body Physics Inspired Inductive Bias,” Feb. 2024. arXiv:2402.10192

    2024

  38. [47]

    Quantum enhancements for deep reinforcement learning in large spaces,

    S. Jerbi, L. M. Trenkwalder, H. P. Nautrup, H. J. Briegel, and V. Dunjko, “Quantum enhancements for deep reinforcement learning in large spaces,” PRX Quantum , vol. 2, p. 010328, Feb. 2021. arXiv:1910.12760

    2021 arXiv

  39. [48]

    Skill Learning by Autonomous Robotic Playing Using Active Learning and Exploratory Behavior Composition,

    S. Hangl, V. Dunjko, H. J. Briegel, and J. Piater, “Skill Learning by Autonomous Robotic Playing Using Active Learning and Exploratory Behavior Composition,” Frontiers in Robotics and AI , vol. 7, Apr. 2020. Publisher: Frontiers

    2020

  40. [49]

    Development of swarm behavior in artificial learning agents that adapt to different foraging environments,

    A. L´ opez-Incera, K. Ried, T. M¨ uller, and H. J. Briegel, “Development of swarm behavior in artificial learning agents that adapt to different foraging environments,” PLOS ONE , vol. 15, p. e0243628, Dec. 2020

    2020

  41. [50]

    Collective defense of honeybee colonies: experimental results and theoretical modeling,

    A. L´ opez-Incera, M. Nouvian, K. Ried, T. M¨ uller, and H. J. Briegel, “Collective defense of honeybee colonies: experimental results and theoretical modeling,” BMC Biology, vol. 19, p. 106, Dec. 2021. arXiv:2010.07326

    2021 arXiv

  42. [51]

    Adaptive quantum computation in changing environments using projective simulation,

    M. Tiersch, E. J. Ganahl, and H. J. Briegel, “Adaptive quantum computation in changing environments using projective simulation,” Scientific Reports, vol. 5, p. 12874, Aug. 2015

    2015

  43. [52]

    Automated Gadget Discovery in Science,

    L. M. Trenkwalder, A. L´ opez-Incera, H. P. Nautrup, F. Flamini, and H. J. Briegel, “Automated Gadget Discovery in Science,” Dec. 2022. arXiv:2212.12743

    2022 arXiv

  44. [53]

    Learning higher-order sequential structure with cloned HMMs,

    A. Dedieu, N. Gothoskar, S. Swingle, W. Lehrach, M. L´ azaro-Gredilla, and D. George, “Learning higher-order sequential structure with cloned HMMs,” May 2019. arXiv:1905.00507

    2019 arXiv

  45. [54]

    Graph schemas as abstractions for transfer learning, inference, and planning,

    J. S. Guntupalli, R. V. Raju, S. Kushagra, C. Wendelken, D. Sawyer, I. Deshpande, G. Zhou, M. L´ azaro- Gredilla, and D. George, “Graph schemas as abstractions for transfer learning, inference, and planning,” Dec

  46. [55]

    Clone-structured graph representations enable flexible learning and vicarious evaluation of cognitive maps,

    D. George, R. V. Rikhye, N. Gothoskar, J. S. Guntupalli, A. Dedieu, and M. L´ azaro-Gredilla, “Clone-structured graph representations enable flexible learning and vicarious evaluation of cognitive maps,” Nature Communi- cations, vol. 12, 12 2021

    2021

  47. [56]

    Free-Energy Minimization and the Dark-Room Problem,

    K. Friston, C. Thornton, and A. Clark, “Free-Energy Minimization and the Dark-Room Problem,” Frontiers in Psychology, vol. 3, 2012

    2012

  48. [57]

    Improving Generalization for Temporal Difference Learning: The Successor Representation,

    P. Dayan, “Improving Generalization for Temporal Difference Learning: The Successor Representation,”Neural Computation, vol. 5, pp. 613–624, July 1993

    1993

  49. [58]

    Predictive representations can link model-based reinforcement learning to model-free mechanisms,

    E. M. Russek, I. Momennejad, M. M. Botvinick, S. J. Gershman, and N. D. Daw, “Predictive representations can link model-based reinforcement learning to model-free mechanisms,” PLOS Computational Biology, vol. 13, Sept. 2017. Publisher: Public Library of Science

    2017

  50. [59]

    The successor representation in human reinforcement learning,

    I. Momennejad, E. M. Russek, J. H. Cheong, M. M. Botvinick, N. D. Daw, and S. J. Gershman, “The successor representation in human reinforcement learning,” Nature Human Behaviour , vol. 1, pp. 680–692, Aug. 2017. 21

    2017

  51. [60]

    Are animals stuck in time?,

    W. A. Roberts, “Are animals stuck in time?,” Psychological Bulletin, vol. 128, pp. 473–489, May 2002

    2002

  52. [61]

    Value of Information and Reward Specification in Active Inference and POMDPs,

    R. Wei, “Value of Information and Reward Specification in Active Inference and POMDPs,” Aug. 2024. arXiv:2408.06542

    2024 arXiv

  53. [62]

    The Nature of Reinforcement in Cerebellar Learning,

    R. F. Thompson, J. K. Thompson, J. J. Kim, D. J. Krupa, and P. G. Shinkman, “The Nature of Reinforcement in Cerebellar Learning,” Neurobiology of Learning and Memory , vol. 70, pp. 150–176, July 1998

    1998

  54. [63]

    Neocortex-Cerebellum Circuits for Cognitive Processing,

    M. J. Wagner and L. Luo, “Neocortex-Cerebellum Circuits for Cognitive Processing,” Trends in Neurosciences, vol. 43, pp. 42–54, Jan. 2020

    2020

  55. [64]

    Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects,

    R. P. N. Rao and D. H. Ballard, “Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects,” Nature Neuroscience, vol. 2, pp. 79–87, Jan. 1999

    1999

  56. [65]

    Curiosity-Bottleneck: Exploration By Distilling Task- Specific Novelty,

    Y. Kim, W. Nam, H. Kim, J.-H. Kim, and G. Kim, “Curiosity-Bottleneck: Exploration By Distilling Task- Specific Novelty,” in Proceedings of the 36th International Conference on Machine Learning , pp. 3379–3388, PMLR, May 2019. ISSN: 2640-3498

    2019

  57. [66]

    Unifying Count-Based Exploration and Intrinsic Motivation,

    M. Bellemare, S. Srinivasan, G. Ostrovski, T. Schaul, D. Saxton, and R. Munos, “Unifying Count-Based Exploration and Intrinsic Motivation,” in Advances in Neural Information Processing Systems, vol. 29, Curran Associates, Inc., 2016

    2016

  58. [67]

    A rubric for human-like agents and NeuroAI,

    I. Momennejad, “A rubric for human-like agents and NeuroAI,” Philosophical Transactions of the Royal Society B: Biological Sciences, vol. 378, p. 20210446, Jan. 2023

    2023

  59. [68]

    How to build a cognitive map,

    J. C. R. Whittington, D. McCaffary, J. J. W. Bakermans, and T. E. J. Behrens, “How to build a cognitive map,” Nature Neuroscience, vol. 25, pp. 1257–1272, Oct. 2022. A Expected free energy in the wandering phase During the wandering phase, the preference distribution is design...

    2022

  60. [2021]

    eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/pcn.13138

    doi

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.