{"id":"48354f10-543b-49d3-9e51-646c7e83ba08","arxiv_id":"2411.14991","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"FEPS agents combine projective simulation with active inference to learn world models and goal-directed policies from prediction accuracy alone, resolving ambiguous observations without external rewards.","lead":"This paper introduces FEPS, an active inference agent that uses a graph-based projective simulation memory instead of a neural network. It learns to disambiguate hidden states and re-target its policy using only prediction accuracy as an internal reward.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The strongest claim is conditional on experimenter-supplied clone counts: N_clones encodes the latent multiplicity that the agent is supposed to discover, and no mechanism is provided to set or grow it.","rationale":"The FEPS construction is internally coherent: the wandering-phase EFE reduces to an information gain in App. A, the update rules are well specified, and the experiments show that given enough clones the model can settle on a consistent clone assignment. The concern raised here is not that the mechanism is wrong but that its advertised scope is broader than what is tested. N_clones is a fixed architectural hyperparameter, not something inferred from prediction errors, so 'fully resolve ambiguity based on prediction accuracy only' is true only relative to a correctly chosen capacity. This is a limitation and an overclaim in the abstract, but it does not invalidate the demonstrated mechanism, so the reader's CONDITIONAL verdict is appropriate rather than a rejection. The missing β in Eq. (18), the untested 'any target observation' claim for aliased targets, and the absence of released code are additional reproducibility concerns that reinforce the conditional verdict.","tokens_in":26320,"tokens_out":9022,"duration_ms":92753,"concrete_test":"Run the timed-response environment of §6.1 with waiting time n=3 and N_clones=2, holding all Table 1 parameters fixed, and record the median trajectory length and VFE over 4000 episodes. If agents do not converge to the wait-three-times-then-press policy while the n=1, N_clones=2 configuration of Fig. 4 does, the successful disambiguation is shown to depend on N_clones being externally set at least as large as the latent multiplicity. A complementary check is to sweep N_clones∈{1,2,3,4} on the grid task and report the minimum N_clones per observation needed for the §6.2 results to reproduce.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's strongest claim—that FEPS agents 'fully resolve' observation ambiguity 'based on prediction accuracy only'—is conditional on a piece of environment knowledge that the agent is never given: the number of hidden states per observation. In both experiments this number is supplied by the experimenter as N_clones (2 in §6.1.1, 3 in §6.2.1), and §6.1.1 concedes that larger waiting times 'might' require more clones. The training update in Eq. (8) can only strengthen or weaken existing clone-to-clone edges; it cannot create new clone clips. Consequently, if N_clones is smaller than the true latent multiplicity behind an observation, the ambiguity is not represented in the state space at all, and no amount of prediction-error training can resolve it. The claim that disambiguation arises from prediction accuracy alone therefore presupposes the very information (the degree of aliasing) that is supposed to be discovered. The Discussion's 'without any prior knowledge about it and independently of any goal' (Sec. 7) is likewise too strong, since the clone count encodes prior knowledge about the environment's latent structure.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Free Energy Projective Simulation (FEPS), an active-inference agent that replaces neural-network world models with a clone-structured projective-simulation graph. The world model is trained by internal reinforcement based on whether predicted observations match perceived observations (Eqs. 7-8), and actions are selected by minimizing the expected free energy (EFE) computed from this model (Eqs. 3-4, 9). The paper proposes a two-phase scheme: a wandering phase in which the EFE with model-marginal preferences reduces to information gain (Eqs. 10-14), and a task phase in which a goal observation is injected into the preference distribution and a look-ahead value heuristic propagates value over belief states (Eqs. 15-19). It also introduces a superposition method for belief-state estimation that maintains multiple candidate clone clips and prunes them on prediction mismatches (Sec. 5.2). The model is evaluated on a timed-response (Skinner-box) task and a 3x3 grid navigation task. The authors report that FEPS disambiguates aliased observations from prediction accuracy alone and recomputes near-optimal policies for new target observations without further environment interaction.","tokens_in":26520,"tokens_out":8572,"duration_ms":80270,"significance":"If the central claims hold, FEPS is a useful addition to interpretable active inference: it demonstrates that a graph-based, non-neural model can learn a POMDP world model from an internal prediction-accuracy signal and re-target policies by changing only the preference distribution. The paper's strongest pieces are concrete and checkable: the derivation that the wandering-phase EFE equals the information gain about belief states (Eqs. 10-14 and Appendix A) is genuine and not circular, and the numerical studies are reasonably extensive, including comparisons of task-oriented versus wandering training, several values of the scaling parameter ζ, and both single and superposition belief-state estimation. The clone-structured representation makes the learned models directly interpretable: the grid-world experiment shows a one-to-one map from clone clips to grid cells (Fig. 6b), which is a valuable demonstration.","major_comments":[{"comment":"The claim in the abstract that FEPS agents 'fully resolve the ambiguity ... based on prediction accuracy only' is conditional on the experimenter-supplied number of clone clips N_clones. The update rule in Eq. (8) can only strengthen or weaken existing edges; it cannot create new clone clips, so the world model cannot represent more distinct contexts per observation than the fixed N_clones. The experiments fix N_clones=2 (Sec. 6.1.1) and N_clones=3 (Sec. 6.2.1), matching the known latent multiplicity, and Sec. 6.1.1 concedes that larger waiting times 'might' require more clones. This means the agent is given prior information about the degree of aliasing, which is inconsistent with the Discussion's statement that the agent adapts 'without any prior knowledge about it' (Sec. 7). The revision should either add a data-driven mechanism for growing or splitting clone clips, or explicitly restate the result as valid for a fixed, experimenter-chosen clone count.","section":"§4.1, Eq. (8), §6.1.1"},{"comment":"The discount factor β in Eq. (18) is not reported anywhere in Table C or the text; only its allowed range 0 ≤ β ≤ 1 is given. The look-ahead preference values v_n, and therefore the policies presented as optimal, depend quantitatively on β and on the horizon Th, so the planning experiments are not reproducible without this value. Please report the values used for both environments and, ideally, a small sensitivity analysis.","section":"§5.1.2, Eq. (18), Table C"},{"comment":"The planning method is heuristic and the paper provides no optimality guarantee for it. Eq. (18) is a monotone value-propagation rule that uses reachability under the current policy r^(k) rather than a Bellman optimality equation, and the final policy comes from a single EFE evaluation rather than an optimization over action sequences. The paper therefore does not establish the abstract's 'optimal policies for any target observation' as a theorem; the grid results show near-optimal behavior with an overhead of up to one step (Sec. 6.2.2, Fig. 6c). Please either present a formal optimality statement under explicit conditions or qualify the claim to 'empirically near-optimal' policies for the tested targets.","section":"§5.1.2, Eqs. (16)-(19)"},{"comment":"The phrase 'optimal policies flexibly for any target observation' is stronger than the evidence. In the grid environment the authors test two target observations (3 and 0), and in Fig. 6c the median times to target are not always equal to the shortest-path distances (an average overhead of 1.5 steps when starting directly under the target is reported). Moreover, when a target observation is emitted by more than one hidden state, a preference over that observation does not specify a unique goal; the paper itself acknowledges this limitation in Sec. 7. The abstract should be tempered to the tested target observations or the conditions under which 'any target' holds should be stated precisely.","section":"Abstract, §6.2.2"}],"minor_comments":[{"comment":"The sentence 'The behavior of FEPS agents does not depend on any reinforcement' is inaccurate as written, since Eq. (8) uses the internal reward R; it should read 'does not depend on external reinforcement.'","section":"Sec. 7"},{"comment":"The text refers to 'Figure 1c' when introducing the superposition method, but Figure 1 has only panels (a) and (b) in the caption; either add the missing panel or correct the reference.","section":"Sec. 5.2"},{"comment":"There is a typo in the sentence 'In the fourth fourth line, we separated the expectation values'; it should read 'In the fourth line.'","section":"Appendix A"},{"comment":"The children set is defined using the inequality r^(k)(b^+|bt) > r^(k)(Bt+1|bt), where Bt+1 is a random variable on the right-hand side; the definition should use a threshold such as the mean of r^(k) over Bt+1, as described in the following sentence, or state this mean explicitly in the equation.","section":"Eq. (19)"},{"comment":"The table reports ζtask=-1 for the Skinner box and -3 for navigation, but the text in Sec. 6.1.1 says the timed task also tested a wandering condition with ζ=0; the table should clarify which rows correspond to which training phase.","section":"Table C"}],"recommendation":"major_revision","confidential_remarks":"The paper makes a solid empirical contribution but the abstract is noticeably stronger than the results and the missing β parameter is a reproducibility gap. These are correctable in revision; I see no basis for rejection. The fit of the paper is good for a journal interested in AI, active inference, and biologically inspired learning."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this paper deserves a serious referee, but the headline claim is stronger than what the experiments support. The stress-test note is on target: N_clones is doing unrecognized work.\n\nWhat's actually new: a projective-simulation agent with a clone-structured HMM world model trained purely on prediction accuracy, plus two genuinely useful algorithmic pieces. The wandering-phase EFE reduction to information gain in Appendix A is a real derivation, not a restatement. The belief-state superposition estimator, which roughly doubles trajectory length in the grid, is a clean, interpretable idea. The look-ahead preference propagation is a reasonable heuristic, close in spirit to successor representation, and the one-shot planning step is a practical choice.\n\nThe soft spots are real but not fatal. The abstract says agents 'fully resolve the ambiguity of both environments based on prediction accuracy only' and 'infer optimal policies flexibly for any target observation.' The experiments show near-optimal paths on two targets, with up to one-step overhead for one of them. 'Any target' is not demonstrated. More importantly, disambiguation capacity is fixed by N_clones, chosen by the experimenter as 2 or 3. The update rule in Eq. (8) can only strengthen or weaken existing clone-to-clone edges; it cannot create new clone clips. If the true latent multiplicity exceeds N_clones, the ambiguity is unrepresentable, so the 'prediction accuracy only' claim presupposes knowledge of the degree of aliasing. The paper itself concedes in Sec. 6.1.1 that larger waiting times may require more clones. That is the load-bearing caveat. Also, the discount factor beta in Eq. (18) is not reported in Table 1, and no code or data is provided.\n\nWhat holds up: the core mechanism is supported in the tested deterministic environments. The learning curves show staged acquisition and convention formation, and the clone-structured mapping onto grid cells is genuinely interpretable. The wandering-phase derivation is sound. The paper is not sloppy about acknowledging limitations in passing, even though the abstract overclaims.\n\nThis is for readers working on active inference, interpretable RL, or cognitive modeling. I'd bring it to a reading group and would cite it. Recommendation: send it to peer review with major revision. Fix the overclaims, add a clone-count sensitivity analysis, report beta, and release code. The architecture is worth engaging with; it just needs to be presented with its actual scope.","headline":"Solid paper with a real derivation and useful agent architecture, but the abstract overstates what the experiments show; the clone count is doing more work than the 'prediction accuracy only' claim admits.","tokens_in":27072,"tokens_out":2538,"would_cite":true,"duration_ms":27783,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Free Energy Projective Simulation claims that a memory-graph agent with no external rewards and no neural network can learn a complete world model of a partially observable environment from prediction accuracy alone, and can then…","keywords":["active inference","free energy principle","projective simulation","clone-structured HMM","partially observable Markov decision process","internal rewards","expected free energy","interpretable reinforcement learning"],"falsifier":"Run the timed-response task with $N_{\\mathrm{clones}} = 1$ for the ambiguous observation 'light on, hungry' and otherwise identical parameters; the paper's account predicts the agent cannot distinguish the two hidden states, so the variational free energy should not reach the low plateau and the optimal wait-before-pressing policy should not emerge.","tokens_in":26107,"feed_emoji":"🧠","tokens_out":10105,"duration_ms":92453,"temperature":0.7,"pith_summary":"The paper introduces Free Energy Projectile Simulation (FEPS), an active-inference agent whose memory is a graph of clone clips and whose only reinforcement signal is whether the next observation was predicted correctly. It claims that this reward-free learning rule is enough for an agent to build an accurate world model of a partially observable environment, including resolving cases where one observation is emitted by several hidden states. The claim is tested in a timed lever-press task and a 3x3 grid navigation task with symmetric smell observations, both inspired by behavioral biology. In both environments the paper reports that trained agents fully disambiguate the ambiguous observations using context alone, and that a single change to the preference distribution re-targets the policy to a different goal without additional environment interaction. If correct, this offers an interpretable, non-neural route to active inference in which goals are specified as preferences rather than learned as rewards.","feed_headline":"Prediction accuracy alone teaches an agent hidden states","feed_subtitle":"A clone-memory graph disambiguates ambiguous observations and retargets policies with one preference change.","key_machinery":"The load-bearing object is the clone-structured Episodic and Compositional Memory (ECM) with confidence-weighted trajectory rewards. Each clone clip is bound to exactly one observation, so the emission likelihood is a delta function; ambiguity therefore lives entirely in the transition edges, which are updated only when predictions are confirmed. The confidence attribute $f$ counts the number of consecutive correct predictions a transition edge enabled, and the $h$-value update distributes a reward proportional to $f$, creating a built-in learning schedule. Planning uses look-ahead preference propagation: an absolute preference $p^*$ on a target observation is propagated backwards through the world model for $T_h$ steps via $v_n(b_{t+1}) = \\max( v_{n-1}(b_{t+1}), \\max_{b^+} \\beta^{n-1} r(b^+|b_{t+1}) v_{n-1}(b^+) )$, and the resulting preference distribution over belief states is fed into the expected free energy $G_{b_t}[a_t] = -H[B_{t+1}|b_t,a_t] + \\mathbb{E}[\\mathcal{S}^{pref}(S_{t+1},B_{t+1}|b_t,a_t)]$ with a softmax policy $\\pi(a_j|b_i) = \\operatorname{softmax}(\\zeta G_{b_i}[a_j])$. During the wandering phase, the preference distribution is taken as the action-marginal world model, which reduces $G$ to the information gain $D_{KL}[p(B_{t+1}|b_t,a_t)\\|p(B_{t+1}|b_t)]$, so exploration is driven by expected model improvement.","core_discovery":"FEPS is a Projective Simulation agent with a clone-structured world model: each of the $N_S$ observations is represented by $N_{\\mathrm{clones}}$ clone clips, the emission function $p(s_t|b_t)$ is deterministic, and the transition edges between clone clips are the only trainable part. Learning is driven by prediction accuracy: when the sampled next belief state $b_{t+1}$ leads to a predicted observation $\\hat{s}_{t+1}$ that matches the environment's signal $s_{t+1}^{\\mathrm{env}}$, the confidence $f$ of every edge in the trajectory is incremented; when a prediction fails, the trajectory ends and each edge receives the update $h_{ij} \\leftarrow h_{ij} - \\gamma(h_{ij} - h_{ij}^0) + f_{ij} R$, after which confidences reset. The paper argues that this signal is sufficient for the clone clips to acquire distinct contextual meanings, so that in the timed-response environment the two clones of 'light on, hungry' come to represent the two hidden states $E_1$ and $E_2$, and in the grid the clones form a consistent topological map of the cells. From this world model the policy is set by the expected free energy $G_{b_t}[a_t]$, using an absolute preference for the target observation and a look-ahead preference over belief states computed by the recursive value update in Eq. (18). The paper's central claim is that this yields optimal policies in both environments, and that retargeting to another observation—for example, from smell 3 to smell 0 in the grid—requires only replacing the absolute preference, with no new interaction with the environment.","pith_inferences":["One consequence the paper leaves implicit: if the required clone count grows with the number of hidden states behind an observation, a self-extending memory that adds clones when prediction errors persist would be the natural next step; the paper fixes $N_{\\mathrm{clones}}$ by hand and notes that more clones may be needed for longer waiting times.","The wandering-phase identity $G = $ information gain suggests a bridge to model-based exploration in standard reinforcement learning: the same clone graph could serve as an intrinsic curiosity signal without a separate exploration bonus, though the paper does not test this.","The tested environments are deterministic; a stochastic environment would probe whether confidence-weighted rewards can still separate true context from noise, which is a direct test of the prediction-accuracy learning rule."],"forward_implications":["Without any external reward, an agent can learn a world model whose prediction accuracy reaches high levels, as shown by the plateau in correct-prediction trajectory lengths in both environments.","Clone clips acquire contextual meaning from the recent past, so observations that are emitted by several hidden states become distinguishable using only the structure of past predictions.","After learning, a new goal costs no retraining: changing the absolute preference distribution and recomputing the look-ahead preferences re-derives a near-optimal policy in one step, even for goals that require opposite policies.","A wandering phase with $\\zeta>0$, where preferences equal the action-marginal world model, turns the expected free energy into an information-gain objective and avoids the dark-room problem while producing a model that predicts better when the policy is uniform.","Estimating belief states in superposition, by carrying all clone clips compatible with the current observation and eliminating those that fail to predict the next observation, doubles the length of correct-prediction trajectories in the grid."],"supporting_citations":[{"why":"It supplies the graph-based Projective Simulation memory and the $h$-value update rule that FEPS inherits.","marker":"[31]"},{"why":"It supplies the cloned-HMM idea of giving each observation multiple clone states to carry hidden context.","marker":"[53]"},{"why":"It supplies the clone-structured graph representation and the demonstration that such clones disambiguate aliased observations in navigation.","marker":"[55]"},{"why":"It supplies the graph-schema framework for transferring clone clips across observation sets and for planning with the learned graph.","marker":"[54]"},{"why":"It supplies the active-inference action-selection loop and the expected-free-energy formulation that FEPS implements.","marker":"[3]"},{"why":"It supplies the analysis of expected-free-energy variants and the exploration-exploitation trade-off that motivates the choice of the scaling parameter $\\zeta$.","marker":"[37]"},{"why":"It frames the dark-room problem that motivates the wandering-phase preference distribution.","marker":"[56]"}],"fun_headline_variants":["Prediction accuracy alone disentangles hidden states in FEPS","No deep nets: FEPS learns world models from prediction errors","Free Energy Projective Simulation: interpretable action via prediction accuracy","FEPS agents infer hidden contexts using only prediction success","Clone-structured world model: prediction accuracy is all you need"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the fixed number of clone clips per observation is large enough to represent every hidden state that emits that observation; choose too few and the agent cannot form the necessary distinct belief states, so the central claim fails.","fun_headline_variants_meta":{"raw":{"variants":["Prediction accuracy alone disentangles hidden states in FEPS","No deep nets: FEPS learns world models from prediction errors","Free Energy Projective Simulation: interpretable action via prediction accuracy","FEPS agents infer hidden contexts using only prediction success","Clone-structured world model: prediction accuracy is all you need"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000292,"raw_usage":{"total_tokens":1814,"prompt_tokens":1166,"completion_tokens":648,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":782,"completion_tokens_details":{"reasoning_tokens":566}},"tokens_in":782,"tokens_out":648,"duration_ms":6430,"temperature":1.0,"reasoning_tokens":566,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:38:39.465676+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the timed-response task with $N_{\\mathrm{clones}} = 1$ for the ambiguous observation 'light on, hungry' and otherwise identical parameters; the paper's account predicts the agent cannot distinguish the two hidden states, so the variational free energy should not reach the low plateau and the optimal wait-before-pressing policy should not emerge.","supporting_citations":[{"cited_title":"Projective simulation for artificial intelligence,","cited_arxiv_id":null,"evidence_quote":"It supplies the graph-based Projective Simulation memory and the $h$-value update rule that FEPS inherits."},{"cited_title":"Clone-structured graph representations enable flexible learning and vicarious evaluation of cognitive maps,","cited_arxiv_id":null,"evidence_quote":"It supplies the clone-structured graph representation and the demonstration that such clones disambiguate aliased observations in navigation."},{"cited_title":"Graph schemas as abstractions for transfer learning, inference, and planning,","cited_arxiv_id":null,"evidence_quote":"It supplies the graph-schema framework for transferring clone clips across observation sets and for planning with the learned graph."},{"cited_title":"The Active Inference Approach to Ecological Percep- tion: General Information Dynamics for Natural and Artificial Embodied Cognition,","cited_arxiv_id":null,"evidence_quote":"It supplies the active-inference action-selection loop and the expected-free-energy formulation that FEPS implements."},{"cited_title":"Whence the Expected Free Energy?","cited_arxiv_id":"2004.08128","evidence_quote":"It supplies the analysis of expected-free-energy variants and the exploration-exploitation trade-off that motivates the choice of the scaling parameter $\\zeta$."},{"cited_title":"Free-Energy Minimization and the Dark-Room Problem,","cited_arxiv_id":null,"evidence_quote":"It frames the dark-room problem that motivates the wandering-phase preference distribution."}],"review_version":1}