{"id":"4875c701-9c80-4862-9c10-b046bd3e7592","arxiv_id":"1908.11494","paper_version":4,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A hybrid agent called RMC, adding an RNN world model and a decaying curiosity reward to Soft Actor-Critic, is presented with performance claims over SAC, TD3, PPO, and DDPG that are not quantitatively documented.","lead":"This RL preprint combines Soft Actor-Critic with a recurrent world model and a curiosity bonus for continuous control and partially observable tasks. The authors claim state-of-the-art sample efficiency, but the paper provides no code, no error bars, and leaves key details unspecified.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The curiosity reward, a named component of RMC, is never added to the stored reward or the Q/policy losses in the paper's equations, so the algorithm as written cannot produce the reported curiosity-driven improvements.","rationale":"The reader's REJECT verdict is well-founded, but the most load-bearing problem is more specific than the reader's stated weakest assumption about representation sufficiency. The curiosity mechanism, which appears in the agent's name, the abstract, and the ablation study, is absent from the algorithmic specification: the intrinsic reward never enters the stored transition or any of the RL loss equations. This is not a matter of missing code or hyperparameters; it is a logical gap in the paper's own formal description. A corrected version that includes the intrinsic reward in the RL objective, together with the missing numerical scores, error bars, and code, would be needed before the state-of-the-art claim can be evaluated. The L1-vs-L2 mismatch in Eq. 7 and the vector-valued r^i_t in Eq. 8 are further signs that the formal description has not been carefully checked. For these reasons, the verdict should remain REJECT.","tokens_in":10215,"tokens_out":9145,"duration_ms":86926,"concrete_test":"Implement Algorithm 1 literally and compute the Q-target in Eq. 9 using the stored reward r. Then recompute the same target with r replaced by r + β·(\\hat{s}_t - s_t), as Eq. 8 dictates. If the two targets are numerically identical for all sampled batches, and likewise for the policy loss, the equations do not use curiosity. Equivalently, derive ∂L_Q/∂β and ∂L_policy/∂β from the printed formulas; both will be zero. The test is passed only if the manuscript is corrected to include the intrinsic reward in the stored transition (e.g., 'r ← r(s_t,a_t) + β·r^i_t' before line 6) and the curiosity ablation is rerun under that correction.","verdict_should_be":"REJECT","load_bearing_attack":"Section 3.3 defines an intrinsic reward r^i_t = \\hat{s}_t - s_t and a combined reward R = β·r^i_t + r^e_t (Eq. 8), and Section 4.3.2 reports that varying β changes performance. However, Algorithm 1 stores only the extrinsic reward r(s_t,a_t) in the replay buffer (line 6), and the Q-loss (Eq. 9) and policy loss (Eq. 11) use that stored reward without adding any β·r^i_t term. The temperature loss (Eq. 12) also contains no reward. Consequently, the derivative of every update objective with respect to β is identically zero: β has no influence on the actor, critic, or temperature updates. The curiosity ablation in Fig. 3c therefore cannot follow from the described method. Even if β were inserted, r^i_t is defined as a vector difference rather than a scalar norm, so Eq. 8 would add a vector to a scalar reward. This is an internal inconsistency in the core algorithm, not a question of tuning or missing baselines; as written, the method does not implement one of its named components.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RMC (Recurrent Model with Curiosity), an actor-critic agent that combines Soft Actor-Critic with an RNN-based world model. The world model is trained jointly with the critic to produce a latent state representation, and the prediction error is used to define an intrinsic curiosity reward intended to encourage exploration. The authors claim state-of-the-art results on several MuJoCo continuous-control tasks, improved robustness on a flickering MuJoCo POMDP, and analyze ablations over six backpropagation schemes and over the curiosity reward weight. The main contribution is the architecture and the joint-training scheme, but the presentation is incomplete and the algorithm as written contains several internal inconsistencies that undermine the reported experiments.","tokens_in":10474,"tokens_out":5419,"duration_ms":51148,"significance":"If the experimental claims were substantiated, RMC would be a useful demonstration that a recurrent world model with a joint model/value representation can improve sample efficiency and partially observable robustness over strong model-free baselines. The architectural idea of blocking policy gradients into the RNN and using both model and value losses for representation learning is interesting and worth investigating. However, the paper does not provide quantitative results, error bars, or statistical tests, and the algorithm description has multiple errors: the curiosity reward is never added to any stored reward or training objective, the policy loss is inverted relative to SAC, and the target-network update in Algorithm 1 is a no-op. These are not presentation issues; they mean the method as written cannot reproduce the claimed results. Without code or complete hyperparameter details, the central claims are not verifiable.","major_comments":[{"comment":"The curiosity reward is not implemented in any update. Section 3.3 defines an intrinsic reward r^i_t = \\hat{s}_t - s_t (a vector) and a combined reward R = \\beta r^i_t + r^e_t (Eq. 8), and Section 4.3.2 reports that varying \\beta changes performance. However, Algorithm 1 line 6 stores only the extrinsic reward r(s_t, a_t), and the Q-loss (Eq. 9), policy loss (Eq. 11), and temperature loss (Eq. 12) all operate on this stored reward with no \\beta r^i_t term. Consequently the derivative of every loss with respect to \\beta is identically zero, so Fig. 3c cannot be produced by the described method. Even if the reward were inserted, r^i_t is a vector difference, so Eq. 8 would add a vector to a scalar reward. The curiosity component of RMC is therefore internally inconsistent as presented.","section":"§3.3, §3.4, Algorithm 1"},{"comment":"The policy loss appears to be negated relative to standard SAC. Eq. (11) defines L_policy(\\theta) = E[Q_{\\phi_1}(s, \\tilde{a}_\\theta) - \\alpha \\log \\pi_\\theta(\\tilde{a}_\\theta|s)], and Algorithm 1 line 15 performs stochastic gradient descent on this loss. In SAC the actor is trained to maximize Q plus the entropy bonus, which corresponds to minimizing E[\\alpha \\log \\pi - Q]. As written, gradient descent on Eq. (11) would push the policy toward minimizing Q-values and decreasing entropy relative to the stated objective, so the policy-improvement step is inverted. This is a load-bearing error in the central algorithm.","section":"§3.4.2, Eq. (11), Algorithm 1 line 15"},{"comment":"The Q-function target is incomplete. The target in Eq. (9) is r + (1-d)(min_{i=1,2} Q_{\\phi_i}(s',\\tilde{a}) - \\alpha \\log \\pi(\\tilde{a}|s')), but no discount factor \\gamma multiplies the continuation term, so the Bellman backup does not match the discounted objective in Eq. (1). Additionally, Algorithm 1 line 18 updates \\phi_i \\leftarrow \\rho \\phi_i + (1-\\rho)\\phi_i. If \\phi_i is the online parameter, this leaves it unchanged; if it is intended to denote the target parameter, the notation is identical to the online parameter, making the update ambiguous. The target-network mechanism is therefore not correctly specified.","section":"§3.4.1, Eq. (9), Algorithm 1 line 18"},{"comment":"The central empirical claim that 'RMC significantly outperforms the baselines' is not backed by quantitative evidence. Section 4.1 presents only unlabeled training curves in Fig. 2 (a)-(e) with the statement that averages over five seeds were used, but no final scores, standard deviations, or statistical significance tests are reported. Similarly, the POMDP generalization results in Fig. 3a are plotted without error bars or confidence intervals. The hyperparameters for the baselines are not fully specified, and no model hyperparameters (e.g., learning rate for the model, \\beta schedule, network sizes beyond the critic/actor) are given. These omissions prevent the reader from assessing the strength or reproducibility of the claimed state-of-the-art results.","section":"§4.1 and §4.2"},{"comment":"The choice of training scheme is justified on a single environment. Section 4.3.1 reports that 'updating RNN on model loss, value loss and not on policy loss gets the best result' based on experiments on HalfCheetah-v2 only. The paper does not show that this conclusion transfers to Ant, Humanoid, Walker2d, or the POMDP setting, yet the method's general claims rely on this scheme. This is a load-bearing gap because the value/model joint representation is the core novelty of RMC.","section":"§4.3.1, Fig. 3b"}],"minor_comments":[{"comment":"Eq. (7) is labeled an L1 model loss but the formula uses the L2 norm \\|\\cdot\\|_2. The text states that L1 outperforms L2 in the authors' experiments, but no ablation supporting this is shown. The equation should be corrected to the L1 norm or the notation clarified.","section":"§3.2, Eq. (7)"},{"comment":"The sentence 'We will discuss the choice of back-propagating value loss and model loss later in section ??' contains an unresolved cross-reference 'section ??'. This indicates the manuscript is incomplete.","section":"§3.1.1, after Eq. (6)"},{"comment":"The stored tuple in line 6, (o_t, s_t, a_t, r(s_t,a_t), o_{t+1}, s_{t+1}), is inconsistent with the sampled batch in line 9, {(s,a,r,s',d)}: the terminal flag d appears only in the latter, and the observations o are not used in the loss definitions. The notation should be aligned.","section":"Algorithm 1, line 6"},{"comment":"The phrase 'tow target Q networks' should be 'two target Q networks'. Also, the entropy term in the target is written with the current policy \\pi_\\theta, but in SAC it should be evaluated with the target policy; the text does not clarify this distinction.","section":"§3.4.1, Eq. (9)"},{"comment":"The reported effect of the curiosity scale \\beta is described qualitatively ('if we use a huge scale of \\beta, the intrinsic reward dominates'), but no numerical values for \\beta or its decay schedule are provided, so the ablation cannot be reproduced.","section":"§4.3.2, Fig. 3c"},{"comment":"The caption of Fig. 2f says 'normalized score for training on Flicker MuJoCo with p = 0.5', and the text compares against 'standard SAC1 combine with frame stack', but the baseline configuration (e.g., number of stacked frames) is not specified. This should be stated for a fair comparison.","section":"§4.2, Fig. 2f"}],"recommendation":"reject","confidential_remarks":"This paper appears to be an incomplete preprint (e.g., the 'section ??' cross-reference, no code release, no quantitative results beyond curves). The algorithm contains internal inconsistencies that are independent of tuning or missing baselines: the curiosity reward is not part of any objective, the policy loss is inverted relative to SAC, and the target-network update is a no-op. These issues preclude reproducing the reported experiments from the text. I see no straightforward revision that would not effectively rewrite the method and the experimental evaluation, so the appropriate decision for a journal submission is rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper has a coherent story and one nice idea, but as written the algorithm does not implement the curiosity bonus it claims to ablate. That alone kills it for me.\n\nWhat's actually new: the specific combination of SAC with an RNN latent-dynamics head, where the RNN is trained on model loss plus value loss but gradients from the policy head are blocked. The scheme ablation in Fig. 3b is a sensible way to test that design, and the new flickering MuJoCo POMDP benchmark is useful. If the reported curves are accurate, the method does seem to help sample efficiency on MuJoCo.\n\nNow the soft spots, in order of severity. First, Section 3.3 defines the intrinsic reward as a vector r^i_t = \\hat{s}_t - s_t, then Eq. 8 adds β·r^i_t to the scalar extrinsic reward — dimension mismatch. More importantly, Algorithm 1 stores only r(s_t,a_t) in the replay buffer, and the Q-loss, policy loss, and temperature loss (Eqs. 9, 11, 12) never use β·r^i_t. So the derivative of every update with respect to β is zero. The curiosity ablation in Fig. 3c cannot be produced by the method as described. This is not a minor omission.\n\nSecond, Eq. 7 is labeled L1 but writes an L2 norm. Third, Algorithm 1's target update line is φi ← ρφi + (1−ρ)φi, which is a tautology; there is no separate target parameter in the notation. Fourth, the empirical claims rest on five-seed curves with no numerical table, no error bars, no statistical tests, and no code release. Fifth, the ablation is on HalfCheetah only, and there is no comparison to Dreamer, PlaNet, or ICM-style curiosity, which were already around.\n\nWhat holds up: the central representation-learning hypothesis is reasonable, and the design choice to block policy gradients is well-motivated. But the evidence cannot support the headline claims without the missing numbers and without fixing the curiosity mechanism.\n\nWho it's for: someone collecting examples of RL preprints that announce SOTA without releasing code or numbers, or a methods reader who wants to see the scheme-ablation idea. Not worth a serious referee slot in current form. I'd desk-reject, and invite resubmission if they release code, exact scores, and a corrected algorithm.","headline":"A plausible SAC+RNN combination that is undone by an unbuildable curiosity reward and unsupported SOTA claims.","tokens_in":10990,"tokens_out":3925,"would_cite":false,"duration_ms":39241,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper asserts that an agent combining Soft Actor-Critic with a recurrent world model and a decaying curiosity bonus beats prior model-free methods on MuJoCo control and loses less performance under partial observation.","keywords":["reinforcement learning","world model","POMDP","curiosity","recurrent neural network","Soft Actor-Critic","sample efficiency","continuous control"],"falsifier":"Train RMC and SAC with the same network sizes, replay settings, and seeds on all five MuJoCo environments, recording final return and total environment steps; the central claim is falsified if RMC does not exceed SAC in final return or reach the same return in fewer steps on most tasks. For the POMDP claim, evaluate an MDP-trained RMC at flicker probabilities from 0.1 to 0.9 and compare the normalized score drop with MDP-trained SAC; the paper predicts a strictly smaller drop at every probability.","tokens_in":9972,"feed_emoji":"🤖","tokens_out":7634,"duration_ms":75381,"temperature":0.7,"pith_summary":"The paper argues that an intelligent reinforcement-learning agent needs intuition, memory, prediction, and curiosity, and that these can be combined in one architecture. It proposes RMC, which pairs Soft Actor-Critic as the decision-making intuition with a recurrent network that learns to predict the next state and supplies a curiosity reward from prediction error. The authors claim RMC reaches new state-of-the-art results on MuJoCo continuous-control benchmarks while keeping sample efficiency and stable training, and that the same agent transfers to partially observed POMDP tasks without reconstructing hidden states by hand. If true, this would mean model-free agents can gain the benefits of a learned world model without sacrificing off-policy sample efficiency, and can tolerate missing observations more gracefully.","feed_headline":"Recurrent world model beats model-free RL on MuJoCo and flicker tests","feed_subtitle":"RMC gives its Soft Actor-Critic agent memory, prediction, and a decaying curiosity bonus.","key_machinery":"The load-bearing object is the recurrent hidden state $s_t$ produced by a GRU from the previous hidden state, the current observation, and the previous action. The paper trains this hidden state with two objective terms: a model head minimizing the residual L1 prediction error $\\|s_{t+1} - \\hat f_\\psi(s_t, a_t)\\|$ and a value head providing the soft Q Bellman loss, while gradients from the policy head are blocked. The same prediction error supplies the intrinsic curiosity reward, scaled by a decaying $\\beta$ and added to the extrinsic reward. This design is meant to push the learned $s_t$ toward a sufficient statistic of the environment so that Soft Actor-Critic's policy can act as if the state were fully observed.","core_discovery":"On its own terms, the paper's discovery is that one recurrent encoder can serve three roles at once: memory through a GRU hidden state, prediction through an L1 residual next-state model, and curiosity through a decaying prediction-error bonus, while the SAC policy acts purely on the learned hidden state. The authors report that RMC outperforms SAC, TD3, PPO, and DDPG on MuJoCo tasks, outperforms SAC with frame stacking on the flickering MuJoCo POMDP, and preserves more performance than SAC when agents trained on MDPs are evaluated under increasing levels of observation obscuring. They attribute the result to the choice of which losses shape the recurrent representation: model loss plus value loss are back-propagated into the RNN, while policy gradients are deliberately blocked for training stability.","pith_inferences":["A testable extension the paper does not pursue is to replace SAC with another off-policy learner such as TD3 or DDPG; if the recurrent representation is the real driver, RMC's gains should persist, separating the contribution of the world model from the contribution of SAC.","The paper reports no performance loss at flicker probability $p=0.5$; evaluating at higher obscuration probabilities or with longer occlusions would show whether the recurrent state truly stores missing information or only tolerates occasional blank frames.","The curiosity bonus is tested only as a decaying scalar on prediction error; an adaptive, state-dependent bonus could keep exploration alive in sparse-reward environments without the need for a fixed decay schedule.","If the learned hidden state is a sufficient statistic, the same encoder could be reused for transfer between MDP and POMDP versions of a task, which is a direct prediction the paper leaves untested."],"forward_implications":["If RMC's central claim holds, an off-policy agent can incorporate a recurrent world model without sacrificing sample efficiency, outperforming SAC, TD3, PPO, and DDPG on MuJoCo control tasks.","In the flickering MuJoCo POMDP, a single observation per time step plus a recurrent hidden state beats SAC with frame stacking, so memory can substitute for explicit observation history.","The best of the six training schemes is to back-propagate model loss and value loss into the RNN while blocking policy gradients, which prescribes how to combine supervised world-model learning with reinforcement learning.","A decaying curiosity bonus based on model prediction error improves early exploration, while setting it to zero or too large hurts performance, supporting the use of a scheduled intrinsic reward.","Because the MDP-trained RMC degrades less than SAC when observations are obscured, the framework suggests partially observed control tasks can be tackled without hand-designed hidden-state inference."],"supporting_citations":[{"why":"It supplies the SAC off-policy actor-critic algorithm that RMC uses as its intuition head.","marker":"[Haarnoja et al., 2018b]"},{"why":"It provides the twin Q-function and minimum-of-targets backup used in the critic.","marker":"[Fujimoto et al., 2018]"},{"why":"It supplies the stored-state with burn-in recurrent experience replay strategy used to train the RNN from replay.","marker":"[Kapturowski et al., 2018]"},{"why":"It motivates predicting the residual state change rather than the full next state in the model head.","marker":"[Kurutach et al., 2018]"},{"why":"It provides the L1 model loss that the paper reports outperforms L2 loss in experiments.","marker":"[Luo et al., 2018]"},{"why":"It supplies the psychological grounding for defining curiosity as prediction error.","marker":"[Loewenstein, 1994]"}],"fun_headline_variants":["RMC: SAC with memory, prediction, and curiosity via world model","Recurrent world model gives SAC memory and curiosity, beating model-free RL","One RNN for memory, prediction, curiosity: RMC tops MuJoCo and flicker tests","World model + SAC: sample-efficient RL with hidden state for POMDPs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The plan works only if the hidden state formed by jointly predicting the next state and estimating future reward is a good enough substitute for the true environment state in every task, yet the paper demonstrates this representation choice on one ablation environment, HalfCheetah-v2.","fun_headline_variants_meta":{"raw":{"variants":["RMC: SAC with memory, prediction, and curiosity via world model","Recurrent world model gives SAC memory and curiosity, beating model-free RL","One RNN for memory, prediction, curiosity: RMC tops MuJoCo and flicker tests","World model + SAC: sample-efficient RL with hidden state for POMDPs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00055,"raw_usage":{"total_tokens":2594,"prompt_tokens":886,"completion_tokens":1708,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":502,"completion_tokens_details":{"reasoning_tokens":1621}},"tokens_in":502,"tokens_out":1708,"duration_ms":13058,"temperature":1.0,"reasoning_tokens":1621,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T10:13:20.683373+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train RMC and SAC with the same network sizes, replay settings, and seeds on all five MuJoCo environments, recording final return and total environment steps; the central claim is falsified if RMC does not exceed SAC in final return or reach the same return in fewer steps on most tasks. For the POMDP claim, evaluate an MDP-trained RMC at flicker probabilities from 0.1 to 0.9 and compare the normalized score drop with MDP-trained SAC; the paper predicts a strictly smaller drop at every probability.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It supplies the stored-state with burn-in recurrent experience replay strategy used to train the RNN from replay."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It supplies the psychological grounding for defining curiosity as prediction error."}],"review_version":1}