{"id":"9b612bc8-f1de-4988-a55f-275467bd0870","arxiv_id":"2607.19749","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Under replay, DreamerV3's world model remembers old tasks but its actor forgets; supervised self-imitation on graded dreams retains behavior where RL-in-imagination fails.","lead":"What this paper found: in a model-based RL agent trained on a task sequence with an endless replay buffer, the world model keeps old-task knowledge while the actor's behavior still collapses. Why read it: it offers a fix—rehearsing the actor on the world model's own graded dreams—that retained all tasks in 3/3 runs where plain replay passed 0/3.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Recovery-race claim is confounded: the two arms do not actually train on identical imagined data (online rollouts diverge; BC uses top-25% graded subset, RL uses all), so 0/3 vs 3/3 may stem from data/filtering, not the learning channel.","rationale":"The paper is unusually transparent: pre-registration, full per-seed reporting, and explicit limitation statements. The component-level localization in §4 is strong and well-supported. However, the central conceptual contribution—that forgetting is a channel problem rather than a memory problem—rests on the §5 recovery race. The reader's weakest assumption correctly identifies that the two arms do not truly receive identical imagined data. I agree, and I would sharpen the concern by noting that even if the rollout distribution were identical, the BC arm's top-25% grading filter is a second, independent data-selection difference from the RL arm's use of all rollouts. The proposed 2×1 control (with RL on the same graded subset) would settle whether the result is due to the learning rule or to data selection/distribution. Until such a control is run, the channel-isolation claim should remain conditional, not accepted as fully established. The paper's §6 interleaved rehearsal results and §7 grading analysis remain valuable regardless of the outcome of this test, so no rejection is warranted.","tokens_in":10301,"tokens_out":10512,"duration_ms":123194,"concrete_test":"Re-run the §5 recovery race with a fixed roll-in policy: pre-generate one set of imagined rollouts from the initial drifted actor on the frozen world model (enough for 20k updates), then train (a) BC on the top-25% realized-first graded subset, (b) RL-in-imagination on the entire fixed set, and (c) RL-in-imagination on the same top-25% graded subset, all with same budget and same starts. If (c) succeeds, the original 0/3 vs 3/3 split is a selection/filtering effect, not a channel effect. If (b) succeeds, the original split is an online-distribution effect. Only if both (b) and (c) fail while (a) succeeds does the 'channel' attribution survive.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing premise of §5 is that the two recovery arms receive \"identical imagined data\" (Abstract; Figure 1 caption), making the 0/3 vs 3/3 split attributable to the learning channel. The protocol does not deliver that. \"Dream self-imitation (roll the current actor with sampling...)\" and the \"standard DreamerV3 actor–critic update\" both use online rollouts from the current actor, so after the first update the two arms' policies differ and future imagined rollouts differ. In addition, the behavior-cloning arm trains only on the top-25% of graded trajectories, while the RL arm trains on all imagined trajectories. Thus the comparison varies three things simultaneously: learning rule, data-generation distribution, and data selection/filtering. The failure of RL could be caused by having to learn from a broader, lower-quality distribution or by lacking the realized-first filter, not by the policy-gradient channel per se. Since the \"channel problem, not memory problem\" framing is the paper's central claim (Abstract, §1, §10), this confound is load-bearing. The paper's own §C lists untested mechanisms but does not test a data-locked or filtered-RL control.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies catastrophic forgetting in DreamerV3-style model-based RL with a never-cleared replay buffer. Using MiniGrid task chains and n=3 seeds, it reports that the world model retains reward discrimination, value estimates, and termination structure for earlier tasks, while the actor's behavior collapses; freezing the actor or the world model alone fails. It then reports a recovery race from a frozen world model: RL-in-imagination fails (0/3) whereas supervised self-imitation on graded imagined rollouts succeeds (3/3), with zero environment interaction. Interleaving this 'dream rehearsal' during training yields 3/3 passes on four- and eight-task chains, outperforming matched real-episode cloning, and the authors identify two failure modes of dream grading and propose a realized-first scoring rule with an offline selection gauge. All experiments are pre-registered and per-seed data are provided. The paper's central framing is that forgetting in this regime is a channel problem, not a memory problem, and that behavior can be read out of a replay-maintained world model via a supervised channel.","tokens_in":10634,"tokens_out":5536,"duration_ms":63009,"significance":"If correct, the component-level localization is an important negative result for the assumption behind replay-based continual MBRL: protecting the world model is not enough. The recovery and rehearsal results are a concrete, falsifiable intervention with practical potential. The manuscript's strengths are real: pre-registered protocols, full per-seed reporting, honest discussion of refuted hypotheses and bugs, direct probes rather than inference from returns, and an offline selection gauge. However, the headline channel-isolation claim rests on a comparison that does not actually hold data fixed; the recovery arms differ in data-generation distribution and in selection/filtering, so the 0/3 vs 3/3 split cannot yet be attributed to the learning channel. The paper's own limitations (§9) honestly delimit scale, n=3, and fixed order, but the channel claim is a central contribution and needs a controlled replication.","major_comments":[{"comment":"The claim that the two recovery arms receive 'identical imagined rollouts' is not supported by the protocol. The RL arm applies the standard DreamerV3 actor-critic update to rollouts generated by the current (drifting) actor, while the dream self-imitation arm behavior-clones the top 25% of trajectories scored by the realized-first rule, also generated by the current actor. After the first update the two actors differ, so the imagined data distributions diverge. Moreover, the BC arm trains on a filtered subset (top-quartile) while RL trains on all rollouts. Thus the 0/3 vs 3/3 split varies the learning rule, the data distribution, and the selection filter simultaneously. This is load-bearing for the 'channel problem' framing and for contribution #2. A data-locked control (e.g., freezing the rollout policy for both arms, or feeding the RL arm the same top-25% graded batch, or an unfiltere","section":"§5 and Fig. 1 caption; Abstract"},{"comment":"The paper's own Appendix C lists candidate mechanisms for the RL channel's failure, including critic-optimism contamination through the bootstrapped return and update-magnitude interference. These are not merely alternative explanations; they are entangled with the recovery-race design itself, because the RL arm uses exactly the ungraded, bootstrapped objective while the BC arm uses realized actions from graded trajectories. The current protocol therefore cannot distinguish 'RL channel is unstable' from 'RL without the grading/filtering prior is unstable.' This weakens the strong conclusion stated at the end of §5 ('the instability is the RL channel') and in the abstract.","section":"§5 with §C"},{"comment":"The comparison between dream rehearsal and real-episode cloning is well designed in spirit, but the headline paired difference (+0.131, bootstrap CI [0.073, 0.238]) is computed on n=3 pairs. With complete seed separation the qualitative claim is credible, yet the bootstrap CI is not a strong inferential statement at this sample size. The authors already state this limitation in §9; I recommend reporting the three raw paired differences explicitly in the main text so readers can assess stability, rather than only the bootstrap aggregate.","section":"§6.2 and Table 2"}],"minor_comments":[{"comment":"The abstract and §4.3 say the world model retains 'essentially everything we can measure,' but the same section reports a frozen-action-margin probe degrading to −0.25, indicating latent-space drift. The caveat that co-trained heads track the drift is fair, but the wording overstates the localization. Since the title and central claim hinge on 'the world model remembers,' the measurement basis should be stated more precisely: what is retained is replay-maintained head knowledge, not stable latent geometry.","section":"§4.3"},{"comment":"The caption says 'identical imagined data'; given the protocol, this should be revised to something like 'matched starts and same frozen world model' to avoid the implication of exactly equal trajectories.","section":"§5 and Fig. 1"},{"comment":"The definition of p_H is implicit (p_t = product_{k<t} c_k, so p_H uses the first H continuation probabilities). Consider writing p_0=1 explicitly, since the terminal bootstrap term γ^H p_H V(s_H) depends on it.","section":"§7, Eq. (1)"},{"comment":"The legend text 'per-seed points over bar means' is unclear; the figure seems to show per-seed points with means as bars, but this should be stated more directly.","section":"Fig. 2"},{"comment":"The 'audition-gated' task selection in §8 is described only informally. Please state the gate criterion (e.g., whether all candidate tasks had to reach 0.6 within the phase budget in a pilot run) so readers can gauge the selection pressure.","section":"§9 and Table 4"}],"recommendation":"major_revision","confidential_remarks":"The paper is carefully conducted and unusually transparent, with pre-registration, full per-seed reporting, and honest documentation of refuted hypotheses. The main risk is the recovery-race confound: the 'identical imagined data' claim is not delivered by the protocol, and this underpins the paper's strongest theoretical claim. I do not see this as fatal, but it requires additional controls or a re-framing before the channel-isolation result can be accepted. This is a fixable issue within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this one. First, the §4 localization—world model under never-clear replay retains reward discrimination, values, and termination structure while the actor's behavior collapses—is a genuinely new measurement. The cited continual-MBRL line assumed protecting the world model preserves policy; nobody actually probed components. The probes are direct, pre-registered, per-seed reported. That part is solid and worth building on.\n\nSecond, the recovery-race result that frames forgetting as a \"channel problem\" is not as clean as the abstract claims. The two arms do not receive identical imagined data. The dream self-imitation arm rolls out with the current actor and clones the top 25% graded trajectories; the RL arm uses the standard Dreamer update over all imagined trajectories. After the first update the policies differ, so subsequent rollouts diverge, and the filtering differs too. So the 0/3 vs 3/3 split could come from data distribution or selection rather than the learning channel per se. The paper's own §C lists untested mechanisms but never runs the data-locked or filtered-RL control that would settle it. That honestly weakens the headline claim.\n\nWhat the paper does well: the dream rehearsal method itself works on these chains—3/3 four-task and eight-task passes, no task labels, no parameter growth, and it beats matched real-episode cloning with complete seed separation. The grading analysis in §7 is careful: two scoring failure modes, an offline gauge, and a realized-first rule that closes them. The transparency is exemplary: pre-registration, per-seed values, refuted hypotheses, even self-reported bugs. That is real evidence.\n\nThe soft spots beyond the confound are minor-to-moderate: n=3 on a single domain with one fixed order, hand-set thresholds without sensitivity analysis, and rehearsal cost scaling linearly with task count. The authors acknowledge most of this.\n\nWho is this for? People working on continual MBRL and replay-based continual learning generally. The localization result will be cited; the \"channel problem\" framing should be treated with caution until the control is run. It deserves a serious referee—the paper is important and honest enough that the confound is exactly the kind of thing a good reviewer would ask to fix.","headline":"A transparent, useful localization of forgetting in replay-protected MBRL, but the paper's signature channel-isolation comparison is confounded by non-identical training data.","tokens_in":11094,"tokens_out":2418,"would_cite":true,"duration_ms":27432,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"In replay-protected world-model agents, forgetting is a channel problem, not a memory problem.","keywords":["continual reinforcement learning","world models","catastrophic forgetting","self-imitation learning","dream rehearsal","model-based RL","channel localization"],"falsifier":"Re-run the recovery race with both arms consuming the same pre-generated, fixed set of imagined trajectories (no on-policy regeneration), so the data distribution cannot diverge between arms. If RL-in-imagination then recovers the lost skill on a substantial fraction of seeds, the claim that the policy-gradient channel itself is the failure would be falsified.","tokens_in":10185,"feed_emoji":"🧠","tokens_out":5186,"duration_ms":50312,"temperature":0.7,"pith_summary":"This paper asks which component of a model-based reinforcement-learning agent actually forgets when tasks are learned sequentially. Using a Dreamer-family agent with a never-cleared replay buffer on MiniGrid environments, it measures the world model, the reward/value heads, and the actor separately. The world model keeps essentially everything: reward discrimination, value estimates, and termination structure for old tasks remain intact. The actor's behavior collapses anyway, and the paper shows the failure is in the policy-gradient channel: reinforcement learning in imagination cannot recover a lost skill, while supervised self-imitation on the same world model's graded dreams can. Interleaving this dream rehearsal during training produces a continual learner that retains all tasks without task labels or added parameters, on four- and eight-task chains—and this reframes what replay-based continual MBRL should protect.","feed_headline":"World model remembers, actor forgets: forgetting is a channel problem","feed_subtitle":"Supervised imitation on graded dreams recovers lost skills where RL in imagination fails, 3/3 vs 0/3 seeds.","key_machinery":"Graded dream rehearsal: after each task phase, the agent keeps that phase's buffered episodes as rehearsal starts; during later-task training it imagines rollouts from those states with the current sampling actor, scores each imagined trajectory with the live reward/continuation/value heads, and behavior-clones the top 25%. The scoring rule (Equation 1) is load-bearing: a realized-first gate that weights each step by reach probability under the continuation head, prioritizes trajectories that actually achieved reward in imagination, and bootstraps only through surviving steps. This makes grading safe for imagined data, where dreams do not stop at their own endings and the critic's optimism i","core_discovery":"Under never-clear replay, the world model of a Dreamer-family agent is a sufficient behavioral memory at the tested scale: reward discrimination, value estimates, and termination structure for old tasks survive (retention ratio near 1.0), and the actor loses old-task behavior anyway. With the world model frozen and identical imagined rollouts, policy-gradient RL in imagination fails to recover a lost skill (0/3 seeds); supervised self-imitation on the world model's own graded dreams recovers it (3/3 seeds, zero environment steps). So forgetting in this regime is a channel problem, not a memory problem. Interleaved graded dream rehearsal—behavior cloning the top 25% of imagined trajectories s","pith_inferences":["Our inference: the paper's candidate explanation—that the actor–critic bootstrap chases unconstrained critic optimism in imagination—is directly testable by penalizing or calibrating the critic during imagination-only recovery; a successful RL recovery under pessimistic bootstrapping would confirm the mechanism.","Our inference: the result suggests that any world model able to generate realistic rollouts could serve as a data source for supervised policy distillation, potentially transferring to settings where reward or continuation heads are unreliable.","Our inference: the reported recovery race may be confounded if the RL arm's imagined data distribution drifts away from demonstrated states after the first update; a fixed pre-generated dream buffer held identical across both arms would settle whether the channel, not the data, is the cause."],"forward_implications":["If the localization is right, replay-based continual MBRL should stop treating the world model as the component needing protection; it is already protected by replay, so effort should go into a stable behavioral readout.","A single actor, with no task labels and no parameter growth, can retain long task chains when periodically imitating graded dreams of prior tasks.","The grading rule, not the rehearsal schedule, is where the difficulty lies: a naive scorer selects near-randomly on short lethal episodes (AUC 0.49–0.56), and the realized-first rule closes both failure modes.","Dream rehearsal outperforms matched real-episode cloning on the hardest task by a paired difference of +0.13 with complete seed separation, so imagination contributes beyond the simplest supervised channel.","At doubled chain length (eight tasks), the mechanism still passes all seeds, with no observed rehearsal dilution; retention settles at each task's own competence ceiling."],"fun_headline_variants":["Actor forgets, world model remembers: graded dreams recover skills","Graded dream imitation revives lost skills where RL fails","Policy collapse is a channel problem, not memory loss","Dream rehearsal, not replay, fixes catastrophic forgetting"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that an imagined rollout from the current actor is a faithful stand-in for real old-task experience, and that the two recovery arms truly train on identical imagined data; if the RL arm's data drifts further from old-task states than the imitation arm's does, the channel-isolation conclusion weakens.","fun_headline_variants_meta":{"raw":{"variants":["Actor forgets, world model remembers: graded dreams recover skills","Graded dream imitation revives lost skills where RL fails","Policy collapse is a channel problem, not memory loss","Dream rehearsal, not replay, fixes catastrophic forgetting"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000704,"raw_usage":{"total_tokens":3062,"prompt_tokens":845,"completion_tokens":2217,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":589,"completion_tokens_details":{"reasoning_tokens":2161}},"tokens_in":589,"tokens_out":2217,"duration_ms":17914,"temperature":1.0,"reasoning_tokens":2161,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T11:48:10.125150+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the recovery race with both arms consuming the same pre-generated, fixed set of imagined trajectories (no on-policy regeneration), so the data distribution cannot diverge between arms. If RL-in-imagination then recovers the lost skill on a substantial fraction of seeds, the claim that the policy-gradient channel itself is the failure would be falsified.","supporting_citations":[],"review_version":1}