{"id":"ad33e451-50dc-4029-8995-31e911731340","arxiv_id":"2506.21899","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A survey that categorizes continual reinforcement learning methods, environments, and evaluation metrics for deep RL, with a focus on robotics.","lead":"This paper reviews how reinforcement learning agents can learn tasks one after another without forgetting old ones, an area called continual reinforcement learning. It organizes existing methods, benchmarks, and metrics, and is aimed at researchers entering the field.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Figure 8, the paper's map for newcomers, omits regularization as a CRL family even though EWC/Online EWC/MAS/VCL appear in the paper's own Table 4; this weakens the 'easily accessible resource' claim.","rationale":"The reader's weakest assumption is that the selected references and Figure 8 categories faithfully represent the CRL field. My analysis sharpens this into a concrete, internally checkable failure: the paper's own organizing tree omits regularization as a family, even though regularization-based methods are central to the text and listed in its own benchmarking table. This is not merely an external coverage concern about a missing systematic search; it is an internal inconsistency in the resource that newcomers are directed to use. A new researcher looking only at Figure 8 would not discover that EWC, Online EWC, MAS, or VCL are distinct CRL approaches, and would be puzzled why the text discusses them so prominently. The paper has genuine strengths: it gathers many relevant works, provides useful background on CL and RL, and includes practical benchmarking tools. Those strengths support the conditional verdict. However, the accessible-map claim depends on the map being faithful, so the taxonomy should be corrected or the advertised claim narrowed. I keep the reader's CONDITIONAL verdict because the concern is addressable in revision and does not invalidate the survey's overall usefulness.","tokens_in":48291,"tokens_out":5091,"duration_ms":57242,"concrete_test":"Use the algorithms listed in Table 4 (EWC, Online EWC, P&C, CLEAR, MAS, VCL, PackNet, Perfect Memory, A-GEM, ClonEx-SAC) and attempt to assign each to a leaf of Figure 8 using only the tree labels. If regularization-based algorithms (EWC, Online EWC, MAS, VCL) have no leaf, then the figure omits a family the paper itself benchmarks, and the accessible-map claim fails. A second pass: independently re-derive the Figure 8 tree from Section 3.2 subsections and from Khetarpal et al. (2022); count how many CRL methods discussed in §3.2 are unplaceable. Report the unplaceable fraction; if it exceeds 10% of the methods covered, the taxonomy needs revision before the paper can serve as a reliable entry point.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's advertised value is an 'easily accessible resource for new researchers' (§1 Scope), and the central mechanism for delivering that is Figure 8's tree of CRL families. That tree is internally incomplete in a way that matters for newcomers: regularization is a first-class CRL method family in the field (EWC, Online EWC, SI, MAS, RWalk, VCL all appear in the paper's own text and in Table 4's benchmark rows), yet Figure 8 has no regularization leaf. It branches only into modular architecture, parameter isolation, memory consolidation, distillation, and hybrid techniques. Consequently, Table 4's own algorithm lists (EWC, Online EWC, MAS, VCL) cannot be placed in the map the paper tells readers to use. Placement inconsistencies reinforce this: P&C is presented under Memory Consolidation (§3.2.1) even though it is a distillation/EWC-based algorithm, while distillation is a separate branch (§3.2.4). The same absence means the taxonomy diverges from the standard CRL taxonomy in Khetarpal et al. (2022), which the paper cites but does not follow. Because no search or inclusion protocol is reported, a newcomer cannot tell whether the omission is deliberate pruning or accidental coverage gap; either way, the 'comprehensive' title and the accessible-map claim are not supported by the paper's own organizing figure.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript is a survey of continual reinforcement learning (CRL). It introduces biological inspiration and continual learning (CL) background, reviews RL basics, categorizes CRL approaches into modular architectures, parameter isolation, memory consolidation, distillation, and hybrids, discusses robotic CRL, summarizes evaluation metrics and benchmark tools, and lists open challenges. The paper states its core goal as providing an easily accessible resource for new researchers and a comprehensive overview of recent CRL developments, rather than presenting a formal taxonomy.","tokens_in":48693,"tokens_out":4431,"duration_ms":48904,"significance":"If the survey were internally consistent and verifiably comprehensive, it would be a useful entry point for newcomers: it compiles a large bibliography, includes pseudo-codes for several standard CL algorithms, and tabulates environments, metrics, and benchmark platforms. The paper's central claim, however, depends on the accuracy and completeness of its organizing map and on a defensible selection of references. The taxonomy in Figure 8 omits regularization as a CRL family even though the paper's own text and Table 4 discuss EWC, Online EWC, SI, MAS, RWalk, and VCL, and the paper reports no systematic literature-search protocol to support the 'comprehensive' title. These issues weaken the accessibility claim but are correctable within the manuscript's scope.","major_comments":[{"comment":"Figure 8 is presented as the paper's map of CRL families, but it omits regularization-based approaches entirely. EWC, Online EWC, SI, MAS, RWalk, and VCL are discussed in Sections 3.2.1 and 3.2.5 and appear in the benchmark rows of Table 4, yet a newcomer using Figure 8 cannot place any of them. This is load-bearing for the 'easily accessible resource' claim in Section 1. The figure should either include a regularization branch or the paper should explicitly justify why its taxonomy departs from standard CRL taxonomies such as Khetarpal et al. (2022), which is cited but not followed.","section":"3.2, Figure 8"},{"comment":"Progress and Compress (P&C) is described in Section 3.2.1 as 'based on knowledge distillation' and using an Online EWC component, but it is categorized in Figure 8 under Memory Consolidation, while distillation is a separate branch (Section 3.2.4). This internal inconsistency makes the taxonomy harder to use. The authors should either cross-reference P&C as a hybrid or explain the assignment rule they use when an algorithm spans multiple families.","section":"3.2.1 vs 3.2.4"},{"comment":"The title and Scope promise a 'comprehensive review' and coverage of 'the latest advancements', but the paper does not report a systematic literature-search procedure, inclusion/exclusion criteria, or any coverage assessment. Without such a methodology, a reader cannot distinguish deliberate pruning from accidental omission, and the comprehensiveness claim is not verifiable. Adding a short methodology paragraph or an explicit coverage discussion would substantiate the paper's central claim.","section":"1, Scope"},{"comment":"The proposed metrics in Section 5.2 are not validated. Transfer efficiency in Eq. (12) uses an undefined performance measure P and is presented without any experimental or theoretical demonstration, and the task-identification efficiency metric is only described verbally. For a paper intended as an accessible resource for newcomers, presenting these as established metrics is misleading. They should be moved to the future-directions section or provided with precise definitions and illustrative validation.","section":"5.2"}],"minor_comments":[{"comment":"The abstract uses the term 'continuous reinforcement learning', while the rest of the paper uses 'continual reinforcement learning'; since 'continuous' also refers to continuous action/state spaces in RL, this inconsistency will confuse newcomers and should be fixed.","section":"Abstract"},{"comment":"There are several copyediting errors in this section, including 'the the need' and 'it's policy' instead of 'its policy'; these should be corrected throughout.","section":"2.3.5"},{"comment":"The citation 'Gulcehre et al., 2006' for RL Unplugged is incorrect; the actual work was published in 2020. The reference should be updated to avoid propagating a wrong year.","section":"2.3.4, References"},{"comment":"The forward-transfer formula in Eq. (8) for robotic environments uses AUC values but does not define AUC or explain the normalization denominator (1 - AUC_Tr(i)); a brief definition would make the metric reproducible for readers new to the area.","section":"5.1, Table 3"},{"comment":"The phrase '2x improvement' should be written as '2× improvement' for consistency with standard notation.","section":"4"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a survey without primary experimental results, so its value rests on the accuracy and internal consistency of its literature map. The Figure 8 omission and the lack of a search protocol are the main concerns for the journal's readership. I recommend asking the authors to address the taxonomy inconsistency and methodology gap before reconsideration; the rest of the revisions are local and should not require another full review round."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a serviceable entry point to continual RL, and the survey works best as orientation for people new to the field. It is not a comprehensive review in the strict sense, and the paper's own organizing figure has a hole that undercuts the 'accessible map' promise.\n\nWhat it does well: the background on CL and RL is solid, the distinctions among on-policy/off-policy/offline learning in Table 1 are clear, and the tables on environments, metrics, and benchmark tools are genuinely handy. The robotics section is a useful specialization that many CRL surveys lack. The metric equations in Table 3 match the cited sources, and the descriptions of EWC, SI, MAS, RWalk, replay, and P&C are accurate. No circular reasoning; self-citations are just background.\n\nThe soft spots, in order of size. First, Figure 8, the tree the paper tells newcomers to use, has no regularization leaf. It branches into modular architecture, parameter isolation, memory consolidation, distillation, and hybrid techniques. But the paper's own text and Table 4 list EWC, Online EWC, MAS, and VCL—regularization methods—as benchmark baselines. Worse, P&C is presented under Memory Consolidation even though it is a distillation/EWC hybrid, while distillation gets its own branch. So the map doesn't fit the content. That is not a minor quibble; it is the paper's central pedagogical device failing on its own terms. Second, the title says 'Comprehensive' but no search protocol or inclusion criteria are reported, so coverage is hard to verify. Third, there is a wrong citation year: Gulcehre et al. (2006) is actually RL Unplugged (2020). Fourth, the abstract and introduction say 'continuous reinforcement learning' while the rest of the paper says 'continual'—a terminology inconsistency a newcomer will trip over. The proposed metrics in Section 5.2 are fine as suggestions but are not validated, and the paper should label them as preliminary.\n\nBottom line: the reader's conditional verdict is fair. The central argument—that this is an accessible resource—holds only partly because of Figure 8 and the missing search protocol. For a newcomer, it is still a useful starting point; for an expert, it mainly serves as a pointer table. I would not cite it as a source of record in my own work, but I would send students to it for initial orientation, and I would send it to peer review because the field needs good surveys and this one is fixable. Recommended action: major revision, with Figure 8 repaired and 'comprehensive' either justified or dropped.","headline":"A useful newcomer's map of continual RL, but the paper's own Figure 8 omits regularization as a family—contradicting its own Table 4—so 'comprehensive' overpromises.","tokens_in":48966,"tokens_out":2467,"would_cite":false,"duration_ms":25732,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A survey maps continual reinforcement learning into five method families and the open problems around them.","keywords":["continual reinforcement learning","catastrophic forgetting","lifelong learning","reinforcement learning survey","robotics","benchmarks","task-agnostic learning","knowledge retention"],"falsifier":"A reader could test the comprehensiveness claim by taking the proceedings of the last few major reinforcement learning conferences, listing every continual RL method introduced, and checking whether each one fits into a branch of Figure 8 or appears in the survey's tables; any prominent missing algorithm or category that does not fit would show that the map is incomplete.","tokens_in":48124,"feed_emoji":"🧠","tokens_out":5018,"duration_ms":49840,"temperature":0.7,"pith_summary":"This survey tries to give newcomers a single readable path into continual reinforcement learning (CRL), the setting where an RL agent must learn a sequence of tasks without forgetting earlier ones. It assembles background on brain-inspired continual learning and reinforcement learning, then organizes CRL algorithms into families such as memory consolidation, modular architectures, parameter isolation, distillation, and hybrid methods. It also catalogs common evaluation environments, benchmarks, and metrics, and reviews recent CRL work in robotics. The paper's goal is descriptive: to map the field and its open problems so that a new researcher can find methods, benchmarks, and gaps quickly.","feed_headline":"Survey organizes continual RL into five method families","feed_subtitle":"Replay, modular nets, isolation, distillation, and hybrids, plus robotics benchmarks, in one newcomer-facing map.","key_machinery":"The organizing device is the taxonomy of CRL knowledge-retention methods (Figure 8), together with the tables of RL environments, benchmarks, and metrics. The taxonomy splits CRL into memory consolidation techniques (rehearsal, generative replay, dual-memory systems, distillation, hybrids), modular architecture techniques (fixed and dynamic networks), and parameter isolation techniques (pruning and masking), and the tables attach concrete algorithms, environments, and evaluation formulas to those categories. This machinery carries the survey's contribution by turning a scattered literature into a map that a newcomer can navigate by method family, application domain, and measurement choice.","core_discovery":"The paper's central claim is that continual reinforcement learning can be presented as a structured landscape with a manageable number of knowledge-retention strategies, and that this organization, paired with a summary of environments and metrics, is enough to orient a newcomer. On its own terms, the survey establishes a vertical taxonomy in which CRL approaches branch into memory consolidation techniques, modular architecture techniques, parameter isolation techniques, distillation-based approaches, and hybrid techniques, and it maps prominent algorithms and benchmarks onto these branches. It further claims that most CRL evaluations rely on a small set of environments, with Atari historically dominant and rehearsal-based methods most common there, and that robotics is a growing application area with its own evaluation conventions. The discovery, if accepted, is that the field is mature enough to be surveyed in one accessible document yet still open on issues such as task-agnostic learning, backward transfer, and realistic benchmarks.","pith_inferences":["If the survey's coverage is representative, then the field's heavy focus on catastrophic forgetting means that other desiderata, such as backward transfer and skill composition, are under-measured; adding standardized backward-transfer metrics to existing benchmarks would likely reshuffle current method rankings.","Because the survey does not describe a systematic search or inclusion criteria, its map is best read as a curated entry point; a living, community-maintained version would be needed to keep the title's promise of completeness true as the field grows.","The proposed skill re-usability and task-identification-efficiency metrics could be operationalized directly in hierarchical and task-agnostic benchmarks, giving reviewers a quantitative way to compare methods that currently report only average reward."],"forward_implications":["A newcomer can use the Figure 8 taxonomy to place any CRL algorithm into one of five method families and then find representative work and environments for that family.","Researchers comparing methods can use the survey's metric summaries (average performance, forward transfer, catastrophic forgetting, backward transfer, continual evaluation) to align their evaluation with common practice in non-robotic and robotic settings.","The review's environment table identifies Atari, Continual World, Minigrid, Procgen, and Meta-World as common testbeds, and shows that rehearsal-based algorithms dominate Atari evaluations.","The robotics section implies that sim-to-real transfer, wear-and-tear non-stationarity, and limited data are the main obstacles that CRL must solve before robots can master multiple tasks on one system.","The future-directions discussion identifies task-agnostic learning, backward transfer, and benchmark realism as the open problems most likely to define the next phase of CRL research."],"supporting_citations":[{"why":"Defines continual learning for robotics and supplies the framework the paper adopts for robotic CRL, its applications, and its open challenges.","marker":"Lesort et al., 2020"},{"why":"Provides the review-and-perspectives framing of continual reinforcement learning that the paper extends with recent robotics work.","marker":"Khetarpal et al., 2022"},{"why":"Introduces Elastic Weight Consolidation, the regularization baseline referenced throughout the CL and CRL method families.","marker":"Kirkpatrick et al., 2017"},{"why":"Contributes Continual World, the robotic benchmark whose metrics (performance, forgetting, forward transfer) the paper uses for non-robotic and robotic evaluation.","marker":"Wolczyk et al., 2021"},{"why":"Supplies the benchmark platform and metric conventions the paper draws on for continual evaluation and forward transfer.","marker":"Powers et al., 2022b"},{"why":"Progressive Networks anchor the modular and parameter-isolation branch and are cited as one of the earliest growing-architecture CRL methods.","marker":"Rusu et al., 2016"},{"why":"Provides the task-agnostic continual RL analysis and the 3RL baseline that the paper cites for task-boundary detection and off-policy replay advantages.","marker":"Caccia et al., 2023"},{"why":"Progress and Compress is the distillation-plus-EWC system that motivates the memory consolidation and distillation branches.","marker":"Schwarz et al., 2018"}],"fun_headline_variants":["Continual RL review maps five retention strategies","CRL survey: five families, one newcomer-friendly map","Five ways agents remember: a continual RL taxonomy","Robotics and benchmarks: continual RL state of the art"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The review's usefulness rests on the assumption that the selected references and the Figure 8 categories faithfully represent the current CRL field, since the paper does not describe a systematic literature search or inclusion criteria.","fun_headline_variants_meta":{"raw":{"variants":["Continual RL review maps five retention strategies","CRL survey: five families, one newcomer-friendly map","Five ways agents remember: a continual RL taxonomy","Robotics and benchmarks: continual RL state of the art"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000167,"raw_usage":{"total_tokens":1201,"prompt_tokens":830,"completion_tokens":371,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":446,"completion_tokens_details":{"reasoning_tokens":309}},"tokens_in":446,"tokens_out":371,"duration_ms":4331,"temperature":1.0,"reasoning_tokens":309,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:15:16.458079+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A reader could test the comprehensiveness claim by taking the proceedings of the last few major reinforcement learning conferences, listing every continual RL method introduced, and checking whether each one fits into a branch of Figure 8 or appears in the survey's tables; any prominent missing algorithm or category that does not fit would show that the map is incomplete.","supporting_citations":[],"review_version":1}