{"id":"8414fc6a-3639-4260-8f59-e150fed52448","arxiv_id":"2411.11188","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Using two-hot classification for value prediction and binary-filtered imitation for policy updates makes multi-task meta-RL training scale-invariant to reward magnitudes, improving performance across five benchmarks without task labels.","lead":"This paper shows that training a Transformer-based reinforcement learning agent with classification-style losses for both the actor and critic lets it learn many different tasks at once without being told which task it is playing. The approach may make it easier to scale RL agents toward general, multi-game behavior with in-context adaptation.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The scale-invariance mechanism is not isolated from other benefits of the two loss changes; the dependent baseline's global PopArt is weaker than the per-task normalization it is meant to stand in for.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the experiments attribute gains to scale invariance without cleanly separating it from other benefits of classification and filtered imitation. My reading agrees, and I consider this the single most important issue because the abstract and conclusion make scale-invariance the explanatory mechanism, not merely an incidental property. The paper has real independent support: the four-way ablation isolates the loss choice, the reward-rescaling experiment in Procgen directly tests scale sensitivity, and the code is released. However, that support is incomplete. The scaling experiment shows that Dep/Dep is hurt by artificial scale changes and Ind/Ind is more robust, but it does not show that the natural scale differences across tasks are why Ind/Ind wins in the unscaled benchmarks. The use of a global, task-agnostic PopArt layer in the dependent baseline is a specific weakness: standard PopArt is per-task, and a label-free method that infers task identity from context could plausibly implement per-task normalization. Without that control, the central causal claim remains underdetermined. The proposed concrete test would settle the question by comparing against per-task normalization with labels as a diagnostic, and then against a label-free inferred-task version. If the gap persists with per-task normalization, the paper's recipe is still valuable but the scale-barrier mechanism is incorrect, which would materially change the conclusion. Since the reader already conditioned the verdict on this uncertainty, no change to the verdict is needed; the conditional status is the appropriate outcome.","tokens_in":33304,"tokens_out":3526,"duration_ms":40243,"concrete_test":"Run the Meta-World ML45 and Multi-Game Atari setups with the dependent actor/critic but with per-task PopArt normalization using ground-truth task IDs during training only (as a diagnostic, not as the final agent), keeping all other hyperparameters fixed. If this baseline closes most of the gap to the independent/independent update, the gains are attributable to return-scale imbalance; if a large gap remains, classification and imitation benefits beyond scale invariance are responsible. Additionally, train a label-free baseline that infers a task embedding from the context window and applies per-task normalization to test whether explicit task labels are truly unnecessary.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that converting actor and critic objectives to classification terms decouples optimization from return scale and thereby breaks the multi-task barrier. The load-bearing condition is that the observed gains are caused by removing scale dependence, not by other properties of the two losses. The four-way ablation (Section 4, Figure 4) holds architecture and replay details fixed, but each loss swap changes two things at once: the loss functional form and its sensitivity to Q-scale. Two-hot classification is known to improve representation learning and robustness to noisy targets even in single-task RL (concurrent work [34] makes exactly this argument), and binary-filtered imitation adds off-policy stabilization. The reward-rescaling experiment in Figure 6 demonstrates that the dependent update is more sensitive to artificially injected scale differences, but it does not establish that natural return-scale imbalance is the active bottleneck in Meta-World, Atari, or BabyAI. Appendix A states that the dependent baseline uses a global, task-agnostic PopArt layer, whereas the PopArt solution cited as the standard approach is per-task normalization with task labels. A global layer normalizes the aggregate scale but leaves per-task relative imbalances intact, so the comparison may overstate the advantage of scale-resistant losses over a stronger label-free baseline. A per-task or inferred-task normalization control is missing, leaving the causal attribution to scale invariance unproven.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a modification to the AMAGO Transformer-based meta-RL agent that replaces the standard scale-dependent actor and critic losses (Eqs. 1-2) with scale-insensitive classification objectives: a two-hot symlog value classifier for the critic (Eq. 3) and an advantage-filtered imitation loss for the actor (Eq. 4). The authors evaluate all four combinations of dependent/independent actor and critic losses on Meta-World ML45, multi-task POPGym, multi-game Procgen, multi-game Atari, and BabyAI, reporting that the fully independent combination improves performance and sample efficiency in most settings. They conclude that scale-invariant objectives break a 'multi-task barrier' in online meta-RL without requiring task labels.","tokens_in":33628,"tokens_out":4739,"duration_ms":49726,"significance":"If the mechanism claim holds, the paper offers a simple, label-free recipe for scaling online meta-RL to heterogeneous task mixtures, with code and per-task learning curves provided in the appendix. The main empirical contribution, namely consistent gains of the fully independent update across five benchmarks, is meaningful and well documented. However, the causal attribution to return-scale invariance is not isolated from other properties of the two loss changes, and the dependent baseline may be weaker than a proper scale-normalized control. These issues leave the title-level claim under-supported, although the practical value of the loss combination is credible.","major_comments":[{"comment":"The central claim that the gains come from decoupling optimization from return scale is not established by the 2x2 ablation. Each loss change alters two things at once: the loss functional form and its sensitivity to Q-scale. Two-hot classification (Eq. 3) is known to improve representation learning and robustness to noisy targets even in single-task RL, as the paper itself notes by citing reference [34], and advantage-filtered imitation (Eq. 4) adds an off-policy stabilization effect. The paper does not include a control that isolates scale-invariance, such as training the dependent losses with per-task or inferred-task return normalization, or comparing the independent update against a task-agnostic normalization that removes global scale differences. Without such a control, the results support the weaker claim that classification-style losses help multi-task meta-RL, but not the stronger claim that scale invariance is the causal mechanism.","section":"Section 3, Eqs. (3)-(4); Figure 4; Appendix A"},{"comment":"The dependent baseline uses a global, task-agnostic PopArt layer, as stated in Appendix A, yet the paper cites per-task PopArt as the standard multi-task solution in Section 2. A global layer normalizes the aggregate scale but leaves per-task relative imbalances intact, so the comparison may overstate the advantage of scale-resistant losses over a stronger label-free baseline. The reward-rescaling experiment in Figure 6 demonstrates that the dependent update is more sensitive to artificially injected scale differences, but it does not quantify the natural return-scale imbalance in Meta-World, Atari, or BabyAI. The paper should either add a stronger normalization baseline or present evidence that natural return-scale imbalance is the active bottleneck in those domains; otherwise the conclusion that scale invariance 'should be a priority' in Section 5 overreaches.","section":"Appendix A; Section 4, Figure 6"},{"comment":"Statistical claims such as 'significant progress' and 'more than double' rest on error bars that are the minimum and maximum over three or four independent trials, and Figure 6 uses only two trials. The paper should report means with standard errors or confidence intervals, state the number of seeds explicitly, and, where possible, include a paired significance test across tasks or seeds. This is necessary to support the strength of the headline empirical claims, especially in comparisons to prior methods such as RL2-PPO and MuZero that use different architectures, budgets, and hyperparameters.","section":"Section 4, Figures 4-6, 9-10"},{"comment":"The paper explicitly states that the trade-offs among bin count B, bin limits, and the use of symlog are 'underexplored in the results of this work.' Because the two-hot critic requires choosing B and the return bounds, and because Figure 7 shows that the multi-task Procgen result depends on the number of frequently used labels, the manuscript should include at least one multi-task sensitivity analysis for these choices or justify the fixed configuration more strongly. Without this, the 'simple and scalable' claim is qualified by an unstudied hyperparameter dependence.","section":"Appendix A, Value Classification Details"}],"minor_comments":[{"comment":"The comparisons to RL2-PPO, MuZero, and HTrMRL are gathered from prior papers with different architectures, budgets, and hyperparameters; the text should label these as reference curves rather than implying a controlled comparison, especially when stating that the proposed method 'more than doubles' the RL2 success rate.","section":"Section 4, Meta-World ML45"},{"comment":"There are typographical errors in the manuscript: Figure 1 contains 'agentso' and Figure 10 is titled 'Multli-Task BabyAI' instead of 'Multi-Task BabyAI'.","section":"Figures 1 and 10"},{"comment":"The statement that the independent actor loss 'becomes IL on expert datasets' is imprecise: the binary filter is based on a learned advantage estimate, not on expert labels, so the imitation is only approximate and depends on the quality of the critic.","section":"Section 3, Eq. (4)"},{"comment":"The use of a global PopArt layer for the dependent baseline is an important design choice that should be highlighted in the main text rather than only in the appendix, because it directly affects the interpretation of the headline ablation in Figure 4.","section":"Appendix A, Base RL Details"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely to be a competitive empirical contribution if the authors either add a control that isolates scale-invariance or soften the mechanism claim. The main risk is overclaiming causality: the demonstrated improvement of classification-style losses over a global-PopArt dependent baseline is not by itself evidence that return-scale imbalance is the causal bottleneck. The self-citations to AMAGO are appropriate as the base agent, and the concurrent work [34] is explicitly acknowledged, so novelty concerns are manageable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this is the most convincing demonstration I've seen that classification-based actor-critic losses scale to online multi-task meta-RL without task labels. Second, the causal story—that removing return-scale dependence is what does the work—is supported but not nailed down, and the dependent baseline is weaker than it could have been.\n\nThe 2x2 ablation is the right design. They hold architecture and replay details fixed and swap actor/critic losses independently, showing the two-hot critic is the main driver and the CRR-style actor adds a consistent boost. The Procgen reward-rescaling experiment directly manipulates scale and shows the dependent update collapses while the independent one stays robust. That is a genuine test of the mechanism, not just of the recipe. They also ship code and are transparent about tuning details in Appendix A, including the under-explored bin-count trade-offs. The evaluation spans five very different benchmarks—robotics, memory POMDPs, Procgen, Atari, BabyAI—which is rare in this area.\n\nThe soft spot is the baseline representing \"dependent\" learning. Appendix A says they use a global, task-agnostic PopArt layer; that normalizes the aggregate scale but leaves per-task imbalances intact. The paper's framing compares against per-task PopArt, which needs labels, but a stronger label-free baseline would infer task identity from context and normalize per task. I suspect the main result would survive such a baseline—this recipe is simple and works—but the title's \"breaking the multi-task barrier\" is stronger than what is proven. Also, error bars are min/max over three or four seeds; common for large-scale RL, but not a substitute for standard errors. The comparisons to RL2-PPO, MuZero, and HTrMRL are reference numbers from other papers with different budgets and architectures; the paper labels them clearly, so I wouldn't fault it much, but they are not controlled.\n\nWho should read this: anyone working on multi-task or meta-RL, especially on label-free generalization. The recipe is simple enough to adopt right away, and the reward-rescaling result is a useful negative control for future work.\n\nIt deserves serious review. A referee should push on the mechanism claim and ask for a per-task normalization control, but the empirical contribution stands on its own.","headline":"A solid, well-validated empirical paper showing two-hot critic classification plus binary advantage-filtered actor updates substantially improve label-free multi-task meta-RL; the scale-invariance mechanism is plausible but not fully isolated.","tokens_in":34101,"tokens_out":2352,"would_cite":true,"duration_ms":24455,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Converting both actor and critic objectives to classification terms that ignore the current scale of returns lets a single Transformer-based agent learn and adapt across many tasks without labels.","keywords":["meta-reinforcement learning","multi-task reinforcement learning","in-context learning","transformers","scale invariance","two-hot value classification","advantage-weighted imitation","long-term memory"],"falsifier":"A decisive test would train the same agent on a set of tasks whose return scales are deliberately equalized and compare the scale-resistant losses to standard regression; if the classification losses still win, their benefit is not scale invariance. The complementary check is to multiply the rewards of a single task by a large constant and observe whether Q-dependent losses degrade much faster than the scale-resistant ones.","tokens_in":33057,"feed_emoji":"🤖","tokens_out":8568,"duration_ms":80771,"temperature":0.7,"pith_summary":"This paper tries to establish that the \"multi-task barrier\" in meta-reinforcement learning is largely a problem of uneven return scales, and that it can be removed without task labels by making both learning objectives scale-resistant. The proposed agent, built on a Transformer policy that conditions on trajectory context, replaces value regression with two-hot classification over symlog-compressed return bins and replaces the Q-weighted policy update with a binary advantage-filtered imitation objective. Because neither loss is scaled by the size of current rewards, tasks with high absolute returns no longer dominate the gradient. Across Meta-World ML45, Multi-Task POPGym, Multi-Game Procgen, Multi-Game Atari, and BabyAI, the scale-resistant combination improves multi-task performance and sample efficiency relative to the same architecture trained with standard actor-critic losses. A sympathetic reading takes the paper's central claim to be that scale invariance is a first-order design requirement for adaptive memory-based agents, not that these are the only useful properties of the two losses.","feed_headline":"Two scale-resistant losses double multi-task meta-RL success","feed_subtitle":"Converting both learning losses to classification lets one agent handle 45 robot skills and 10 Atari games without task labels.","key_machinery":"The load-bearing mechanism is a pair of scale-resistant classification updates. The critic's two-hot classification converts a scalar TD target $y_t$ into probabilities over $B$ fixed bins through $\\mathrm{twohot}_B(\\mathrm{symlog}(y_t))$, with a symlog transform $\\mathrm{symlog}(y)=\\mathrm{sign}(y)\\ln(|y|+1)$ compressing a wide return range into a bounded label space and symexp recovering the scalar. This makes the loss depend on bin-count resolution rather than on the absolute size of returns. The actor's binary advantage filter turns policy improvement into imitation of actions whose one-step advantage estimate is positive, so the update is a maximum-likelihood classification over a dynamically selected subset of the replay buffer. The shared Transformer consumes trajectories of observations, actions, rewards, and reset signals, so task identity must be inferred from context rather than provided; an ensemble of critics and multiple discount factors are used as in the AMAGO baseline. The machinery's role is to keep the optimization process insensitive to each task's current return scale while preserving the memory and adaptation behavior of sequence-model RL.","core_discovery":"The paper's central claim is that a single Transformer-based meta-RL agent can learn and adapt across qualitatively different tasks without task labels if its actor and critic updates depend on Q-values only through classification. Concretely, the critic loss is $L_{\\text{Critic-Ind}}(t) = -\\mathrm{twohot}_B(y_t)^T \\log Q_B(h_t, a_t)$, where the TD target $y_t$ is mapped with a symlog transform into two adjacent bins, and the actor loss is $L_{\\text{Actor-Ind}}(t) = -\\mathbf{1}\\{Q(h_t,a_t)-\\mathbb{E}_{a'\\sim\\pi(h_t)}[Q(h_t,a')] > 0\\}\\log\\pi(a_t\\mid h_t)$, so the policy imitates only actions whose estimated advantage over the current policy is positive. Both losses keep the one-step TD structure and can be computed in parallel over a context window, but they do not grow or shrink with the absolute magnitude of returns in any task. The paper reports that this pair of updates more than doubles ML45 success over the original RL2 result at 100M timesteps, matches MuZero's reported 50M-timestep result with a simpler one-step Q-learning, improves 8 of 10 unclipped Atari games, and remains largely unaffected by deliberately rescaling rewards in Procgen by factors of 10 and 100. The paper reads this evidence as supporting the hypothesis that multi-task training is bottlenecked by return-scale imbalance.","pith_inferences":["If the scale-invariance mechanism is the true driver, the same two losses should transfer to non-Transformer sequence models such as recurrent or state-space policies; this is not tested in the paper and would be a direct way to check the claim's scope.","The paper's binary actor filter means the policy update is automatically ignoring a changing fraction of the replay buffer; an untested corollary is that the method should be robust to strongly imbalanced replay data without explicit rebalancing, since it learns from the positive-advantage subset.","The dependent-dependent baseline uses a global, task-agnostic normalization layer rather than the per-task normalization PopArt originally proposed, so the reported gap may overstate the value of scale invariance relative to a label-free per-task solution; comparing against an oracle that recovers task identity from context would clarify the margin.","A clean single-task control with artificially separated return scales could separate scale-invariance from other benefits of two-hot classification such as representation learning and noisy-target robustness; the paper does not run this control."],"forward_implications":["The same Transformer architecture, with no task-specific heads or labels, can be trained across heterogeneous task suites; the paper demonstrates this on ML45, 27 POPGym tasks, 16 Procgen games, 10 Atari games, and 68 BabyAI task configurations.","Because the losses do not depend on return scale, deliberate reward rescaling (Coinrun rewards multiplied by 100 and Climber rewards divided by 10) barely hurts the scale-resistant agent, while the Q-dependent baseline loses most of its performance in the unaffected games.","Scale-resistant critics and actors improve not just aggregate scores but coverage: in ML45 the number of manipulation skills mastered roughly doubles compared with Q-dependent updates and the original RL2 reference.","Long-context memory settings benefit as well: in the Procgen memory mode and two-episode BabyAI evaluations, the agent improves from the first to the second attempt on unseen levels and held-out tasks, showing adaptation and memory can coexist with multi-task training.","The recipe reduces RL updates to two supervised-learning-style classification heads, pointing to a simpler path for scaling online RL with recycled data."],"supporting_citations":[{"why":"Supplies the multi-task scale-imbalance hypothesis and the per-task PopArt normalization that the paper contrasts with its label-free approach.","marker":"[25]"},{"why":"Provides the symlog compression and wide-limit two-hot binning used in the scale-resistant critic.","marker":"[31]"},{"why":"Provides the binary advantage-filtered imitation objective (CRR binary mean) used as the scale-resistant actor update.","marker":"[32]"},{"why":"Is the AMAGO baseline whose Transformer actor-critic, replay, ensemble-critic, and multi-discount details the paper inherits and ablates.","marker":"[57]"},{"why":"Is the concurrent work showing two-hot critic classification improves RL more generally; the paper distinguishes its scale-invariance attribution from that alternative explanation.","marker":"[34]"},{"why":"Defines the Meta-World ML45 benchmark and the RL2-PPO reference success rate that the paper's main robotic manipulation results improve on.","marker":"[17]"},{"why":"Defines the POPGym benchmark and supplies the PPO-GRU baseline that the multi-task POPGym results are measured against.","marker":"[27]"}],"fun_headline_variants":["Classification losses let one RL agent master 45 skills and Atari","No task labels needed: AMAGO-2 scales meta-RL via classification losses","RL agent adapts across tasks by treating Q-values as classification","AMAGO-2: Two classification losses break multi-task RL bottleneck"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the two losses' measured gains come from ignoring return scale; the paper does not isolate that property from other benefits of two-hot classification and advantage filtering, so if those other benefits do most of the work, the scale-barrier mechanism would be wrong even if the recipe still works.","fun_headline_variants_meta":{"raw":{"variants":["Classification losses let one RL agent master 45 skills and Atari","No task labels needed: AMAGO-2 scales meta-RL via classification losses","RL agent adapts across tasks by treating Q-values as classification","AMAGO-2: Two classification losses break multi-task RL bottleneck"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000607,"raw_usage":{"total_tokens":2883,"prompt_tokens":1056,"completion_tokens":1827,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":672,"completion_tokens_details":{"reasoning_tokens":1748}},"tokens_in":672,"tokens_out":1827,"duration_ms":12280,"temperature":1.0,"reasoning_tokens":1748,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:48:10.494286+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A decisive test would train the same agent on a set of tasks whose return scales are deliberately equalized and compare the scale-resistant losses to standard regression; if the classification losses still win, their benefit is not scale invariance. The complementary check is to multiply the rewards of a single task by a large constant and observe whether Q-dependent losses degrade much faster than the scale-resistant ones.","supporting_citations":[],"review_version":1}