Pith. sign in

REVIEW 3 major objections 7 minor 2 cited by

Bigger, Regularized, Categorical: High-Capacity Value Functions are Efficient Multi-Task Learners

T0 review · 3 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that high-capacity value functions trained online with temporal-difference learning and a categorical cross-entropy loss scale to a billion parameters and beat single-task specialists across more than 280 tasks.

desk verdict Strong empirical scaling results for online multi-task value-based RL, but the reward-normalization guarantee in Appendix C.2 is mathematically wrong and the code is not out yet; still worth a serious referee. read the letter →

arxiv 2505.23150 v1 pith:TGBTB2W2 submitted 2025-05-29 cs.LG

Michal Nauman , Marek Cygan , Carmelo Sferrazza , Aviral Kumar , Pieter Abbeel This is my paper
classification cs.LG
keywords multi-taskreinforcementlearningscalingvaluefunctionscategoricaldistributionalRLcross-entropylosstaskembeddingsonlinetemporal-differencetransferbillion-parametercritic
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that value-based reinforcement learning can be scaled the way language and vision models are scaled: make the value model much bigger, train it online on many tasks at once, and the multi-task learner not only matches but outperforms strong single-task specialists. The authors show this on 283 tasks across five benchmarks, including high-degree-of-freedom humanoid control and vision-based Atari, using a single hyperparameter configuration. The load-bearing change is to train high-capacity critic networks with a categorical cross-entropy loss (distributional RL) and condition them on learnable task embeddings, which stabilizes the gradient conflicts and reward-scale imbalances that have made online multi-task temporal-difference learning brittle. If correct, this suggests that online TD learning, not expert distillation or offline datasets, can be the route to generalist control agents.

What carries the argument

The carrying mechanism is BRC, a combination of three components: (1) a scaled Q-value network using the BroNet residual architecture with layer normalization, which the paper shows is the most important ingredient (about 62% of the improvement by Shapley values); (2) a cross-entropy loss, implemented through categorical distributional RL with a fixed support of 101 atoms over $[-10, 10]$, with per-task reward normalization by a running maximum Monte Carlo return; and (3) learnable task embeddings of dimension 32, concatenated to the state (or to the image encoder's output), trained end-to-end by backpropagating the TD loss. This combination converts the multi-task problem into a single conditioned MDP, balances gradient magnitudes across tasks, and reduces gradient conflict rates as capacity grows.

What would settle it

Take a benchmark task whose return scale grows throughout training, for example by multiplying rewards by an increasing coefficient, and run BRC; if performance collapses once the observed maximum $G_i$ saturates the fixed support at $V_{\max}=10$, that shows the fixed-support reward normalization is load-bearing rather than incidental.

Watch

Extended reading notes

Core claim

The central claim is that value models trained online with temporal-difference learning scale to the billion-parameter range, and that a single such model trained jointly on many tasks surpasses strong single-task specialists, establishing a generalist advantage in value-based RL. Behind this are three design choices used together: a regularized residual critic architecture (BroNet) whose capacity can scale to about 1B parameters; a cross-entropy loss via categorical Q-learning with per-task return normalization, which removes the implicit task prioritization that MSE loss creates when reward scales differ; and learnable task embeddings, optimized by backpropagating the TD loss, in place of separate per-task heads. The paper reports that multi-task BRC achieves state-of-the-art results on 283 tasks from MetaWorld, DeepMind Control, HumanoidBench, ShadowHand, and Atari 100k, reaches the final performance of single-task experts with 40x fewer gradient updates, and transfers to new tasks with improved sample efficiency, including a frozen-model variant that adapts only the task embedding and surpasses 90% success on new ShadowHand objects.

Load-bearing premise

The reward normalization factor $G_i$, defined as the largest absolute Monte Carlo return seen so far in a task, must track the true return range closely enough that clipping to the fixed support $[-10, 10]$ never distorts or saturates the categorical value targets.

Editorial extensions

If this is right

  • Value-based RL can be scaled by model capacity and task diversity in the same way supervised learning is scaled, without expert data or behavioral cloning.
  • Multi-task online training with one shared critic can outperform single-task specialists while using 40x fewer gradient updates for the same performance.
  • Pretrained multi-task value models can be fine-tuned on new tasks with better sample efficiency than learning from scratch, and this transfer improves with both model size and the number of pretraining tasks.
  • Freezing the entire network and adapting only a low-dimensional task embedding can produce more than 90% success on out-of-distribution object manipulation, showing that the pretrained model encodes a transferable task space.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the per-task reward normalization could be replaced by a learned or adaptive support; if that worked as well, the fixed-support assumption of categorical RL would be a convenience rather than a hard requirement.
  • Beyond the paper: the observation that gradient-conflict rates fall as parameter count rises suggests that conflict-resolution heuristics such as gradient surgery may matter mostly at small scale, which is an extrapolation the paper gestures at but does not fully test.
  • Beyond the paper: if TD-learned task embeddings genuinely cluster similar embodiments, the embedding geometry could be used to predict which task pairs benefit from multi-task pretraining, turning an empirical observation into a forward planning signal.
  • Beyond the paper: the authors report that mixed-embodiment suites sometimes lack synergy, so a natural test is to measure per-task embedding distances and shared dynamics to predict when multi-task pretraining will accelerate or slow learning on a new task.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper proposes BRC (Bigger, Regularized, Categorical), a value-based RL method that combines high-capacity normalized residual Q-networks (BroNet), categorical TD learning with cross-entropy loss, and learnable task embeddings for online multi-task learning. The authors report experiments on over 280 tasks from five benchmarks and claim, for the first time, that value models trained with online TD learning scale to the billion-parameter range, that multi-task training with BRC surpasses strong single-task specialists, and that the resulting value models transfer efficiently to new tasks. The method is presented as a simple combination of existing ingredients (BroNet, C51, task embeddings) whose synergy stabilizes multi-task gradient conflict and reward-scale imbalance.

Significance. If the claims hold, the contribution is significant: it challenges the common assumption that online value-based RL does not scale with model capacity and task diversity, and it provides a concrete recipe (capacity + categorical loss + task embeddings) that is easy to implement and broadly applicable. The empirical scope is unusually broad, and the paper includes useful ablations, exact Shapley-value attribution of design choices, and honest limitations in Appendix A. The main load-bearing technical component, however, is the reward normalization scheme that underpins the categorical fixed-support representation, and that scheme has a correctness gap that directly affects the interpretation of the experimental results. Until that gap is resolved, the empirical phenomena may have a different explanation than the one claimed.

major comments (3)
  1. [Appendix C.2, Eqs. (2)-(4)] The claim that 'Normalizing rewards according to Equation 3 guarantees that the modelled returns are bounded by Vmax' does not follow from the equations as written. The soft return modeled by the categorical critic necessarily includes the entropy term αH/(1−γ), which is exactly what λ_i is supposed to represent; Eq. (3), however, divides only the reward by (Ḡ_i + λ_i), leaving the entropy contribution unscaled. The maximum modeled soft return is therefore Vmax·G/(G+λ) + λ, which exceeds Vmax whenever λ > 0. Early in training, with high exploration entropy and small Ḡ_i, a substantial fraction of targets can fall outside the support and be clipped by the C51 projection, so the claimed absence of 'regret from using categorical bounded Q-value representation' is unsupported. I ask the authors to correct the normalization so that the entire soft-target return (reward plus entropy) is rescaled, and to report the fraction of targets that fall outside [−Vmax, Vmax] together with the learning curves, to establish that the clipping is empirically negligible.
  2. [Appendix C.2, Eq. (2)] The per-task normalization factor Ḡ_i is a monotone running maximum of bootstrapped Monte-Carlo returns, and for truncated episodes those returns are bootstrapped with the critic. A single early overestimate therefore permanently inflates Ḡ_i; because the running maximum never decreases, the error never self-corrects, and all subsequent rewards for that task are scaled down. This can silently reweight tasks and produce an unintended curriculum, which would confound the claim that cross-entropy stabilization is the cause of the observed gains. The manuscript does not report the initialization of Ḡ_i, does not compare Ḡ_i against true return ranges, and provides no sensitivity analysis. I request a plot of Ḡ_i over training for representative tasks, and at least one ablation that replaces the running max with a robust estimator (e.g., a quantile or an exponential moving average) to show that the main results are not artifacts of the monotone max.
  3. [Section 4, Training and Baselines] The comparison underlying the central 'generalist advantage' and 'state-of-the-art' claims is weakened by the mixed use of previously reported numbers and locally rerun baselines. The text states that 'Whenever possible, we show previously reported results, otherwise we run official repositories,' but the figures (e.g., Figure 1) do not distinguish which baseline numbers were reproduced under the same evaluation protocol and which were quoted from other papers. Since normalization schemes, evaluation timesteps, and environment wrappers differ across papers, the reported improvements over TD-MPC2, BRO, SimBaV2, and others may partly reflect protocol differences. I ask the authors to provide a table that marks each baseline as reproduced versus cited, and to rerun at least the strongest single-task competitors (TD-MPC2 and BRO or SimBaV2) under the exact evaluation pipeline used for BRC, so that the headline improvements are established under identical conditions.
minor comments (7)
  1. [Abstract and Section 4] The abstract says 'over 280 unique tasks' while Section 4 states 'a total of 283'; the counts should be made consistent, and the definition of 'unique' (after accounting for overlap between MW+DMC and the individual MW and DMC sets) should be clarified.
  2. [Appendix E.3] In the score normalization section, the text 'we report returns, which we normalize according to Equation 3' appears to be a cross-reference error: Equation 3 is the training-time reward normalization, whereas the evaluation normalization is Equation 5. Please correct the reference.
  3. [Section 4 and Appendix F] The text claims a 'single set of hyperparameters across all benchmarks, in both continuous and discrete action experiments,' but Tables 3 and 4 give different hyperparameters for SAC+BRC and DrQ+BRC. The claim should be phrased as one set per base algorithm, or the tables should be reconciled.
  4. [Section 4, Code availability] The code repository URL is a placeholder (https://github.com/release_soon); a working link with the actual implementation is required for reproducibility of the empirical claims.
  5. [Section 5, Figure 9] The '40× fewer gradient updates' comparison should state explicitly that multi-task BRC consumes more environment steps per gradient update, and should report the total compute (GPU hours) in addition to gradient-step counts, to avoid overstating the efficiency gain.
  6. [Figure 12, right panel] The claim that logarithmic curves model sample-efficiency scaling 'well' should be supported by a goodness-of-fit measure or at least by the fitted exponents and their confidence intervals.
  7. [Section 3, return normalization description] The prose says rewards are normalized by 'dividing them by the maximal Monte Carlo return,' which omits the λ_i term that appears in Eq. (3); the text and equation should be aligned.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: BRC's scaling and transfer claims are empirical, benchmarked against external baselines, and its components (BroNet, C51, task embeddings) are prior methods used as building blocks rather than derived from the target result.

full rationale

The paper's central claims are empirical: online TD value models scale to 1B parameters and multi-task BRC surpasses single-task specialists. There is no derivation chain linking first principles to these results; support comes from controlled experiments on 283 tasks against external baselines (TD-MPC2, DreamerV3, BRO, SimBa, PCGrad, etc.). The reward normalization in Appendix C.2 defines G_i as a running maximum of Monte Carlo returns and rescales rewards in Eq. 3. This is a data-dependent preprocessing step, not a fitted parameter renamed as a prediction: it is not optimized to match the measured outcomes, and the statement that returns are bounded by Vmax is a designed inequality following from the definition of G_i and lambda_i. Any concern that the running max can be stale or inflated is a stability/correctness risk, not circularity. The Shapley analysis in Figure 6 is a post-hoc attribution of ablations, not a separate prediction. Self-citations (BroNet architecture from Nauman et al. [74], design choices from [72,73]) are present, but they are not load-bearing in a circular sense: the paper re-runs and ablates the architecture against vanilla and SimBa baselines (Figure 4), and no uniqueness theorem or forced choice is imported from prior work. The claims are falsifiable and self-contained against external benchmarks, so no circular step is exhibited.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The method depends on several empirical assumptions, chiefly the adaptive reward normalization and the transferability of embeddings. These assumptions are not derived from first principles and their failure could undermine the results.

free parameters (2)
  • per-task reward scale G_i = online maximum absolute Monte Carlo return per task
    Defined in Eq. (2) and used in Eq. (3) to rescale rewards. This adaptively fitted quantity is essential for the fixed categorical support to work.
  • categorical support bound Vmax = 10
    Hand-chosen hyperparameter (Appendix F) that defines the reward range after normalization. If the true normalized returns exceed this, the representation truncates; its adequacy is not validated.
assumptions (4)
  • standard math The categorical distributional Bellman operator (Bellemare et al., 2017) with a fixed support is a valid learning target for value distributions.
    Invoked in Section C.2 to justify using cross-entropy loss; the paper relies on this known result without proof.
  • domain assumption The per-task reward normalization in Eq. (3) preserves optimal policies and does not break non-stationarity of TD learning.
    The normalizer G_i is updated online using observed maximum returns, so the reward scale changes over time. This is assumed to be stable and beneficial, but no analysis is given.
  • domain assumption TD learning with function approximation (SAC/DrQ) converges to a useful policy despite distribution shift and large capacity.
    The paper uses SAC and DrQ as base algorithms and assumes their convergence with BroNet and cross-entropy loss; Section C.1 notes modifications without convergence guarantees.
  • domain assumption Learned task embeddings generalize to new tasks so that a pretrained embedding can be selected or tuned for unseen tasks.
    The transfer experiments (Figures 10, 11) assume the embedding space is semantically organized; no theoretical argument is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Bigger, Regularized, Categorical: High-Capacity Value Functions are Efficient Multi-Task Learners." pith.science (2026). https://pith.science/paper/TGBTB2W2

@misc{pith2026250523150,
  author       = {Pith},
  title        = {Pith review of: Bigger, Regularized, Categorical: High-Capacity Value Functions are Efficient Multi-Task Learners},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TGBTB2W2}},
  note         = {Machine review of arXiv:2505.23150}
}
read the original abstract

Recent advances in language modeling and vision stem from training large models on diverse, multi-task data. This paradigm has had limited impact in value-based reinforcement learning (RL), where improvements are often driven by small models trained in a single-task context. This is because in multi-task RL sparse rewards and gradient conflicts make optimization of temporal difference brittle. Practical workflows for generalist policies therefore avoid online training, instead cloning expert trajectories or distilling collections of single-task policies into one agent. In this work, we show that the use of high-capacity value models trained via cross-entropy and conditioned on learnable task embeddings addresses the problem of task interference in online RL, allowing for robust and scalable multi-task training. We test our approach on 7 multi-task benchmarks with over 280 unique tasks, spanning high degree-of-freedom humanoid control and discrete vision-based RL. We find that, despite its simplicity, the proposed approach leads to state-of-the-art single and multi-task performance, as well as sample-efficient transfer to new tasks.

Figures

Figures reproduced from arXiv: 2505.23150 by the authors.

Figure 1
Figure 1. Scaling multi-task training leads to state-of-the-art performance. Naïve scaling of SAC to multi-task decreases the aggregate performance (left). Our proposed method (BRC) works both in single and multi-task learning and provides a pronounced performance improvement over previous approaches, including optimized single-task learners (right). We denote multi-task agents with ⋆. Large-scale neural networks trained on l… view at source ↗
Figure 2
Figure 2. Scaling multi-task training allows for sample-efficient transfer to new tasks. We compare the performance of single-task BRC agent trained from scratch (green), to an agent initialized with our pretrained multi-task BRC agent trained on different tasks (blue). We find that transferring a multi-task BRC model to new tasks leads to better sample efficiency than learning from scratch. Y-axis denotes the average final p… view at source ↗
Figure 3
Figure 3. Cross-entropy loss stabilizes online multi-task learning. We investigate BRC with naive application of MSE loss (purple), MSE loss paired with return normalization (green) and cross-entropy paired with return normalization (blue) on HB-MEDIUM. Varying reward magnitudes in multi-task learning can destabilize learning of certain tasks, which translates to high variance of signals between tasks (left). Stabilizing this… view at source ↗
Figures from the paper (26 more)
Figure 4
Figure 4. Figure 4: BroNet paired with cross-entropy loss scales in both single and multi-task RL. We compare scaling behavior of different architectures in single (left) and multi-task (right) when solving the HB-MEDIUM benchmark. We pair SAC with the vanilla [36], SimBa [60], BroNet wit…
Figure 5
Figure 5. Figure 5: Using task embeddings is preferable to separate heads design. We compare the performance (left) and gradient similarity [114] (right) of different approaches for multi-task learning on HB-MEDIUM. We consider single-task, multi-task via separate heads [43, 58] and via t…
Figure 6
Figure 6. Figure 6: Scaled Q-value model is the most important design choice. (Left) We investigate the final performance of the base model paired with various combinations of scaled Q-value model (SQ), cross-entropy loss via categorical RL (CE), and learnable task-embedding module (TE) o…
Figure 7
Figure 7. Figure 7: Learnable task embeddings discover the underlying dynamics structure. We graph the first two principal components of the task embeddings learned online on MW+DMC and find that the dis￾covered embeddings cluster similar embodiments. Task embeddings. To accommodate scala…
Figure 8
Figure 8. Figure 8: We consider 283 tasks from 5 simulation benchmarks. We test our approach with SAC [36] on MetaWorld, DeepMind Control, HumanoidBench and ShadowHand, and with DrQ-ϵ [55] on vision-based Atari. In both approaches we use single set of hyperparameters across all tasks and …
Figure 9
Figure 9. Figure 9: Multi-task learning is sample and compute-efficient. We compare the performance of our proposed single and multi-task BRC agents on HB-HARD benchmark and find that multi-task learning leads 2× improvement in sample efficiency (left) while performing 40× less gradient u…
Figure 10
Figure 10. Figure 10: Increased data and compute improves pretraining and transfer performance. We train our multi-task agent on HB-MEDIUM using 3, 6 or 9 tasks for 1M environment steps. We find that the final performance on 3 shared tasks improves when increasing the number of pretraining…
Figure 11
Figure 11. Figure 11: Value-based RL agents are transferable. (Left) We compare the performance of model transfer with the BRC agent to a fresh agent initialized with the buffer used in pretraining of the transferred BRC model (data transfer). We consider 15 transfer tasks from MW, HB-MEDI…
Figure 12
Figure 12. Figure 12: Scaling online multi-task TD learning past 1B parameters. We investigate the performance of online (BRC) and offline (BRC+BC) agents on the multi-task MW+DMC benchmark consisting of 12 different robot embodiments [38]. (Left) Online multi-task learning is significantl…
Figure 13
Figure 13. Figure 13: BroNet used in the SAC+BRC method. We use the exact residual architecture as presented in Nauman et al. [74]. Image-based control. In image-based experiments, we use the scaled Impala architecture [58, 87]. Following previous work [59, 74], we add a layer normalizatio…
Figure 14
Figure 14. Figure 14: Impala architecture used in the DRQ+BRC method. The architecture is based on observations made in previous works [58, 87, 74]. C.4 Task embeddings We use an embeddings module1 which we adjust by backpropagating the temporal difference loss. Following prior works [38],…
Figure 15
Figure 15. Figure 15: Our approach for concatenating observations with the learned task embeddings. We use slightly different approach for proprioceptive (left) and vision-based (right) control. In particular, in vision-based tasks, we learn a task-agnostic encoder by concatenating task em…
Figure 1
Figure 1. Figure 1: We report the average final performance on every considered benchmark. We take average with respect to random seeds (we use a minimum of 5 random seeds per method) and different tasks. We denote the training lengths directly on the graph. We normalize scores according …
Figure 4
Figure 4. Figure 4: We report final performance on the HumanoidBench-Medium benchmark detailed in Appendix E.1 after 1M environment steps per task. We consider single and multi-task BRC models with different critic architectures and varying width according to [PITH_FULL_IMAGE:figures/ful…
Figure 7
Figure 7. Figure 7: We present the learned BRC task embeddings after 500k steps of online training on the MW+DMC benchmark. We take the 32-dimensional embeddings and extract first two principal components using the PCA algorithm [PITH_FULL_IMAGE:figures/full_fig_p021_7.png]
Figure 11
Figure 11. Figure 11: On the left Figure, we show the training curves averaged over 15 transfer tasks from HumanoidBench-Medium, HumanoidBench-Hard and MetaWorld listed in Appendix E.2. We report the first 500k steps of training, and show three algorithms: single-task BRC trained from scra…
Figure 12
Figure 12. Figure 12: Here, we focus on the MW+DMC benchmark. In the left figure, we report the final performance resulting from offline training according to Hansen et al. [38]. Additionally, we report the GPU days required to finish the training, assuming an 80 GB A100 graphics unit. On …
Figure 16
Figure 16. Figure 16: Aggregate training curves. Y-axis denotes the performance metric, and X-axis denotes environment steps. We report 95% confidence interval calculated via bootstrapping. 26 [PITH_FULL_IMAGE:figures/full_fig_p026_16.png]
Figure 17
Figure 17. Figure 17: Unnormalized training curves for the HumanoidBench-Medium benchmark. Y-axis denotes sum of episodic returns and X-axis denotes environment steps. We report 95% confidence interval calculated via bootstrapping. 0.4 0.3 0.2 0.1 0.0 0.1 0.2 0.3 0.4 0.04 0.02 0.00 0.02 0.…
Figure 18
Figure 18. Figure 18: Unnormalized training curves for the HumanoidBench-Hard benchmark. Y-axis denotes sum of episodic returns and X-axis denotes environment steps. We report 95% confidence interval calculated via bootstrapping. 27 [PITH_FULL_IMAGE:figures/full_fig_p027_18.png]
Figure 19
Figure 19. Figure 19: Unnormalized training curves for the DMC-Hard benchmark. Y-axis denotes sum of episodic returns and X-axis denotes environment steps. We report 95% confidence interval calculated via bootstrapping [PITH_FULL_IMAGE:figures/full_fig_p028_19.png]
Figure 20
Figure 20. Figure 20: Unnormalized training curves for the MetaWorld benchmark. Y-axis denotes the success rate and X-axis denotes environment steps. We report 95% confidence interval calculated via bootstrapping. 29 [PITH_FULL_IMAGE:figures/full_fig_p029_20.png]
Figure 21
Figure 21. Figure 21: Unnormalized training curves for the ShadowHand benchmark. Y-axis denotes the success rate and X-axis denotes environment steps. We report 95% confidence interval calculated via bootstrapping. 30 [PITH_FULL_IMAGE:figures/full_fig_p030_21.png]
Figure 22
Figure 22. Figure 22: Unnormalized training curves for the MW+DMC benchmark. For MetaWorld tasks, Y-axis denotes the success rate whereas for DMC tasks it denotes sum of episodic returns. X-axis denotes environment steps. We report 95% confidence interval calculated via bootstrapping. 31 …
Figure 23
Figure 23. Figure 23 [PITH_FULL_IMAGE:figures/full_fig_p032_23.png]
Figure 24
Figure 24. Figure 24: Effectiveness of pretrained embeddings. We show the performance of using pretrained embeddings to manipulate new shapes. Interestingly, most of the testing shapes can be manipulated via majority of the pretrained embeddings, showcasing that the individual policies do …

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. RN-D: Discretized Categorical Actors for On-Policy Reinforcement Learning

    cs.LG 2026-01 conditional novelty 5.0 of 10

    A discretized categorical actor paired with a pre-LayerNorm residual MLP (RN-D) improves PPO's final performance and sample efficiency on continuous-control benchmarks.

  2. Scaling DRL for Decision Making: A Survey on Data, Network, and Training Budget Strategies

    cs.LG 2025-08 conditional novelty 4.0 of 10

    A survey that categorizes deep reinforcement learning scaling strategies into data, network, and training budget dimensions and outlines challenges for scaling DRL systems.

Reference graph

Works this paper leans on

118 extracted references · 41 canonical work pages · cited by 2 Pith papers

  1. [1]

    L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    Provable benefits of representational transfer in reinforcement learning

    Agarwal, A., Song, Y ., Sun, W., Wang, K., Wang, M., and Zhang, X. Provable benefits of representational transfer in reinforcement learning. In The Thirty Sixth Annual Conference on Learning Theory, pp. 2114–2187. PMLR, 2023

    2023

  3. [3]

    S., Courville, A., and Bellemare, M

    Agarwal, R., Schwarzer, M., Castro, P. S., Courville, A., and Bellemare, M. G. Deep reinforcement learning at the edge of the statistical precipice. Advances in Neural Information Processing Systems, 2021

    2021

  4. [4]

    S., Courville, A

    Agarwal, R., Schwarzer, M., Castro, P. S., Courville, A. C., and Bellemare, M. Deep rein- forcement learning at the edge of the statistical precipice. Advances in neural information processing systems, 34:29304–29320, 2021

    2021

  5. [5]

    Hindsight experience replay.Advances in neural information processing systems, 30, 2017

    Andrychowicz, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., Pieter Abbeel, O., and Zaremba, W. Hindsight experience replay.Advances in neural information processing systems, 30, 2017

    2017

  6. [6]

    M., Baker, B., Chociej, M., Jozefowicz, R., McGrew, B., Pachocki, J., Petron, A., Plappert, M., Powell, G., Ray, A., et al

    Andrychowicz, O. M., Baker, B., Chociej, M., Jozefowicz, R., McGrew, B., Pachocki, J., Petron, A., Plappert, M., Powell, G., Ray, A., et al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3–20, 2020

    2020

  7. [7]

    Video pretraining (vpt): Learning to act by watching unlabeled online videos

    Baker, B., Akkaya, I., Zhokov, P., Huizinga, J., Tang, J., Ecoffet, A., Houghton, B., Sampedro, R., and Clune, J. Video pretraining (vpt): Learning to act by watching unlabeled online videos. Advances in Neural Information Processing Systems, 35:24639–24654, 2022

    2022

  8. [8]

    J., Smith, L., Kostrikov, I., and Levine, S

    Ball, P. J., Smith, L., Kostrikov, I., and Levine, S. Efficient online reinforcement learning with offline data. In International Conference on Machine Learning, pp. 1577–1594. PMLR, 2023

    2023

Show all 118 references
  1. [9]

    J., Schaul, T., van Hasselt, H

    Barreto, A., Dabney, W., Munos, R., Hunt, J. J., Schaul, T., van Hasselt, H. P., and Silver, D. Successor features for transfer in reinforcement learning. Advances in neural information processing systems, 30, 2017

    2017

  2. [10]

    G., Naddaf, Y ., Veness, J., and Bowling, M

    Bellemare, M. G., Naddaf, Y ., Veness, J., and Bowling, M. The arcade learning environment: An evaluation platform for general agents. J. Artif. Int. Res., 47(1):253–279, May 2013. ISSN 1076-9757

    2013

  3. [11]

    G., Dabney, W., and Munos, R

    Bellemare, M. G., Dabney, W., and Munos, R. A distributional perspective on reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pp. 449–458. JMLR. org, 2017

    2017

  4. [12]

    Dynamic Programming

    Bellman, R. Dynamic Programming. Princeton University Press, 1957

    1957

  5. [13]

    Bishop, C. M. and Nasrabadi, N. M. Pattern recognition and machine learning, volume 4. Springer, 2006. 10

    2006

  6. [14]

    P., and Weinberger, K

    Bjorck, N., Gomes, C. P., and Weinberger, K. Q. Towards deeper deep reinforcement learning with spectral normalization. Advances in neural information processing systems, 34:8242– 8255, 2021

    2021

  7. [15]

    J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., VanderPlas, J., Wanderman-Milne, S., et al

    Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., VanderPlas, J., Wanderman-Milne, S., et al. Jax: composable transformations of python+ numpy programs. 2018

    2018

  8. [16]

    Rt-1: Robotics transformer for real-world control at scale

    Brohan, A., Brown, N., Carbajal, J., Chebotar, Y ., Dabis, J., Finn, C., Gopalakrishnan, K., Hausman, K., Herzog, A., Hsu, J., et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817, 2022

    2022 arXiv

  9. [17]

    D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020

    1901

  10. [18]

    Decision transformer: Reinforcement learning via sequence modeling

    Chen, L., Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., Abbeel, P., Srinivas, A., and Mordatch, I. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:15084–15097, 2021

    2021

  11. [19]

    A system for general in-hand object re-orientation

    Chen, T., Xu, J., and Agrawal, P. A system for general in-hand object re-orientation. In Conference on Robot Learning, pp. 297–307. PMLR, 2022

    2022

  12. [20]

    Gradnorm: Gradient normaliza- tion for adaptive loss balancing in deep multitask networks

    Chen, Z., Badrinarayanan, V ., Lee, C.-Y ., and Rabinovich, A. Gradnorm: Gradient normaliza- tion for adaptive loss balancing in deep multitask networks. In International conference on machine learning, pp. 794–803. PMLR, 2018

    2018

  13. [21]

    Just pick a sign: Optimizing deep multitask models with gradient sign dropout

    Chen, Z., Ngiam, J., Huang, Y ., Luong, T., Kretzschmar, H., Chai, Y ., and Anguelov, D. Just pick a sign: Optimizing deep multitask models with gradient sign dropout. Advances in Neural Information Processing Systems, 33:2039–2050, 2020

    2020

  14. [22]

    Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., Schuh, P., Shi, K., Tsvyashchenko, S., Maynez, J., Rao, A., Barnes, P., Tay, Y ., Shazeer, N., Prabhakaran, V ., Reif, E., Du, N., Hutchinson, B., Pop...

    2023

  15. [23]

    F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D

    Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017

    2017

  16. [24]

    Better exploration with optimistic actor critic

    Ciosek, K., Vuong, Q., Loftin, R., and Hofmann, K. Better exploration with optimistic actor critic. Advances in Neural Information Processing Systems, 32, 2019

    2019

  17. [25]

    Magnetic control of tokamak plasmas through deep reinforcement learning

    Degrave, J., Felici, F., Buchli, J., Neunert, M., Tracey, B., Carpanese, F., Ewalds, T., Hafner, R., Abdolmaleki, A., de Las Casas, D., et al. Magnetic control of tokamak plasmas through deep reinforcement learning. Nature, 602(7897):414–419, 2022

    2022

  18. [26]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. North American Chapter of the Association for Computational Linguistics, 2019. doi: 10.18653/v1/N19-1423

    2019 doi

  19. [27]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., De- hghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. International Conference on ...

    2020

  20. [28]

    S., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q., Yu, T., et al

    Driess, D., Xia, F., Sajjadi, M. S., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q., Yu, T., et al. Palm-e: An embodied multimodal language model. InInternational Conference on Machine Learning, pp. 8469–8488. PMLR, 2023

    2023

  21. [29]

    Eysenbach, B., Zhang, T., Levine, S., and Salakhutdinov, R. R. Contrastive learning as goal- conditioned reinforcement learning. Advances in Neural Information Processing Systems, 35: 35603–35620, 2022

    2022

  22. [30]

    A., Chebotar, Y ., Xiao, T., Irpan, A., Levine, S., Castro, P

    Farebrother, J., Orbay, J., Vuong, Q., Taiga, A. A., Chebotar, Y ., Xiao, T., Irpan, A., Levine, S., Castro, P. S., Faust, A., et al. Stop regressing: Training value functions via classification for scalable deep rl. In Forty-first International Conference on Machine Learning

  23. [31]

    Model-agnostic meta-learning for fast adaptation of deep networks

    Finn, C., Abbeel, P., and Levine, S. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pp. 1126–1135. PMLR, 2017

    2017

  24. [32]

    and Gu, S

    Fujimoto, S. and Gu, S. S. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132–20145, 2021

    2021

  25. [33]

    Addressing function approximation error in actor-critic methods

    Fujimoto, S., Hoof, H., and Meger, D. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pp. 1587–1596. PMLR, 2018

    2018

  26. [34]

    S., Precup, D., and Meger, D

    Fujimoto, S., Chang, W.-D., Smith, E., Gu, S. S., Precup, D., and Meger, D. For sale: State- action representation learning for deep reinforcement learning. Advances in neural information processing systems, 36:61573–61624, 2023

    2023

  27. [35]

    Divide-and-conquer reinforcement learning

    Ghosh, D., Singh, A., Rajeswaran, A., Kumar, V ., and Levine, S. Divide-and-conquer reinforcement learning. In International Conference on Learning Representations, 2018

    2018

  28. [36]

    Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor

    Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pp. 1861–1870. PMLR, 2018

    2018

  29. [37]

    Mastering diverse domains through world models

    Hafner, D., Pasukonis, J., Ba, J., and Lillicrap, T. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104, 2023

    2023 arXiv

  30. [38]

    Td-mpc2: Scalable, robust world models for continuous control

    Hansen, N., Su, H., and Wang, X. Td-mpc2: Scalable, robust world models for continuous control. arXiv preprint arXiv: 2310.16828, 2023

    2023 arXiv

  31. [39]

    R., Millman, K

    Harris, C. R., Millman, K. J., Van Der Walt, S. J., Gommers, R., Virtanen, P., Cournapeau, D., Wieser, E., Taylor, J., Berg, S., Smith, N. J., et al. Array programming with numpy.Nature, 585(7825):357–362, 2020

    2020

  32. [40]

    T., Wang, Z., Heess, N., and Riedmiller, M

    Hausman, K., Springenberg, J. T., Wang, Z., Heess, N., and Riedmiller, M. Learning an embedding space for transferable robot skills. In International Conference on Learning Representations, 2018

    2018

  33. [41]

    Efficient multi-task reinforcement learning with cross-task policy guidance

    He, J., Li, K., Zang, Y ., Fu, H., Fu, Q., Xing, J., and Cheng, J. Efficient multi-task reinforcement learning with cross-task policy guidance. Advances in Neural Information Processing Systems, 37:117997–118024, 2024

    2024

  34. [42]

    B., Dhariwal, P., Gray, S., et al

    Henighan, T., Kaplan, J., Katz, M., Chen, M., Hesse, C., Jackson, J., Jun, H., Brown, T. B., Dhariwal, P., Gray, S., et al. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010.14701, 2020

    2010 arXiv

  35. [43]

    Multi-task deep reinforcement learning with popart

    Hessel, M., Soyer, H., Espeholt, L., Czarnecki, W., Schmitt, S., and van Hasselt, H. Multi-task deep reinforcement learning with popart. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 3796–3803, 2019

    2019

  36. [44]

    Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022

    2022 arXiv

  37. [45]

    Otter: A vision-language-action model with text-aware visual feature extraction

    Huang, H., Liu, F., Fu, L., Wu, T., Mukadam, M., Malik, J., Goldberg, K., and Abbeel, P. Otter: A vision-language-action model with text-aware visual feature extraction. arXiv preprint arXiv:2503.03734, 2025. 12

    2025

  38. [46]

    Generalization in dexterous manipulation via geometry-aware multi-task learning

    Huang, W., Mordatch, I., Abbeel, P., and Pathak, D. Generalization in dexterous manipulation via geometry-aware multi-task learning. arXiv preprint arXiv:2111.03062, 2021

    2021 arXiv

  39. [47]

    Hunter, J. D. Matplotlib: A 2d graphics environment. Computing in science & engineering, 9 (03):90–95, 2007

    2007

  40. [48]

    H., Czechowski, K., Erhan, D., Finn, C., Kozakowski, P., Levine, S., et al

    Kaiser, Ł., Babaeizadeh, M., Miłos, P., Osi´nski, B., Campbell, R. H., Czechowski, K., Erhan, D., Finn, C., Kozakowski, P., Levine, S., et al. Model based reinforcement learning for atari. In International Conference on Learning Representations, 2019

    2019

  41. [49]

    B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D

    Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020

    2001 arXiv

  42. [50]

    Champion-level drone racing using deep reinforcement learning

    Kaufmann, E., Bauersfeld, L., Loquercio, A., Müller, M., Koltun, V ., and Scaramuzza, D. Champion-level drone racing using deep reinforcement learning. Nature, 620(7976):982–987, 2023

    2023

  43. [51]

    J., Pertsch, K., Karamcheti, S., Xiao, T., Balakrishna, A., Nair, S., Rafailov, R., Foster, E., Lam, G., Sanketi, P., et al

    Kim, M. J., Pertsch, K., Karamcheti, S., Xiao, T., Balakrishna, A., Nair, S., Rafailov, R., Foster, E., Lam, G., Sanketi, P., et al. Openvla: An open-source vision-language-action model. arXiv preprint arXiv:2406.09246, 2024

    2024 arXiv

  44. [52]

    and Ba, J

    Kingma, D. and Ba, J. Adam: A method for stochastic optimization. International Conference on Learning Representations (ICLR), 2015

    2015

  45. [53]

    C., Lo, W.-Y ., et al

    Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y ., et al. Segment anything. InProceedings of the IEEE/CVF international conference on computer vision, pp. 4015–4026, 2023

    2023

  46. [54]

    Konda, V . R. and Tsitsiklis, J. N. Actor-critic algorithms. InAdvances in neural information processing systems, pp. 1008–1014, 2000

    2000

  47. [55]

    Image augmentation is all you need: Regularizing deep reinforcement learning from pixels

    Kostrikov, I., Yarats, D., and Fergus, R. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. arXiv preprint arXiv:2004.13649, 2020

    2004 arXiv

  48. [56]

    DR3: Value- Based Deep Reinforcement Learning Requires Explicit Regularization

    Kumar, A., Agarwal, R., Ma, T., Courville, A., Tucker, G., and Levine, S. DR3: Value- Based Deep Reinforcement Learning Requires Explicit Regularization. arXiv preprint arXiv:2112.04716, 2021

    2021 arXiv

  49. [57]

    Rma: Rapid motor adaptation for legged robots

    Kumar, A., Fu, Z., Pathak, D., and Malik, J. Rma: Rapid motor adaptation for legged robots. arXiv preprint arXiv:2107.04034, 2021

    2021 arXiv

  50. [58]

    Offline q-learning on diverse multi-task data both scales and generalizes

    Kumar, A., Agarwal, R., Geng, X., Tucker, G., and Levine, S. Offline q-learning on diverse multi-task data both scales and generalizes. InThe Eleventh International Conference on Learn- ing Representations, 2023. URL https://openreview.net/forum?id=4-k7kUavAj

    2023

  51. [59]

    Reinforcement learning with augmented data

    Laskin, M., Lee, K., Stooke, A., Pinto, L., Abbeel, P., and Srinivas, A. Reinforcement learning with augmented data. Advances in neural information processing systems, 33:19884–19895, 2020

    2020

  52. [60]

    J., Subramanian, K., Wurman, P

    Lee, H., Hwang, D., Kim, D., Kim, H., Tai, J. J., Subramanian, K., Wurman, P. R., Choo, J., Stone, P., and Seno, T. Simba: Simplicity bias for scaling up parameters in deep reinforcement learning. arXiv preprint arXiv:2410.09754, 2024

    arXiv

  53. [61]

    Hyperspherical normalization for scalable deep reinforcement learning

    Lee, H., Lee, Y ., Seno, T., Kim, D., Stone, P., and Choo, J. Hyperspherical normalization for scalable deep reinforcement learning. arXiv preprint arXiv:2502.15280, 2025

    2025 arXiv

  54. [62]

    End-to-end training of deep visuomotor policies

    Levine, S., Finn, C., Darrell, T., and Abbeel, P. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1):1334–1373, 2016

    2016

  55. [63]

    Deepseek-v3 technical report

    Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  56. [64]

    Conflict-averse gradient descent for multi-task learning

    Liu, B., Liu, X., Jin, X., Stone, P., and Liu, Q. Conflict-averse gradient descent for multi-task learning. Advances in Neural Information Processing Systems, 34:18878–18890, 2021. 13

    2021

  57. [65]

    Moka: Open-vocabulary robotic manipulation through mark-based visual prompting

    Liu, F., Fang, K., Abbeel, P., and Levine, S. Moka: Open-vocabulary robotic manipulation through mark-based visual prompting. In First Workshop on Vision-Language Models for Navigation and Manipulation at ICRA 2024

    2024

  58. [66]

    Scaling laws for fine-grained mixture of experts

    Ludziejewski, J., Krajewski, J., Adamczewski, K., Pióro, M., Krutul, M., Antoniak, S., Ciebiera, K., Król, K., Odrzygó´ zd´ z, T., Sankowski, P., et al. Scaling laws for fine-grained mixture of experts. In Forty-first International Conference on Machine Learning, 2024

    2024

  59. [67]

    S., Gao, T., Sampaio, G

    Mark, M. S., Gao, T., Sampaio, G. G., Srirama, M. K., Sharma, A., Finn, C., and Kumar, A. Policy agnostic rl: Offline rl and online rl fine-tuning of any class and backbone. arXiv preprint arXiv:2412.06685, 2024

    2024 arXiv

  60. [68]

    McCandlish, S., Kaplan, J., Amodei, D., and Team, O. D. An empirical model of large-batch training. arXiv preprint arXiv:1812.06162, 2018

    2018 arXiv

  61. [69]

    A., Veness, J., Bellemare, M

    Mnih, V ., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., and Hassabis, D. Human-level con...

    2015

  62. [70]

    P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., and Kavukcuoglu, K

    Mnih, V ., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., and Kavukcuoglu, K. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pp. 1928–1937. PMLR, 2016

    1928

  63. [71]

    Tactical optimism and pessimism for deep reinforcement learning

    Moskovitz, T., Parker-Holder, J., Pacchiano, A., Arbel, M., and Jordan, M. Tactical optimism and pessimism for deep reinforcement learning. Advances in Neural Information Processing Systems, 34:12849–12863, 2021

    2021

  64. [72]

    and Cygan, M

    Nauman, M. and Cygan, M. On the theory of risk-aware agents: Bridging actor-critic and economics. In ICML 2024 Workshop: Aligning Reinforcement Learning Experimentalists and Theorists, 2023

    2024

  65. [73]

    Overestimation, overfitting, and plasticity in actor-critic: the bitter lesson of reinforcement learning

    Nauman, M., Bortkiewicz, M., Miło ´s, P., Trzcinski, T., Ostaszewski, M., and Cygan, M. Overestimation, overfitting, and plasticity in actor-critic: the bitter lesson of reinforcement learning. In Proceedings of the 41st International Conference on Machine Learning, 2024. URL ...

    2024 arXiv

  66. [74]

    Bigger, regularized, optimistic: scaling for compute and sample-efficient continuous control

    Nauman, M., Ostaszewski, M., Jankowski, K., Miło´s, P., and Cygan, M. Bigger, regularized, optimistic: scaling for compute and sample-efficient continuous control. arXiv preprint arXiv:2405.16158, 2024

    2024 arXiv

  67. [75]

    Dinov2: Learning robust visual features without supervision

    Oquab, M., Darcet, T., Moutakanni, T., V o, H., Szafraniec, M., Khalidov, V ., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al. Dinov2: Learning robust visual features without supervision. Transactions on Machine Learning Research Journal, 2024

    2024

  68. [76]

    Training language models to follow instructions with human feedback

    Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022

    2022

  69. [77]

    L., and Salakhutdinov, R

    Parisotto, E., Ba, J. L., and Salakhutdinov, R. Actor-mimic: Deep multitask and transfer reinforcement learning. arXiv preprint arXiv:1511.06342, 2015

    2015 arXiv

  70. [78]

    Is value learning really the main bottleneck in offline rl? arXiv preprint arXiv:2406.09329, 2024

    Park, S., Frans, K., Levine, S., and Kumar, A. Is value learning really the main bottleneck in offline rl? arXiv preprint arXiv:2406.09329, 2024

    2024 arXiv

  71. [79]

    Flow q-learning

    Park, S., Li, Q., and Levine, S. Flow q-learning. arXiv preprint arXiv:2502.02538, 2025

    2025 arXiv

  72. [80]

    Puterman, M. L. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014

    2014

  73. [81]

    W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al

    Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748–8763. PmLR, 2021. 14

    2021

  74. [82]

    Efficient off-policy meta- reinforcement learning via probabilistic context variables

    Rakelly, K., Zhou, A., Finn, C., Levine, S., and Quillen, D. Efficient off-policy meta- reinforcement learning via probabilistic context variables. In International conference on machine learning, pp. 5331–5340. PMLR, 2019

    2019

  75. [83]

    G., Novikov, A., Barth-Maron, G., Gimenez, M., Sulsky, Y ., Kay, J., Springenberg, J

    Reed, S., Zolna, K., Parisotto, E., Colmenarejo, S. G., Novikov, A., Barth-Maron, G., Gimenez, M., Sulsky, Y ., Kay, J., Springenberg, J. T., et al. A generalist agent. arXiv preprint arXiv:2205.06175, 2022

    2022 arXiv

  76. [84]

    A., Colmenarejo, S

    Rusu, A. A., Colmenarejo, S. G., Gulcehre, C., Desjardins, G., Kirkpatrick, J., Pascanu, R., Mnih, V ., Kavukcuoglu, K., and Hadsell, R. Policy distillation. arXiv preprint arXiv:1511.06295, 2015

    2015 arXiv

  77. [85]

    Value-based deep rl scales predictably

    Rybkin, O., Nauman, M., Fu, P., Snell, C., Abbeel, P., Levine, S., and Kumar, A. Value-based deep rl scales predictably. arXiv preprint arXiv:2502.04327, 2025

    2025 arXiv

  78. [86]

    D., Courville, A., and Bachman, P

    Schwarzer, M., Anand, A., Goel, R., Hjelm, R. D., Courville, A., and Bachman, P. Data- efficient reinforcement learning with self-predictive representations. In International Confer- ence on Learning Representations, 2020

    2020

  79. [87]

    Schwarzer, M., Ceron, J. S. O., Courville, A., Bellemare, M. G., Agarwal, R., and Castro, P. S. Bigger, better, faster: Human-level atari with human-level efficiency. In International Conference on Machine Learning, pp. 30365–30380. PMLR, 2023

    2023

  80. [88]

    and Koltun, V

    Sener, O. and Koltun, V . Multi-task learning as multi-objective optimization. Advances in neural information processing systems, 31, 2018

    2018

  81. [89]

    Humanoidbench: Simu- lated humanoid benchmark for whole-body locomotion and manipulation

    Sferrazza, C., Huang, D.-M., Lin, X., Lee, Y ., and Abbeel, P. Humanoidbench: Simu- lated humanoid benchmark for whole-body locomotion and manipulation. arXiv preprint arXiv:2403.10506, 2024

    2024 arXiv

  82. [90]

    Shapley, L. S. et al. A value for n-person games. 1953

    1953

  83. [91]

    Deterministic policy gradient algorithms

    Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., and Riedmiller, M. Deterministic policy gradient algorithms. In International conference on machine learning, pp. 387–395. PMLR, 2014

    2014

  84. [92]

    Mastering the game of go without human knowledge

    Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., et al. Mastering the game of go without human knowledge. nature, 550(7676):354–359, 2017

    2017

  85. [93]

    Multi-task reinforcement learning with context-based representations

    Sodhani, S., Zhang, A., and Pineau, J. Multi-task reinforcement learning with context-based representations. In International Conference on Machine Learning, pp. 9767–9779. PMLR, 2021

    2021

  86. [94]

    T., Abdolmaleki, A., Zhang, J., Groth, O., Bloesch, M., Lampe, T., Brakel, P., Bechtle, S

    Springenberg, J. T., Abdolmaleki, A., Zhang, J., Groth, O., Bloesch, M., Lampe, T., Brakel, P., Bechtle, S. M. E., Kapturowski, S., Hafner, R., et al. Offline actor-critic reinforcement learning scales to large models. In International Conference on Machine Learning, pp. 46323...

    2024

  87. [95]

    Paco: Parameter-compositional multi-task reinforcement learning

    Sun, L., Zhang, H., Xu, W., and Tomizuka, M. Paco: Parameter-compositional multi-task reinforcement learning. Advances in Neural Information Processing Systems, 35:21495–21507, 2022

    2022

  88. [96]

    Sutton, R. S. and Barto, A. G. Reinforcement learning: An introduction. MIT press, 2018

    2018

  89. [97]

    Tassa, Y ., Doron, Y ., Muldal, A., Erez, T., Li, Y ., Casas, D. d. L., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq, A., et al. Deepmind control suite. arXiv preprint arXiv:1801.00690, 2018

    2018 arXiv

  90. [98]

    M., Hauth, A., Millican, K., et al

    Team, G., Anil, R., Borgeaud, S., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  91. [99]

    M., Ghosh, D., Walke, H., Pertsch, K., Black, K., Mees, O., Dasari, S., Hejna, J., Kreiman, T., Xu, C., et al

    Team, O. M., Ghosh, D., Walke, H., Pertsch, K., Black, K., Mees, O., Dasari, S., Hejna, J., Kreiman, T., Xu, C., et al. Octo: An open-source generalist robot policy. arXiv preprint arXiv:2405.12213, 2024. 15

    2024 arXiv

  92. [100]

    M., Quan, J., Kirkpatrick, J., Hadsell, R., Heess, N., and Pascanu, R

    Teh, Y ., Bapst, V ., Czarnecki, W. M., Quan, J., Kirkpatrick, J., Hadsell, R., Heess, N., and Pascanu, R. Distral: Robust multitask reinforcement learning. Advances in neural information processing systems, 30, 2017

    2017

  93. [101]

    Llama 2: Open foundation and fine-tuned chat models

    Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y ., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  94. [102]

    Pixel recurrent neural networks

    Van Den Oord, A., Kalchbrenner, N., and Kavukcuoglu, K. Pixel recurrent neural networks. In International conference on machine learning, pp. 1747–1756. PMLR, 2016

    2016

  95. [103]

    Deep reinforcement learning and the deadly triad

    Van Hasselt, H., Doron, Y ., Strub, F., Hessel, M., Sonnerat, N., and Modayil, J. Deep reinforcement learning and the deadly triad. arXiv preprint arXiv:1812.02648, 2018

    2018 arXiv

  96. [104]

    P., Hessel, M., and Aslanides, J

    Van Hasselt, H. P., Hessel, M., and Aslanides, J. When to use parametric models in reinforce- ment learning? Advances in Neural Information Processing Systems, 32, 2019

    2019

  97. [105]

    L., et al

    Van Rossum, G., Drake, F. L., et al. Python reference manual, volume 111. Centrum voor Wiskunde en Informatica Amsterdam, 1995

    1995

  98. [106]

    E., Haberland, M., Reddy, T., Cournapeau, D., Burovski, E., Peterson, P., Weckesser, W., Bright, J., et al

    Virtanen, P., Gommers, R., Oliphant, T. E., Haberland, M., Reddy, T., Cournapeau, D., Burovski, E., Peterson, P., Weckesser, W., Bright, J., et al. Scipy 1.0: fundamental algorithms for scientific computing in python. Nature methods, 17(3):261–272, 2020

    2020

  99. [107]

    Unidexgrasp++: Improving dexterous grasping policy learning via geometry-aware curriculum and iterative generalist-specialist learning

    Wan, W., Geng, H., Liu, Y ., Shan, Z., Yang, Y ., Yi, L., and Wang, H. Unidexgrasp++: Improving dexterous grasping policy learning via geometry-aware curriculum and iterative generalist-specialist learning. In Proceedings of the IEEE/CVF International Conference on Computer Vi...

    2023

  100. [108]

    Unidexgrasp: Universal robotic dexterous grasping via learning diverse proposal generation and goal-conditioned policy

    Xu, Y ., Wan, W., Zhang, J., Liu, H., Shan, Z., Shen, H., Wang, R., Geng, H., Weng, Y ., Chen, J., et al. Unidexgrasp: Universal robotic dexterous grasping via learning diverse proposal generation and goal-conditioned policy. In Proceedings of the IEEE/CVF Conference on Comput...

    2023

  101. [109]

    Knowledge transfer in multi-task deep reinforce- ment learning for continuous control

    Xu, Z., Wu, K., Che, Z., Tang, J., and Ye, J. Knowledge transfer in multi-task deep reinforce- ment learning for continuous control. arXiv preprint arXiv:2010.07494, 2020

    2010 arXiv

  102. [110]

    Adatask: A task-aware adaptive learning rate approach to multi-task learning

    Yang, E., Pan, J., Wang, X., Yu, H., Shen, L., Chen, X., Xiao, L., Jiang, J., and Guo, G. Adatask: A task-aware adaptive learning rate approach to multi-task learning. In Proceedings of the AAAI conference on artificial intelligence, volume 37, pp. 10745–10753, 2023

    2023

  103. [111]

    H., Sadigh, D., and Finn, C

    Yang, J. H., Sadigh, D., and Finn, C. Polybot: Training one policy across robots while embracing variability. In Conference on Robot Learning, pp. 2955–2974. PMLR, 2023

    2023

  104. [112]

    Image augmentation is all you need: Regularizing deep reinforcement learning from pixels

    Yarats, D., Kostrikov, I., and Fergus, R. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In International Conference on Learning Representations,

  105. [113]

    Y ., et al

    Ye, S., Jang, J., Jeon, B., Joo, S., Yang, J., Peng, B., Mandlekar, A., Tan, R., Chao, Y .-W., Lin, B. Y ., et al. Latent action pretraining from videos.arXiv preprint arXiv:2410.11758, 2024

    2024 arXiv

  106. [114]

    Gradient surgery for multi-task learning

    Yu, T., Kumar, S., Gupta, A., Levine, S., Hausman, K., and Finn, C. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems , 33:5824–5836, 2020

    2020

  107. [115]

    Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning

    Yu, T., Quillen, D., He, Z., Julian, R., Hausman, K., Finn, C., and Levine, S. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning, pp. 1094–1100. PMLR, 2020

    2020

  108. [116]

    Efficient online reinforcement learning fine-tuning need not retain offline data

    Zhou, Z., Peng, A., Li, Q., Levine, S., and Kumar, A. Efficient online reinforcement learning fine-tuning need not retain offline data. arXiv preprint arXiv:2412.07762, 2024

    2024 arXiv

  109. [117]

    hard manipulation

    Zitkovich, B., Yu, T., Xu, S., Xu, P., Xiao, T., Xia, F., Wu, J., Wohlhart, P., Welker, S., Wahid, A., et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. In Conference on Robot Learning, pp. 2165–2183. PMLR, 2023. 16 Societal Impact Our work ...

    2023

  110. [2021]

    URL https://openreview.net/forum?id=GY6-6sTvGaf

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.