Pith. sign in

REVIEW 3 major objections 5 minor 60 references

Skill Expansion and Composition in Parameter Space

T0 review · 3 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that blending LoRA skill modules in parameter space with state-dependent weights outperforms composing their noisy outputs or generated actions, and it demonstrates this across safe offline RL, continual policy shift, and…

desk verdict Solid paper with a genuinely useful controlled comparison of composition levels, but the continual-evolution result in Table 2 has a capacity confound that needs an ablation. read the letter →

arxiv 2502.05932 v2 pith:IKCCCKVS submitted 2025-02-09 cs.LG cs.AIcs.RO

Tenglong Liu , Jianxiong Li , Yinan Zheng , Haoyi Niu , Yixing Lan , Xin Xu , Xianyuan Zhan This is my paper · ORCID
classification cs.LGcs.AIcs.RO
keywords skillcompositionparameterspaceLoRAlow-rankadaptationdiffusionpoliciesofflinereinforcementlearningcontinualcontext-aware
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that an autonomous agent can grow its abilities by storing each learned skill as a small low-rank (LoRA) module attached to a frozen base policy, then synthesize new skills by merging those modules directly in parameter space. The central claim is that this parameter-level composition is more sample-efficient and yields better task performance than composing predicted noises or generated actions, because it lets skills share structure before any action is produced. The authors support this with controlled comparisons where only the composition stage changes, plus evaluations on safe offline RL, continual policy shift, and dynamics-shift benchmarks. If correct, the framework offers a practical path to lifelong skill libraries that expand and recombine without catastrophic forgetting.

What carries the argument

The load-bearing object is the LoRA interpolation identity $W_0 + \sum_i \alpha_i \Delta W_i$, where each $\Delta W_i = B_i A_i$ is a low-rank module encoding one skill and $\alpha(s;\theta)$ is a small state-conditioned network trained by the diffusion denoising loss while all skill modules and the base policy stay frozen. This is early fusion: it blends feature transformations before actions are produced, in contrast to late fusion in noise space or action space. Each primitive is a diffusion policy, and the paper provides t-SNE visualizations showing that skill features are better separated and more shareable in parameter space than in noise or action space.

What would settle it

Train two identical skill libraries for a held-out task, composing them only in parameter space versus only in noise space with the same context-aware network and the same small dataset, across several DSRL tasks; if the noise-level variant matches or beats PSEC on both reward and cost, the paper's central claim is falsified.

Watch

Extended reading notes

Core claim

The paper claims that the right place to compose skills is the parameter space, not the noise or action space. Each skill is encoded as a LoRA module $\Delta W_i$ on a frozen pretrained policy, and a lightweight context-aware network $\alpha(s;\theta)$ produces state-dependent weights so the effective policy weights become $W = W_0 + \sum_i \alpha_i(s;\theta)\Delta W_i$. Because the merge happens before any action is generated, the composed network can exploit shared and complementary structure across skills. The paper reports that this parameter-level composition outperforms noise-level composition ($\epsilon = \sum_i \alpha_i \epsilon_i$) and action-level composition ($a = \sum_i \alpha_i a_i$) under identical training budgets, enables multi-objective reward/safety balancing, and lets the skill library grow so that adding a walking skill makes learning to run easier.

Load-bearing premise

The pretrained base policy must already cover a sufficiently diverse distribution of states and actions, so that a small low-rank module can express each new skill; without that coverage, the LoRA adaptation and hence the entire composition mechanism cannot succeed.

Editorial extensions

If this is right

  • New skills can be added as plug-and-play LoRA modules using only tens of demonstrations and a few thousand gradient steps, and the skill library can be pruned by deleting modules without retraining.
  • Parameter-level composition with a context-aware weighting network yields better reward/safety trade-offs on safe offline RL tasks than noise-level or action-level composition with the same training budget.
  • The library grows in capability over time: in the continual policy shift experiments, composing stand plus walk to learn run outperforms composing stand only to learn run.
  • Under dynamics or morphology shift, reusing a source policy through LoRA plus composition outperforms training on pooled source and target data, especially when target data are scarce.
  • Context-aware composition outperforms carefully tuned fixed composition weights, indicating that real-time state-dependent blending is important for decision-making tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The parameter-space advantage suggests that representations shared across skills are more linearly interpolable than their sampling outputs; a testable prediction is that this advantage grows with the number of skills and with task similarity.
  • The same composition machinery could plausibly transfer to other parameter-efficient adapters, such as adapters or prefixes, and to compositional generation beyond decision making, though the paper only demonstrates LoRA.
  • Because only the composition network is trained on a new task, the learned weights $\alpha(s;\theta)$ could serve as an interpretable indicator of which prior skill is active at each state, although the paper does not formally analyze this interpretability.
  • The framework might be combined with automated skill-library pruning based on diversity metrics, which the authors mention as future work but do not implement.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes PSEC, a framework for continual skill acquisition and composition in the parameter space of diffusion policies. Skills are stored as LoRA modules attached to a frozen base policy π0, and a state-conditioned context-aware network outputs scalar weights that merge the LoRA modules directly in parameter space (Eq. 7). The authors compare this parameter-level composition with noise-level composition (NSEC, Eq. 18) and action-level composition (ASEC, Eq. 19) under controlled conditions where only the composition stage differs. The method is evaluated on multi-objective safe offline RL (DSRL), continual policy shift on DMC (stand→walk→run), and dynamics shift on D4RL. The central empirical claim is that composing in parameter space yields better sample efficiency and safety/reward trade-offs than composing at the noise or action level, and that the skill library can be expanded and reused to progressively improve performance.

Significance. If the controlled comparisons are taken at face value, the paper makes a useful and practical contribution: it shows that, for LoRA-encoded diffusion policies, the stage at which skills are composed matters, and that parameter-space merging with a context-aware weight network outperforms noise- and action-space merging under otherwise identical training conditions. The strengths of the paper are the controlled PSEC/NSEC/ASEC comparisons in §4.1, the broad evaluation across three distinct scenarios, the additional Meta-World experiments in Appendix F, and the explicit discussion of limitations in Appendix A. The main weaknesses are the absence of variance reporting in the headline tables and a confounded continual-evolution comparison, which currently prevent the strongest claims from being fully supported.

major comments (3)
  1. [§4.2, Table 2, Appendix C.2] The claim that S+W→R (247) outperforms S→R (221) demonstrates continual evolution is confounded. In S→R the composition weight is α(s;θ2) ∈ R^1, while in S+W→R it is α(s;θ) ∈ R^2, so the two conditions differ not only in the presence of the walk LoRA ΔW1 but also in the capacity of the context-aware network (an additional output unit and an additional frozen module in the composed network). The 26-point improvement could therefore come from the larger composition network rather than from the walk knowledge. Please add an ablation that isolates the walk skill, e.g., S+W→R with a randomly initialized or zeroed ΔW1, or S→R with a two-output α whose second branch is fixed to zero, and report the resulting scores.
  2. [Tables 1 and 2, Figure 7] The main results are reported without standard deviations: Table 1 states 'averaged over 20 evaluation episodes and 4 seeds' and Table 2 'averaged over 10 episodes and 5 seeds', yet no variance or significance measure is given. This makes it impossible to judge whether the reported improvements, especially the modest S+W→R vs S→R gap in Table 2, are statistically reliable. Please report mean ± std (or confidence intervals) across seeds in these tables, consistent with Table 8 in the appendix, and, where margins are small, provide a significance test.
  3. [Appendix B] The sentence 'We conduct empirical evaluations in our rebuttal to demonstrate this' refers to evidence that is not part of the manuscript. An archival version must be self-contained; either include those evaluations in the paper or remove the sentence.
minor comments (5)
  1. [§4.1] The text says 'We denote them as NESC and ASEC respectively', but the acronym NESC does not appear elsewhere; it should be NSEC throughout.
  2. [Appendix C.3] There is a typo: 'halcheetah-medium-v2' should be 'halfcheetah-medium-v2'.
  3. [§3.3, Eq. (19)] The comparison with MCP (Peng et al., 2019) in Eq. (8) uses a product-of-Gaussians composition, whereas the ASEC baseline in Eq. (19) uses additive action composition. Clarifying this distinction would avoid the impression that action-level composition is deliberately weakened.
  4. [Figure 7] The bar chart in the main text does not show error bars even though Appendix Table 8 reports standard deviations. Adding error bars to Figure 7 would make the dynamics-shift results more interpretable.
  5. [§4.2] The statement 'we train less than 50k gradient steps for almost all tasks, while previous methods typically require millions of gradient steps and data' is not directly quantified against the baselines; a concrete comparison of training steps or wall-clock time would strengthen the efficiency claim.

Circularity Check

0 steps flagged · score 2.0 of 10

No load-bearing circularity: the central parameter-vs-noise-vs-action composition claim rests on controlled in-paper comparisons, and self-citations are not load-bearing; the Table 2 continual-evolution gain is a capacity confound (a correctness risk), not a constructed reduction.

full rationale

The paper's central claim — that composing LoRA-encoded diffusion policies in parameter space beats composing at the noise or action level — is supported by controlled in-paper experiments rather than by a derivation that reduces to its own inputs. In the multi-objective setting (Section 4.1, Appendix C.1), NSEC and ASEC are built by changing only the composition stage (Eqs. 18-19) while keeping the base policy, the LoRA modules, the advantage weighting, the composition network, the training data, and the training loss identical; the comparison of composition stages is therefore self-contained, and standard external baselines (BC, CDT, COptiDICE, CPQ, FISOR) plus the external DSRL, D4RL, and DMC benchmarks provide independent grounding. The context-aware weights alpha(s;theta) are trained on the target task's own data in all three composition variants, so the PSEC-vs-NSEC-vs-ASEC comparison does not favor PSEC by construction; it is a standard train-and-evaluate setup, and the Meta-World zero-shot experiment (Appendix F.2, Table 7) tests genuine transfer. Several cited results are co-authored by PSEC's authors (FISOR, Zheng et al. 2024; TSRL, Cheng et al. 2023; DOGE, Li et al. 2023; OMPO, Luo et al. 2024a), but none is load-bearing in a circular sense: FISOR supplies the advantage-weighted objective for pi_1 and pi_2 and is used identically for PSEC, NSEC, and ASEC, while baseline numbers sourced from TSRL and FISOR are standard reporting of published results. No uniqueness theorem or ansatz is imported from co-authored work; the LoRA interpolation property is cited to external work (Hu et al. 2021; Clark et al. 2024; Zhang et al. 2023b). The weakest point is Section 4.2's continual-evolution comparison: S+W->R (247) and S->R (221) share the same run LoRA Delta W_2 trained on the same 10 trajectories, and the +26 gain could in principle come from the larger R^2 composition network rather than from the walk knowledge. This is a capacity confound and an evaluation-design risk, not a circularity: the S+W->R score is a measured rollout result, no equation in the paper forces it to exceed S->R, and no fitted parameter is renamed as a prediction. The stated limitation in Appendix A — that pi_0 must cover a diverse distribution for small LoRA modules to be expressive — is a genuine structural assumption, distinct from the composition claim, and is flagged rather than assumed into the result.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The framework depends on a small number of hyperparameters (LoRA rank, scale, diffusion steps) and a data selection threshold, plus three domain assumptions: coverage of π0, validity of LoRA interpolation for policies, and fixed state-action spaces. No new physical or conceptual entities are introduced.

free parameters (4)
  • LoRA rank n = 8 (default); 16 in multi-objective setting
    Selected by the ablation in Section 4.4 (Figure 8); performance varies with rank and the best rank is chosen for all experiments.
  • LoRA scale = 16
    Fixed interpolation weight used when training skill modules (Eq. 5, Appendix C.1: W1 = W0 + 16ΔW1). No sensitivity analysis reported.
  • D-star filter = top-30 trajectories, cost < 5
    Ad hoc selection of near-expert demonstrations for training the composer; influences all multi-objective composition results (Appendix C.1).
  • Diffusion steps T = 5
    Number of denoising steps, fixed for all experiments.
assumptions (4)
  • domain assumption The pretrained policy π0 has sufficient state-action coverage for low-rank adaptation to represent new skills.
    Stated as the main limitation in Appendix A; if false, LoRA modules cannot encode new behaviors and the framework's expansion mechanism breaks.
  • domain assumption Linearly interpolating LoRA parameter matrices produces a policy that meaningfully interpolates the corresponding skills.
    Invoked in Section 3.2 around Eq. (6), citing Clark et al. (2024) in image generation; the paper assumes this property transfers to decision-making policies.
  • domain assumption State and action spaces are unchanged across tasks.
    Assumed in Section 3.1 problem setup; needed for the same base policy and LoRA modules to be composable across tasks.
  • domain assumption Diffusion models can represent policies and predict noise gradients that are composable as energy functions.
    Standard in diffusion policy literature (Section 3.1, Eq. 2-3).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Skill Expansion and Composition in Parameter Space." pith.science (2026). https://pith.science/paper/IKCCCKVS

@misc{pith2026250205932,
  author       = {Pith},
  title        = {Pith review of: Skill Expansion and Composition in Parameter Space},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IKCCCKVS}},
  note         = {Machine review of arXiv:2502.05932}
}
read the original abstract

Humans excel at reusing prior knowledge to address new challenges and developing skills while solving problems. This paradigm becomes increasingly popular in the development of autonomous agents, as it develops systems that can self-evolve in response to new challenges like human beings. However, previous methods suffer from limited training efficiency when expanding new skills and fail to fully leverage prior knowledge to facilitate new task learning. In this paper, we propose Parametric Skill Expansion and Composition (PSEC), a new framework designed to iteratively evolve the agents' capabilities and efficiently address new challenges by maintaining a manageable skill library. This library can progressively integrate skill primitives as plug-and-play Low-Rank Adaptation (LoRA) modules in parameter-efficient finetuning, facilitating efficient and flexible skill expansion. This structure also enables the direct skill compositions in parameter space by merging LoRA modules that encode different skills, leveraging shared information across skills to effectively program new skills. Based on this, we propose a context-aware module to dynamically activate different skills to collaboratively handle new tasks. Empowering diverse applications including multi-objective composition, dynamics shift, and continual policy shift, the results on D4RL, DSRL benchmarks, and the DeepMind Control Suite show that PSEC exhibits superior capacity to leverage prior knowledge to efficiently tackle new challenges, as well as expand its skill libraries to evolve the capabilities. Project website: https://ltlhuuu.github.io/PSEC/.

Figures

Figures reproduced from arXiv: 2502.05932 by the authors.

Figure 1
Figure 1. PSEC framework and its application in diverse scenarios. (a) We maintain a skill library that contains many skills primitives and can progressively expand by adding new LoRA modules. (b) Then we train a context-aware compositional network to adaptively compose different elements in the skill library to solve new tasks. (c) PSEC framework is versatile to diverse applications where reusing prior knowledge is crucial. … view at source ↗
Figure 2
Figure 2. (a) Each skill is encoded in separate LoRA modules respectively. (b) By adjusting the composing weights αi, different LoRA modules can merge together to interpolate new skills. the state space S and action space A remain unchanged during training, which is a mild assumption in many relevant works (Peng et al., 2019; Ajay et al., 2023; Nair et al., 2020; Liu et al., 2024b; Dai et al., 2024). We consider an agent with… view at source ↗
Figure 3
Figure 3. Comparison between parameter-, noise-, and action-level composition. Parameter-level composition offers more flexibility to leverage the shared or complementary structure across skills to compose new skills. Noise- and action-level composition, however, is too late to benefit from this information. management by removing suboptimal primitives and retaining the effective ones. More importantly, by adjusting the value… view at source ↗
Figures from the paper (17 more)
Figure 4
Figure 4. Figure 4: t-SNE projections of samples from different skills in parameter, noise, and action space. The parameter space exhibits a good structure for skill composition, where skills share common knowledge while retaining their unique features to avoid confusion. Noise and action…
Figure 5
Figure 5. Figure 5: Output weights of context-aware modular evaluated on the MetaDrive-easymean task. The network dynamically adjusts the weights to handle real-time demands: It prioritizes safety policies when the vehicle approaches obstacles or navigates a turn while avoiding boundary l…
Figure 6
Figure 6. Figure 6: Comparisons on sample and training efficiency and the effectiveness of context-aware modular. S, W, R denote stand, walk and run, respectively. Each value is averaged over 10 episodes and 5 seeds. compositions and LoRA module. 3) We evaluate different PSEC variants wit…
Figure 7
Figure 7. Figure 7: Results in the dynamics shift setting over 10 episodes and 5 seeds. -m, -mr and -me refer to D P1 o sampling from medium, medium-replay and medium-expert-v2 data in D4RL (Fu et al., 2020), respectively. eralizable offline RL methods including DOGE (Li et al., 2023) and…
Figure 8
Figure 8. Figure 8: Ablations on LoRA ranks. We primarily ablate on different LoRA ranks n to assess the robustness of our methods in continual policy shift setting [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]
Figure 9
Figure 9. Figure 9: Illustrative comparisons between PSEC and other modularized multitask learning frameworks when deployed to continual learning settings. Modularized skills for multitask learning. Multitask learning methods attempt to leverage the complementary benefits and commonalitie…
Figure 10
Figure 10. Figure 10: Results in the policy shift setting. Each value is averaged over 10 episodes and 5 seeds. Stand Walk Run [PITH_FULL_IMAGE:figures/full_fig_p025_10.png]
Figure 11
Figure 11. Figure 11: Continual evolution on DeepMind Control Suite for Continual policy shift. 10 30 50 100 Number of Trajectories 0 200 400 600 Scores S R with different data quantity of R task Scratch ASEC NSEC PSEC (a) Sample efficiency. 0k 2k 4k 6k 8k 10k Training Steps 100 200 300 Sc…
Figure 12
Figure 12. Figure 12: Comparisons on sample and training efficiency and the effectiveness of context-aware modular. S, R denote stand, run, respectively. Each value is averaged over 10 episodes and 5 seeds. C.3 DYNAMIC SHIFT To further validate the versatility of PSEC, we conduct experimen…
Figure 13
Figure 13. Figure 13: The illustration of the source and target domains for the dynamic shift setting. tuples (s, a1) into the trained networks and get the output features of the middle LoRA augmented layers. Then, we project these features using t-SNE in [PITH_FULL_IMAGE:figures/full_fig…
Figure 14
Figure 14. Figure 14: Outputs weights of the context-aware modular on DeepMind Control [PITH_FULL_IMAGE:figures/full_fig_p027_14.png]
Figure 15
Figure 15. Figure 15: Comparison of Model Parameters: The parameter count for PSEC is approximately 7.58% of Scratch, demonstrating a significantly smaller model size while maintaining strong parameter efficiency, effec￾tively leveraging a smaller number of parameters for superior task per…
Figure 16
Figure 16. Figure 16: Visualization of the simulation environments and representative tasks of MetaDrive and Bullet￾Safety-Gym. The figure is credited to Liu et al. (2023a). E.3 ADVANTAGE OF THE BENCHMARK By generating diverse datasets across many environments with systematically varied co…
Figure 17
Figure 17. Figure 17: Visualization of the simulation environments and representative tasks of Meta-World. 29 [PITH_FULL_IMAGE:figures/full_fig_p029_17.png]
Figure 18
Figure 18. Figure 18: Illustration of the cost-reward plot for datasets from MetaDrive and Bullet-Safety-Gym. 33 [PITH_FULL_IMAGE:figures/full_fig_p033_18.png]
Figure 19
Figure 19. Figure 19: We evaluate the final running policies of PSEC, NSEC and ASEC with the “stand,” “walk,” and “run” rewards with 10 episodes and 3 random seeds. Then we plot the reward distribution by kernel density estimation (KDE). Each curve represents the probability density of rew…
Figure 20
Figure 20. Figure 20: Results of performance conducted on dynamic shift and body shift tasks. The lines and [PITH_FULL_IMAGE:figures/full_fig_p035_20.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

60 extracted references · 37 canonical work pages

  1. [1]

    Road” encompasses three levels of difficulty for self-driving cars: easy, medium, and hard, while “Vehicle

    and achieves higher rewards. MetaDrive. It leverages the Panda3D game engine to simulate realistic driving scenarios. The tasks are categorized as {Road}{Vehicle}, where “Road” encompasses three levels of difficulty for self-driving cars: easy, medium, and hard, while “Vehicle” represents four levels of surrounding traffic density: sparse, mean, and dense...

    work page 2023

  2. [3]

    Thigh: the thigh is enlarged to double its original size to produce a morphology gap on the embodiment. The target datasets DP1 o are sampled from the D4RL benchmark with un-modified dynamics P1, includ- ing 6 types: halcheetah-medium-v2, halfcheetah-medium-replay-v2, halfcheetah-medium-expert- v2, walker2d-medium-v2, walker2d-medium-replay-v2, walker2d-m...

    work page 2025

  3. [4]

    Offline reinforcement learning via high-fidelity generative behavior modeling

    Huayu Chen, Cheng Lu, Chengyang Ying, Hang Su, and Jun Zhu. Offline reinforcement learning via high-fidelity generative behavior modeling. arXiv preprint arXiv:2209.14548,

    arXiv

  4. [5]

    Modularized skills for multitask learning

    Figure 9: Illustrative comparisons between PSEC and other modularized multitask learning frameworks when deployed to continual learning settings. Modularized skills for multitask learning. Multitask learning methods attempt to leverage the complementary benefits and commonalities across different tasks to enhance the cross-task general- ization and capabi...

    work page 2020

  5. [6]

    Is Mamba Compatible with Trajectory Optimization in Offline Reinforcement Learning?

    Yang Dai, Oubo Ma, Longfei Zhang, Xingxing Liang, Shengchao Hu, Mengzhu Wang, Shouling Ji, Jincai Huang, and Li Shen. Is mamba compatible with trajectory optimization in offline rein- forcement learning? arXiv preprint arXiv:2405.12094,

    work page Pith review arXiv

  6. [7]

    Interestingly, even without access to unseen task data during training, PSEC demonstrates strong performance on several tasks. Notably, PSEC substantially outperforms NSEC and ASEC on this zero-shot transfer setting, highlighting the advantages of skill compositions in parameter spaces over noise and action spaces. Overall, the results demonstrate PSEC’s ...

    work page 2025

  7. [8]

    D4rl: Datasets for deep data-driven reinforcement learning

    Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219,

    arXiv 2004

  8. [9]

    All these methods demonstrate a modularized structure, where separate parameters are required to solve different tasks. Not only enjoying the benefits of multitask learning, this modularized design allows for efficient adaptation to new tasks by exploiting the shareable knowledge stored in different modules (Happel & Murre, 1994; SHARKEY, 1996; Auda & Kam...

    work page 1994

Show all 60 references
  1. [10]

    Philippe Hansen-Estruch, Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine

    URL https://openreview.net/forum?id=HkxStoC5F7. Philippe Hansen-Estruch, Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. Idql: Implicit q-learning as an actor-critic method with diffusion policies. arXiv preprint arXiv:2304.10573,

    arXiv

  2. [11]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685,

    arXiv

  3. [13]

    Solving continual offline rein- forcement learning with decision transformer

    Kaixin Huang, Li Shen, Chen Zhao, Chun Yuan, and Dacheng Tao. Solving continual offline rein- forcement learning with decision transformer. arXiv preprint arXiv:2401.08478,

    arXiv

  4. [14]

    Unsupervised-to-online reinforcement learning

    13 Published as a conference paper at ICLR 2025 Junsu Kim, Seohong Park, and Sergey Levine. Unsupervised-to-online reinforcement learning. arXiv preprint arXiv:2408.14785,

    2025 arXiv

  5. [15]

    Adam: A method for stochastic optimization

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980,

    arXiv

  6. [16]

    Daniel Lawson and Ahmed H Qureshi

    doi: 10.1016/J.KNOSYS.2022.108221. Daniel Lawson and Ahmed H Qureshi. Merging decision transformers: Weight averaging for form- ing multi-task policies. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 12942–12948. IEEE,

    2022

  7. [19]

    Adaptive advantage-guided policy regularization for offline reinforcement learning

    Tenglong Liu, Yang Li, Yixing Lan, Hao Gao, Wei Pan, and Xin Xu. Adaptive advantage-guided policy regularization for offline reinforcement learning. In International Conference on Machine Learning, volume 235, pp. 31406 – 31424. PMLR, 2024b. 14 Published as a conference paper ...

    2025 arXiv

  8. [20]

    OMPO: A unified framework for RL under policy and dynamics shifts

    Yu Luo, Tianying Ji, Fuchun Sun, Jianwei Zhang, Huazhe Xu, and Xianyuan Zhan. OMPO: A unified framework for RL under policy and dynamics shifts. In Forty-first International Conference on Machine Learning, 2024a. URL https://openreview.net/forum?id= R83VIZtHXA. Yunhao Luo, Che...

    arXiv

  9. [21]

    Meta-neural networks that learn by learning

    Devang K Naik and Richard J Mammone. Meta-neural networks that learn by learning. In[Proceed- ings 1992] IJCNN International Joint Conference on Neural Networks , volume 1, pp. 437–442. IEEE,

    1992

  10. [23]

    6892–6903

    In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 6892–6903. IEEE,

    2024

  11. [24]

    Modular deep belief networks that do not forget

    Leo Pape, Faustino Gomez, Mark Ring, and J ¨urgen Schmidhuber. Modular deep belief networks that do not forget. In The 2011 International joint conference on neural networks, pp. 1191–1198. IEEE,

    2011

  12. [26]

    Lora soups: Merging loras for practical skill composition tasks

    Akshara Prabhakar, Yuanzhi Li, Karthik Narasimhan, Sham Kakade, Eran Malach, and Samy Jelassi. Lora soups: Merging loras for practical skill composition tasks. arXiv preprint arXiv:2410.13025,

    arXiv

  13. [27]

    Diffusion policy policy optimiza- tion

    Allen Z Ren, Justin Lidard, Lars L Ankile, Anthony Simeonov, Pulkit Agrawal, Anirudha Majum- dar, Benjamin Burchfiel, Hongkai Dai, and Max Simchowitz. Diffusion policy policy optimiza- tion. arXiv preprint arXiv:2409.00588,

    arXiv

  14. [28]

    An overview of multi-task learning in deep neural networks

    S Ruder. An overview of multi-task learning in deep neural networks. arXiv preprint arXiv:1706.05098,

    arXiv

  15. [29]

    Continual diffusion: Continual customization of text-to-image diffusion with c-lora

    James Seale Smith, Yen-Chang Hsu, Lingyu Zhang, Ting Hua, Zsolt Kira, Yilin Shen, and Hongxia Jin. Continual diffusion: Continual customization of text-to-image diffusion with c-lora. arXiv preprint arXiv:2304.06027,

    arXiv

  16. [30]

    An introduction to lifelong supervised learning

    Shagun Sodhani, Mojtaba Faramarzi, Sanket Vaibhav Mehta, Pranshu Malviya, Mohamed Abdel- salam, Janarthanan Janarthanan, and Sarath Chandar. An introduction to lifelong supervised learning. arXiv preprint arXiv:2207.04354,

    arXiv

  17. [31]

    Paco: Parameter-compositional multi-task reinforcement learning

    16 Published as a conference paper at ICLR 2025 Lingfeng Sun, Haichao Zhang, Wei Xu, and Masayoshi Tomizuka. Paco: Parameter-compositional multi-task reinforcement learning. Advances in Neural Information Processing Systems , 35: 21495–21507,

    2025

  18. [32]

    Deepmind control suite

    Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Bud- den, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801.00690,

    arXiv

  19. [34]

    Are expressive models truly necessary for offline rl? arXiv preprint arXiv:2412.11253, 2024a

    Guan Wang, Haoyi Niu, Jianxiong Li, Li Jiang, Jianming Hu, and Xianyuan Zhan. Are expressive models truly necessary for offline rl? arXiv preprint arXiv:2412.11253, 2024a. Lirui Wang, Jialiang Zhao, Yilun Du, Edward H Adelson, and Russ Tedrake. Poco: Policy compo- sition from ...

    arXiv

  20. [35]

    Multi-task reinforcement learning with soft modularization

    17 Published as a conference paper at ICLR 2025 Ruihan Yang, Huazhe Xu, Yi Wu, and Xiaolong Wang. Multi-task reinforcement learning with soft modularization. Advances in Neural Information Processing Systems, 33:4767–4777,

    2025

  21. [36]

    Trace norm regularised deep multi-task learning

    Yongxin Yang and Timothy M Hospedales. Trace norm regularised deep multi-task learning. arXiv preprint arXiv:1606.04038,

    arXiv

  22. [39]

    • Assumption on the expressiveness of the pretrain policy

    18 Published as a conference paper at ICLR 2025 A L IMITATIONS AND FUTURE WORKS In this section, we provide detailed discussions about the limitations and their potential solutions. • Assumption on the expressiveness of the pretrain policy. The main limitation of PSEC is the a...

    2025

  23. [40]

    also employ LoRA to encode skills. However, they solely investigate the parameter-isolation property of LoRA to prevent catastrophic forgetting, while overlooking the potential to merge different LoRA modules to interpolate new skills. Moreover, TAIL only studies the IL domain...

    2023

  24. [43]

    aim to learn a shared feature that is strong and generalizable enough to capture the commonalities across all different tasks. This is achieved by developing a multi-head style structure, where different heads solve different tasks and all heads share some common layers (Sun e...

    2022

  25. [44]

    are similar to the hard ones, with the differ- ences primarily in the shared features. Instead of directly employing shared layers (Bakker & Hes- kes, 2003; Caruana, 1993; Sun et al., 2022; Le ´on et al., 2021), soft parameter sharing approaches adopt regularizations to enforc...

    2024

  26. [45]

    The modularization method, however, can address this problem fundamen- tally by learning new parameters without disrupting pretrained ones

    often exhibit some catastrophic forgetting. The modularization method, however, can address this problem fundamen- tally by learning new parameters without disrupting pretrained ones. Along this line, numerous works also adopt modularized structure in a hard or soft manner as ...

    1994

  27. [46]

    or the parameter space (Huang et al., 2023; Prabhakar et al., 2024; Pape et al., 2011; Zhong et al., 2024). PSEC systematically investigates the advantages of skill compo- sitions in parameter space over the noise space and action space, offering clear guidance for future rese...

    2023

  28. [47]

    Additionally, Araki et al

    but require significant human effort for skill management, limiting scalability. Additionally, Araki et al. (2021) focuses on efficient pretraining, not on fast adaptation/continual improvement. In contrast, PSEC targets the later setups and minimizes human effort by incorpora...

    2021

  29. [48]

    For FISOR (Zheng et al., 2024), CDT (Liu et al., 2023b), COptiDICE (Lee et al., 2022a), CPQ (Xu et al.,

    Baselines. For FISOR (Zheng et al., 2024), CDT (Liu et al., 2023b), COptiDICE (Lee et al., 2022a), CPQ (Xu et al.,

    2024

  30. [49]

    For NSEC and ASEC results, we only change the compositional stages, and meanwhile keep all other training details the same to ensure a fair comparison

    and BC, we adopt the results from FISOR (Zheng et al., 2024). For NSEC and ASEC results, we only change the compositional stages, and meanwhile keep all other training details the same to ensure a fair comparison. Specifically, the context-aware modular for NSEC is trained via...

    2024

  31. [50]

    (2024) for the policy learning

    We use three expert datasets including walker-stand DT0 e , walker-walk DT1 e , and walker-run DT2 e , released by Bai et al. (2024) for the policy learning. Specifically,DT0 e , DT1 e and DT2 e contains 1000, 10 and 10 trajectories, respectively. DT1 e and DT2 e contain only ...

    2024

  32. [51]

    We compare PSEC with other composition methods NSEC and ASEC, the Scratch method, and the variant PSEC (MLP)

    Baselines. We compare PSEC with other composition methods NSEC and ASEC, the Scratch method, and the variant PSEC (MLP). NSEC and ASEC train the context-aware modular repre- sented by Eq. (18) and Eq. (19), respectively. Scratch method means training a policy from scratch by I...

    2023

  33. [53]

    Baselines. We compare PSEC with other methods in dynamic shift settings, including behavioral cloning (BC), offline RL approaches like CQL (Kumar et al., 2020), IQL (Kostrikov et al., 2022), and model-based methods such as MOPO (Yu et al., 2020c). Additionally, we evaluate mor...

    2020

  34. [54]

    The baseline results for comparison are sourced from the TSRL paper (Cheng et al., 2023), which reports state-of-the-art performance in these regimes

    and TSRL (Cheng et al., 2023), which have demonstrated superiority in small sample regimes. The baseline results for comparison are sourced from the TSRL paper (Cheng et al., 2023), which reports state-of-the-art performance in these regimes. Furthermore, we assess policies tr...

    2023

  35. [57]

    The training datasets are the same as the datasets collected by L2M

    and L2M (Schmied et al., 2024), we split the 50 tasks into 40 pre-training tasks and 10 fine-tuning unseen tasks (CW10). The training datasets are the same as the datasets collected by L2M. We train 10K steps per task in CW10, which is only 10% training steps of L2M, with a ba...

    2024

  36. [59]

    stand,” “walk,

    The results show that PSEC achieves high rewards across all tasks, whereas NSEC and ASEC cannot, demonstrating that the PSEC’s running skill retains behaviors from walking and standing and suggesting superior skill sharing of PSEC compared to NSEC and ASEC. 31 Published as a c...

    2025

  37. [1992]

    Awac: Accelerating online rein- forcement learning with offline datasets

    Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online rein- forcement learning with offline datasets. arXiv preprint arXiv:2006.09359,

    2006 arXiv

  38. [1997]

    Dota 2 with large scale deep reinforcement learning

    11 Published as a conference paper at ICLR 2025 Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemysław Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprin...

    2025 arXiv

  39. [2006]

    Coptidice: Offline constrained reinforcement learning via stationary distribution correction estimation

    Jongmin Lee, Cosmin Paduraru, Daniel J Mankowitz, Nicolas Heess, Doina Precup, Kee-Eung Kim, and Arthur Guez. Coptidice: Offline constrained reinforcement learning via stationary distribution correction estimation. In International Conference on Learning Representations, 2022a...

    arXiv

  40. [2008]

    $ 0.25 ×FrictionSource domain Target domain Thigh Size ×

    projection of data samples in different spaces. Specifically, for each dataset DT0 e , DT1 e , DT2 e in the continual policy shift setting in Section C.2, we randomly sample 512 data samples (s, a), which forms three types of data that encode the standing, walking and running ...

    2025

  41. [2011]

    The ideal continual learner: An agent that never forgets

    15 Published as a conference paper at ICLR 2025 Liangzu Peng, Paris Giampouras, and Ren´e Vidal. The ideal continual learner: An agent that never forgets. In International Conference on Machine Learning, pp. 27585–27610. PMLR,

    2025

  42. [2012]

    Policy expansion for bridging offline-to-online reinforce- ment learning

    Haichao Zhang, Wei Xu, and Haonan Yu. Policy expansion for bridging offline-to-online reinforce- ment learning. In The Eleventh International Conference on Learning Representations, 2023a. Jinghan Zhang, Junteng Liu, Junxian He, et al. Composing parameter-efficient modules wit...

    arXiv

  43. [2014]

    The lines and shaded areas indicate the averages and standard deviations calculated over 5 random seeds

    Training steps 1e6 π1 Q∗ r(s, a) hidden dim 256 Q∗ r(s, a) hidden layers 2 Q∗ r(s, a) activation function ReLU V ∗ r (s) hidden dim 256 V ∗ r (s) hidden layers 2 V ∗ r (s) activation function ReLU Actor hidden dim 256 Actor hidden layers 2 Actor Activation function ReLU Mini-b...

    2025

  44. [2017]

    Intelligent problem-solving as integrated hierarchical reinforcement learning

    12 Published as a conference paper at ICLR 2025 Manfred Eppe, Christian Gumbsch, Matthias Kerzel, Phuong DH Nguyen, Martin V Butz, and Stefan Wermter. Intelligent problem-solving as integrated hierarchical reinforcement learning. Nature Machine Intelligence, 4(1):11–20,

    2025

  45. [2018]

    Do- main randomization for transferring deep neural networks from simulation to the real world

    Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Do- main randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS) , pp. 23–...

    2017

  46. [2019]

    Mod- ular meta-learning in abstract graph networks for combinatorial generalization

    Ferran Alet, Maria Bauza, Alberto Rodriguez, Tomas Lozano-Perez, and Leslie P Kaelbling. Mod- ular meta-learning in abstract graph networks for combinatorial generalization. arXiv preprint arXiv:1812.07768,

    arXiv

  47. [2020]

    Single-task continual offline reinforcement learning

    Sibo Gai and Donglin Wang. Single-task continual offline reinforcement learning. arXiv preprint arXiv:2404.12639,

    arXiv

  48. [2021]

    Contin- ual offline reinforcement learning via diffusion-based dual generative replay

    Jinmei Liu, Wenbin Li, Xiangyu Yue, Shilin Zhang, Chunlin Chen, and Zhi Wang. Contin- ual offline reinforcement learning via diffusion-based dual generative replay. arXiv preprint arXiv:2404.10662, 2024a. Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B Tenenbaum. ...

    arXiv

  49. [2022]

    Lorahub: Effi- cient cross-task generalization via dynamic lora composition

    Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. Lorahub: Effi- cient cross-task generalization via dynamic lora composition. arXiv preprint arXiv:2307.13269,

    arXiv

  50. [2023]

    Scaling offline model-based rl via jointly-optimized world-action model pretrain- ing

    Jie Cheng, Ruixi Qiao, Gang Xiong, Qinghai Miao, Yingwei Ma, Binhua Li, Yongbin Li, and Yisheng Lv. Scaling offline model-based rl via jointly-optimized world-action model pretrain- ing. arXiv preprint arXiv:2410.00564, 2024a. Jie Cheng, Gang Xiong, Xingyuan Dai, Qinghai Miao,...

  51. [2024]

    Solving rubik’s cube with a robot hand

    Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, et al. Solving rubik’s cube with a robot hand. arXiv preprint arXiv:1910.07113,

    1910 arXiv

  52. [2025]

    Ming Zhong, Yelong Shen, Shuohang Wang, Yadong Lu, Yizhu Jiao, Siru Ouyang, Donghan Yu, Jiawei Han, and Weizhu Chen

    URL https://openreview.net/forum?id=wM2sfVgMDH. Ming Zhong, Yelong Shen, Shuohang Wang, Yadong Lu, Yizhu Jiao, Siru Ouyang, Donghan Yu, Jiawei Han, and Weizhu Chen. Multi-lora composition for image generation. arXiv preprint arXiv:2402.16843,

    arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.