Pith. sign in

REVIEW 3 major objections 5 minor 6 cited by

Policy Decorator: Model-Agnostic Online Refinement for Large Policy Model

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A small residual policy trained online with sparse rewards can refine frozen imitation policies to near-perfect performance on eight manipulation tasks while preserving smooth motion.

desk verdict Solid empirical paper on residual RL for large imitation policies; the per-task hyperparameters and softened RLPD baseline are the main soft spots. read the letter →

arxiv 2412.13630 v1 pith:RIWLAU73 submitted 2024-12-18 cs.RO cs.AIcs.LG

Xiu Yuan , Tongzhou Mu , Stone Tao , Yunhao Fang , Mengke Zhang , Hao Su This is my paper · ORCID
classification cs.ROcs.AIcs.LG
keywords residualpolicyonlinereinforcementlearningimitationbehaviorcloningmodel-agnosticrefinementcontrolledexplorationsparserewardsrobotmanipulation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Policy Decorator claims that a large, offline-trained imitation policy can be improved through online interaction without changing any of its weights. The paper's proposal is to learn a small residual policy—a per-step correction added to the base policy's action—using reinforcement learning with only the task's sparse success signal. Its argument rests on the observation that imitation policies fail mainly through compounding error: small deviations that a bounded correction can pull back onto the demonstration-covered path. With residual actions squashed to $(-\alpha,\alpha)$ and introduced through a schedule that starts from the base policy alone, the method reaches near-perfect success on the tasks it evaluates with low-dimensional state observations, for both Behavior Transformer and Diffusion Policy bases, while preserving the smooth, multi-modal motion of the imitation policy.

What carries the argument

The load-bearing object is the bounded residual policy combined with the progressive exploration schedule. The residual is a small Gaussian network (MLP for state observations, CNN for images) trained by SAC; after a $\tanh$ squashing function and scaling by $\alpha$, its output lies in $(-\alpha,\alpha)$, so the correction is small by construction. The behavior policy executes $\pi_{\text{base}}(s)$ with probability $1-\epsilon$ and $\pi_{\text{base}}(s)+\pi_{\text{res}}(s)$ otherwise, with $\epsilon$ increasing linearly from 0 to 1 over $H$ environment steps. This design keeps the agent inside states where the sparse success signal is still reachable, which is what makes online refinement sample-efficient; because the base policy is only queried and never differentiated, the method works for any policy architecture that maps observations to actions.

What would settle it

Take a base policy whose rollouts on a target task land mostly in states far from the demonstration state distribution, measurable by the density of demonstrations around the states the policy visits. Run Policy Decorator with the paper's settings: if success does not rise substantially, or if the learned residual saturates at the $\alpha$ bound on most steps while reward stays near zero, the central 'small adjustments bring deviated trajectories back on track' premise is falsified for that policy.

Watch

Extended reading notes

Core claim

The central claim is that a frozen imitation policy $\pi_{\text{base}}$ can be wrapped, decorator-style, by a small learnable residual policy $\pi_{\text{res}}$ trained with Soft Actor-Critic on the environment's sparse success reward; the action executed in the environment is $\pi_{\text{base}}(s)+\pi_{\text{res}}(s)$. Two controls make this stable: the residual output is $\tanh$-squashed and scaled by $\alpha$ so that the combined trajectory stays near the base trajectory, and a progressive schedule mixes the residual in gradually, using the base policy alone at the start and increasing the probability of adding the residual over time. The paper reports near-perfect success on every task it evaluates with low-dimensional state observations (for Diffusion Policy, StackCube and Adroit Door were skipped because the base policy was already near-optimal), strong results with image observations, and consistent superiority over fine-tuning baselines (LoRA-based SAC, DIPO, Cal-QL, RLPD, ROT) and non-fine-tuning baselines (JSRL, Residual RL, FISH). The refined policy also keeps the base policy's smooth motion and, in the Behavior Transformer case, its multi-modal action distribution.

Load-bearing premise

The base policy must already be reasonably competent, since the method only learns small per-step corrections: if the policy has drifted into states far outside its demonstration distribution, a bounded action-level patch cannot recover and no sparse success signal will be available to train the residual.

Editorial extensions

If this is right

  • Any frozen policy with reasonable initial performance—Behavior Transformer, Diffusion Policy, MLP, RNN, or CNN—can be improved online using only sparse success signals, without gradient access to the base model.
  • Fine-tuning large imitation policies, which the paper finds prone to early unlearning and long-horizon reward starvation, can be replaced by forward-only refinement that costs a fraction of the wall-clock time (7h23m vs 33h52m on StackCube with Behavior Transformer).
  • Refinement transfers to novel objects: on TurnFaucet and PushChair, base policies trained on source objects improved online on target objects not present in the demonstrations.
  • The refined policy keeps the smooth, multi-modal motion of imitation learning rather than the jerky motion of pure RL policies, so large improvements in success need not sacrifice motion quality.
  • The recipe is not tied to one RL algorithm or observation modality: it also works with PPO and with image observations, and on low-performing base checkpoints.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's assumptions imply a cheap rejection test: measure how far a base policy's rollouts wander from the demonstration manifold before investing in online refinement; success is only promised for policies whose failures are small local deviations.
  • A natural extension is state-dependent residual bounds: replacing the global $\alpha$ with per-state or per-action-dimension caps could improve sample efficiency on high-precision tasks without changing the method.
  • Because the base policy is never modified, the decorator framing suggests compositional refinement—several residuals trained for different objectives or constraints could be stacked or combined without retraining the base.
  • The preservation of multi-modality should degrade as $\alpha$ grows large relative to the mode separation of the base policy; the paper's $\alpha$ ablations are consistent with this prediction, and a direct test would compute the combined action distribution's modality for increasing $\alpha$.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Policy Decorator, a method for improving a frozen, offline-trained imitation-learning policy through online reinforcement learning of a small residual policy. The residual action is added to the base-policy action, bounded by a hyperparameter alpha, and introduced through a progressive schedule that interpolates between the base policy alone and the combined policy. The method is evaluated on eight ManiSkill and Adroit tasks with Behavior Transformer and Diffusion Policy base policies, against fine-tuning and non-fine-tuning baselines, with ten seeds and extensive ablations. The headline claim is that Policy Decorator improves both base policies to near-perfect success on all evaluated tasks while preserving the smooth motion of the imitation policy.

Significance. If the empirical results hold, the paper makes a practically useful contribution: large, non-differentiable imitation policies can be improved online without backpropagating through the large model, which sidesteps both architectural incompatibility and the computational cost of fine-tuning. The evaluation is unusually thorough for this area: eight tasks, two base policy families, ten seeds, several strong baselines, a comparison with DPPO, visual-observation experiments, and detailed appendices analyzing why fine-tuning and non-fine-tuning baselines fail. The paper also provides concrete hyperparameter tables and a careful validation of the Cal-QL baseline implementation in Appendix F.3. The main caveats are the per-task tuning of the two key hyperparameters and the unquantified boundary of the assumption on base-policy quality; these limit the strength of the general claim but do not undermine the core demonstration.

major comments (3)
  1. [Sec. 5.4.2, Table 8] The headline results in Figs. 6-8 are produced with per-task values of the two key hyperparameters, the residual bound alpha and the schedule length H. Table 8 lists alpha from 0.03 to 0.8 and H from 30K to 8M across tasks, so there is no cross-task fixed setting. The ablations in Figs. 10-11 show robustness only on StackCube and PushChair, and even there the robust ranges differ between the two tasks. The abstract and Sec. 5.3 state that Policy Decorator 'improves the performance of both offline-trained policies to a near-perfect level on all tasks,' which is stronger than the evidence: the results demonstrate per-task tuned success, not general success without task-specific tuning. Please add an experiment with a fixed (alpha, H) across all tasks, or explicitly qualify the claim to per-task tuned settings.
  2. [Sec. 3, Assumption 3] The method's load-bearing premise is that base-policy failures are correctable by a bounded, per-step action residual. Assumption 3 formalizes this only as the base policy exhibiting 'reasonable initial performance,' and the paper never measures where base-policy failures actually occur relative to the demonstration-covered state manifold. Appendix D.2 tests one low-performing BeT checkpoint on TurnFaucet, which is a useful start, but it does not systematically vary base-policy quality or quantify the state deviation that an alpha-bounded residual can recover. The claim 'near-perfect on all tasks' is therefore demonstrated only for the particular base policies and tasks selected, and its boundary remains uncharacterized. Please add a systematic study of base-policy quality (e.g., checkpoints of varying success rates, with analysis of failure-state distances and required correction magnitudes), and soften the general claim accordingly.
  3. [Sec. 5.5] The paper advertises the preservation of smooth, natural motion as a key advantage over pure RL policies, but the evidence is qualitative: videos and the multi-modality analysis in Appendix J. No quantitative smoothness metric is reported across tasks or seeds. Since this property is part of the paper's central contribution, please add a measurable proxy such as action jerk, action total variation, or a power-spectral measure, and compare the refined policy with both the base policy and a pure-RL baseline.
minor comments (5)
  1. [Appendix J.1] The listed parameters for the illustrative bimodal distribution, mu1 = mu2 = 0.5 and sigma1 = sigma2 = 1, make the two Gaussian components identical, so the 'bimodal' base distribution is actually unimodal. Please correct the parameter values or the description so that the example matches the claim.
  2. [Fig. 17] The label 'Resiudual RL' contains a typo and should read 'Residual RL.'
  3. [Appendix B.2, Table 8] The row 'Adroit: Door (Diffusion Policy, visual)' has no corresponding curve in Fig. 8. Please clarify whether this experiment was run and, if so, where it is reported.
  4. [Abstract and Sec. 5.3] The phrase 'both offline-trained policies to a near-perfect level on all tasks' is too broad given that for Diffusion Policy, StackCube and Door were not tested because the base policy already achieves near-optimal performance. Please state explicitly that the claim concerns the tasks on which improvement was evaluated, or distinguish 'improved' from 'maintained near-perfect.'
  5. [Sec. 5.4.3] The bullet list references Appendix D.1, D.2, and D.3 without their titles; including the titles in the bullets would improve readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's claims are empirical comparisons against measured environment success rates, not quantities derived from fitted parameters or self-citations.

full rationale

The derivation chain in Policy Decorator is an experiment, not a deduction: a frozen base policy is wrapped by a small residual policy trained online by SAC with sparse rewards, and the paper's central claims are success rates measured in the ManiSkill and Adroit environments. The tuned hyperparameters alpha and H affect learning dynamics, but they are not used to define or compute the success metric, so the near-perfect results are not forced by construction. The paper's premise that the base policy exhibits reasonable initial performance (Sec. 3, Assumption 3) is an explicitly stated scope assumption, and the bounded-residual design follows from that assumption; it is a limitation on applicability, not a circular derivation. The only self-referential elements are the authors' own ManiSkill benchmark and demonstration sets, which serve as a standard evaluation testbed and are not invoked as mathematical evidence for the method. No self-citation chain is used to justify the central mechanism, and no prediction is renamed from a fitted value. The unquantified boundary of Assumption 3 is a correctness and scope risk, not circularity.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

No new physical or methodological entities are postulated. The framework combines existing algorithmic components: residual policy, SAC, action bound, and an epsilon-style schedule. The listed free parameters are the main hand-tuned inputs; no fitted constants enter a derivation because there is no derivation.

free parameters (2)
  • alpha (residual action bound) = per-task values in Table 8, from 0.03 to 0.8
    Scales the tanh output of the residual policy. Chosen per task, usually close to the demonstration action scale; ablation shows broad ranges work but extreme values hurt. It is a free parameter that affects the final result.
  • H (progressive exploration schedule length) = per-task values in Table 8, from 30K to 8M environment steps
    Controls how quickly epsilon increases from 0 to 1. Tuned per task; too small H can cause failure, while too large H costs sample efficiency. It is a hand-chosen hyperparameter.
assumptions (4)
  • domain assumption An online environment with a success signal (sparse reward) is available for interaction.
    Sec 3 Assumption 1 states this. All experiments use simulators; real-world applicability depends on having such an environment.
  • domain assumption The base policy has reasonable initial performance and can make progress toward task completion.
    Sec 3 Assumption 3. The method needs enough success signals during early training to learn the residual.
  • ad hoc to paper Small action-level corrections are sufficient to bring deviated trajectories back into demonstration-covered states.
    Sec 1 and Fig. 3. This is the central modeling premise of residual refinement; if the base policy drifts far outside the covered manifold, bounded corrections cannot recover.
  • domain assumption SAC is a stable and sample-efficient RL algorithm for the combined policy.
    Sec 4.1 selects SAC based on these properties; the paper also shows PPO compatibility in Appendix D.3.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Policy Decorator: Model-Agnostic Online Refinement for Large Policy Model." pith.science (2026). https://pith.science/paper/RIWLAU73

@misc{pith2026241213630,
  author       = {Pith},
  title        = {Pith review of: Policy Decorator: Model-Agnostic Online Refinement for Large Policy Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RIWLAU73}},
  note         = {Machine review of arXiv:2412.13630}
}
read the original abstract

Recent advancements in robot learning have used imitation learning with large models and extensive demonstrations to develop effective policies. However, these models are often limited by the quantity, quality, and diversity of demonstrations. This paper explores improving offline-trained imitation learning models through online interactions with the environment. We introduce Policy Decorator, which uses a model-agnostic residual policy to refine large imitation learning models during online interactions. By implementing controlled exploration strategies, Policy Decorator enables stable, sample-efficient online learning. Our evaluation spans eight tasks across two benchmarks-ManiSkill and Adroit-and involves two state-of-the-art imitation learning models (Behavior Transformer and Diffusion Policy). The results show Policy Decorator effectively improves the offline-trained policies and preserves the smooth motion of imitation learning models, avoiding the erratic behaviors of pure RL policies. See our project page (https://policydecorator.github.io) for videos.

Figures

Figures reproduced from arXiv: 2412.13630 by the authors.

Figure 1
Figure 1. Policy Decorator improves base policy to near [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Our framework (Policy Decorator) improves large policy models through online inter￾actions. We learn a residual policy via RL using controlled exploration strategies (Sec. 4.2). Once learned, it functions similarly to Python decorators—wrapping the base policy with an additional function to boost performance. learning models have specific designs to accommodate the multimodal action distributions in the demonstratio… view at source ↗
Figure 3
Figure 3. Small adjustments can bring deviated trajectories back on track. While learning a residual policy through online RL (Jo￾hannink et al., 2019; Alakuijala et al., 2021; Silver et al., 2018; Zhang et al., 2019) can, in principle, refine a base policy, practical implementation is still challenging. As demonstrated in our experiments (Sec. 5), without con￾straints, random exploration during RL training often leads to fai… view at source ↗
Figures from the paper (25 more)
Figure 5
Figure 5. Figure 5: Tasks Visualizations. ManiSkill (left four figures) and Adroit (right four figures). tasks. To reflect this spirit, we bound the output of the residual policy within a certain scale. Since we use SAC as our backbone RL algorithm, the output of the policy is naturally b…
Figure 4
Figure 4. Figure 4: Progressive Explo￾ration Schedule. Inspired by the ϵ-greedy strategy used in DQN (Mnih et al., 2015), we propose to progressively introduce actions from the residual policy into the agent’s behavior policy. Specifically, the behavior policy will use actions from the re…
Figure 6
Figure 6. Figure 6: Results (with Behavior Transformer): During training, we evaluate the agent for 50 episodes every 50K environment steps. The curves depict the evaluation success rates averaged over ten seeds, and the shaded areas represent standard deviations. Our method consistently …
Figure 7
Figure 7. Figure 7: Results (with Diffusion Policy): The setup is similar to [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Results on Image Observations (with Diffusion Policy): Similar to [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 9
Figure 9. Figure 9: The importance of each component. We examined the relative importance of Policy Decorator’s main components: 1) residual pol￾icy learning; 2) progressive exploration sched￾ule; and 3) bounded residual action. We thor￾oughly evaluated all possible combinations of these …
Figure 10
Figure 10. Figure 10: Different values of the bound α for Residual Actions. 0.0 0.5 1.0 1.5 2.0 2.5 3.0 Environment Steps (millions) 0 20 40 60 80 100 Success Rate % ManiSkill: StackCube H = 0 H = 100K H = 300K H = 1M H = 3M 0 1 2 3 4 5 6 Environment Steps (millions) 0 20 40 60 80 100 Mani…
Figure 12
Figure 12. Figure 12: For the Turn Faucet and Push Chair tasks in the ManiSkill benchmark, [PITH_FULL_IMAGE:figures/full_fig_p017_12.png]
Figure 13
Figure 13. Figure 13: Policy Decorator (ours) vs. DPPO on two most challenging robotic manipulation tasks in [PITH_FULL_IMAGE:figures/full_fig_p024_13.png]
Figure 14
Figure 14. Figure 14: Similar to Fig [PITH_FULL_IMAGE:figures/full_fig_p024_14.png]
Figure 15
Figure 15. Figure 15: Policy Decorator with more base policies (MLP, BC-RNN, CNN) on TurnFaucet task. [PITH_FULL_IMAGE:figures/full_fig_p025_15.png]
Figure 16
Figure 16. Figure 16: Policy Decorator with a low-performance BeT checkpoint. [PITH_FULL_IMAGE:figures/full_fig_p025_16.png]
Figure 17
Figure 17. Figure 17: Use PPO as the backbone RL algorithm in our method, RL fine-tuning, and Residual RL. [PITH_FULL_IMAGE:figures/full_fig_p026_17.png]
Figure 18
Figure 18. Figure 18: Comparison of GAIL + MLP and Policy Decorator. [PITH_FULL_IMAGE:figures/full_fig_p026_18.png]
Figure 19
Figure 19. Figure 19: Different variants of input of residual policy. [PITH_FULL_IMAGE:figures/full_fig_p027_19.png]
Figure 20
Figure 20. Figure 20: Different variants of input of critic. 27 [PITH_FULL_IMAGE:figures/full_fig_p027_20.png]
Figure 21
Figure 21. Figure 21: Pre-training a critic by Cal-QL on demo-only datasets does not help online fine-tuning. 0.0 0.5 1.0 1.5 2.0 2.5 3.0 Environment Steps (millions) 0 20 40 60 80 100 Success Rate % Adroit: Pen A: MLP + Mixed Dataset + online Cal-QL (Cal-QL's original setting) B: MLP + De…
Figure 23
Figure 23. Figure 23: Fine-tuning Behavior Transformer using SAC with different effective task horizons. [PITH_FULL_IMAGE:figures/full_fig_p030_23.png]
Figure 24
Figure 24. Figure 24: Fine-tuning with different critic architectures. [PITH_FULL_IMAGE:figures/full_fig_p031_24.png]
Figure 25
Figure 25. Figure 25: Critic warm start results in alpha (the learnable entropy coefficient in SAC) and critic loss [PITH_FULL_IMAGE:figures/full_fig_p031_25.png]
Figure 26
Figure 26. Figure 26: Entropy coefficient and critic loss. We fixed the entropy coefficient alpha during the warm-start phase (0.2M steps) and unfreeze it during fine-tuning. We merge six independent runs into two groups: three of them blow up while the other three remain stable. 31 [PITH…
Figure 27
Figure 27. Figure 27: Fine-tuning with warm-starting and fixed alpha. [PITH_FULL_IMAGE:figures/full_fig_p032_27.png]
Figure 28
Figure 28. Figure 28: Running time comparison of forward and backward passes of the Behavior Transformer [PITH_FULL_IMAGE:figures/full_fig_p034_28.png]
Figure 29
Figure 29. Figure 29: Illustrative Example. Adding a Gaussian distribution to a multi-modal distribution typically maintains its multi-modal property. As demonstrated in [PITH_FULL_IMAGE:figures/full_fig_p035_29.png]
Figure 30
Figure 30. Figure 30: Real Case Study from Our Experiments. Applying a small residual action to correct a multi-modal base policy typically matains its multi-modal property. We can see that the base policy exhibits a clear bimodal distribution. When combined with the residual policy, the s…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. VINE: Taming Generative Control Policies for Reinforcement Learning

    cs.RO 2026-07 conditional novelty 7.0 of 10

    Reconstructing a fresh noisy interpolation state at every denoising step stabilizes end-to-end value-gradient training of multi-step flow-matching policies and yields state-of-the-art offline and real-robot results.

  2. From Prior to Pro: Efficient Skill Mastery via Distribution Contractive RL Finetuning

    cs.RO 2026-03 accept novelty 6.0 of 10

    Residual off-policy RL with selective BC regularization and value-guided sampling contracts a pretrained generative robot policy around successful actions, reaching high success on hard long-horizon tasks from pixels ...

  3. LodeStar: Long-horizon Dexterity via Synthetic Data Augmentation from Human Demonstrations

    cs.RO 2025-08 conditional novelty 6.0 of 10

    LodeStar combines automatic skill segmentation with simulation-based reinforcement learning augmentation and a learned routing transformer to let a robotic hand complete long-horizon dexterous tasks from a few human demos.

  4. Touch begins where vision ends: Generalizable policies for contact-rich manipulation

    cs.RO 2025-06 conditional novelty 6.0 of 10

    A localize-then-execute policy that combines vision-language reaching, semantic background augmentation, and residual reinforcement learning with tactile sensing reaches about 90% success on millimeter-precision manip...

  5. Efficient Online RL Fine Tuning with Offline Pre-trained Policy Only

    cs.LG 2025-05 conditional novelty 6.0 of 10

    PORL fine-tunes a pre-trained policy online by first collecting epsilon-greedy interaction data and learning a fresh Q-function from scratch, removing the need for pre-trained critics or offline data.

  6. SIME: Enhancing Policy Self-Improvement with Modal-level Exploration

    cs.RO 2025-05 conditional novelty 6.0 of 10

    Injecting annealed noise into the observation-encoder latent of a diffusion policy during inference increases rollout diversity and, combined with success- and value-based data selection, improves imitation-learned ro...

Reference graph

Works this paper leans on

14 extracted references · 13 canonical work pages · cited by 6 Pith papers

  1. [1]

    Such a noisy gradient can easily cause the policy to deviate significantly from the initial weights

    Random Critic Initialization: We believe the randomly initialized critic network cannot provide meaningful gradients to guide the policy. Such a noisy gradient can easily cause the policy to deviate significantly from the initial weights. Once the unlearning happens, it becomes very hard to relearn the policy since it cannot get the sparse reward signal a...

    work page 2024

  2. [2]

    As the task horizon increases, the agent’s likelihood of discovering sparse rewards through random exploration diminishes

    Long Task Horizon: Long task horizon also significantly increases the difficulty of fine-tuning, particularly in sparse reward settings. As the task horizon increases, the agent’s likelihood of discovering sparse rewards through random exploration diminishes. Additionally, the sparse reward signal requires more time to propagate through longer trajectorie...

    work page 2024

  3. [3]

    Online Algorithm: The original Cal-QL paper uses Cal-QL algorithm in both offline and online stage. However, computing critic loss in Cal-QL algorithm requires querying the actor 20 times in each update, which is extremely time-consuming given that the actor is a large model in our settings. Therefore, we use SAC in the online phase instead of Cal-QL. To ...

    work page

  4. [4]

    Our Cal-QL baseline uses only 25 human demonstrations, ensuring fair comparison with other learning-from-demo baselines that only utilize demonstrations

    Offline Dataset: The original Cal-QL paper uses an offline dataset consisting of 25 human teleoperation demonstrations and additional trajectories from a BC policy. Our Cal-QL baseline uses only 25 human demonstrations, ensuring fair comparison with other learning-from-demo baselines that only utilize demonstrations. We also made this assumption in Sec. 3

    work page

  5. [5]

    Actor Architecture: The original Cal-QL paper employs a small MLP as the actor, while we use a pre-trained Behavior Transformer (BeT) to align with our goal of improving the pre-trained base policy

    work page

  6. [7]

    Use a randomly initialized MLP

    work page

  7. [8]

    Use a randomly initialized GPT

    work page

  8. [9]

    24, we experimented with all the aforementioned Q-function architectures in SAC fine-tuning experiments

    Use the pre-trained GPT backbone from the actor, and add a randomly initialized value head As shown in Fig. 24, we experimented with all the aforementioned Q-function architectures in SAC fine-tuning experiments. The results indicate that fine-tuning with an MLP Q-function slightly improves the base policy, whereas fine-tuning with the other two Q-functio...

    work page

Show all 14 references
  1. [10]

    These deviations prevent the agent from receiving success signals necessary for guiding learning (see this video for an example)

    During early training stages, random residual actions cause significant deviations from the base policy’s trajectory. These deviations prevent the agent from receiving success signals necessary for guiding learning (see this video for an example)

  2. [11]

    two-layer

    The residual policy is designed to make minor corrections to the base policy. However, without explicit constraints, the magnitude of residual actions often exceeds that of the base policy actions, destroying the base policy’s behavior. Our ablation study (Fig. 10 and 11) supp...

    2024

  3. [13]

    • The PDF of the Gaussian distribution (orange): fGaussian(x) = N (x; µ3, σ2 3)

    + w2 · N(x; µ2, σ2 2), 35 where N represents the Gaussian distribution. • The PDF of the Gaussian distribution (orange): fGaussian(x) = N (x; µ3, σ2 3). • The PDF of the sum of the two distributions (green) can be computed analytically: fsum(x) = w1 · N(x; µ4, σ2

  4. [14]

    • The parameters used in the plot are: w1 = 0.5, w 2 = 0.5, µ 1 = 0.5, µ 2 = 0.5, µ 3 = 3, σ 1 = 1, σ 2 = 1, σ 3 = 1

    + w2 · N(x; µ5, σ2 5), where: µ4 = µ1 + µ3, σ 4 = q σ2 1 + σ2 3, µ 5 = µ2 + µ3, σ 5 = q σ2 2 + σ2 3. • The parameters used in the plot are: w1 = 0.5, w 2 = 0.5, µ 1 = 0.5, µ 2 = 0.5, µ 3 = 3, σ 1 = 1, σ 2 = 1, σ 3 = 1. J.2 R EAL CASE STUDY FROM OUR EXPERIMENTS To demonstrate t...

    2018

  5. [2022]

    However, its online performance is poor, as reported by Ren et al

    represents a basic version of these methods, primarily used in offline RL settings. However, its online performance is poor, as reported by Ren et al. (2024). Consistency AC (Ding & Jin, 2023) distills diffusion models into consistency models, significantly shortening the grad...

    2024

  6. [2066]

    calibrates

    PMLR, 2015. Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shu- ran Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.04137, 2023. Open X-Embodiment Collaboration. Open X-Embodiment: Roboti...

    2023 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.