Preference VLM: Leveraging VLMs for Scalable Preference-Based Reinforcement Learning

Amit Roy-Chowdhury; Dripta S. Raychaudhuri; Jiachen Li; Konstantinos Karydis; Udita Ghosh

REVIEW 4 major objections 5 minor 15 references

Preference VLM: Leveraging VLMs for Scalable Preference-Based Reinforcement Learning

T0 review · 4 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read PrefVLM claims that pairing VLM-generated trajectory preferences with a KL-divergence filter for selective human feedback can match human-only preference-based RL with half the annotations.

desk verdict A plausible efficiency gain from combining VLM preferences with selective human feedback, but the KL filter's theoretical justification is wrong and its effectiveness is unmeasured. read the letter →

arxiv 2502.01616 v1 pith:J5535U7K submitted 2025-02-03 cs.LG

Udita Ghosh , Dripta S. Raychaudhuri , Jiachen Li , Konstantinos Karydis , Amit Roy-Chowdhury This is my paper · ORCID

classification cs.LG

keywords preference-basedreinforcementlearningvision-languagemodelsselectivehumanfeedbacknoisylabelfilteringrewardinversedynamicsroboticmanipulationMeta-World

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

PrefVLM is a framework for preference-based reinforcement learning that replaces most human preference labels with trajectory preferences generated by a vision-language model (a model trained to align images with text), reserving human annotations for the pairs the VLM and reward model are least certain about. The paper's central claim is that on five Meta-World manipulation tasks this selective combination reaches the same or better success rates as the human-only preference-based method PEBBLE while using about half the human feedback, and that adapting the VLM on one task transfers to a related task with up to four times fewer annotations. The mechanism has two parts: a CLIP-style VLM scores trajectory segments by similarity to the task description, and a KL-divergence thresholding rule splits VLM labels into clean, relabeled, and uncertain groups, with only the uncertain pairs sent to a human. A self-supervised inverse-dynamics loss keeps the adapted VLM aligned with the changing data distribution as the policy improves. The paper itself reports that VLM preferences plus filtering but no human feedback do not work, so the contribution is the synergy, not VLM-only reward learning.

What carries the argument

The machinery has three connected parts. The VLM is a CLIP-style model whose language and image encoders are topped with two small trainable layers; a trajectory segment's return is the summed cosine similarity between the task description and each frame, and the segment with higher return is preferred (Eqs. 3-5). A self-supervised inverse-dynamics loss (Eq. 6) trains a linear layer to predict the action between consecutive adapted image embeddings. The third part is a noise filter: after training the reward model, the KL divergence between the predicted preference distribution and the VLM's label is compared to two thresholds, $\tau_{lower}$ and $\tau_{upper}$; low-divergence pairs are treated as clean, high-divergence pairs are relabeled by flipping the label, and the uncertain middle band is where the human budget is spent. The load-bearing identity is the bound in Eq. (7), which claims that the clean-sample preference loss is small enough for a threshold on KL divergence to separate reliable from unreliable VLM labels.

What would settle it

Collect a held-out set of trajectory pairs with human labels, run PrefVLM's KL filter over them, and compare the fraction of correct labels in the clean set ($D_{\tau_l}$), the relabeled set ($D_{\tau_u}$), and the uncertain set sent to humans. If the clean set's label accuracy is not substantially higher than the uncertain set's, or if the flipped labels in $D_{\tau_u}$ are not more accurate than the originals, then the selection mechanism is not doing the work and the reported feedback savings would not transfer to new tasks.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that a pretrained VLM can act as a coarse preference teacher for preference-based RL, and that a small amount of targeted human feedback is enough to turn those noisy labels into a reward model that trains policies as good as those trained entirely on human preferences. Concretely, PrefVLM uses an adapted language-image representation to rank trajectory segments, threshold-based selection sends only high-uncertainty pairs to a human, and the reward model is trained on the union of VLM and human labels. On door, drawer, and window tasks in Meta-World, the paper reports that PrefVLM with 1,000 human comparisons matches PEBBLE with 2,000, and that adapting the VLM on a source task lets it match PEBBLE with 2,000 comparisons using only 500 on the target task. The paper's reported caveat is equally explicit: without the initial human feedback to adapt the VLM, filtering by itself does not produce usable policies.

Load-bearing premise

The load-bearing premise is that the KL-divergence threshold derived from clean-sample loss bounds still separates correct VLM-generated preference labels from incorrect ones once the reward model is trained on those labels; if that separation fails, the human budget is spent on effectively random pairs and the claimed 2x efficiency gain is unsupported.

Editorial extensions

If this is right

A practitioner who currently pays for 2,000 human preference comparisons per Meta-World-style task could spend 1,000 and keep the same final success rate.
Starting from a VLM adapted on one source task cuts the annotation budget further: 500 comparisons on a related task match 2,000 comparisons of human-only PEBBLE.
The adapted VLM embeddings double as a dense reward signal that tracks task progress better after fine-tuning, so the method provides a useful reward-shaping signal during training.
Human annotators no longer label random pairs; they only see pairs the model cannot resolve, so annotation effort is spent where it changes the learned reward.
Because the pipeline runs on visual observations and task-language descriptions, it extends to any manipulation task with a describable goal, without hand-designed reward functions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The claimed 2x gain is likely sensitive to the base VLM's zero-shot competence on the task family; on tasks where the VLM reward carries no signal, the KL filter has no reliable clean group to lean on and the method would degrade toward the filtering-alone failure mode the paper reports.
An immediate testable extension is using the VLM's own confidence or calibration to select human queries instead of the learned reward model's KL divergence; comparing the two selectors on the same human budget would isolate whether the gain comes from the robust-training trick or from the selection rule itself.
Since the inverse-dynamics objective is task-agnostic, the adaptation could in principle be pretrained on unlabeled interaction data from a target environment before any human feedback arrives, which the paper does not explore.
The paper reuses RIME's threshold schedule without validating the clean-loss bound for reward models trained on VLM labels, so the efficiency claim rests on that transfer being valid; a sensitivity study over $\beta_{min}$, $k$, and $\tau_{upper}$ would show how much the result depends on those hyperparameters.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's my read. PrefVLM is a sensible mash-up of things that already work: VLM trajectory preferences (from LIV-style similarity), inverse-dynamics adaptation, and RIME-style KL filtering with a human budget spent on the uncertain leftovers. The combination is new, and on five Meta-World tasks it consistently shows PrefVLM with 1000 human labels tracking PEBBLE at 2000. That's a real, useful result if it holds.

What the paper does well: the empirical setup is honest. Five seeds, standard errors, learning curves for all methods. The ablation on the inverse dynamics loss (Fig. 8) is informative: without it, performance degrades as the policy shifts, and with it, training is stable. The transfer experiments are a nice extra, even though they only cover two settings. The authors also show that VLM-only preferences fail without human feedback, which sets a fair baseline for their method.

Now the soft spots. The KL-thresholding in Sec. 4.3 has a real problem. For a one-hot label, D_KL(ỹ || P_θ) is exactly the per-sample cross-entropy loss ρ. Clean samples have small ρ and small KL. But Eq. (7) states D_KL ≥ -ln ρ + ρ/2, which for small ρ is a large positive number. That inequality is reversed; the right statement would be D_KL ≤ something like -ln ρ? Actually the correct bound for a one-hot label is D_KL = ρ, so the threshold τlower = -ln ρ + αρ makes no sense as a lower bound on clean-sample KL. The stress-test note is correct on this. The paper then uses τlower as a cutoff: samples with KL < τlower are 'clean.' Given the formula, if ρ is small, τlower is large, so almost everything would be clean, which undermines the whole selection mechanism. I suspect this is a typo or a mis-transcription from RIME, but it needs to be fixed and the filter's behavior needs empirical validation. There is no direct measurement of label accuracy on the 'clean' vs 'uncertain' sets, and the paper's own Fig. 8 shows filtering without human feedback fails. So the 2x efficiency claim rests on a mechanism that is currently unjustified.

Other issues: the baseline set is thin—only PEBBLE, no SURF or newer preference-based methods. No numeric tables or significance tests, just learning curves. There is a task-list inconsistency (text lists drawer open, figure shows door close). No code is released. None of these are fatal, but they make the paper weaker than it could be.

Who is this for? People working on preference-based RL and VLM reward models will want to read it. It deserves a serious referee; the core idea is sound and the results are suggestive. But the reviewer should push hard on the KL threshold derivation and demand either a corrected bound or an empirical demonstration that the filter selects higher-quality labels than random selection. If the authors fix that and add a couple more baselines, this becomes a solid paper. As is, it's a conditional accept at best.

Recommendation: send to peer review, but with the expectation that the theoretical section needs rewriting.

Referee Report

4 major / 5 minor

Summary. The paper proposes PrefVLM, a preference-based RL method in which a pretrained CLIP-style VLM (LIV) assigns coarse preference labels to trajectory pairs by comparing summed per-step cosine similarities (Eqs. 3-4); a KL-divergence filter (Eqs. 7-9) is used to split samples into clean, noisy, and uncertain sets, with a small human budget spent on the uncertain set; the VLM's two learned adapter layers are fine-tuned with human preferences and an inverse-dynamics loss (Eqs. 5-6). Experiments on five Meta-World tasks compare PrefVLM with 1000 human labels against PEBBLE with 1000 and 2000 labels, VLM-as-reward, VLM-pref-reward, and an ablation without selection, and report matching or better success rates with half the human feedback, plus a transfer experiment that cuts feedback by 4x.

Significance. If the claimed 2x feedback reduction holds, PrefVLM would be practically valuable for preference-based RL, combining scalable machine labels with targeted human oversight. The inverse-dynamics adaptation is a sensible, testable idea, and the paper includes pseudocode and hyperparameters. However, the current manuscript does not substantiate the filtering mechanism: the key KL bound in Eq. (7) is misstated, the VLM label orientation in Eq. (4) is inconsistent with the defined preference convention, and no quantitative or statistical validation of the central efficiency claim is provided. These issues must be resolved before the headline claim can be accepted.

major comments (4)

[Section 4.3, Eq. (7)] The stated KL bound has the wrong direction. Since the preference label is one-hot, D_KL(ỹ || P_θ(σ0, σ1)) equals -log P_θ(chosen segment), which is exactly the per-sample cross-entropy loss in Eq. (2). If clean samples have loss at most ρ, then we get D_KL ≤ ρ, not D_KL ≥ -ln ρ + ρ/2 + O(ρ²). For small ρ, -ln ρ + ρ/2 is large, so the threshold τlower = -ln ρ + αρ is not a lower bound on clean-sample KL; the set Dτl in Eq. (8) can then contain essentially all samples and the selection mechanism becomes vacuous. The paper provides no measurement of label accuracy on the 'clean' versus 'uncertain' sets, so the 2x efficiency claim is not backed by a validated filtering step.
[Section 4.1, Eq. (4)] The label assignment in Eq. (4) is reversed relative to the convention defined in Section 3.1. There, y = (1, 0) means σ0 is preferred over σ1. Eq. (4) assigns y = (0, 1) when R0 > R1, i.e., it labels the lower-return segment as preferred. If implemented as written, the VLM provides inverted preferences and the reward model is trained against the wrong ordering; if this is a typo, it must be fixed because the subsequent filtering in Eqs. (8-9) and the reward training in Eq. (2) all depend on label orientation.
[Section 5.3, Figure 4] The central claim that PrefVLM 'matches PEBBLE's performance while requiring only half the human feedback' is supported only by learning curves with shaded standard error; no final success-rate table, per-task numerical values, or significance tests are given. With five seeds and visibly overlapping shaded regions, the 'across all tasks' claim cannot be verified. Please add a table of mean ± standard error at the end of training, a paired comparison, and a precise statement of what the 1000/2000 counts include (human labels only, or also VLM-generated labels under the 30,000-sample budget in Appendix C). The baseline set also omits feedback-efficient methods such as SURF and RIME, so the state-of-the-art claim is not demonstrated.
[Section 5.4, Figure 8] The ablation results show that filtering alone (VLM-generated preferences with selection, but no human feedback) fails, and that PrefVLM without selection also performs reasonably. This leaves the contribution of the sample-selection mechanism itself unclear; the only evidence that selection helps is the qualitative gap between PrefVLM and PrefVLM w/o selection, which is not statistically tested. Please report a direct validation of the Eqs. (7-9) mechanism, for example by measuring agreement of the 'clean' machine-labeled set with held-out human labels versus agreement of the 'uncertain' set, so the reader can see that the filter is actually separating reliable from unreliable VLM labels.

minor comments (5)

[Section 5.3] The text refers to 'VLM-as-pref' but the baseline is named 'VLM-pref-reward' in Section 5.2; please unify the name.
[Algorithm 1, line 18] The expression |Dh| = min(|B| - |Dτl| - |Dτu|, 0.05 * N) uses N both for the number of newly sampled preference pairs and for the feedback batch size, and the argument may be negative if the clean and noisy sets exhaust the buffer; please clarify the units and clipping behavior.
[Section 5.1] The phrase 'reference feedback' should be 'preference feedback'.
[Figure 2] The reward curve lacks axis labels and units, making the claim of misalignment with task progress difficult to evaluate.
[Appendix C] The hyperparameters α = 0.5, βmin = 1, βmax = 3, k = 1/300, and τupper = 3 ln 10 are said to follow Cheng et al. (2024), but Eq. (7) as stated does not match the bound derived for cross-entropy loss; please provide a derivation or a precise citation of the source bound.

Circularity Check

0 steps flagged · score 2.0 of 10

No load-bearing circularity; the 2x annotation-efficiency claim is benchmarked against an external baseline and does not reduce to a fitted input. A single non-load-bearing self-citation and an unvalidated imported KL bound are noted but are not circular reductions.

full rationale

PrefVLM's central comparison is against PEBBLE, an external preference-based RL method using only human feedback, and the VLM prior is the externally pretrained LIV model, so the reported 2x human-annotation reduction is not manufactured by construction. The KL-divergence selection mechanism (Eqs. 7-9) is imported from RIME (Cheng et al. 2024) with fixed hyperparameters, not fitted to the target tasks; it could be miscalibrated or misstated, but an imported calibration error is a correctness risk, not a self-referential reduction. The paper's own Fig. 8 caption states 'filtering alone is ineffective without initial human feedback,' which undermines the mechanism's independent contribution but does not make the headline result equal to its inputs. The only self-citation is Raychaudhuri et al. (2021), cited in Sec. 4.2 only to motivate the existence of a domain gap; it is not load-bearing for the method or the efficiency claim. No fitted parameter is renamed as a prediction, and no uniqueness/force theorem is invoked. Score 2 reflects the single minor, non-load-bearing self-citation; the derivation chain otherwise is externally anchored.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The framework relies on several assumptions imported from prior work: the VLM cosine-similarity reward, the inverse dynamics objective, and the RIME sample-selection theory. These are plausible and partially supported by ablations, but they are not re-established from first principles here. No new entities are introduced; the only tunable quantities are standard hyperparameters or thresholds inherited from RIME.

free parameters (3)

alpha = 0.5
Parameter in the lower-bound threshold tau_base in Section 4.3, adopted from RIME; controls how aggressively samples are deemed clean.
beta_min, beta_max, k = 1, 3, 1/300
Parameters of the linear decay schedule for the uncertainty term tau_unc in Section 4.3, set to RIME values.
tau_upper = 3 ln(10)
Upper threshold for relabeling noisy samples in Eq. (9), inherited from RIME.

assumptions (3)

domain assumption Small-loss trick: a reward model trained on preference loss learns clean patterns first, so low KL samples are clean and the bound in Eq. (7) holds.
This is the basis for defining D_tl and D_tu in Section 4.3; it is imported from RIME and not re-derived or verified for VLM-generated preferences.
domain assumption Inverse dynamics loss in Eq. (6) aligns VLM embeddings with environment dynamics, improving the quality of preference labels.
Central to the VLM adaptation in Section 4.2; the paper provides an empirical ablation but no formal justification.
domain assumption CLIP-style cosine similarity in Eq. (3)/(5) provides a meaningful per-step reward signal for the selected tasks.
Assumed throughout Section 3.2; the paper shows the signal is noisy but does not prove it is informative enough for the method to work.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Preference VLM: Leveraging VLMs for Scalable Preference-Based Reinforcement Learning." pith.science (2026). https://pith.science/paper/J5535U7K

@misc{pith2026250201616,
  author       = {Pith},
  title        = {Pith review of: Preference VLM: Leveraging VLMs for Scalable Preference-Based Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/J5535U7K}},
  note         = {Machine review of arXiv:2502.01616}
}

read the original abstract

Preference-based reinforcement learning (RL) offers a promising approach for aligning policies with human intent but is often constrained by the high cost of human feedback. In this work, we introduce PrefVLM, a framework that integrates Vision-Language Models (VLMs) with selective human feedback to significantly reduce annotation requirements while maintaining performance. Our method leverages VLMs to generate initial preference labels, which are then filtered to identify uncertain cases for targeted human annotation. Additionally, we adapt VLMs using a self-supervised inverse dynamics loss to improve alignment with evolving policies. Experiments on Meta-World manipulation tasks demonstrate that PrefVLM achieves comparable or superior success rates to state-of-the-art methods while using up to 2 x fewer human annotations. Furthermore, we show that adapted VLMs enable efficient knowledge transfer across tasks, further minimizing feedback needs. Our results highlight the potential of combining VLMs with selective human supervision to make preference-based RL more scalable and practical.

Figures

Figures reproduced from arXiv: 2502.01616 by the authors.

**Figure 1.** PrefVLM leverages a VLM to obtain preference labels over pairs of the agent’s trajectory segments. These preference labels are then used to train a reward function. In scenarios where the VLM exhibits high uncertainty, PrefVLM can seamlessly incorporate human feedback to refine its understanding and adapt the VLM to the specific environment. By combining machine-generated and expert-guided feedback, PrefVLM learns… view at source ↗

**Figure 2.** VLM reward (Eqn. 3) for an optimal trajectory given the task description “Open a door with a revolving joint.” Although the reward reflects partial task progression, it is noisy and poorly aligned with the actual task progress, as evident from the image observations. reward rt and transitions the agent to the next state st+1. The return Rt = P∞ k=0 γ k rt+k represents the discounted sum of future rewards starting at… view at source ↗

**Figure 3.** Overview of our approach. Given a task description, PrefVLM iteratively updates the policy πϕ via reinforcement learning using the reward model rθ. Trajectory segments from the replay buffer are sampled and labeled with VLM-generated preferences. These samples are then classified as clean or noisy using thresholds τupper and τlower. A budgeted subset of noisy samples is sent for human annotation. The reward model is… view at source ↗

Figures from the paper (6 more)

**Figure 4.** Figure 4: Learning curves for all methods on the 5 Meta-World tasks. PrefVLM consistently outperforms all baselines with minimal human feedback and matches or exceeds PEBBLE’s performance while using 2× fewer annotations. Results are averaged over 5 seeds, with shaded regions …

**Figure 5.** Figure 5: Success rate as a function of human feedback. PrefVLM leverages human feedback more efficiently by complementing it with VLM-based feedback, resulting in higher success rates with fewer human annotations. Results are averaged over 5 seeds, with shaded regions represen…

**Figure 6.** Figure 6: Learning curves for knowledge transfer experiments. PrefVLM achieves comparable or superior performance to PEBBLE while requiring 4× fewer annotations, demonstrating its ability to transfer knowledge across both same task, different object (left) and same object, dif…

**Figure 8.** Figure 8: Success rates for (i) VLM adaptation without inverse dynamics loss and (ii) VLM-generated preferences with noise filtering. Results show that inverse dynamics loss stabilizes learning while filtering alone is ineffective without initial human feedback. Results are ave…

**Figure 7.** Figure 7: VLM reward on Door Open and Drawer Close before (top) and after (bottom) adaptation, averaged across the same five expert trajectories. The adapted VLM better aligns with groundtruth task progress. 5.4. Analysis Impact of VLM adaptation [PITH_FULL_IMAGE:figures/full_…

**Figure 9.** Figure 9: Example expert trajectories demonstrating the tasks we test on. C. Implementation details In our experiments, we begin by collecting initial data using a random policy for the first 1000 interactions with the environment. Following this, we conduct unsupervised explora…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

15 extracted references · 5 canonical work pages

[7]

Pebble: Feedback- efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training

Lee, K., Smith, L., and Abbeel, P. Pebble: Feedback- efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. arXiv preprint arXiv:2106.05091,

arXiv
[9]

Observations are captured from Camera 2 and rendered as 300 × 300 images

For each of the environment, the state s ∈ R39, and the action a ∈ R4. Observations are captured from Camera 2 and rendered as 300 × 300 images. Task descriptions are sourced directly from the Meta-World paper (Yu et al., 2020). The corresponding prompts for each environment are provided below: • door-open-v2 : Open a door with a revolving joint • door-cl...

work page 2020
[12]

A., Khashabi, D., and Hajishirzi, H

Wang, Y ., Kordi, Y ., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., and Hajishirzi, H. Self-instruct: Aligning language models with self-generated instructions. arXiv preprint arXiv:2212.10560,

arXiv
[13]

G., Chiang, H.-T

Yu, W., Gileadi, N., Fu, C., Kirmani, S., Lee, K.-H., Are- nas, M. G., Chiang, H.-T. L., Erez, T., Hasenclever, L., 10 Minimizing Human Feedback in Reinforcement Learning using Vision-Language Models Humplik, J., et al. Language to rewards for robotic skill synthesis. arXiv preprint arXiv:2306.08647,

arXiv
[1952]

Rime: Robust preference-based reinforcement learning with noisy preferences

Cheng, J., Xiong, G., Dai, X., Miao, Q., Lv, Y ., and Wang, F.- Y . Rime: Robust preference-based reinforcement learning with noisy preferences. arXiv preprint arXiv:2402.17257,

arXiv
[2004]

Language reward modulation for pretraining reinforcement learning

Adeniji, A., Xie, A., Sferrazza, C., Seo, Y ., James, S., and Abbeel, P. Language reward modulation for pretraining reinforcement learning. arXiv preprint arXiv:2308.12270,

arXiv
[2009]

Avid: Learning multi-stage tasks via pixel-level trans- lation of human videos

Smith, L., Dhawan, N., Zhang, M., Abbeel, P., and Levine, S. Avid: Learning multi-stage tasks via pixel-level trans- lation of human videos. arXiv preprint arXiv:1912.04443,

arXiv 1912
[2016]

FuRL: Visual-Language Models as Fuzzy Rewards for Reinforcement Learning

Fu, Y ., Zhang, H., Wu, D., Xu, W., and Boulet, B. Furl: Visual-language models as fuzzy rewards for reinforce- ment learning. arXiv preprint arXiv:2406.00645,

work page Pith review arXiv

Show all 15 references

[2018]

Hejna III, D. J. and Sadigh, D. Few-shot preference learn- ing for human-in-the-loop rl. In Conference on Robot Learning, pp. 2014–2025. PMLR,

2014
[2020]

J., Kumar, V ., Zhang, A., Bastani, O., and Jayaraman, D

9 Minimizing Human Feedback in Reinforcement Learning using Vision-Language Models Ma, Y . J., Kumar, V ., Zhang, A., Bastani, O., and Jayaraman, D. Liv: Language-image representations and rewards for robotic control. In International Conference on Machine Learning, pp. 23301–...

arXiv
[2021]

Vision-language models are zero-shot re- ward models for reinforcement learning

Rocamonde, J., Montesinos, V ., Nava, E., Perez, E., and Lindner, D. Vision-language models are zero-shot re- ward models for reinforcement learning. arXiv preprint arXiv:2310.12921,

arXiv
[2022]

Surf: Semi-supervised reward learning with data augmentation for feedback-efficient preference-based re- inforcement learning

Park, J., Seo, Y ., Shin, J., Lee, H., Abbeel, P., and Lee, K. Surf: Semi-supervised reward learning with data augmentation for feedback-efficient preference-based re- inforcement learning. arXiv preprint arXiv:2203.10050,

arXiv
[2023]

Kingma, D. P. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980,

arXiv
[2024]

L., Faust, A., Fiser, M., and Francis, A

Chiang, H.-T. L., Faust, A., Fiser, M., and Francis, A. Learn- ing navigation behaviors end-to-end with autorl. IEEE Robotics and Automation Letters, 4(2):2007–2014,

2007
[3000]

The reward model is trained with a learning rate of 0.0003, a batch size of 128, and 200 update steps per iteration

For the reward model, we use an ensemble of 3 MLPs, each with 3 hidden layers of 256 nodes and Leaky ReLU activation, while the final layer applies a tanh activation. The reward model is trained with a learning rate of 0.0003, a batch size of 128, and 200 update steps per iter...

2024

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.

[1] [7]

Pebble: Feedback- efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training

Lee, K., Smith, L., and Abbeel, P. Pebble: Feedback- efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. arXiv preprint arXiv:2106.05091,

arXiv

[2] [9]

Observations are captured from Camera 2 and rendered as 300 × 300 images

For each of the environment, the state s ∈ R39, and the action a ∈ R4. Observations are captured from Camera 2 and rendered as 300 × 300 images. Task descriptions are sourced directly from the Meta-World paper (Yu et al., 2020). The corresponding prompts for each environment are provided below: • door-open-v2 : Open a door with a revolving joint • door-cl...

work page 2020

[3] [12]

A., Khashabi, D., and Hajishirzi, H

Wang, Y ., Kordi, Y ., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., and Hajishirzi, H. Self-instruct: Aligning language models with self-generated instructions. arXiv preprint arXiv:2212.10560,

arXiv

[4] [13]

G., Chiang, H.-T

Yu, W., Gileadi, N., Fu, C., Kirmani, S., Lee, K.-H., Are- nas, M. G., Chiang, H.-T. L., Erez, T., Hasenclever, L., 10 Minimizing Human Feedback in Reinforcement Learning using Vision-Language Models Humplik, J., et al. Language to rewards for robotic skill synthesis. arXiv preprint arXiv:2306.08647,

arXiv

[5] [1952]

Rime: Robust preference-based reinforcement learning with noisy preferences

Cheng, J., Xiong, G., Dai, X., Miao, Q., Lv, Y ., and Wang, F.- Y . Rime: Robust preference-based reinforcement learning with noisy preferences. arXiv preprint arXiv:2402.17257,

arXiv

[6] [2004]

Language reward modulation for pretraining reinforcement learning

Adeniji, A., Xie, A., Sferrazza, C., Seo, Y ., James, S., and Abbeel, P. Language reward modulation for pretraining reinforcement learning. arXiv preprint arXiv:2308.12270,

arXiv

[7] [2009]

Avid: Learning multi-stage tasks via pixel-level trans- lation of human videos

Smith, L., Dhawan, N., Zhang, M., Abbeel, P., and Levine, S. Avid: Learning multi-stage tasks via pixel-level trans- lation of human videos. arXiv preprint arXiv:1912.04443,

arXiv 1912

[8] [2016]

FuRL: Visual-Language Models as Fuzzy Rewards for Reinforcement Learning

Fu, Y ., Zhang, H., Wu, D., Xu, W., and Boulet, B. Furl: Visual-language models as fuzzy rewards for reinforce- ment learning. arXiv preprint arXiv:2406.00645,

work page Pith review arXiv