Pith. sign in

REVIEW 3 major objections 6 minor 47 references

RFTF: Reinforcement Fine-tuning for Embodied Agents with Temporal Feedback

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read RFTF fine-tunes embodied agents with dense rewards from a temporal value model, setting a new CALVIN ABC-D record of 4.296 average success length.

desk verdict RFTF is a plausible dense-reward scheme for VLA fine-tuning, but the SOTA claim rests on a 0.013 Avg. Len. gap without reported variance. read the letter →

arxiv 2505.19767 v1 pith:G4WWGBLJ submitted 2025-05-26 cs.RO

Junyang Shu , Zhiwei Lin , Yongtao Wang This is my paper
classification cs.RO
keywords reinforcementfine-tuningvaluemodeldenserewardtemporalfeedbackembodiedagentsCALVINbenchmarkvision-language-actionmodelsshaping
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that reinforcement fine-tuning of vision-language-action (VLA) models for embodied manipulation is held back by sparse, outcome-only rewards, which cannot credit or blame individual actions within a long episode. To fix this, RFTF trains a value model that predicts how close the current state is to completing the instructed task, using only temporal ordering in expert demonstrations and no robot action labels. This value model supplies a dense reward for every step through potential-based reward shaping and generalized advantage estimation. In experiments on CALVIN ABC-D, fine-tuning with RFTF raises the average number of consecutively completed tasks to 4.296 for Seer-Large, the best reported result, and also improves rapid adaptation to an unseen environment.

What carries the argument

The load-bearing component is the temporally trained value model $V_{\phi}(s_t, l)$, initialized from a VLA model and trained by contrastive learning over time-ordered state pairs from expert demonstrations. The paper assumes monotonic value increase along successful demonstrations and optimizes a sigmoid contrastive loss. During fine-tuning, this value model converts sparse outcome feedback into per-step dense rewards via the reward-shaping term, amplified by GAE to propagate credit across all intermediate states; a balancing coefficient ($\eta=0.25$ on success, $1$ on failure) and an indicator term $I(\text{success})$ further shape the advantage.

What would settle it

Construct a long-horizon manipulated task where the agent must temporarily move away from the goal (e.g., pull an object out from under an obstacle before pushing it home). Train the RFTF value model on expert demos of this task and fine-tune the policy with it. If the resulting success rate is no better than—or worse than—sparse-reward fine-tuning, the monotonicity assumption fails to generalize to non-monotonic progress.

Watch

Extended reading notes

Core claim

The central claim is that a value model trained on temporal information alone—without action labels—can generate dense rewards that make reinforcement fine-tuning of embodied agents both more effective and more stable than sparse-reward fine-tuning. The value model is trained with a contrastive loss enforcing that later states in expert demonstrations receive higher values than earlier states. Once trained, it provides a reward $R_t = \gamma V(s_{t+1}, l) - V(s_t, l)$ at each non-terminal step, and this dense signal is folded into a PPO objective with GAE, a success/failure term, and positive-negative sample balancing. The paper reports that this yields state-of-the-art average success length on CALVIN ABC-D and better adaptation to a new environment, with ablations showing that replacing the dense reward with a sparse reward consistently degrades performance.

Load-bearing premise

The value model is trained on the assumption that in a successful expert demonstration the state value always increases with time; if real task progress is non-monotonic, the learned dense reward can become misleading and may hurt fine-tuning.

Editorial extensions

If this is right

  • If the result holds, dense-reward RL fine-tuning can extract meaningful learning signal from unlabeled trajectories, reducing the dependence of embodied agents on expensive action annotations.
  • The value model doubles as a progress estimator, which could be reused for other purposes such as failure detection or automated curriculum design.
  • The method's success on CALVIN ABC-D suggests that similar temporal-feedback fine-tuning may lift other VLA baselines beyond what sparse-reward RL achieves.
  • Because the value model is trained only on expert demonstrations, it could be ported across embodiments or tasks without retraining on action data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The monotonicity assumption is the fragile point: real manipulation often involves temporary regressions (e.g., reaching around obstacles, repositioning a grasped object), and the paper's own Figure 6 shows non-monotonic value curves in agent-collected episodes. The method may need a relaxation that tolerates local non-monotonicity.
  • The value model could be evaluated head-to-head against other dense-reward sources (e.g., goal-progress metrics or learned dynamics models) to isolate whether temporal ordering alone is what helps, an ablation the paper does not run.
  • Since the approach is validated only in simulation, a natural testable extension is to deploy RFTF on a real robot; the authors explicitly flag this as future work.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes RFTF, a two-stage reinforcement fine-tuning method for Vision-Language-Action (VLA) embodied agents. In the first stage, a value model is trained on expert demonstrations using a temporal contrastive objective, under the assumption that state value increases monotonically as a task progresses; the value model requires no robot action labels. In the second stage, the value model is used to provide dense reward-shaped signals, combined with PPO, GAE, a success/failure indicator, and a sample-balancing coefficient, to fine-tune VLA policies. Experiments on CALVIN ABC-D report an average success length of 4.296 for RFTF(Seer-Large), claimed as new state-of-the-art, and 4.301 in an adaptation setting after fine-tuning on environment D. An ablation compares dense rewards against sparse rewards.

Significance. If the reported improvements are statistically reliable, RFTF would be a valuable contribution to embodied RL fine-tuning: it offers a way to generate dense rewards without action labels, which is a practical advantage over sparse-reward RL fine-tuning, and it demonstrates that a temporal contrastive value model can be used for reward shaping. The paper also reports a consistent advantage of dense rewards over sparse rewards in the ablation, and it honestly acknowledges that the method is only verified in simulation. However, the central SOTA claim rests on a very small improvement over the baseline and is not supported by any measure of variance; the value-model evaluation only checks pairwise ordering on expert frames, not the quality of the resulting reward signal; and the advantage formula in Eq. (3) is asserted without derivation. These issues are load-bearing for the paper's main claims and need to be addressed before the results can be taken as established.

major comments (3)
  1. [Section 4.2.1, Tables 1-3] The central claim of new state-of-the-art performance rests on a difference of 0.013 in Avg. Len. (RFTF(Seer-Large) 4.296 vs. Seer-Large 4.283 in Table 1) and 0.018 in the adaptation table (Table 2: 4.301 vs. 4.283). The paper states that each experiment was evaluated using three different seeds, with the mean value reported as the final result, but it provides no per-seed values, standard deviations, confidence intervals, or significance tests. Given that CALVIN evaluation is based on 1000 rollouts and typical seed-to-seed variation in Avg. Len. is on the order of 0.01 to 0.05, the reported differences are not distinguishable from noise. Please report per-seed results with a statistical test or effect-size measure, and soften the SOTA claim unless the improvement is shown to be statistically significant. The same issue affects Table 3: the Seer-Large generalization gap of 0.071 (4.296 vs. 4.225) is larger and more reassuring, but it also lacks any variance information.
  2. [Section 3.2, Eq. (1)] The entire dense reward signal derives from the value model, which is trained under the assumption that in expert demonstrations the state value increases monotonically with time (vt < vt+1 < ... < vt+n-1). This assumption is load-bearing. Real manipulation episodes can contain temporary regressions, such as reaching around an obstacle or correcting a failed grasp, and if the value function cannot represent non-monotonic progress, the shaped reward may be biased. Section 4.3 and Figure 6 demonstrate that the value model does produce non-monotonic values on agent-collected rollouts, but that does not address whether the training-time monotonicity assumption distorts the reward signal for trajectories that are not monotonically progressing. Please provide evidence that the learned value function yields a useful reward for RL, for example by correlating value predictions with task progress on non-expert rollouts, or by comparing the proposed shaping reward against an alternative dense reward definition.
  3. [Section 3.3, Eq. (3)] The final advantage function in Eq. (3) is asserted without derivation. Standard GAE applied to the reward-shaped reward Rt = gamma V(st+1,l) - V(st,l) (with terminal reward 0) leads to a telescoping sum that simplifies in a specific way, and it is not obvious why the final form should include an additional I(success) term weighted by eta at every time step, nor how the values eta = 0.25 (success) and eta = 1 (failure) were chosen. Please provide a full derivation of Eq. (3), clarify exactly when and how the success indicator is added as a reward, and justify the sample-balancing coefficient. Without this, it is unclear whether the policy-gradient estimator is unbiased and whether the reported gains come from the reward-shaping term, the success indicator, or the balancing coefficient. The notation in Eq. (3) also needs to be made precise, including the summation index and range.
minor comments (6)
  1. [Section 2.1, last paragraph] The sentence that says RFTF helps the model adapt to novel environments with any action labels appears to mean without any action labels; please correct the wording.
  2. [Section 3.2, Eq. (1)] The notation in Eq. (1) is unclear: C 2n should presumably be C_n^2, the range of Delta t is given as [1, n-t) but t is not defined, and the expectation is written over (s_t,a_t) though the contrastive term involves s_{t+Delta t}. Please define all quantities and the sampling procedure explicitly.
  3. [Section 4.2.1, text after Table 1] The text says GR-MG fine-tuned by RFTF achieved a score of 4.081, surpassing the baseline of 4.043, but Table 1 reports the GR-MG baseline as 4.047. Please correct the inconsistency.
  4. [References] References 33 and 34 are the same paper (Open X-Embodiment), and references 44 and 45 are the same paper (Wu et al., Embodied task planning with large language models). Please consolidate the duplicates.
  5. [Section 4.3, Figure 5] The value model accuracy plot in Figure 5 appears to have no error bars and no explicit description of the number of validation samples or the evaluation protocol beyond pairwise ordering; please state how many pairs were used and whether the accuracy is averaged over seeds.
  6. [Section 4.1.4, implementation details] The description that the model output is discretized with 1000 bins to obtain the probability term in the PPO objective is ambiguous for continuous-action VLA models; please clarify how the discretized probability is computed and how this interacts with the continuous action outputs of the base models.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the value model is trained independently and the RL objective is not defined in terms of the reported benchmark results.

full rationale

RFTF's derivation chain is a standard reward-learning-then-RL pipeline. The value model is trained on expert demonstrations via the contrastive loss in Eq. (1) under the stated monotonicity assumption (Section 3.2), with no use of the policy being fine-tuned and no action labels. The dense reward (Eq. 2) and GAE advantage (Eq. 3) are then computed from this fixed value model during PPO fine-tuning. Because the policy's rollouts are not fed back into value-model training, the reported CALVIN ABC-D gains are not forced by construction: the reward is a learned surrogate, not a fitted copy of the evaluation metric. Hyperparameters such as eta and lambda are selected a priori and are not fit to the benchmark target. There are no load-bearing self-citations; references to prior RL fine-tuning methods are external works, and the paper does not invoke any uniqueness theorem from its own authors. The monotonic-value assumption is a modeling assumption with potential correctness risk, but it is not circularity. The paper even reports (Figure 6) that the value model assigns non-monotonic values to agent rollouts, showing the reward signal is not merely reinforcing the policy's original behavior. No quoted reduction of a prediction to an input was found.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central claim depends on the monotonicity assumption, the transfer of the value model across environments, and the comparability of baseline numbers. No new physical entities are introduced.

free parameters (6)
  • eta (sample balance coefficient) = 0.25 for success, 1 for failure
    Set in Eq. 3 to balance positive and negative samples; affects the scale of the advantage and thus policy updates.
  • lambda (GAE hyperparameter) = unspecified
    Controls bias-variance trade-off in generalized advantage estimation in Eq. 3.
  • beta (KL penalty coefficient) = unspecified
    Weights the adaptive KL divergence term in the PPO objective, Eq. 4.
  • gamma (discount factor) = unspecified
    Discount factor in the POMDP formulation and Eq. 2.
  • epsilon (PPO clip range) = unspecified
    Clipping threshold in Eq. 4 to limit policy updates.
  • Value model training epoch = 1
    The value model was selected after the first epoch to avoid overfitting (Section 4.3).
assumptions (5)
  • domain assumption In expert demonstrations, state value increases monotonically with time (vt < vt+1 < ... < vt+n-1).
    Stated in Section 3.2 and Figure 2; the entire value model training relies on this ordering.
  • standard math Potential-based reward shaping (gamma V(s') - V(s)) preserves the optimal policy.
    Invoked in Eq. 2 based on reference [31].
  • domain assumption The learned value model trained on ABC environments transfers to the D environment.
    Used in generalization experiments on CALVIN ABC-D without retraining the value model on D.
  • domain assumption Baseline numbers from prior papers are computed under the same evaluation protocol (1000 rollouts, top-3 checkpoints).
    The comparison in Tables 1 relies on numbers from prior works.
  • standard math PPO with GAE and KL penalty provides stable RL fine-tuning.
    Standard RL machinery used in Eqs. 3 and 4.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RFTF: Reinforcement Fine-tuning for Embodied Agents with Temporal Feedback." pith.science (2026). https://pith.science/paper/G4WWGBLJ

@misc{pith2026250519767,
  author       = {Pith},
  title        = {Pith review of: RFTF: Reinforcement Fine-tuning for Embodied Agents with Temporal Feedback},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/G4WWGBLJ}},
  note         = {Machine review of arXiv:2505.19767}
}
read the original abstract

Vision-Language-Action (VLA) models have demonstrated significant potential in the field of embodied intelligence, enabling agents to follow human instructions to complete complex tasks in physical environments. Existing embodied agents are often trained through behavior cloning, which requires expensive data and computational resources and is constrained by human demonstrations. To address this issue, many researchers explore the application of reinforcement fine-tuning to embodied agents. However, typical reinforcement fine-tuning methods for embodied agents usually rely on sparse, outcome-based rewards, which struggle to provide fine-grained feedback for specific actions within an episode, thus limiting the model's manipulation capabilities and generalization performance. In this paper, we propose RFTF, a novel reinforcement fine-tuning method that leverages a value model to generate dense rewards in embodied scenarios. Specifically, our value model is trained using temporal information, eliminating the need for costly robot action labels. In addition, RFTF incorporates a range of techniques, such as GAE and sample balance to enhance the effectiveness of the fine-tuning process. By addressing the sparse reward problem in reinforcement fine-tuning, our method significantly improves the performance of embodied agents, delivering superior generalization and adaptation capabilities across diverse embodied tasks. Experimental results show that embodied agents fine-tuned with RFTF achieve new state-of-the-art performance on the challenging CALVIN ABC-D with an average success length of 4.296. Moreover, RFTF enables rapid adaptation to new environments. After fine-tuning in the D environment of CALVIN for a few episodes, RFTF achieved an average success length of 4.301 in this new environment.

Figures

Figures reproduced from arXiv: 2505.19767 by the authors.

Figure 1
Figure 1. Comparison between sparse reward and dense reward. In typical reinforcement fine￾tuning methods for embodied agents, only sparse, outcome-based rewards are provided, which can confuse embodied agents when encountering partially correct or incorrect episodes. In contrast, RFTF leverages a value model trained with temporal information to predict the value of each state within an episode, providing embodied agents with… view at source ↗
Figure 2
Figure 2. Training procedure of the value model. We assume that during an episode of a human￾demonstrated successful embodied task, the state value increases monotonically over time. 3.1 Notation and preliminary Due to challenges such as limited camera coverage and occlusions between objects, we model each robot task as a Partially Observable Markov Decision Process (POMDP), defined by the tuple (S, A,P, R, O,L, γ). Here, S a… view at source ↗
Figure 3
Figure 3. Illustration of RL fine-tuning pipeline. RFTF utilizes a value model trained with temporal information to predict the value of each state in episodes of interaction between the embodied agent and the environment, thereby providing guidance for each action in episodes to fine-tune the embodied agent. function as the optimization objective: loss(ϕ) = − 1 C2 n E(st,at)∼P [log(σ(Vϕ(st+∆t, l) − Vϕ(st, l))], (1) where C 2… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Visualization of the CALVIN benchmark. The CALVIN benchmark includes four distinct environments, differing in the positions of the LED, light bulb, slider, drawer, switch, and button, as well as the material of the table. information about task completion, instead of s…
Figure 5
Figure 5. Figure 5: Train curves of the value model. We show the loss curve of the value model during training and evaluation results of the value model across different epochs. them in the D environment. As shown in [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: An example of a state value curve. As depicted, the curve exhibits a decline midway due to an incorrect grasping action by the embodied agent [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

47 extracted references · 16 canonical work pages

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    The claude 3 model family: Opus, sonnet, haiku

    AI Anthropic. The claude 3 model family: Opus, sonnet, haiku. Claude-3 Model Card, 2024

    2024

  3. [3]

    Paligemma: A versatile 3b vlm for transfer

    Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, et al. Paligemma: A versatile 3b vlm for transfer. arXiv preprint arXiv:2407.07726, 2024

    arXiv 2024

  4. [4]

    π0: A vision-language-action flow model for general robot control

    Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. π0: A vision-language-action flow model for general robot control. arXiv preprint arXiv:2410.24164, 2024

    arXiv

  5. [5]

    Do as i can, not as i say: Grounding language in robotic affordances

    Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, et al. Do as i can, not as i say: Grounding language in robotic affordances. In Conference on Robot Learning (CoRL) , 2023

    work page 2023

  6. [6]

    Agibot world colosseo: A large-scale manipulation platform for scalable and intelligent embodied systems

    Qingwen Bu, Jisong Cai, Li Chen, Xiuqi Cui, Yan Ding, Siyuan Feng, Shenyuan Gao, Xindong He, Xu Huang, Shu Jiang, et al. Agibot world colosseo: A large-scale manipulation platform for scalable and intelligent embodied systems. arXiv preprint arXiv:2503.06669, 2025

    arXiv 2025

  7. [7]

    Closed-loop visuomotor control with generative expectation for robotic manipulation

    Qingwen Bu, Jia Zeng, Li Chen, Yanchao Yang, Guyue Zhou, Junchi Yan, Ping Luo, Heming Cui, Yi Ma, and Hongyang Li. Closed-loop visuomotor control with generative expectation for robotic manipulation. arXiv preprint arXiv:2409.09016, 2024

    arXiv 2024

  8. [8]

    Pali-x: On scaling up a multilingual vision and language model

    Xi Chen, Josip Djolonga, Piotr Padlewski, Basil Mustafa, Soravit Changpinyo, Jialin Wu, Carlos Riquelme Ruiz, Sebastian Goodman, Xiao Wang, Yi Tay, et al. Pali-x: On scaling up a multilingual vision and language model. arXiv preprint arXiv:2305.18565, 2023

    arXiv 2023

Show all 47 references
  1. [9]

    Con- rft: A reinforced fine-tuning method for vla models via consistency policy

    Yuhui Chen, Shuai Tian, Shugao Liu, Yingting Zhou, Haoran Li, and Dongbin Zhao. Con- rft: A reinforced fine-tuning method for vla models via consistency policy. arXiv preprint arXiv:2502.05450, 2025

    2025 arXiv

  2. [10]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In IEEE Conference on Computer Vision and Pattern Recogni...

    2024

  3. [11]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  4. [12]

    Improving vision-language-action model with online reinforcement learning

    Yanjiang Guo, Jianke Zhang, Xiaoyu Chen, Xiang Ji, Yen-Jen Wang, Yucheng Hu, and Jianyu Chen. Improving vision-language-action model with online reinforcement learning. arXiv preprint arXiv:2501.16664, 2025

    2025 arXiv

  5. [13]

    Diffusion transformer policy

    Zhi Hou, Tianyi Zhang, Yuwen Xiong, Hengjun Pu, Chengyang Zhao, Ronglei Tong, Yu Qiao, Jifeng Dai, and Yuntao Chen. Diffusion transformer policy. arXiv preprint arXiv:2410.15959, 2024

    2024 arXiv

  6. [14]

    Flare: Achieving masterful and adaptive robot policies with large-scale reinforcement learning fine-tuning

    Jiaheng Hu, Rose Hendrix, Ali Farhadi, Aniruddha Kembhavi, Roberto Martín-Martín, Peter Stone, Kuo-Hao Zeng, and Kiana Ehsani. Flare: Achieving masterful and adaptive robot policies with large-scale reinforcement learning fine-tuning. arXiv preprint arXiv:2409.16578, 2024

    2024 arXiv

  7. [15]

    Look before you leap: Unveiling the power of gpt-4v in robotic vision-language planning

    Yingdong Hu, Fanqi Lin, Tong Zhang, Li Yi, and Yang Gao. Look before you leap: Unveiling the power of gpt-4v in robotic vision-language planning. arXiv preprint arXiv:2311.17842 , 2023

    2023 arXiv

  8. [16]

    Inner monologue: Embodied reasoning through planning with language models

    Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.05608, 2022

    2022 arXiv

  9. [17]

    Robobrain: A unified brain model for robotic manipulation from abstract to concrete

    Yuheng Ji, Huajie Tan, Jiayu Shi, Xiaoshuai Hao, Yuan Zhang, Hengyuan Zhang, Pengwei Wang, Mengdi Zhao, Yao Mu, Pengju An, et al. Robobrain: A unified brain model for robotic manipulation from abstract to concrete. arXiv preprint arXiv:2502.21257, 2025. 10

    2025 arXiv

  10. [18]

    Prismatic vlms: Investigating the design space of visually-conditioned language models

    Siddharth Karamcheti, Suraj Nair, Ashwin Balakrishna, Percy Liang, Thomas Kollar, and Dorsa Sadigh. Prismatic vlms: Investigating the design space of visually-conditioned language models. In International Conference on Machine Learning (ICML) , 2024

    2024

  11. [19]

    3d diffuser actor: Policy diffusion with 3d scene representations

    Tsung-Wei Ke, Nikolaos Gkanatsios, and Katerina Fragkiadaki. 3d diffuser actor: Policy diffusion with 3d scene representations. arXiv preprint arXiv:2402.10885, 2024

    2024 arXiv

  12. [20]

    Openvla: An open-source vision-language-action model

    Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-action model. arXiv preprint arXiv:2406.09246, 2024

    2024 arXiv

  13. [21]

    Gr-mg: Leveraging partially-annotated data via multi-modal goal-conditioned policy

    Peiyan Li, Hongtao Wu, Yan Huang, Chilam Cheang, Liang Wang, and Tao Kong. Gr-mg: Leveraging partially-annotated data via multi-modal goal-conditioned policy. IEEE Robotics and Automation Letters, 2025

    2025

  14. [22]

    Manipllm: Embodied multimodal large language model for object-centric robotic manipulation

    Xiaoqi Li, Mingxu Zhang, Yiran Geng, Haoran Geng, Yuxing Long, Yan Shen, Renrui Zhang, Jiaming Liu, and Hao Dong. Manipllm: Embodied multimodal large language model for object-centric robotic manipulation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024

    2024

  15. [23]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Neural Information Processing Systems (NeurIPS), 2023

    2023

  16. [24]

    Rdt-1b: a diffusion foundation model for bimanual manipulation

    Songming Liu, Lingxuan Wu, Bangguo Li, Hengkai Tan, Huayu Chen, Zhengyi Wang, Ke Xu, Hang Su, and Jun Zhu. Rdt-1b: a diffusion foundation model for bimanual manipulation. arXiv preprint arXiv:2410.07864, 2024

    2024 arXiv

  17. [25]

    Aligning cyber space with physical world: A comprehensive survey on embodied ai

    Yang Liu, Weixing Chen, Yongjie Bai, Xiaodan Liang, Guanbin Li, Wen Gao, and Liang Lin. Aligning cyber space with physical world: A comprehensive survey on embodied ai. arXiv preprint arXiv:2407.06886, 2024

    2024 arXiv

  18. [26]

    Exploring the limit of outcome reward for learning mathematical reasoning

    Chengqi Lyu, Songyang Gao, Yuzhe Gu, Wenwei Zhang, Jianfei Gao, Kuikun Liu, Ziyi Wang, Shuaibin Li, Qian Zhao, Haian Huang, et al. Exploring the limit of outcome reward for learning mathematical reasoning. arXiv preprint arXiv:2502.06781, 2025

    2025 arXiv

  19. [27]

    A survey on vision- language-action models for embodied ai

    Yueen Ma, Zixing Song, Yuzheng Zhuang, Jianye Hao, and Irwin King. A survey on vision- language-action models for embodied ai. arXiv preprint arXiv:2405.14093, 2024

    2024 arXiv

  20. [28]

    Robomatrix: A skill-centric hierarchical framework for scalable robot task planning and execution in open-world

    Weixin Mao, Weiheng Zhong, Zhou Jiang, Dong Fang, Zhongyue Zhang, Zihan Lan, Haosheng Li, Fan Jia, Tiancai Wang, Haoqiang Fan, et al. Robomatrix: A skill-centric hierarchical framework for scalable robot task planning and execution in open-world. arXiv preprint arXiv:2412.00171, 2024

    2024 arXiv

  21. [29]

    Genrl: Multimodal-foundation world models for generalization in embodied agents.Neural Information Processing Systems (NeurIPS), 2024

    Pietro Mazzaglia, Tim Verbelen, Bart Dhoedt, Aaron Courville, and Sai Rajeswar. Genrl: Multimodal-foundation world models for generalization in embodied agents.Neural Information Processing Systems (NeurIPS), 2024

    2024

  22. [30]

    Calvin: A benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks

    Oier Mees, Lukas Hermann, Erick Rosete-Beas, and Wolfram Burgard. Calvin: A benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks. IEEE Robotics and Automation Letters, 2022

    2022

  23. [31]

    Policy invariance under reward transfor- mations: Theory and application to reward shaping

    Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transfor- mations: Theory and application to reward shaping. In International Conference on Machine Learning (ICML), 1999

    1999

  24. [32]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Neural Information Processing Systems (NeurIPS) , 2022

    2022

  25. [33]

    Open x- embodiment: Robotic learning datasets and rt-x models: Open x-embodiment collaboration 0

    Abby O’Neill, Abdul Rehman, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, et al. Open x- embodiment: Robotic learning datasets and rt-x models: Open x-embodiment collaboration 0. In International Con...

    2024

  26. [34]

    Open x- embodiment: Robotic learning datasets and rt-x models: Open x-embodiment collaboration 0

    Abby O’Neill, Abdul Rehman, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, et al. Open x- embodiment: Robotic learning datasets and rt-x models: Open x-embodiment collaboration 0. In International Con...

    2024

  27. [35]

    Diffusion policy policy optimization

    Allen Z Ren, Justin Lidard, Lars L Ankile, Anthony Simeonov, Pulkit Agrawal, Anirudha Majumdar, Benjamin Burchfiel, Hongkai Dai, and Max Simchowitz. Diffusion policy policy optimization. arXiv preprint arXiv:2409.00588, 2024

    2024 arXiv

  28. [36]

    High- dimensional continuous control using generalized advantage estimation

    John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438, 2015

    2015 arXiv

  29. [37]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  30. [38]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  31. [39]

    Large language models as general- izable policies for embodied tasks

    Andrew Szot, Max Schwarzer, Harsh Agrawal, Bogdan Mazoure, Rin Metcalf, Walter Talbott, Natalie Mackraz, R Devon Hjelm, and Alexander T Toshev. Large language models as general- izable policies for embodied tasks. In International Conference on Learning Representations (ICLR), 2023

    2023

  32. [40]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024

    2024 arXiv

  33. [41]

    Predictive inverse dynamics models are scalable learners for robotic manipulation

    Yang Tian, Sizhe Yang, Jia Zeng, Ping Wang, Dahua Lin, Hao Dong, and Jiangmiao Pang. Predictive inverse dynamics models are scalable learners for robotic manipulation. arXiv preprint arXiv:2412.15109, 2024

    2024 arXiv

  34. [42]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  35. [43]

    Reft: Rea- soning with reinforced fine-tuning

    Luong Trung, Xinbo Zhang, Zhanming Jie, Peng Sun, Xiaoran Jin, and Hang Li. Reft: Rea- soning with reinforced fine-tuning. In Annual Meeting of the Association for Computational Linguistics (ACL), 2024

    2024

  36. [45]

    Embodied task planning with large language models

    Zhenyu Wu, Ziwei Wang, Xiuwei Xu, Jiwen Lu, and Haibin Yan. Embodied task planning with large language models. arXiv preprint arXiv:2307.01848, 2023

    2023 arXiv

  37. [46]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  38. [47]

    Fine-tuning large vision-language models as decision-making agents via reinforcement learning

    Simon Zhai, Hao Bai, Zipeng Lin, Jiayi Pan, Peter Tong, Yifei Zhou, Alane Suhr, Saining Xie, Yann LeCun, Yi Ma, et al. Fine-tuning large vision-language models as decision-making agents via reinforcement learning. Neural Information Processing Systems (NeurIPS) , 2024

    2024

  39. [48]

    Rt-2: Vision-language-action models transfer web knowledge to robotic control

    Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. In Conference on Robot Learning (CoRL) , 2023. 12

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.