Pith. sign in

REVIEW 4 major objections 6 minor 53 references

TCPO: Thought-Centric Preference Optimization for Effective Embodied Decision-making

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Training the chain of thought lifts embodied AI to 26.7% success

desk verdict A plausible incremental DPO variant for embodied VLMs with a real but modest ALFWorld gain; the theoretical justification for the weighted loss is circular, yet the empirical claim deserves replication and the paper merits review with revisions. read the letter →

arxiv 2509.08500 v1 pith:7ITAMAUP submitted 2025-09-10 cs.AI

Kechen Jiao , Zhirui Fang , Jiahao Liu , Bei Li , Qifan Wang , Xinyu Liu , Junhao Ruan , Zhongjian Qiao
show 4 more authors
Yifan Zhu Yaxin Xu Jingang Wang Xiu Li
This is my paper · ORCID
classification cs.AI
keywords Thought-CentricPreferenceOptimizationembodieddecision-makingchain-of-thoughtreasoninglearningvision-languagemodelsactionpolicyconsistencyALFWorldmodeldegradation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that aligning an embodied vision-language agent’s chain-of-thought reasoning, rather than only its final actions, yields better and more stable decision-making in dynamic environments. It proposes TCPO, which converts sparse rewards into stepwise preference pairs and adds a consistency constraint that keeps the action distribution close to the reference model. On ALFWorld, TCPO achieves a 26.7% average success rate versus 20.0% for the RL4VLM baseline, a 6-point improvement, while also mitigating the model degradation that online fine-tuning typically causes. If correct, this means preference-based stepwise optimization over reasoning traces is a more sample-efficient and stable alternative to PPO-style reinforcement learning for embodied agents.

What carries the argument

The central object is the TCPO loss in Eq. 8, $\tilde{L} + \kappa \cdot L_2(\pi_\theta(a_1|T_1), \pi_{\mathrm{ref}}(a_1|T_1))$, where $\tilde{L}$ is the action-probability-weighted stepwise preference loss of Eq. 6. APW reweights the log-probability of each reasoning step by $p(a_t|T_t)$, the conditional probability of the action given the thought, suppressing highly random positive samples and promoting deterministic thought-aligned actions. APC adds an L2 penalty pulling the fine-tuned action distribution toward the reference model’s thought-to-action mapping. Preference pairs are built from sparse-reward trajectories by turning success or failure into contrastive stepwise rankings, which supplies dense supervision and higher sample efficiency.

What would settle it

Train the same model with the unweighted loss of Eq. 4 and the weighted loss of Eq. 6 in a setting where initial action probabilities are spread out, such as a randomly initialized policy or an unfamiliar task, and record $p(a_i|T_i)$ during the first 500 training steps; if a substantial fraction of chosen actions have $p < 0.9$, the approximation underpinning APW is violated and the weighted loss should diverge from the intended DPO objective.

Watch

Extended reading notes

Core claim

The central claim is that optimizing a stepwise DPO-style preference loss on thought-action pairs, reweighted by the probability that the action follows the thought (APW), and regularizing the action distribution against a reference model (APC), improves embodied decision-making and prevents degradation during online fine-tuning. The paper reports an average success rate of 26.67% on ALFWorld, compared with 20.0% for RL4VLM and 22.1% for D3PO, and shows that jointly optimizing APW and APC outperforms either component alone. The claim is that thought-centric alignment, not action-only optimization, is what drives both higher task success and more deterministic, legal action generation.

Load-bearing premise

The method relies on the assumption that the probability of the action given the thought is close to 1 during optimization, and the only evidence offered is the action-token distribution produced by the method itself; if that approximation fails in a new environment, the weighted loss changes the objective and the reported gains may not transfer.

Editorial extensions

If this is right

  • TCPO achieves higher average task success than PPO and DPO baselines in both ALFWorld and GymCards, suggesting that thought-centric preference optimization generalizes beyond a single benchmark.
  • The action probability weighting pushes decision action token probabilities toward 1, making the policy more deterministic and reducing the incidence of illegal actions during embodied interaction.
  • The APC regularization with $\kappa \approx 0.1$ stabilizes online fine-tuning and mitigates model degradation; both too little and too much constraint reduce performance.
  • TCPO reaches the same success-rate thresholds in fewer training steps than PPO or DPO, indicating a practical sample-efficiency gain in sparse-reward interactive environments.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The weighting scheme could transfer to any hybrid reasoning-and-acting model, not only VLMs, wherever a conditional probability $p(\text{action}|\text{thought})$ is computable, making it a general regularizer for chain-of-thought policies.
  • The approximation $p(a|T) \approx 1$ may be partially self-fulfilling: APW itself drives action probabilities toward 1, so the evidence in Figure 5c is generated by the loss it is meant to justify; a clean test would measure the approximation error under an unweighted baseline at the same training steps.
  • Because the paper notes the Markovian assumption, extending TCPO with explicit memory or temporal modeling could retain its stability gains in long-horizon, non-Markovian tasks.
  • The APC term could serve as a deployment safety mechanism, keeping an online-adapted agent’s actions close to a vetted reference policy in real-world settings where distribution shift is costly.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Thought-Centric Preference Optimization (TCPO), an online preference-based fine-tuning method for vision-language model (VLM) embodied agents. TCPO constructs stepwise preference pairs from interaction trajectories, derives a DPO-style loss (Eq. 4) over thought-action pairs, introduces an Action Probability Weighting (APW) loss (Eq. 6) intended to approximate it, and adds an Action Policy Consistency (APC) term that applies L2 regularization between the fine-tuning model's action distribution and a reference model's action distribution (Eq. 8). Experiments on GymCards and ALFWorld report average success rates of 42.9% and 26.7%, respectively, compared with 42.7% and 20.0% for the RL4VLM baseline. The paper also includes ablations over the APC weight κ, a comparison of APW with classic DPO, a joint-optimization study, and a sample-efficiency table.

Significance. If established, the paper's central claim—that stepwise preference optimization of chain-of-thought reasoning, combined with an action-consistency regularizer, improves embodied decision-making and mitigates model degradation—would be a useful contribution to VLM-based embodied agents. The manuscript provides algorithmic pseudocode, a parameter study, and multiple ablations, which are strengths. However, the theoretical justification of the APW loss as a faithful approximation of the stepwise DPO objective is not sound as presented: the gradient derivation omits model-dependent terms, and the only empirical support for the key approximation is measured after training with the very loss that induces the approximation. The experimental reporting also needs improvement, as the main results table lacks variance information and contains inconsistencies with the baseline comparison table. The significance of the empirical result is moderate, but the manuscript's theoretical framing is substantially overclaimed relative to the evidence provided.

major comments (4)
  1. [Section 3.2, Eq. 5] The gradient in Eq. 5 is derived as if the action probabilities p(a_t^i|T_t^i) were independent of the policy parameters θ, but these probabilities are outputs of the same model π_θ. The gradient of Eq. 4 with respect to θ should therefore contain terms of the form ∇_θ log p(a_t^i|T_t^i) as well as gradients of the sigmoid argument through the thought probabilities. These terms are omitted without justification, so Eq. 5 does not establish the claimed 'elimination of direct action probability influences.' If p(a|T) is intended to be held constant under a stop-gradient, that must be stated explicitly, and the resulting bias in the update should be analyzed. This issue is load-bearing because it underlies the derivation of the APW loss in Eq. 6.
  2. [Section 3.2, Eq. 7 and Figure 5c] The validation of the key approximation p(a|T)≈1 is circular. The only empirical evidence, Figure 5c, plots the action-token probability distribution after training with TCPO itself; since the APW loss reweights the objective toward high-probability actions, the loss actively compresses the distribution toward 1, so the post-training measurement cannot independently confirm that p(a|T) was close to 1 during the initial or early optimization phases when the approximation was first applied. The authors should measure p(a|T) on the initial replay buffer and at early training steps, before the loss has substantially shaped the distribution, and compare those values with the assumption.
  3. [Section 3.2, Eq. 7 and Section 3.1] The approximation error Δ in Eq. 7 is not bounded. The second term, (1−p(a|T))·log(π_θ(T|τ)/π_ref(a,T|τ)), contains a log-ratio that can be large during online fine-tuning when the current policy diverges from the reference model. Even for p(a|T)=0.99, a large log-ratio can make the error non-negligible. This is particularly concerning because the replay buffer deliberately includes low-reward trajectories (Section 3.1), for which p(a|T) is most likely to be far from 1. Without a quantitative bound that accounts for the log-ratio, Eq. 6 cannot be considered a reliable surrogate for Eq. 4.
  4. [Section 4.1, Tables 1 and 2] The empirical comparison is not fully reported. Table 1 lists only point estimates, although the text and Appendix F state that results across at least 5 seeds with mean and variance were computed; without error bars or confidence intervals, the claimed 6% improvement over RL4VLM cannot be statistically assessed. Additionally, the GymCards result for the PPO/RL4VLM baseline is inconsistent: Table 1 reports 42.7% for RL4VLM, while Table 2 reports 32.8% for PPO. Since the text identifies PPO as the RL4VLM baseline, this apparent contradiction must be resolved by clarifying the experimental conditions or correcting the table.
minor comments (6)
  1. [Table 1] Several cells in Table 1 are concatenated without separators (e.g., '28.60', '40.370.0', '28.65.9'), making the table difficult to read and interpret correctly.
  2. [Abstract and Section 1] The phrase 'achieving a 6% improvement' is ambiguous: the reported difference between 26.7% and 20.0% is 6.7 percentage points, or about 33% relative improvement. The paper should state which quantity is meant.
  3. [Section 3.2, Eq. 4] The sentence 'where we have Λ instead of σ(Q̂_θ(...) − Q̂_θ(...))' does not define Λ. Please give an explicit definition (e.g., Λ = 1 − σ(...) or similar) so that Eq. 5 can be checked.
  4. [Section 4.3 and Figure 5c] Figure 5c shows a density over action-token probabilities, yet the text says it 'quantitatively characterizes the action probability evolution during the initial 2000 training iterations.' A static density does not show evolution; either clarify how aggregation is done or provide a time-resolved plot.
  5. [Appendix D, Algorithm 1] The phrase 'past trajectories wises buffer' appears to contain a typo; it should likely be 'replay buffer' or 'wise buffer.'
  6. [Section 4.1, Eq. 9] The notation ⊮{invalid} is nonstandard; it should be written as an indicator function, e.g., 1[action not admissible], for clarity.

Circularity Check

1 steps flagged · score 4.0 of 10

The p(a|T)≈1 premise in Eq. 7 is validated only by action-token probabilities produced by training with Eq. 6, making the APW approximation self-confirming; the benchmark success-rate claim remains an external result.

  1. self definitional [See Section 3.2, Eq. 7, Figure 5c, and Appendix F.]
    "Assuming that the pre-trained model has achieved good alignment, so p(a|T) will be close to 1. We have the following: Δ(p(ati|Tti)) = log p(ati|Tti)πθ(Tti|τt−1 i)/πref(ati,Tti|τt−1 i) − p(ati|Tti) log πθ(Tti|τt−1 i)/πref(ati,Tti|τt−1 i) = log p(ati|Tti) + (1−p(ati|Tti)) log πθ(Tti|τt−1 i)/πref(ati,Tti|τt−1 i) (7) This variable will approach zero as p(a|T) approaches 1. In practice, we have calculated the approximate distribution of action probabilities and demonstrated that our assumption is well-founded, which is illustrated in Figure 5c."

    Equation 7 is used to justify replacing the stepwise DPO loss (Eq. 4) by the APW loss (Eq. 6) on the premise p(a|T)≈1 at the policies on which the loss is applied. The only empirical support offered is Figure 5c, which plots action-token probabilities after or during training with TCPO itself (Section 4.3: 'Figure 5c quantitatively characterizes the action probability evolution during the initial 2000 training iterations'; Appendix F: 'the TCPO method shows a probability distribution ... concentrated around 1'). Since Eq.

full rationale

TCPO's headline empirical improvement over RL4VLM (26.7% vs 20.0% on ALFWorld) comes from benchmark evaluations, not from the APW derivation, so the central empirical claim is not circular. The one genuinely circular step is the validation of the APW approximation: Section 3.2 assumes p(a|T)≈1 and cites Figure 5c, but Figure 5c reports action-token probabilities after training with TCPO/APW, a distribution the loss itself compresses toward 1. This does not establish the premise at initialization or during early optimization, when the replay buffer deliberately contains low-reward trajectories. The derivation also leans on a Bradley-Terry/Q-value formula cited from Yang et al. (2024), a prior work sharing an author; however, that formula is a standard, externally checkable identity (σ(Q1−Q2)=exp(Q1)/(exp Q1+exp Q2)) and is not the target result, so it does not add circularity. Overall, the empirical success-rate claims are self-contained and independently testable, while the theoretical justification of Eq. 6 is partially circular; hence a score of 4.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central claim rests on several weakly justified assumptions: the p(a|T)≈1 approximation, the design of the preference scoring function, and the Markovian environment assumption. Several hyperparameters (κ, γ, success-rate weight) are chosen by hand or tuned on the evaluation benchmark. No new entities are introduced.

free parameters (6)
  • APC weight κ = 0.1
    Tuned by sweeping κ over {0.001, 0.01, 0.1, 1} on ALFWorld (Table 5); the best value is used in final results.
  • Discount factor γ for step-wise preference scores = 0.99
    Chosen as in RL; used to credit-assign trajectory success to individual steps (Appendix F).
  • Success-rate weight in preference score = 50
    Equation 9 sets P = 50 * success - invalid; the 50 weight is hand-chosen relative to the penalty.
  • Invalid action penalty = 1
    Indicator penalty for illegal actions in Eq. 9; a hand-chosen constant relative to the success weight.
  • DPO temperature β = not stated
    Appears in Eq. 2-6 but no value or schedule is reported; a free parameter affecting the optimization.
  • Start training samples = 1,000
    Number of online samples before updates (Table 4); chosen by the authors.
assumptions (5)
  • domain assumption The environment is Markovian and the model's state representation via observations and recent trajectory context suffices.
    The loss conditions on τ_{1:t-1} but the method is framed as Markovian; the authors acknowledge this in the Limitations section.
  • domain assumption The pre-trained model's conditional action distribution π_ref(a|T) is a good target for preserving consistency.
    The APC term (Eq. 8) penalizes deviation of π_θ(a|T) from π_ref(a|T), assuming the reference mapping is desirable.
  • ad hoc to paper The action probability p(a|T) is close to 1 during training, so the weighted loss Eq. 6 approximates Eq. 4.
    Stated in Section 3.2; the only support is Figure 5c, which measures the distribution after training with the loss.
  • standard math The Bradley-Terry preference model applies to the trajectory-segment comparisons.
    Adopted from Yang et al. (2024) in Appendix B to derive Eq. 13 from the Q-values.
  • domain assumption Preference scores computed from task success and invalid action counts (Eq. 9) faithfully rank trajectory segments.
    This defines the training signal; it is hand-designed and may not reflect true task progress.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TCPO: Thought-Centric Preference Optimization for Effective Embodied Decision-making." pith.science (2026). https://pith.science/paper/7ITAMAUP

@misc{pith2026250908500,
  author       = {Pith},
  title        = {Pith review of: TCPO: Thought-Centric Preference Optimization for Effective Embodied Decision-making},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7ITAMAUP}},
  note         = {Machine review of arXiv:2509.08500}
}
read the original abstract

Using effective generalization capabilities of vision language models (VLMs) in context-specific dynamic tasks for embodied artificial intelligence remains a significant challenge. Although supervised fine-tuned models can better align with the real physical world, they still exhibit sluggish responses and hallucination issues in dynamically changing environments, necessitating further alignment. Existing post-SFT methods, reliant on reinforcement learning and chain-of-thought (CoT) approaches, are constrained by sparse rewards and action-only optimization, resulting in low sample efficiency, poor consistency, and model degradation. To address these issues, this paper proposes Thought-Centric Preference Optimization (TCPO) for effective embodied decision-making. Specifically, TCPO introduces a stepwise preference-based optimization approach, transforming sparse reward signals into richer step sample pairs. It emphasizes the alignment of the model's intermediate reasoning process, mitigating the problem of model degradation. Moreover, by incorporating Action Policy Consistency Constraint (APC), it further imposes consistency constraints on the model output. Experiments in the ALFWorld environment demonstrate an average success rate of 26.67%, achieving a 6% improvement over RL4VLM and validating the effectiveness of our approach in mitigating model degradation after fine-tuning. These results highlight the potential of integrating preference-based learning techniques with CoT processes to enhance the decision-making capabilities of vision-language models in embodied agents.

Figures

Figures reproduced from arXiv: 2509.08500 by the authors.

Figure 1
Figure 1. Comparison results of our TCPO and PPO methods. In our TCPO method, we emphasize the logi￾cal consistency of actions generated by rationale and in￾corporate an Action Probability Consistency constraint (APC). In contrast, traditional PPO methods may com￾promise consistency during training, resulting in the generation of illegal actions, as shown on the right. tional capabilities in natural language understand￾ing an… view at source ↗
Figure 2
Figure 2. Overview of TCPO framework. The upper stage implements preference-driven CoT fine-tuning: The VLM [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Prompt used in ALFWorld tasks. reasoning. This environment is prioritized for its comprehensive benchmarking that better reflects real-world challenges compared to foundational GymCards tasks. Prompt Design Our chain-of-thought prompting strategy integrates three core components through natural language instructions. First, we formal￾ize ALFWorld’s semantic instructions into struc￾tured objectives, mapping paraphras… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Training curves in the ALFWorld environment. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: The experimental result of TCPO. (a) Impact of different values of [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

53 extracted references · 13 canonical work pages

  1. [1]

    Josh Abramson, Arun Ahuja, Iain Barr, Arthur Brussee, Federico Carnevale, Mary Cassin, Rachita Chhaparia, Stephen Clark, Bogdan Damoc, Andrew Dudzik, et al. 2020. Imitating interactive intelligence. arXiv preprint arXiv:2012.05672

    arXiv 2020

  2. [2]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  3. [3]

    Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. 2022. Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691

    arXiv 2022

  4. [4]

    Tom B Brown. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165

    arXiv 2020

  5. [5]

    Thomas Carta, Cl \'e ment Romac, Thomas Wolf, Sylvain Lamprier, Olivier Sigaud, and Pierre-Yves Oudeyer. 2023. Grounding large language models in interactive environments with online reinforcement learning. In International Conference on Machine Learning, pages 3676--3713. PMLR

    work page 2023

  6. [6]

    Lichang Chen, Jiuhai Chen, Chenxi Liu, John Kirchenbauer, Davit Soselia, Chen Zhu, Tom Goldstein, Tianyi Zhou, and Heng Huang. 2024. https://arxiv.org/abs/2406.07657 Optune: Efficient online preference tuning . Preprint, arXiv:2406.07657

    arXiv 2024

  7. [7]

    Maxime Chevalier-Boisvert, Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. 2018. Babyai: A platform to study the sample efficiency of grounded language learning. arXiv preprint arXiv:1810.08272

    arXiv 2018

  8. [8]

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2023. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1--113

    2023

Show all 53 references
  1. [9]

    Ishita Dasgupta, Christine Kaeser-Chen, Kenneth Marino, Arun Ahuja, Sheila Babayan, Felix Hill, and Rob Fergus. 2023. Collaborating with language models for embodied reasoning. arXiv preprint arXiv:2302.00763

    2023 arXiv

  2. [10]

    Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. 2023. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378

    2023 arXiv

  3. [11]

    Yicong Hong, Qi Wu, Yuankai Qi, Cristian Rodriguez-Opazo, and Stephen Gould. 2021. Vln bert: A recurrent vision-and-language bert for navigation. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pages 1643--1653

    2021

  4. [12]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. https://arxiv.org/abs/2106.09685 Lora: Low-rank adaptation of large language models . Preprint, arXiv:2106.09685

    2021 arXiv

  5. [13]

    Hengyuan Hu and Dorsa Sadigh. 2023. Language instructed reinforcement learning for human-ai coordination. In International Conference on Machine Learning, pages 13584--13598. PMLR

    2023

  6. [14]

    Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. 2022 a . Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International conference on machine learning, pages 9118--9147. PMLR

    2022

  7. [15]

    Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. 2022 b . Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.05608

    2022 arXiv

  8. [16]

    Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. 2022. Vima: General robot manipulation with multimodal prompts. arXiv preprint arXiv:2210.03094, 2(3):6

    2022 arXiv

  9. [17]

    Siddharth Karamcheti, Megha Srivastava, Percy Liang, and Dorsa Sadigh. 2022. Lila: Language-informed latent actions. In Conference on Robot Learning, pages 1379--1390. PMLR

    2022

  10. [18]

    Geunwoo Kim, Pierre Baldi, and Stephen McAleer. 2024 a . Language models can solve computer tasks. Advances in Neural Information Processing Systems, 36

    2024

  11. [19]

    Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. 2024 b ...

    2024 arXiv

  12. [20]

    Xin Lai, Zhuotao Tian, Yukang Chen, Senqiao Yang, Xiangru Peng, and Jiaya Jia. 2024. https://arxiv.org/abs/2406.18629 Step-dpo: Step-wise preference optimization for long-chain reasoning of llms . Preprint, arXiv:2406.18629

    2024 arXiv

  13. [21]

    Shuang Li, Xavier Puig, Chris Paxton, Yilun Du, Clinton Wang, Linxi Fan, Tao Chen, De-An Huang, Ekin Aky \"u rek, Anima Anandkumar, et al. 2022. Pre-trained language models for interactive decision-making. Advances in Neural Information Processing Systems, 35:31199--31212

    2022

  14. [22]

    Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. 2023. Code as policies: Language model programs for embodied control. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 9493--9500. IEEE

    2023

  15. [23]

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024. https://arxiv.org/abs/2310.03744 Improved baselines with visual instruction tuning . Preprint, arXiv:2310.03744

    2024 arXiv

  16. [24]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. https://arxiv.org/abs/2304.08485 Visual instruction tuning . Preprint, arXiv:2304.08485

    2023 arXiv

  17. [25]

    Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. 2024. Chameleon: Plug-and-play compositional reasoning with large language models. Advances in Neural Information Processing Systems, 36

    2024

  18. [26]

    Arjun Majumdar, Ayush Shrivastava, Stefan Lee, Peter Anderson, Devi Parikh, and Dhruv Batra. 2020. Improving vision-and-language navigation with image-text pairs from the web. In Computer Vision--ECCV 2020: 16th European Conference, Glasgow, UK, August 23--28, 2020, Proceeding...

    2020

  19. [27]

    Yao Mu, Qinglong Zhang, Mengkang Hu, Wenhai Wang, Mingyu Ding, Jun Jin, Bin Wang, Jifeng Dai, Yu Qiao, and Ping Luo. 2023. https://arxiv.org/abs/2305.15021 Embodiedgpt: Vision-language pre-training via embodied chain of thought . Preprint, arXiv:2305.15021

    2023 arXiv

  20. [28]

    Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. 2024. https://arxiv.org/abs/2402.13228 Smaug: Fixing failure modes of preference optimisation with dpo-positive . Preprint, arXiv:2402.13228

    2024 arXiv

  21. [29]

    Simone Parisi, Aravind Rajeswaran, Senthil Purushwalkam, and Abhinav Gupta. 2022. The unsurprising effectiveness of pre-trained vision models for control. In international conference on machine learning, pages 17359--17371. PMLR

    2022

  22. [30]

    Manning, and Chelsea Finn

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. 2024. https://arxiv.org/abs/2305.18290 Direct preference optimization: Your language model is secretly a reward model . Preprint, arXiv:2305.18290

    2024 arXiv

  23. [31]

    Allen Z Ren, Bharat Govil, Tsung-Yen Yang, Karthik R Narasimhan, and Anirudha Majumdar. 2023. Leveraging language for accelerated learning of tool manipulation. In Conference on Robot Learning, pages 1531--1541. PMLR

    2023

  24. [32]

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. https://arxiv.org/abs/1707.06347 Proximal policy optimization algorithms . Preprint, arXiv:1707.06347

    2017 arXiv

  25. [33]

    Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. 2024. Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face. Advances in Neural Information Processing Systems, 36

    2024

  26. [34]

    Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. https://arxiv.org/abs/2303.11366 Reflexion: Language agents with verbal reinforcement learning . Preprint, arXiv:2303.11366

    2023 arXiv

  27. [35]

    Mohit Shridhar, Lucas Manuelli, and Dieter Fox. 2022. Cliport: What and where pathways for robotic manipulation. In Conference on robot learning, pages 894--906. PMLR

    2022

  28. [36]

    Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. 2020. https://arxiv.org/abs/1912.01734 Alfred: A benchmark for interpreting grounded instructions for everyday tasks . Preprint, arXiv:1912.01734

    2020 arXiv

  29. [37]

    Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. 2023. Progprompt: Generating situated robot task plans using large language models. In 2023 IEEE International Conference on Robotics and Aut...

    2023

  30. [38]

    Sadler, Wei-Lun Chao, and Yu Su

    Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M. Sadler, Wei-Lun Chao, and Yu Su. 2023. https://arxiv.org/abs/2212.04088 Llm-planner: Few-shot grounded planning for embodied agents with large language models . Preprint, arXiv:2212.04088

    2023 arXiv

  31. [39]

    Richard S Sutton. 2018. Reinforcement learning: An introduction. A Bradford Book

    2018

  32. [40]

    Andrew Szot, Max Schwarzer, Harsh Agrawal, Bogdan Mazoure, Walter Talbott, Katherine Metcalf, Natalie Mackraz, Devon Hjelm, and Alexander Toshev. 2024. https://arxiv.org/abs/2310.17722 Large language models as generalizable policies for embodied tasks . Preprint, arXiv:2310.17722

    2024 arXiv

  33. [41]

    Weihao Tan, Wentao Zhang, Shanqi Liu, Longtao Zheng, Xinrun Wang, and Bo An. 2024. https://arxiv.org/abs/2401.14151 True knowledge comes from practice: Aligning llms with embodied environments via reinforcement learning . Preprint, arXiv:2401.14151

    2024 arXiv

  34. [42]

    Gemini Team. 2024. https://arxiv.org/abs/2312.11805 Gemini: A family of highly capable multimodal models . Preprint, arXiv:2312.11805

    2024 arXiv

  35. [43]

    Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. 2023 a . Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291

    2023 arXiv

  36. [44]

    Zihao Wang, Shaofei Cai, Guanzhou Chen, Anji Liu, Xiaojian Ma, and Yitao Liang. 2023 b . Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents. arXiv preprint arXiv:2302.01560

    2023 arXiv

  37. [45]

    Zhiyuan Xu, Kun Wu, Junjie Wen, Jinming Li, Ning Liu, Zhengping Che, and Jian Tang. 2024. https://arxiv.org/abs/2402.02385 A survey on robotics with foundation models: toward embodied ai . Preprint, arXiv:2402.02385

    2024 arXiv

  38. [46]

    Kai Yang, Jian Tao, Jiafei Lyu, Chunjiang Ge, Jiaxin Chen, Weihan Shen, Xiaolong Zhu, and Xiu Li. 2024. https://doi.org/10.1109/CVPR52733.2024.00854 Using human feedback to fine-tune diffusion models without any reward model . In IEEE/CVF Conference on Computer Vision and Patt...

    2024

  39. [47]

    Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang. 2023. https://arxiv.org/abs/2309.17421 The dawn of lmms: Preliminary explorations with gpt-4v(ision) . Preprint, arXiv:2309.17421

    2023 arXiv

  40. [49]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. https://arxiv.org/abs/2210.03629 React: Synergizing reasoning and acting in language models . Preprint, arXiv:2210.03629

    2023 arXiv

  41. [50]

    Andy Zeng, Maria Attarian, Brian Ichter, Krzysztof Choromanski, Adrian Wong, Stefan Welker, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, et al. 2022. Socratic models: Composing zero-shot multimodal reasoning with language. arXiv preprint arXiv:2204.00598

    2022 arXiv

  42. [51]

    Yuexiang Zhai, Hao Bai, Zipeng Lin, Jiayi Pan, Shengbang Tong, Yifei Zhou, Alane Suhr, Saining Xie, Yann LeCun, Yi Ma, et al. 2024. Fine-tuning large vision-language models as decision-making agents via reinforcement learning. arXiv preprint arXiv:2405.10292

    2024 arXiv

  43. [52]

    Longtao Zheng, Rundong Wang, Xinrun Wang, and Bo An. 2023. Synapse: Trajectory-as-exemplar prompting with memory for computer control. In The Twelfth International Conference on Learning Representations

    2023

  44. [53]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  45. [54]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.