Pith. sign in

REVIEW 5 major objections 4 minor 53 references

Making VLMs More Robot-Friendly: Self-Critical Distillation of Low-Level Procedural Reasoning

T0 review · 5 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Small VLMs beat models 100X larger by critiquing their own plans

desk verdict The self-refinement loop is real and the embodied results are credible, but the headline win rates lean on a GPT-4o judge whose human grounding is too thin to carry the 'beats 100X larger models' claim. read the letter →

arxiv 2507.08224 v2 pith:EBTOWITL submitted 2025-07-11 cs.RO

Chan Young Park , Jillian Fisher , Marius Memmel , Dipika Khullar , Seoho Yun , Abhishek Gupta , Yejin Choi This is my paper
classification cs.RO
keywords self-distillationvision-languagemodelsproceduralplanningself-critiqueself-refinementroboticLLM-as-judgeembodiedagents
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SelfReVision claims that small vision-language models can make their own robot-oriented plans substantially better by running them through a loop of self-critique, revision, and verification, with no teacher model or external labels. Tested on models from 3B to 72B, the loop raises plan quality enough that, on the paper's GPT-4o-based win-rate evaluation, the refined plans beat plans from GPT-4o itself, a model roughly 100 times larger. The same refined plans improve success in downstream simulated and real-world embodied tasks, suggesting the quality gain is not just textual. If true, this means lightweight models can close much of the planning gap with far larger systems using only their own iterative reasoning.

What carries the argument

The carrying mechanism is the Criticize–Revise–Verify (CRV) loop. The model generates an initial plan, writes an open-ended critique that must identify concrete execution flaws (missing steps, unrealistic assumptions, vague actions), revises the plan on the basis of that critique, and then compares the two plans in a verification step, keeping the revised plan only if it judges it better and otherwise trying again. The Verify step is what stops the loop from drifting into speculative elaboration: removing it lowers the win-rate improvement by roughly 9 percentage points. The accepted plans can be used immediately or distilled into the model weights through supervised fine-tuning, making the loop both an inference-time and a training-time device.

What would settle it

Substitute execution-based scoring for the judge: take the same 100 PLACES and 100 simulation scenes, have a human or a real robot execute the base, SelfReVision, and GPT-4o plans, and compare success rates. If SelfReVision plans no longer beat GPT-4o plans under execution scoring, the reported win rates measure judge preference rather than plan quality.

Watch

Extended reading notes

Core claim

The central claim is that a weak VLM can act as its own teacher for visual procedural planning. Given an image and a user instruction, the model writes an initial plan p0, produces a self-critique aimed at execution-relevant flaws, revises the plan in response, and then verifies which of the two plans is better, iterating until it accepts an improvement. This Criticize–Revise–Verify loop, used either at inference or to generate fine-tuning data, produces plans that beat the base plans on coverage, ordering, completeness, image groundedness, and overall quality, with an average 68% win rate on a real-world dataset and 72% on a simulation dataset. For models of 12B parameters and larger, the refined plans also beat GPT-4o's own plans in a majority of matchups, and the improvements carry into embodied execution: successful block manipulation rises by 17–26 percentage points and successful hierarchical trace generation rises from 61% to 70%.

Load-bearing premise

The load-bearing premise is that GPT-4o's plan-quality judgments reflect what actually makes a plan executable and preferable, yet the paper's own 30-sample calibration found human–model agreement of only 0.442 and human–human agreement of only 0.25 on the overall criterion.

Editorial extensions

If this is right

  • Models from 3B to 72B can raise their visual planning quality through their own critique, so a weak model is not stuck with its first-pass plan.
  • The same loop works at inference and as a fine-tuning data generator, giving deployers a choice between overhead and latency.
  • Plans improved by SelfReVision transfer to embodied control, raising simulated block manipulation success and hierarchical trace success in real-world scenes.
  • Within the tested range, larger base models benefit more, suggesting that the loop amplifies existing reasoning rather than compensating for its absence.
  • If the judge-based results hold, a 12B model with SelfReVision can match or beat a closed 100X-larger model on visually grounded procedural planning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An untested consequence is that the same CRV loop might improve other perceptual planning outputs such as navigation directions or repair guides, wherever a model can compare its own candidate outputs.
  • A direct way to stress-test the headline is to execute the refined plans in a real robot setting and compare success rates with the judge's win rates; the paper's downstream experiments are a partial version of this but cover only two model sizes and small n.
  • The paper's own limitation about flawed internal reward suggests a threshold effect: below some base capability, the critique may be too weak to produce improvements, so the method could fail for models smaller than 3B.
  • Because the judge is also the model class being compared, part of the reported advantage may come from GPT-4o rewarding the kinds of cautious, detailed language that self-critique produces; the embodied wins are the unaffected core.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper introduces SelfReVision, a self-improvement loop in which small vision-language models (3B–72B) critique, revise, and verify their own robot procedural plans without external supervision. The refined plans can be used at inference time or as self-distillation data for supervised fine-tuning. The main evidence is a set of GPT-4o-as-judge win-rate comparisons against the model's own initial plans, against GPT-4o, and against baselines such as Best-of-N and PaliGemma, across a new PLACES-based test set and a hand-modified SIMULATION test set derived from MFE-ETP. The paper additionally reports downstream embodied results on Ravens-style block manipulation and HAMSTER hierarchical plan execution, where SelfReVision plans outperform base-model plans on small sample sizes.

Significance. If the central claims hold, the paper would make a useful practical contribution: a lightweight, teacher-free inference-time refinement loop that converts weak open VLMs into more execution-oriented procedural planners, with a substantial ablation isolating the roles of criticism, revision, and verification. The paper has real strengths: it evaluates seven open models across two families, reports ablations of each pipeline stage, provides a scaling analysis of refinement rounds, releases code, and includes independent embodied evaluations that do not rely on LLM-as-judge. The embodied results are the most convincing part of the paper and justify a weaker claim that SelfReVision improves plan executability in some settings.

major comments (5)
  1. [§4.1 and Appendix B, Table 5] The central quantitative claim—that SelfReVision plans beat GPT-4o plans by large margins—rests almost entirely on GPT-4o-as-judge, and the validation in Appendix B is weaker than the main text suggests. The main text states that average human-model agreement (0.52) is 'in line with' human-human agreement, but on the headline Overall criterion the human-human Brennan-Prediger agreement is 0.250 while human-model agreement is 0.442, and all three humans agree on only 27% of the 30 samples. This means the construct 'which plan is better overall' has no stable human ground truth in the validation set; the judge cannot be distinguished from one idiosyncratic but highly self-consistent rater (model-model agreement 0.95). The 60–80% win rates, including the 'outperforms models 100X the size' claim, therefore require either a larger and more reliable human validation, an executability-based evaluation, or a substantial softening of the claims.
  2. [§3, Algorithm 1, Prompt 6, Prompt 9, and Figure 3] The self-verification step and the external judge share the same reward geometry: both reward Coverage and Completeness, which longer, more detailed text satisfies mechanically. The internal verifier terminates the loop only when the weak base model approves a revision, and Figure 3 shows that additional rounds increase Coverage and Completeness while Ordering and Image Groundedness drift downward (e.g., +11/+10 vs. -5/-3 on PLACES). This is consistent with the loop optimizing judge-preferred verbosity rather than visual grounding or executability. The paper should report whether the plans selected by the internal verifier at different rounds differ in downstream embodied success, or otherwise provide evidence that the verification signal tracks executability rather than length and detail.
  3. [§4.1 and Appendix A.2] The SIMULATION test set was constructed by hand-modifying the MFE-ETP samples: the authors selected the 'best' image when multiple images were available and wrote new tasks when no image sufficed. This manual curation is a potential source of selection bias for all SIMULATION results, and no inter-annotator reliability or audit procedure is reported for these modifications. The SIMULATION win rates should either be re-generated from an unmodified benchmark with single-image prompts, or the curation process should be documented and validated so readers can assess how representative the 100 samples are.
  4. [§4.2, Table 4] The downstream embodied experiments are the strongest external evidence, but they are based on small samples: n=112 for block manipulation and n=10 for HAMSTER trace success. The reported 70% vs. 61% HAMSTER difference comes from only 10 scenarios, and no significance tests, confidence intervals, or per-scenario breakdowns are provided. The claims in the abstract and conclusion that SelfReVision 'yield[s] improved control in downstream embodied tasks' should be qualified to reflect the limited scale of these experiments, or supplemented with additional scenarios and uncertainty quantification.
  5. [§4.1, Tables 1 and 2] The comparison against GPT-4o is not matched in inference cost or generation procedure: GPT-4o is evaluated with a few-shot prompt that may underuse its capabilities, while SelfReVision receives multiple rounds of critique and revision. The claim that small models 'outperform models 100X the size' is therefore an apples-to-oranges comparison. Reporting GPT-4o with the same SelfReVision loop, or at least with best-of-N and self-critique variants, would make the scaling claim fair and more informative.
minor comments (4)
  1. [§4.1, 'SelfReVision Implementation Details'] The hyperparameters for the loop (number of revisions per round = 2, maximum rounds = 5, temperatures) are reported as set choices, but no sensitivity analysis is shown beyond the scaling experiment in Figure 3; a short paragraph on how sensitive the main win rates are to these values would improve reproducibility.
  2. [Appendix A.5] The hardware description says '24 NVIDIA A100 GPUs with 80B memory'; this should read '80GB memory'.
  3. [§4.2 and Appendix A.3] There are several typos, including 'enviornments' in §4.2 and Appendix A.3, 'SelfReVi+SFT' for 'SelfReVision+SFT' in §4.1, and 'teh best image' in Appendix A.2; these should be corrected.
  4. [Figure 7] The caption says 'Section A.3 shows the 10 realistic task with corresponding goals', but the list of 10 tasks is not included in the appendix text; the full scenario list should be added to the appendix.

Circularity Check

0 steps flagged · score 1.0 of 10

No circular derivation: the self-revision loop's outputs are tested by an external GPT-4o judge and by embodied execution, so improvements are not forced by construction; the shared rubric between the internal verifier and the external judge is a validity concern rather than a by-construction equivalence.

full rationale

The paper's derivation chain is self-contained. Algorithm 1 defines an initial plan p0, a critique, a revision, and a verify step; the loop terminates when the base model's verifier selects the revised plan. This termination condition does not by construction imply the GPT-4o-judged improvements in Tables 1-3: the judge is a different, larger model, and the win rates are empirical outcomes that could have gone the other way. Independent support exists in Table 4 (block success 0.16 to 0.45 for Gemma-12B) and the HAMSTER trace success (70% vs 61%), which do not depend on an LLM judge. The ablation in Table 3 and the trade-offs in Figure 3 (coverage and completeness improve while image grounding declines with more rounds) likewise show the loop is not a tautology. The main non-circularity concern is evaluative self-reference: GPT-4o generates the PLACES user inputs and also serves as judge, and Prompt 6 (internal verifier) is given the same rubric (coverage, ordering, completeness, image grounding) as Prompt 9 (external judge). This could inflate win rates if the weak verifier's preferences align with GPT-4o's, but it is not a by-construction equivalence because the two models are different and their judgments could disagree. The paper itself flags the risk in the Limitations section: 'if the model's internal reward signal or critique mechanism is flawed, this could reinforce incorrect behaviors or lead to overfitting on superficial plan heuristics.' Self-citations (e.g., Brahman et al. for evaluation criteria; HAMSTER for downstream traces) are not load-bearing, as the criteria are standard and the embodied benchmarks are external. Overall, no circular derivation step is exhibited, so the score is low.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The method introduces no new entities or fitted physical constants. It relies on hyperparameters (rounds, revisions, temperature) and on domain assumptions about the self-critique capability, the validity of the LLM judge, and the faithfulness of the embodied simulators.

free parameters (5)
  • number of revisions per round = 2
    Selected via scaling experiment (Figure 3); the paper reports that most gains occur within 2-3 rounds.
  • maximum rounds = 5
    Set as stopping condition; ablations show diminishing returns beyond 5.
  • temperature for critique/revise = 0.5
    Chosen for sampling diversity; greedy decoding is used for initial planning and validation.
  • best-of-N sample count = 5
    Baseline matched to approximately the inference budget of SelfReVision.
  • SFT learning rate = best of 1e-5, 3e-5, 5e-5
    Chosen per model by dev-set loss; reported results use the best-performing one.
assumptions (4)
  • domain assumption The base VLMs can produce meaningful self-critiques and revisions of their own plans under the provided prompts.
    Invoked in Section 3 (Criticize/Revise/Verify); the entire method depends on this capability.
  • domain assumption GPT-4o-as-judge scores correlate with plan executability and human preference.
    Used as the primary evaluation metric; validation on n=30 shows only moderate agreement with humans.
  • domain assumption The code-as-policies simulator and HAMSTER action model faithfully convert plans into successful robot actions.
    Used in embodied experiments (Section 4.2) to claim downstream improvements.
  • domain assumption The hand-curated test sets (100 PLACES pairs, 100 modified MFE-ETP pairs) are representative of real procedural-planning requests.
    The PLACES user inputs are GPT-4o-generated; the SIMULATION set is hand-modified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Making VLMs More Robot-Friendly: Self-Critical Distillation of Low-Level Procedural Reasoning." pith.science (2026). https://pith.science/paper/EBTOWITL

@misc{pith2026250708224,
  author       = {Pith},
  title        = {Pith review of: Making VLMs More Robot-Friendly: Self-Critical Distillation of Low-Level Procedural Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EBTOWITL}},
  note         = {Machine review of arXiv:2507.08224}
}
read the original abstract

Large language models (LLMs) have shown promise in robotic procedural planning, yet their human-centric reasoning often omits the low-level, grounded details needed for robotic execution. Vision-language models (VLMs) offer a path toward more perceptually grounded plans, but current methods either rely on expensive, large-scale models or are constrained to narrow simulation settings. We introduce SelfReVision, a lightweight and scalable self-improvement framework for vision-language procedural planning. SelfReVision enables small VLMs to iteratively critique, revise, and verify their own plans-without external supervision or teacher models-drawing inspiration from chain-of-thought prompting and self-instruct paradigms. Through this self-distillation loop, models generate higher-quality, execution-ready plans that can be used both at inference and for continued fine-tuning. Using models varying from 3B to 72B, our results show that SelfReVision not only boosts performance over weak base VLMs but also outperforms models 100X the size, yielding improved control in downstream embodied tasks.

Figures

Figures reproduced from arXiv: 2507.08224 by the authors.

Figure 1
Figure 1. Overview of SelfReVision. VLMs tend to generate human-readable plans that are not detailed enough for robotic execution. SelfReVision employs an iterative self-critique, revision, and verification process, to transform initial plans into actionable steps. details that a robot would need to execute them. As a result, their plans often omit low-level, spa￾tially grounded details essential for execution in the physical… view at source ↗
Figure 2
Figure 2. Evaluation examples from the real-world PLACES dataset (Zhou et al., 2017) (right) and from the SIMULATION dataset, VirtualHome (Puig et al., 2018) and BEHAVIOR-100 (Srivastava et al., 2022) (left). ence time or as training data for finetuning. Using SelfReVision at inference time requires no model updates and allows fast deployment, but may incur computational overhead or complexity in orches￾tration. In contrast, … view at source ↗
Figure 3
Figure 3. Average winrate difference (winrate of Self￾ReVision - p0) over number of refinement per round. OF-N sampling and PaliGemma, SelfReVision shows clear superiority. While Best-of-N offers modest improvements for small models (8% − 38%), SelfReVision provides substantially higher gains (60% across most settings). Somewhat unexpectedly, PaliGemma—a strong pretrained VLM—consistently underperforms, losing over 90% of mat… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Examples from two embodied agent tasks: (a) block-building goals, initial setting, and then finalized setting after running P0, and SelfReVision plans. The first two rows show examples from Gemma 12B and the last row is from Gemma 27B.; (b) examples of correct addition…
Figure 5
Figure 5. Figure 5: Initial plan, feedback, and refined plan generated by Gemma-27B for the example in Figure 1. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Eight initial scenes used for the block manip [PITH_FULL_IMAGE:figures/full_fig_p018_6.png]
Figure 7
Figure 7. Figure 7: Images used in the real-world simulation experiments with corresponding goals. [PITH_FULL_IMAGE:figures/full_fig_p019_7.png]
Figure 8
Figure 8. Figure 8: The instruction given to the human annotators [PITH_FULL_IMAGE:figures/full_fig_p020_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

53 extracted references · 19 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Adilzhan Adilkhanov, Amir Yelenov, Assylkhan Seitzhanov, Ayan Mazhitov, Azamat Abdikarimov, Danissa Sandykbayeva, Daryn Kenzhebek, Daulet Baimukashev, Dinmukhammed Mukashev, Ilyas Umurbekov, Jabrail Chumakov, Kamila Spanova, Karina Burunchina, Rasul Yermagambet, Rustam Chibar, Saltanat Seitzhan, Soibkhon Khajikhanov, Tasbolat Taunyazov, Temirlan Galimzhan...

    arXiv doi:10.48550/arxiv.2502.06851 2025

  4. [4]

    Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alexander Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario M Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, and 24 others. 2022. https://api.semanticscholar.org/CorpusID:2479397...

    work page 2022

  5. [5]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, and 8 others. 2025. https://arxiv.org/abs/2502.13923 Qwen2.5-vl technical report . Preprint, arXiv:2502.13923

    arXiv 2025

  6. [6]

    Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, Thomas Unterthiner, Daniel Keysers, Skanda Koppula, Fangyu Liu, Adam Grycner, Alexey Gritsenko, Neil Houlsby, Manoj Kumar, Keran Rong, and 16 others. 2024. https://arxiv.org/abs/2407...

    arXiv 2024

  7. [7]

    PlaSma: Making Small Language Models Better Procedural Knowledge Models for (Counterfactual) Planning

    Faeze Brahman, Chandra Bhagavatula, Valentina Pyatkin, Jena D. Hwang, Xiang Lorraine Li, Hirona Jacqueline Arai, Soumya Sanyal, Keisuke Sakaguchi, Xiang Ren, and Yejin Choi. 2023. https://api.semanticscholar.org/CorpusID:258987371 Plasma: Making small language models better procedural knowledge models for (counterfactual) planning . ArXiv, abs/2305.19472

    work page Pith review arXiv 2023

  8. [8]

    Brennan and Dale J

    Robert L. Brennan and Dale J. Prediger. 1981. Coefficient kappa: Some uses, misuses, and alternatives. Educational and Psychological Measurement, 41(3):687--699

    work page 1981

Show all 53 references
  1. [9]

    Hongyi Chen, Yunchao Yao, Ruixuan Liu, Changliu Liu, and Jeffrey Ichnowski. 2024. https://doi.org/10.48550/arXiv.2409.03966 Automating robot failure recovery using vision-language models with optimized prompts

    2024 doi arXiv

  2. [10]

    An-Chieh Cheng, Yandong Ji, Zhaojing Yang, Zaitian Gongye, Xueyan Zou, Jan Kautz, Erdem Bıyık, Hongxu Yin, Sifei Liu, and Xiaolong Wang. 2025. https://arxiv.org/abs/2412.04453 Navila: Legged robot vision-language-action model for navigation . Preprint, arXiv:2412.04453

    2025 arXiv

  3. [11]

    Kanzhi Cheng, Yantao Li, Fangzhi Xu, Jianbing Zhang, Hao Zhou, and Yang Liu. 2024. https://arxiv.org/abs/2411.00855 Vision-language models can self-improve reasoning via reflection . Preprint, arXiv:2411.00855

    2024 arXiv

  4. [12]

    Yihe Deng, Pan Lu, Fan Yin, Ziniu Hu, Sheng Shen, Quanquan Gu, James Zou, Kai-Wei Chang, and Wei Wang. 2024. https://proceedings.neurips.cc/paper_files/paper/2024/file/ed45d6a03de84cc650cae0655f699356-Paper-Conference.pdf Enhancing large vision language models with self-traini...

    2024

  5. [13]

    Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint...

    2023 arXiv

  6. [14]

    Yunhao Fang, Ligeng Zhu, Yao Lu, Yan Wang, Pavlo Molchanov, Jang Cho, Marco Pavone, Song Han, and Hongxu Yin. 2024. https://doi.org/10.48550/arXiv.2407.17453 vila^2 : Vila augmented vila

    2024 doi arXiv

  7. [15]

    Zhibin Gou, Zhihong Shao, Yeyun Gong, yelong shen, Yujiu Yang, Nan Duan, and Weizhu Chen. 2024. https://openreview.net/forum?id=Sx038qxjek CRITIC : Large language models can self-correct with tool-interactive critiquing . In The Twelfth International Conference on Learning Rep...

    2024

  8. [16]

    Doucet, Orhan Firat, and Nando de Freitas

    Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alexa Ahern, Miaosen Wang, Chenjie Gu, Wolfgang Macherey, A. Doucet, Orhan Firat, and Nando de Freitas. 2023. https://api.semanticscholar.org/CorpusID:26103...

    2023 arXiv

  9. [17]

    Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.67 Large language models can self-improve . In EMNLP 2023 - 2023 Conference on Empirical Methods in Natural Language Processing, Proce...

    2023 doi

  10. [18]

    https://par.nsf.gov/biblio/10366294 Language models as zero-shot planners: Extracting actionable knowledge for embodied agents

    Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. https://par.nsf.gov/biblio/10366294 Language models as zero-shot planners: Extracting actionable knowledge for embodied agents . International Conference on Machine Learning

  11. [19]

    Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Noah Brown, Tomas Jackson, Linda Luu, Sergey Levine, Karol Hausman, and Brian Ichter. 2022. https://doi.org/10.48550/ARXIV.2...

    2022 doi arXiv

  12. [20]

    Xu Huang, Weiwen Liu, Xiaolong Chen, Xingmei Wang, Hao Wang, Defu Lian, Yasheng Wang, Ruiming Tang, and Enhong Chen. 2024. https://api.semanticscholar.org/CorpusID:267411892 Understanding the planning of llm agents: A survey . ArXiv, abs/2402.02716

    2024 arXiv

  13. [21]

    Muhammet Ilaslan, Ali Koksal, Kevin Qinghong Lin, Burak Satar, Mike Zheng Shou, and Qianli Xu. 2024. https://api.semanticscholar.org/CorpusID:274776716 Vg-tvp: Multimodal procedural planning via visually grounded text-video prompting . In AAAI Conference on Artificial Intelligence

    2024

  14. [22]

    Jaehun Jung, Peter West, Liwei Jiang, Faeze Brahman, Ximing Lu, Jillian Fisher, Taylor Sorensen, and Yejin Choi. 2024. https://doi.org/10.18653/v1/2024.naacl-long.250 Impossible distillation for paraphrasing and summarization: How to make high-quality lemonade out of small, lo...

    2024 doi

  15. [23]

    Namasivayam Kalithasan, Himanshu Gaurav Singh, Vishal Bindal, Arnav Tuli, Vishwajeet Agrawal, Rahul Jain, Parag Singla, and Rohan Paul. 2022. https://api.semanticscholar.org/CorpusID:253180551 Learning neuro-symbolic programs for language guided robot manipulation . 2023 IEEE ...

    2022

  16. [24]

    Qixiu Li, Yaobo Liang, Zeyu Wang, Lin Luo, Xi Chen, Mozheng Liao, Fangyun Wei, Yu Deng, Sicheng Xu, Yizhong Zhang, Xiaofan Wang, Bei Liu, Jianlong Fu, Jianmin Bao, Dong Chen, Yuanchun Shi, Jiaolong Yang, and Baining Guo. 2024 a . https://doi.org/10.48550/arXiv.2411.19650 Cogac...

    2024 doi arXiv

  17. [25]

    Yi Li, Yuquan Deng, Jesse Zhang, Joel Jang, Marius Memmel, Caelan Garrett, Fabio Ramos, Dieter Fox, Anqi Li, Abhishek Gupta, and Ankit Goyal. 2024 b . Hamster: Hierarchical action models for open-world robot manipulation. In CoRL 2024 Workshop on Language and Robot Learning: L...

    2024

  18. [26]

    Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. 2023 a . https://doi.org/10.1109/ICRA48891.2023.10160591 Code as policies: Language model programs for embodied control . In 2023 IEEE International Conference on Robotics ...

    2023

  19. [27]

    Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. 2023 b . Code as policies: Language model programs for embodied control. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 9493--9500. IEEE

    2023

  20. [28]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. https://arxiv.org/abs/2304.08485 Visual instruction tuning . Preprint, arXiv:2304.08485

    2023 arXiv

  21. [29]

    Yujie Lu, Pan Lu, Zhiyu Chen, Wanrong Zhu, Xin Eric Wang, and William Yang Wang. 2023. https://api.semanticscholar.org/CorpusID:258461059 Multimodal procedural planning via dual text-image prompting . ArXiv, abs/2305.01795

    2023 arXiv

  22. [30]

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. https://proceed...

    2023

  23. [31]

    Amir Moslemi, Anna Briskina, Zubeka Dang, and Jason Li. 2024. https://doi.org/10.1016/j.mlwa.2024.100605 A survey on knowledge distillation: Recent advancements . Machine Learning with Applications, 18:100605

    2024

  24. [32]

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff ...

    2024 arXiv

  25. [33]

    Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Ting Wang, Sanja Fidler, and Antonio Torralba. 2018. https://virtual-home.org Virtualhome: Simulating household activities via programs . In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Pre...

    2018

  26. [34]

    Lucy Xiaoyang Shi, Brian Ichter, Michael Equi, Liyiming Ke, Karl Pertsch, Quan Vuong, James Tanner, Anna Walling, Haohuan Wang, Niccolo Fusai, Adrian Li-Bell, Danny Driess, Lachy Groom, Sergey Levine, and Chelsea Finn. 2025. https://api.semanticscholar.org/CorpusID:276618098 H...

    2025 arXiv

  27. [35]

    Sadler, Wei-Lun Chao, and Yu Su

    Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M. Sadler, Wei-Lun Chao, and Yu Su. 2023. Llm-planner: Few-shot grounded planning for embodied agents with large language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)

    2023

  28. [36]

    Sanjana Srivastava, Chengshu Li, Michael Lingelbach, Roberto Mart\'in-Mart\'in, Fei Xia, Kent Elliott Vainio, Zheng Lian, Cem Gokmen, Shyamal Buch, Karen Liu, Silvio Savarese, Hyowon Gweon, Jiajun Wu, and Li Fei-Fei. 2022. https://proceedings.mlr.press/v164/srivastava22a.html ...

    2022

  29. [37]

    Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan. 2023. Principle-driven self-alignment of language models from scratch with minimal human supervision. In Proceedings of the 37th International Conference on Neural Inf...

    2023

  30. [38]

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex...

    2024 arXiv

  31. [39]

    Smith, Daniel Khashabi, and Hannaneh Hajishirzi

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. https://doi.org/10.18653/v1/2023.acl-long.754 Self-instruct: Aligning language models with self-generated instructions . In Proceedings of the 61st Annual Mee...

    2023 doi

  32. [40]

    Guande Wu, Huan Song, Yawei Wang, Qiaojing Yan, Yijun Tian, Lin Lee Cheong, and Panpan Xu. 2025. https://arxiv.org/abs/2503.01754 Sdrt: Enhance vision-language models by self-distillation with diverse reasoning traces . Preprint, arXiv:2503.01754

    2025 arXiv

  33. [41]

    Zhiheng Xi, Dingwen Yang, Jixuan Huang, Jiafu Tang, Guanyu Li, Yiwen Ding, Wei He, Boyang Hong, Shihan Dou, Wenyu Zhan, Xiao Wang, Rui Zheng, Tao Ji, Xiaowei Shi, Yitao Zhai, Rongxiang Weng, Jingang Wang, Xunliang Cai, Tao Gui, and 5 others. 2024. https://doi.org/10.48550/arXi...

    2024 doi arXiv

  34. [42]

    Yuxi Xie, Kenji Kawaguchi, Yiran Zhao, James Xu Zhao, Min-Yen Kan, Junxian He, and Michael Xie. 2023. https://proceedings.neurips.cc/paper_files/paper/2023/file/81fde95c4dc79188a69ce5b24d63010b-Paper-Conference.pdf Self-evaluation guided beam search for reasoning . In Advances...

    2023

  35. [43]

    Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng, Jinyang Li, Can Xu, Dacheng Tao, and Tianyi Zhou. 2024. https://api.semanticscholar.org/CorpusID:267760021 A survey on knowledge distillation of large language models . ArXiv, abs/2402.13116

    2024 arXiv

  36. [44]

    Kevin Yang, Dan Klein, Asli Celikyilmaz, Nanyun Peng, and Yuandong Tian. 2024 a . https://arxiv.org/abs/2307.12950 Rlcd: Reinforcement learning from contrastive distillation for language model alignment . Preprint, arXiv:2307.12950

    2024 arXiv

  37. [45]

    Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. 2023. https://api.semanticscholar.org/CorpusID:257637012 Mm-react: Prompting chatgpt for multimodal reasoning and action . ArXiv, abs/2303.11381

    2023 arXiv

  38. [46]

    Zhutian Yang, Caelan Garrett, Dieter Fox, Tomás Lozano-Pérez, and Leslie Pack Kaelbling. 2024 b . https://arxiv.org/abs/2410.02193 Guiding long-horizon task and motion planning with vision language models . Preprint, arXiv:2410.02193

    2024 arXiv

  39. [47]

    https://par.nsf.gov/biblio/10451467 React: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. https://par.nsf.gov/biblio/10451467 React: Synergizing reasoning and acting in language models . International Conference on Learning Representations (ICLR)

  40. [48]

    Xiao Yu, Baolin Peng, Michel Galley, Jianfeng Gao, and Zhou Yu. 2024. https://doi.org/10.18653/v1/2024.naacl-long.287 Teaching language models to self-improve through interactive demonstrations . In Proceedings of the 2024 Conference of the North American Chapter of the Associ...

    2024 doi

  41. [49]

    Andy Zeng, Pete Florence, Jonathan Tompson, Stefan Welker, Rowan Armstrong, Eric Tzeng, and Lerrel Pinto. 2020. https://transporter-net.github.io/ Transporter networks: Rearranging the visual world for robotic manipulation . In Conference on Robot Learning (CoRL). Project Webs...

    2020

  42. [50]

    Min Zhang, Jianye Hao, Xian Fu, Peilong Han, Hao Zhang, Lei Shi, Hongyao Tang, and Yan Zheng. 2024. https://doi.org/10.48550/arXiv.2407.05047 Mfe-etp: A comprehensive evaluation benchmark for multi-modal foundation models on embodied task planning

    2024 doi arXiv

  43. [51]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. In Proceedings of the 37th Internat...

    2023

  44. [52]

    Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. 2017. Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence

    2017

  45. [53]

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. https://arxiv.org/abs/2304.10592 Minigpt-4: Enhancing vision-language understanding with advanced large language models . Preprint, arXiv:2304.10592

    2023 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.