Pith. sign in

REVIEW 4 major objections 6 minor 47 references

ACTLLM: Action Consistency Tuned Large Language Model

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read ACTLLM proposes that robot manipulation policies should be trained on structured language descriptions of scenes, with actions tied to described scene changes.

desk verdict A plausible architecture with genuinely new integration, but the reported gains rest on a train/test gap the paper never closes. read the letter →

arxiv 2506.21250 v1 pith:VH25WKDA submitted 2025-06-26 cs.RO

Jing Bi , Lianggong Bruce Wen , Zhang Liu , Chenliang Xu This is my paper
classification cs.RO
keywords actionconsistencylossstructuredscenedescriptionrobotmanipulationlargelanguagemodelvisualdialoguecompositionalgeneralizationin-contextimitationlearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ACTLLM tries to establish that a robot manipulation policy can be built on structured language descriptions of scenes rather than hidden visual features. The paper claims that replacing raw observations with text descriptions of objects, colors, and coordinates gives a single interface for spatial reasoning and action generation, and that a new action-consistency loss ties each predicted action to the change between two scene descriptions. It also recasts the manipulation Markov decision process as a multi-turn visual dialogue so that actions are informed by the history of described scene states. On CLIPORT and VIMA-BENCH, the reported success rates are higher than those of the baselines the paper compares against, which is the evidence the paper offers for the approach.

What carries the argument

The load-bearing object is the structured scene description: a text tuple $(o,c,p)$ listing each visible object, its color, and its normalized bounding-box center, produced under a predefined JSON schema that pre-fills fixed tokens during decoding. Spatial tokens are modeled with an Adapter mechanism, and object-to-text matching is trained with the contrastive loss from GLIP. The action-consistency loss couples two terms, a scene-description loss $L(x_t,x_{t+1},\tilde{x}_{t+1},\tilde{x}_t)$ and an action loss $L(a_t,\hat{a}_t)$, where the action is computed as $\hat{a}_t = \mathrm{MLP}(x_{\mathrm{agg}})$ from an attention aggregate of token embeddings, tying action selection to predicted state transitions. Multi-turn dialogue tuning extends the one-step objective to a sum over $T-1$ steps, so historical context shapes each action.

What would settle it

Run ACTLLM on CLIPORT or VIMA-BENCH at test time under two conditions: once with model-generated descriptions as the policy input and once with simulator ground-truth descriptions substituted in. If success rates are nearly identical, description quality is not the bottleneck; if they diverge sharply, description accuracy carries the result.

Watch

Extended reading notes

Core claim

The central claim is that actions can be generated from the aggregated token embedding of structured scene descriptions, and that explicitly supervising description generation together with an action-consistency loss produces policies that generalize better to novel objects, shapes, colors, and task templates. The observation model $M$ and forward model $F$ are both implemented by a single instruction-tuned LLM that outputs JSON-like entries $(o,c,p)$ for object, color, and coordinate; the policy head is a lightweight attention-and-MLP module that consumes the aggregated token and outputs a primitive action with start and target poses. At test time no external detector is used: the model generates the descriptions itself. On the CLIPORT put-shapes-in-bowls protocol the paper reports $64.0\%$ success versus $51.0\%$ for PAFF, and on pack-unseen-objects $85.8\%$ versus $72.8\%$; on VIMA-BENCH it reports level averages of $90.5\%$, $90.9\%$, and $93.4\%$ for L1--L3, above the VIMA baseline's $87.2\%$, $87.0\%$, and $84.0\%$.

Load-bearing premise

The method assumes that, at test time, the model's own generated scene descriptions are accurate enough to be the only state input for action prediction, even though training uses simulator ground-truth descriptions and description quality is never measured separately from task success.

Editorial extensions

If this is right

  • If the central claim holds, robot manipulation can be trained with text-only state representations, making the policy head small and interchangeable while the LLM handles perception.
  • Compositional generalization to novel objects and shapes follows from scene descriptions that enumerate all objects rather than only task-relevant ones, as the paper argues.
  • Multi-turn dialogue tuning supplies a mechanism for long-horizon tasks: each action is conditioned on the history of generated scene states.
  • The method removes the need for an external object detector at test time, since the LLM itself produces the structured description.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that description quality is a separate evaluation axis; measuring object detection and coordinate error directly would show how much of the reported success comes from perception accuracy.
  • A testable extension is to feed simulator ground-truth descriptions at test time instead of generated ones; sharply higher success would indicate description errors are the bottleneck, while similar success would point to the action-consistency mechanism as the cause.
  • The same action-consistency objective could be applied to 3D scene graphs or point-cloud descriptions, which would directly address the paper's stated limitation that 2D images lack precise 3D locations.
  • The visual-dialogue formulation suggests an untested extension to mid-trajectory task switching or user corrections, since multi-turn context is already part of the training objective.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes ACTLLM, a method for robotic manipulation that uses a large language model to generate structured JSON-formatted scene descriptions from visual observations, and uses an action-consistency loss to train a policy head on these descriptions. The method also reformulates the manipulation task as a multi-turn visual dialogue and reports strong results on CLIPORT and VIMA-BENCH. The claimed contributions are a structured scene-description state representation, a joint action-perception learning objective, and multi-turn dialogue tuning for long-horizon tasks.

Significance. If the reported results are reproducible, ACTLLM would be a useful step toward interpretable, language-centric visuomotor policies, and the structured description interface is an appealing way to regularize policy learning. The paper's strengths are the clear architectural concept, the explicit use of simulator ground truth to supervise structured descriptions, and the broad evaluation across two benchmarks. However, the paper currently lacks code, error bars, full per-task results, and a quantitative analysis of the quality of the self-generated descriptions at test time, so the central claim of 'significant' improvement is not yet fully supported.

major comments (4)
  1. [Section III-C, Section IV] The policy is trained on ground-truth scene descriptions but evaluated on self-generated descriptions. Section III-C explicitly avoids using generated descriptions during training because of 'potential inaccuracies in scene descriptions during initial training stages,' yet Section IV states that at test time no state representation is provided and the model generates descriptions. This distribution shift is never measured: no object recall, coordinate error, or parse validity of generated descriptions is reported, and the policy head receives only the attention-aggregated token of whatever descriptions are generated. Please either report description-quality metrics, train with the same description-generation procedure used at inference, or otherwise quantify how much of the reported gains survives with imperfect descriptions. This is necessary to support the claim that ACTLLM outperforms baselines without oracle state access during training.
  2. [Tables I-III, Section IV-A] The empirical evaluation reports only single success rates with no error bars, standard deviations, or number of seeds. Section IV-A says the CLIPORT evaluation uses 100 evaluation instances across 10 scenes, and Table II says 200 episodes were sampled per task, but no variance is reported. Without repeated seeds or confidence intervals, the claim in Section IV-A that ACTLLM 'significantly outperforms the baseline by a substantial margin' is not statistically supported. Please provide means and standard deviations across at least several seeds, or per-task results with confidence intervals, and make code and data available to verify the numbers.
  3. [Section III-E, Eq. (1)-(2)] The training objective is underspecified. Equation (1) defines the joint loss as L(xt,xt+1,~xt+1,~xt) + L(at, ât), but Section III-B only describes the scene-description loss via GLIP-style contrastive and box-regression losses, and Section III-C says the action loss 'includes both the classification loss of primitive skill and the loss of the scene description' without giving explicit forms or relative weights. The loss weighting is a free parameter and no ablation over it is shown. Without a precise definition, Eq. (1) and Eq. (2) cannot be reproduced and the action-consistency mechanism cannot be isolated from the description loss. Please provide exact loss terms, weighting coefficients, and an ablation over the weighting.
  4. [Section III-E] Implementation details are deferred to a missing supplementary: the section says 'Further details will be provided in the supplementary material,' but no supplementary is included with this version, and no code is released. Since the empirical evaluation is the main contribution, essential details such as the LLM fine-tuning protocol, the adapter architecture, the tokenization of coordinates, the training data, learning-rate schedules, hardware, and the exact evaluation protocol are needed for reproducibility. Please include these in the paper or a cited appendix, and ideally release code.
minor comments (6)
  1. [Section III-A, Eq. (1)] The constraint in Eq. (1) lists ~xt, ~xt+1 = Fβ(ot, It), but the text defines only ~xt+1 = Fβ(ot, It). Please clarify whether Fβ predicts both the current and the future state, or whether this is a typo.
  2. [Section III-C] The attention expression Q = EW, xagg = softmax(QQ^T)Q appears to use the same Q for the query and the key. Please define separate query, key, and value projections and clarify the dimensions of the matrices.
  3. [Section IV] The sentence 'we do not need to rely on any object detector to provide a structured screen description' contains a typo ('screen' should be 'scene'), and it is unclear whether the depth channel of the CLIPORT RGB-D observations is used anywhere, despite Section II.A claiming the model relies solely on RGB images.
  4. [Table III] In Table III, the entry '6%' should be '6.0%' for consistency, and the rows for the ablations mix decimal and percentage formats.
  5. [Throughout] There are several typos and informal phrases, including 'Noval Approach' in the contributions list, 'This all us to deduce' in Section III-A, 'fintuning' in Section III-D, and 'save T9 ... as a novel task' in Section IV-B. These should be corrected.
  6. [References] Reference formatting is inconsistent, with duplicate entries for CLIPORT ([2] and [29]) and minor citation style inconsistencies; the reference list should be harmonized with the journal or conference style.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the derivation chain is an empirical learning pipeline with no fitted input renamed as prediction and no load-bearing self-citation.

full rationale

ACTLLM is an empirical method paper whose claims are evaluated against the CLIPORT and VIMA-BENCH benchmarks. The paper's formal contribution is a joint training objective (Eq. 1 and Eq. 2) in which the scene-description loss L(xt, xt+1, ~xt+1, ~xt) is supervised by simulator ground-truth structured descriptions, and the action loss L(at, ât) supervises the MLP policy head on expert actions. The predicted future state ~xt+1 = F_beta(ot, It) is a learned function, and the policy ât = MLP(xagg) is applied to embeddings of ground-truth xt, xt+1 during training. No parameter is fitted to the test-set success rate and then reported as a prediction; the headline numbers are external benchmark results. The ablations in Table III remove modules and evaluate on held-out novel tasks, so the reported gains are not identical to the training objective by construction. The only substantive concern is that test-time scene descriptions are generated by the model rather than provided as ground truth, while training uses simulator ground truth for the policy input; that is a train/test distribution gap, not a circularity, because the model output is not used to define the training target. I also checked the reference list and found no self-citations by the authors, no invoked uniqueness theorem from prior work, and no ansatz smuggled in via citation. Accordingly, the circularity score is 0.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The ledger shows that ACTLLM's central claim rests on simulator-provided ground truth for scene descriptions, a text-coordinate state representation, and an unvalidated forward model, plus a frozen pretrained LLM. These are domain assumptions rather than physical axioms, and no new particles, forces, or dimensions are postulated.

free parameters (1)
  • loss weighting between scene description loss and action loss = 1.0 (implicit, not reported)
    Equation (1) sums L(xt,xt+1,~xt+1,~xt) and L(at,^at) without a learned or reported coefficient. The relative weight controls how strongly description accuracy versus action accuracy is prioritized, and the paper does not state whether this weight was tuned.
assumptions (5)
  • domain assumption The simulator's internal state yields exact, complete ground-truth scene descriptions (object, color, coordinates) at every step.
    Section III-B: "by leveraging access to the simulator's internal state, we can precisely obtain the condition of all objects." This provides the supervision for the description model.
  • domain assumption A text-based set of (object, color, coordinates) triples is a sufficient state representation for action generation.
    Section III-B defines the state representation and the policy head consumes only the token embedding of this text, so any task-relevant information outside this description is discarded.
  • domain assumption The forward model F_β can predict the next state description from the current observation and instruction with sufficient accuracy for the policy.
    Section III-A introduces ~x_{t+1}=F_β(o_t,I_t); training uses ground-truth x_{t+1} while testing uses the predicted ~x_{t+1}, and the paper gives no error bound or accuracy analysis for F_β.
  • domain assumption A frozen LLaMA-3-8B retains enough world knowledge to generate useful scene descriptions and support action inference without updating its weights.
    Section III-E freezes the visual encoder and the LLM, updating only adapters and the alignment layer, and relies on the pretrained model's priors.
  • domain assumption Top-down 2D RGB observations are sufficient to specify the manipulation tasks.
    The method inputs only RGB images and the paper's own limitation statement says "2D images often fail to capture the precise 3D locations and desired states essential for effective robot control" (Section V), yet the method depends on 2D coordinates for actions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ACTLLM: Action Consistency Tuned Large Language Model." pith.science (2026). https://pith.science/paper/VH25WKDA

@misc{pith2026250621250,
  author       = {Pith},
  title        = {Pith review of: ACTLLM: Action Consistency Tuned Large Language Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VH25WKDA}},
  note         = {Machine review of arXiv:2506.21250}
}
read the original abstract

This paper introduces ACTLLM (Action Consistency Tuned Large Language Model), a novel approach for robot manipulation in dynamic environments. Traditional vision-based systems often struggle to learn visual representations that excel in both task execution and spatial reasoning, thereby limiting their adaptability in dynamic environments. ACTLLM addresses these challenges by harnessing language to craft structured scene descriptors, providing a uniform interface for both spatial understanding and task performance through flexible language instructions. Moreover, we introduce a novel action consistency constraint that aligns visual perception with corresponding actions, thereby enhancing the learning of actionable visual representations. Additionally, we have reformulated the Markov decision process for manipulation tasks into a multi-turn visual dialogue framework. This approach enables the modeling of long-term task execution with enhanced contextual relevance derived from the history of task execution. During our evaluation, ACTLLM excels in diverse scenarios, proving its effectiveness on challenging vision-based robot manipulation tasks.

Figures

Figures reproduced from arXiv: 2506.21250 by the authors.

Figure 1
Figure 1. We reformulated the Markov decision process for manipulation tasks [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. In comparison to previous models: (a) Traditional forward models [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Comparison among our vision-language manipulation approach and existing solutions: [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: The illustration of our model. Our model consists of three main components: a visual encoder, a visual-language alignment layer, and a decoder-only [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

47 extracted references · 24 canonical work pages

  1. [1]

    Vima: General robot manip- ulation with multimodal prompts,

    Y . Jiang, A. Gupta, Z. Zhang, G. Wang, Y . Dou, Y . Chen, L. Fei-Fei, A. Anandkumar, Y . Zhu, and L. Fan, “Vima: General robot manip- ulation with multimodal prompts,” arXiv preprint arXiv:2210.03094 , 2022

    arXiv 2022

  2. [2]

    Cliport: What and where pathways for robotic manipulation,

    M. Shridhar, L. Manuelli, and D. Fox, “Cliport: What and where pathways for robotic manipulation,” in CoRL. PMLR, 2022, pp. 894–906

    work page 2022

  3. [3]

    Mastering robot manipulation with multimodal prompts through pretraining and multi-task fine-tuning,

    J. Li, Q. Gao, M. Johnston, X. Gao, X. He, S. Shakiah, H. Shi, R. Ghanadan, and W. Y . Wang, “Mastering robot manipulation with multimodal prompts through pretraining and multi-task fine-tuning,” arXiv preprint arXiv:2310.09676 , 2023

    arXiv

  4. [4]

    Calvin: A benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks,

    O. Mees, L. Hermann, E. Rosete-Beas, and W. Burgard, “Calvin: A benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks,” IEEE Robotics and Automation Letters , vol. 7, no. 3, pp. 7327–7334, 2022

    2022

  5. [6]

    Palm-e: An embodied multimodal language model,

    D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu,et al., “Palm-e: An embodied multimodal language model,” arXiv preprint arXiv:2303.03378, 2023

    arXiv 2023

  6. [7]

    Rt-1: Robotics transformer for real-world control at scale,

    A. Brohan, N. Brown, J. Carbajal, Y . Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, J. Hsu, et al. , “Rt-1: Robotics transformer for real-world control at scale,” arXiv preprint arXiv:2212.06817, 2022

    arXiv 2022

  7. [8]

    Understanding natural language commands for robotic navigation and mobile manipulation,

    S. Tellex, T. Kollar, S. Dickerson, M. Walter, A. Banerjee, S. Teller, and N. Roy, “Understanding natural language commands for robotic navigation and mobile manipulation,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 25, no. 1, 2011, pp. 1507– 1514

    work page 2011

  8. [9]

    Instruct2act: Mapping multi-modality instructions to robotic actions with large language model,

    S. Huang, Z. Jiang, H. Dong, Y . Qiao, P. Gao, and H. Li, “Instruct2act: Mapping multi-modality instructions to robotic actions with large language model,” arXiv preprint arXiv:2305.11176 , 2023

    arXiv 2023

Show all 47 references
  1. [10]

    R3m: A universal visual representation for robot manipulation,

    S. Nair, A. Rajeswaran, V . Kumar, C. Finn, and A. Gupta, “R3m: A universal visual representation for robot manipulation,” arXiv preprint arXiv:2203.12601, 2022

    2022 arXiv

  2. [11]

    Policy adaptation from foundation model feedback,

    Y . Ge, A. Macaluso, L. E. Li, P. Luo, and X. Wang, “Policy adaptation from foundation model feedback,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 19 059–19 069

    2023

  3. [12]

    Programmat- ically grounded, compositionally generalizable robotic manipulation,

    R. Wang, J. Mao, J. Hsu, H. Zhao, J. Wu, and Y . Gao, “Programmat- ically grounded, compositionally generalizable robotic manipulation,” arXiv preprint arXiv:2304.13826 , 2023

    2023 arXiv

  4. [13]

    Sprint: Scalable policy pre-training via language instruction relabeling,

    J. Zhang, K. Pertsch, J. Zhang, and J. J. Lim, “Sprint: Scalable policy pre-training via language instruction relabeling,” arXiv preprint arXiv:2306.11886, 2023

    2023 arXiv

  5. [14]

    Liv: Language-image representations and rewards for robotic control,

    Y . J. Ma, W. Liang, V . Som, V . Kumar, A. Zhang, O. Bastani, and D. Jayaraman, “Liv: Language-image representations and rewards for robotic control,” arXiv preprint arXiv:2306.00958 , 2023

    2023 arXiv

  6. [15]

    Reward design with language models,

    M. Kwon, S. M. Xie, K. Bullard, and D. Sadigh, “Reward design with language models,” in The 11th International Conference on Learning Representations, 2023

    2023

  7. [16]

    Language to rewards for robotic skill synthesis,

    W. Yu, N. Gileadi, C. Fu, S. Kirmani, K.-H. Lee, M. G. Arenas, H.- T. L. Chiang, T. Erez, L. Hasenclever, J. Humplik, et al., “Language to rewards for robotic skill synthesis,” arXiv preprint arXiv:2306.08647, 2023

    2023 arXiv

  8. [17]

    Learning reward functions from diverse sources of human feedback: Optimally integrating demonstrations and preferences,

    E. Bıyık, D. P. Losey, M. Palan, N. C. Landolfi, G. Shevchuk, and D. Sadigh, “Learning reward functions from diverse sources of human feedback: Optimally integrating demonstrations and preferences,” The International Journal of Robotics Research , vol. 41, no. 1, pp. 45–67, 2022

    2022

  9. [18]

    React: Synergizing reasoning and acting in language models,

    S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y . Cao, “React: Synergizing reasoning and acting in language models,” arXiv preprint arXiv:2210.03629, 2022

    2022 arXiv

  10. [19]

    Self-supervised 6d object pose estimation for robot manipulation,

    X. Deng, Y . Xiang, A. Mousavian, C. Eppner, T. Bretl, and D. Fox, “Self-supervised 6d object pose estimation for robot manipulation,” in 2020 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2020, pp. 3665–3671

    2020

  11. [20]

    The best of both modes: Separately leveraging rgb and depth for unseen object instance segmentation,

    C. Xie, Y . Xiang, A. Mousavian, and D. Fox, “The best of both modes: Separately leveraging rgb and depth for unseen object instance segmentation,” in Conference on robot learning . PMLR, 2020, pp. 1369–1378

    2020

  12. [21]

    Qt- opt: Scalable deep reinforcement learning for vision-based robotic manipulation,

    D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V . Vanhoucke, et al. , “Qt- opt: Scalable deep reinforcement learning for vision-based robotic manipulation,” arXiv preprint arXiv:1806.10293 , 2018

    2018 arXiv

  13. [22]

    Learning efficient illumination multiplexing for joint capture of reflectance and shape

    K. Kang, C. Xie, C. He, M. Yi, M. Gu, Z. Chen, K. Zhou, and H. Wu, “Learning efficient illumination multiplexing for joint capture of reflectance and shape.” ACM Trans. Graph. , vol. 38, no. 6, pp. 165–1, 2019

    2019

  14. [23]

    Coarse-to- fine q-attention: Efficient learning for visual robotic manipulation via discretisation,

    S. James, K. Wada, T. Laidlow, and A. J. Davison, “Coarse-to- fine q-attention: Efficient learning for visual robotic manipulation via discretisation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 13 739–13 748

    2022

  15. [24]

    Transporter networks: Rearranging the visual world for robotic manipulation,

    A. Zeng, P. Florence, J. Tompson, S. Welker, J. Chien, M. Attarian, T. Armstrong, I. Krasin, D. Duong, V . Sindhwani, et al., “Transporter networks: Rearranging the visual world for robotic manipulation,” in Conference on Robot Learning . PMLR, 2021, pp. 726–747

    2021

  16. [25]

    Guiding multi-step rearrangement tasks with natural language instructions,

    E. Stengel-Eskin, A. Hundt, Z. He, A. Murali, N. Gopalan, M. Gom- bolay, and G. Hager, “Guiding multi-step rearrangement tasks with natural language instructions,” in Conference on Robot Learning . PMLR, 2022, pp. 1486–1501

    2022

  17. [26]

    Do as i can, not as i say: Grounding language in robotic affordances,

    A. Brohan, Y . Chebotar, C. Finn, K. Hausman, A. Herzog, D. Ho, J. Ibarz, A. Irpan, E. Jang, R. Julian, et al., “Do as i can, not as i say: Grounding language in robotic affordances,” in Conference on Robot Learning. PMLR, 2023, pp. 287–318

    2023

  18. [27]

    Con- cept2robot: Learning manipulation concepts from instructions and hu- man demonstrations,

    L. Shao, T. Migimatsu, Q. Zhang, K. Yang, and J. Bohg, “Con- cept2robot: Learning manipulation concepts from instructions and hu- man demonstrations,” The International Journal of Robotics Research, vol. 40, no. 12-14, pp. 1419–1434, 2021

    2021

  19. [28]

    Interactive visual grounding of re- ferring expressions for human-robot interaction,

    M. Shridhar and D. Hsu, “Interactive visual grounding of re- ferring expressions for human-robot interaction,” arXiv preprint arXiv:1806.03831, 2018

    2018 arXiv

  20. [29]

    Cliport: What and where pathways for robotic manipulation,

    M. Shridhar, L. Manuelli, and D. Fox, “Cliport: What and where pathways for robotic manipulation,” in Conference on Robot Learning. PMLR, 2022, pp. 894–906

    2022

  21. [30]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark,et al., “Learning transferable visual models from natural language supervision,” in International conference on machine learning . PMLR, 2021, pp. 8748–8763

    2021

  22. [31]

    Perceiver-actor: A multi- task transformer for robotic manipulation,

    M. Shridhar, L. Manuelli, and D. Fox, “Perceiver-actor: A multi- task transformer for robotic manipulation,” in Conference on Robot Learning. PMLR, 2023, pp. 785–799

    2023

  23. [32]

    Code as policies: Language model programs for embodied control,

    J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng, “Code as policies: Language model programs for embodied control,” arXiv preprint arXiv:2209.07753 , 2022

    2022 arXiv

  24. [33]

    Language models as zero-shot planners: Extracting actionable knowledge for embodied agents,

    W. Huang, P. Abbeel, D. Pathak, and I. Mordatch, “Language models as zero-shot planners: Extracting actionable knowledge for embodied agents,” in International Conference on Machine Learning . PMLR, 2022, pp. 9118–9147

    2022

  25. [34]

    Inner monologue: Embodied reasoning through planning with language models,

    W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang, P. Florence, A. Zeng, J. Tompson, I. Mordatch, Y . Chebotar, et al. , “Inner monologue: Embodied reasoning through planning with language models,” arXiv preprint arXiv:2207.05608, 2022

    2022 arXiv

  26. [35]

    Do as i can, not as i say: Grounding language in robotic affordances,

    M. Ahn, A. Brohan, N. Brown, Y . Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakrishnan, K. Hausman, et al. , “Do as i can, not as i say: Grounding language in robotic affordances,” arXiv preprint arXiv:2204.01691, 2022

    2022 arXiv

  27. [36]

    Learning language-conditioned robot behavior from offline data and crowd- sourced annotation,

    S. Nair, E. Mitchell, K. Chen, S. Savarese, C. Finn, et al., “Learning language-conditioned robot behavior from offline data and crowd- sourced annotation,” in Conference on Robot Learning. PMLR, 2022, pp. 1303–1315

    2022

  28. [37]

    Socratic models: Composing zero-shot multimodal reasoning with language,

    A. Zeng, A. Wong, S. Welker, K. Choromanski, F. Tombari, A. Purohit, M. Ryoo, V . Sindhwani, J. Lee, V . Vanhoucke,et al., “Socratic models: Composing zero-shot multimodal reasoning with language,” arXiv preprint arXiv:2204.00598, 2022

    2022 arXiv

  29. [38]

    Reward design with language models,

    M. Kwon, S. M. Xie, K. Bullard, and D. Sadigh, “Reward design with language models,” arXiv preprint arXiv:2303.00001 , 2023

    2023 arXiv

  30. [39]

    A survey of imitation learning: Algorithms, recent developments, and challenges,

    M. Zare, P. M. Kebria, A. Khosravi, and S. Nahavandi, “A survey of imitation learning: Algorithms, recent developments, and challenges,” 2023

    2023

  31. [40]

    Parameter-efficient transfer learning for nlp,

    N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. de Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly, “Parameter-efficient transfer learning for nlp,” 2019

    2019

  32. [41]

    Grounded language-image pre-training,

    L. H. Li, P. Zhang, H. Zhang, J. Yang, C. Li, Y . Zhong, L. Wang, L. Yuan, L. Zhang, J.-N. Hwang, K.-W. Chang, and J. Gao, “Grounded language-image pre-training,” in 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . IEEE, June 2022. [Online]. Availab...

    2022

  33. [42]

    Vicuna: An open- source chatbot impressing gpt-4 with 90%* chatgpt quality,

    W.-L. Chiang, Z. Li, Z. Lin, Y . Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y . Zhuang, J. E. Gonzalez, et al. , “Vicuna: An open- source chatbot impressing gpt-4 with 90%* chatgpt quality,” See https://vicuna. lmsys. org (accessed 14 April 2023) , 2023

    2023

  34. [43]

    What makes for good visual tokenizers for large language models?

    G. Wang, Y . Ge, X. Ding, M. Kankanhalli, and Y . Shan, “What makes for good visual tokenizers for large language models?” arXiv preprint arXiv:2305.12223, 2023

    2023 arXiv

  35. [44]

    The llama 3 herd of models,

    A. Dubey, A. Jauhri, and A. e. a. Pandey, “The llama 3 herd of models,” 2024. [Online]. Available: https://arxiv.org/abs/2407.21783

    2024 arXiv

  36. [45]

    Mdetr-modulated detection for end-to-end multi-modal understand- ing,

    A. Kamath, M. Singh, Y . LeCun, G. Synnaeve, I. Misra, and N. Carion, “Mdetr-modulated detection for end-to-end multi-modal understand- ing,” in ICCV, 2021, pp. 1780–1790

    2021

  37. [46]

    Learning to augment synthetic images for sim2real policy transfer,

    A. Pashevich, R. Strudel, I. Kalevatykh, I. Laptev, and C. Schmid, “Learning to augment synthetic images for sim2real policy transfer,” in IROS). IEEE, 2019, pp. 2651–2657

    2019

  38. [47]

    A generalist agent,

    S. Reed, K. Zolna, E. Parisotto, S. G. Colmenarejo, A. Novikov, G. Barth-Maron, M. Gimenez, Y . Sulsky, J. Kay, J. T. Springenberg, T. Eccles, J. Bruce, A. Razavi, A. Edwards, N. Heess, Y . Chen, R. Hadsell, O. Vinyals, M. Bordbar, and N. de Freitas, “A generalist agent,” 2022

    2022

  39. [48]

    Flamingo: a visual language model for few-shot learning,

    J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y . Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, R. Ring, E. Ruther- ford, S. Cabi, T. Han, Z. Gong, S. Samangooei, M. Monteiro, J. Menick, S. Borgeaud, A. Brock, A. Nematzadeh, S. Sharifzadeh, M. Binkowski, R. Bar...

    2022

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.