GhostShell: Streaming LLM Function Calls for Concurrent Embodied Programming

Bingchao Huang; Bo Yuan; Hang Shu; Jian Gong; Jianhang Liang; Jinke Wang; Juncheng Zhan; Mingyue Xiong; Ming Zhu; Runjie Ban

REVIEW 5 major objections 4 minor 70 references

GhostShell: Streaming LLM Function Calls for Concurrent Embodied Programming

T0 review · 5 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read GhostShell claims that an LLM can program a robot while it is still talking, parsing XML function calls from the streaming output and executing them on separate channels, and reports 15/15 task success versus 6/15 for native function callin

desk verdict GhostShell's streaming function-token idea is plausible and worth a real look, but the provided full text is mojibake and the key confound—where channel assignments come from—is unresolved in the abstract. read the letter →

arxiv 2508.05298 v3 pith:X76W26PV submitted 2025-08-07 cs.RO

Jian Gong , Youwei Huang , Bo Yuan , Ming Zhu , Zhou Liao , Jianhang Liang , Juncheng Zhan , Jinke Wang

show 11 more authors

Hang Shu Zihao Xu Mingyue Xiong Yanjun Ye Yufan Zu Yang Zhou Yihan Ding Xuannian Chen Xingyu Lu Runjie Ban Bingchao Huang

This is my paper

classification cs.RO

keywords GhostShellstreamingLLMfunctioncallingembodiedagentsXMLtokensmulti-channelschedulinghuman-robotinteractionconcurrentbehaviorrobotcontrol

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

GhostShell tries to show that embodied agents need not wait for a complete plan before acting. It lets an LLM invoke robot functions incrementally while it is still generating text, parsing XML-style function tokens out of the token stream and executing them on the fly. A multi-channel scheduler runs calls on the same channel in order and calls on different channels concurrently, so speech, head motion, body rotation, and music can overlap as a human would expect. On 30 grounded human-robot interaction tasks the approach scores 0.83 on the authors' Directed Structured Behavior Correctness metric with Claude-Sonnet-4, and on concurrent talk-and-move tasks it completes 15/15 versus 6/15 for native LLM function calling. If correct, this means robots can reason while acting rather than planning first, which matters for any system that must talk and move at the same time.

What carries the argument

The load-bearing piece is the function token schema: an XML-based representation of callable robot functions, such as <speak>...</speak> or <play_bgm/>, embedded inline with natural-language tokens. GhostShell parses these tokens from the LLM streaming output, maps them to actual functions, and feeds them into a multi-channel scheduler. The scheduler is what makes concurrency possible: intra-channel synchronous dispatch preserves order within one robot component, while inter-channel asynchronous dispatch lets different components act simultaneously.

What would settle it

Give GhostShell a task in which speech must finish before rotation starts, with the two functions assigned to different channels. If the inter-channel asynchronous dispatch starts rotation mid-sentence, the claim that the scheduler captures task ordering fails.

Watch

Extended reading notes

Core claim

The central claim is that the LLM's output token stream can be treated as a live program. GhostShell defines function tokens as XML-based function-call fragments, parses them as they appear during generation, maps them to callable robot functions, and dispatches them before the model has finished responding. This removes the plan-then-execute round trip and replaces it with reasoning-while-acting. The multi-channel scheduler then coordinates the parsed calls: calls assigned to the same channel execute synchronously, while calls on different channels execute asynchronously and in parallel, which is how a robot can stand up, say 'Let's go!', play music, and rotate while counting from 1 to 8 as

Load-bearing premise

The result rests on the assumption that every task's real ordering constraints can be represented by the authors' channel assignment, so that actions on different channels are always safe to run asynchronously.

Editorial extensions

If this is right

If this holds, LLM-based embodied control does not need to wait for a complete tool-call response before starting physical and verbal actions.
Concurrent linguistic and physical behavior can be specified by channel assignment rather than by explicitly writing parallel control code.
The same XML function-token schema works across nine LLM providers, suggesting it is a portable interface rather than a provider-specific tool-call format.
Native LLM function calling, which serializes calls, may be a poor fit for embodied tasks; GhostShell's 15/15 versus 6/15 on concurrent tasks is direct evidence for that limitation.
The DSBC scoring scheme gives a concrete way to measure ordered, structured behavior correctness in embodied tasks, provided the rubric is validated.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

One consequence the paper leaves implicit is that hand-designed channel assignments carry the concurrency model: if a task needs ordering across channels, the asynchronous policy could silently reorder actions unless an explicit dependency mechanism is added.
The token-stream idea probably extends beyond robots to any agent that must interleave long-running tool calls and user-facing output, but the paper only demonstrates it on the CoCo robotic platform.
The DSBC metric is author-defined, so a natural next test is measuring inter-rater agreement and comparing it against human judgment of task success.
A testable extension would be to let the LLM itself declare channel dependencies, rather than relying on experimenters to assign functions to channels, which would remove the main manual step in the pipeline.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know two things up front. First, the abstract describes a concrete systems contribution that I take seriously: parse XML function tokens out of the LLM's token stream and dispatch them through a multi-channel scheduler with intra-channel synchronous and inter-channel asynchronous execution. That is a real departure from plan-then-execute, and it's the kind of mechanism that could plausibly let a robot act before generation finishes. Second, the full text we have is unreadable mojibake. So everything below is abstract-plus-figure-fragment level, and the paper is not currently refereeable in the form I received.

What's genuinely good: the evaluation design is more ambitious than most papers of this type—33 real-world tasks, nine providers, and a direct head-to-head against native LLM function calling. Reporting a specific DSBC number (0.83 for Claude-Sonnet-4) and a 15/15 versus 6/15 comparison gives you a falsifiable claim rather than a vibe. The function-token schema is a concrete interface, not a prompt trick. That earns credit.

The soft spots are real but I can't confirm them as flaws from the abstract. The biggest one matches the stress-test note: the figure fragments show functions mapped to channels named head, body, bgm, main, and we don't know where those channel assignments come from. If they're hand-coded per task, then the LLM isn't doing the concurrency reasoning—it's filling in a schedule the designer already made, and the 15/15 result would largely measure the schema, not the streaming. The asynchronous inter-channel policy could also silently reorder cross-channel dependencies, and the tasks may have been selected to avoid that. The DSBC rubric is not shown, there are no confidence intervals or variances, and the human evaluation covers only three tasks. These are concerns, not convictions, but they are the questions I would want answered.

Who this is for: people building LLM-driven robot controllers, and anyone working on streaming function calling. The idea is substantive enough that a clean, readable version deserves serious referee attention. But what I was sent is not that version. I would tell the editor to get a usable PDF before sending it out, and to ask specifically about channel-assignment provenance and cross-channel ordering constraints. My own verdict is not on the merits—it's unverdict due to an unreadable file.

Referee Report

5 major / 4 minor

Summary. The paper presents GhostShell, an LLM-based behavioral programming framework for an embodied robot. It defines XML-style function tokens that GhostShell parses from the streaming LLM output and dispatches through a multi-channel scheduler with intra-channel synchronous and inter-channel asynchronous execution. The claimed contribution is 'reasoning-while-acting': incremental function invocation during generation, as opposed to plan-then-execute or native function calling. Evaluation is on a custom quadruped robot (CoCo) with 33 tasks: 30 grounded HRI tasks scored by a new DSBC metric (best 0.83 for Claude-Sonnet-4) and 3 long-horizon tasks scored by humans (best 7.0/10 for GPT-4.1). The paper further reports 15/15 task completion with the function-token schema versus 6/15 with native LLM function calling on tasks involving concurrent linguistic and physical actions.

Significance. If valid, the work is a useful engineering contribution: it demonstrates that streaming XML function tokens can be parsed and dispatched while an LLM is still generating, and it provides a concrete multi-channel execution model for concurrent robot behavior. The evaluation across nine LLM providers and real robot tasks is commendable, and the supplementary videos provide a useful sanity check. The central claim is plausible, but the current evidence is not sufficient to establish that streaming parsing, rather than the channel topology or task selection, is what drives the reported gains. The DSBC metric and the small human evaluation also need validation. The work would be of interest to the embodied-agent and human-robot interaction communities if these gaps are closed.

major comments (5)

[Abstract; Figure 1 (scheduler/channel mapping)] It is not specified where the channel assignments (head, body, bgm, main) come from. If they are manually fixed per function or per task, the scheduler's 'intra-channel synchronous and inter-channel asynchronous dispatch' encodes part of the temporal plan, so the comparison against native function calling is confounded. In the running example, speak (main) must precede play_bgm (bgm), but asynchronous inter-channel dispatch would start both when generated; nothing in the described policy prevents play_bgm from overtaking speak. Please (a) state whether channel assignments are task-specific or fixed in the schema, (b) describe how cross-channel precedence constraints are represented and enforced, and (c) provide an ablation that isolates the streaming parser from the channel assignment and dispatch policy.
[§Evaluation, DSBC metric (abstract)] The headline DSBC scores are computed with a metric whose rubric, annotation procedure, and reliability are not reported. Without inter-annotator agreement and per-task score distributions, DSBC cannot be distinguished from an author-designed rubric that may reward the framework's expected outputs. Please include the full DSBC rubric, scoring examples, agreement statistics, and per-task scores for all models.
[§Evaluation, native function calling comparison (abstract)] The 15/15 vs 6/15 comparison is reported without variance, confidence intervals, or significance testing. With 15 tasks, the difference is plausibly real, but a single run per model and task is not enough. Please report repeated trials (e.g., 3-5 runs/model), statistical tests, and the exact task list. Also clarify whether the channel assignments for GhostShell were hand-tuned per task, since the native baseline lacks such an external channel topology.
[§Evaluation, human study (abstract)] The human evaluation covers only three long-horizon tasks, with a single 7.0/10 score reported. There is no information on number of raters, rater instructions, score variance, or whether raters were blind to condition. This evidence is too limited to support the claim that GhostShell enables better long-horizon multimodal behavior. Please expand the human study or soften the claim accordingly.
[§Method (reasoning-while-acting)] The central mechanism is 'incrementally invoking functions during streaming generation,' but the evaluation does not measure streaming behavior directly: no latency until first action, no inter-action interval, no comparison to plan-then-execute under matched function schemas. Task completion could be driven by the schema and scheduler rather than by incremental parsing. Add measurements of parse-to-dispatch timing and an ablation that waits for full generation.

minor comments (4)

[Figure 1] The channel labels head/body/bgm/main are not defined in the caption; clarify the channel mapping and which functions belong to each channel.
[Figure 4 / Table 3] Model names are truncated in the tables (e.g., 'Grok Code Fast', 'GLM-4.5'), making reproduction difficult; spell out provider and model names.
[§Method] The XML function-token grammar appears only as a figure fragment. Place the grammar and parser pseudocode in an appendix so the schema is fully specified.
[Related work] The comparison to prior streaming/function-calling baselines is limited to native function calling. Discuss or compare to ReAct-style interleaving and structured tool-calling APIs.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the claim is an empirical system evaluation, not a derivation from its own inputs.

full rationale

GhostShell is presented as an engineering system with an empirical evaluation: an XML function-token schema is parsed from an LLM stream and dispatched by a multi-channel scheduler. The main evidence is task-completion rates (15/15 vs 6/15) against native LLM function calling, scored by DSBC and human raters. The DSBC metric is author-defined, but it is an evaluation rubric, not a parameter fitted to the outcome; human raters and the native-function-calling baseline are external anchors. The multi-channel scheduler's channel assignments are a design choice, not a quantity derived from the target result; any task-specificity would be an experimental confound for generalization, not circularity. No load-bearing step reduces to a self-citation or to a definitional equivalence: the paper does not fit a parameter to the reported scores and then rename that fit a prediction. The 15/15 result is an observed empirical outcome, not an analytic consequence of the schema. Therefore no circular step is present.

Assumptions & free parameters 0 free parameters · 3 assumptions · 2 invented entities

The central claim rests on three domain assumptions made in the abstract (streaming parse reliability, channel-assignment adequacy, DSBC validity) plus an empirical evaluation that is not inspectable in this version. No numeric parameters are fitted in the abstract; any scheduler thresholds, prompt constants, or parser tolerances could not be audited because the body is corrupted. No invented physical entities (particles, forces, dimensions) are introduced; the system, the schema, and the metric are the invented artifacts.

assumptions (3)

domain assumption LLM providers emit parseable XML function tokens incrementally during streaming, with parse failures rare enough not to dominate outcomes.
The central mechanism "parses from the LLM generation stream and maps to callable functions" (abstract) presupposes mid-stream tokens are reliably well-formed across nine providers; robustness is asserted but not evidenced in the abstract.
domain assumption The channel mapping of robot components (intra-channel synchronous, inter-channel asynchronous) matches the dependency structure of all 33 tasks.
Quoted from the abstract: "intra-channel synchronous and inter-channel asynchronous dispatch." If cross-channel order dependencies exist, concurrent dispatch yields incorrect behavior and the headline gains would not generalize.
ad hoc to paper DSBC (Directed Structured Behavior Correctness) is a valid and repeatable measure of task correctness.
DSBC is introduced by this paper as its headline quantitative metric; the abstract provides no definition, rubric, or reliability statistics, and no external validation is referenced.

invented entities (2)

GhostShell framework and XML function-token schema independent evidence
purpose: Streaming interface that maps LLM output tokens to callable robot functions so behavior starts before generation completes
Evaluated against an external baseline (native LLM function calling) and demonstrated on physical robot tasks at a project page with videos, but the schema itself is hand-designed by the authors and its superiority is only as strong as the evaluation.
DSBC metric
purpose: Headline scoring metric for the 30 grounded HRI tasks
Author-defined metric with no rubric or agreement statistics visible in the abstract; if the metric encodes the authors' notion of correctness, the 0.83 headline is only as valid as that definition.

how reviews work

0 comments

Cite this review

Pith. "Pith review of GhostShell: Streaming LLM Function Calls for Concurrent Embodied Programming." pith.science (2026). https://pith.science/paper/X76W26PV

@misc{pith2026250805298,
  author       = {Pith},
  title        = {Pith review of: GhostShell: Streaming LLM Function Calls for Concurrent Embodied Programming},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/X76W26PV}},
  note         = {Machine review of arXiv:2508.05298}
}

read the original abstract

We present GhostShell, a novel approach that leverages Large Language Models (LLMs) for streaming and concurrent behavioral programming in embodied systems. In contrast to predefined behavioral structures and plan-then-execute paradigms, GhostShell enables reasoning-while-acting by incrementally invoking functions during LLM streaming generation. We define function tokens as an XML-based function-call representation that GhostShell parses from the LLM generation stream and maps to callable functions. A multi-channel scheduling algorithm further orchestrates these calls with intra-channel synchronous and inter-channel asynchronous dispatch, coordinating sequential-parallel behavior execution across multiple robotic components. We evaluate GhostShell on our robotic prototype CoCo across 33 real-world tasks with LLMs from nine providers. On 30 grounded Human-Robot Interaction (HRI) tasks, our approach achieves the highest Directed Structured Behavior Correctness (DSBC) score of 0.83 with Claude-Sonnet-4, while on three long-horizon multimodal tasks, GPT-4.1 attains a top human evaluation score of 7.0/10. Compared to native LLM function calling, our function token schema achieves a 15/15 task completion rate versus 6/15, particularly in coordinating concurrent linguistic and physical actions. Supplementary materials, including videos, are available at https://coco-robot.github.io/GhostShell.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

70 extracted references · 43 canonical work pages

[1]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...
[2]

Goertzel

B. Goertzel. Artificial general intelligence: Concept, state of the art, and future prospects. Journal of Artificial General Intelligence, 5 0 (1): 0 1, 2014

work page 2014
[3]

M. R. Morris, J. Sohl-Dickstein, N. Fiedel, T. Warkentin, A. Dafoe, A. Faust, C. Farabet, and S. Legg. Levels of agi for operationalizing progress on the path to agi. arXiv preprint arXiv:2311.02462, 2023

arXiv 2023
[4]

Bubeck, V

S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712, 2023

arXiv 2023
[5]

What is Multimodality?

L. Parcalabescu, N. Trost, and A. Frank. What is multimodality? arXiv preprint arXiv:2103.06304, 2021

work page Pith review arXiv 2021
[6]

J. Lin, R. Men, A. Yang, C. Zhou, Y. Zhang, P. Wang, J. Zhou, J. Tang, and H. Yang. M6: Multi-modality-to-multi-modality multitask mega-transformer for unified pretraining. In Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining, pages 3251--3261, 2021

work page 2021
[7]

T. B. Sheridan. Human--robot interaction: status and challenges. Human factors, 58 0 (4): 0 525--532, 2016

work page 2016
[8]

Bartneck, T

C. Bartneck, T. Belpaeme, F. Eyssel, T. Kanda, M. Keijsers, and S. S abanovi \'c . Human-robot interaction: An introduction. Cambridge University Press, 2024

work page 2024

Show all 70 references

[9]

C. Wang, S. Hasler, D. Tanneberg, F. Ocker, F. Joublin, A. Ceravola, J. Deigmoeller, and M. Gienger. Lami: Large language models for multi-modal human-robot interaction. In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, pages 1--10, 2024

2024
[10]

Koubaa, A

A. Koubaa, A. Ammar, and W. Boulila. Next-generation human-robot interaction with chatgpt and robot operating system. Software: Practice and Experience, 55 0 (2): 0 355--382, 2025

2025
[11]

Colledanchise and P

M. Colledanchise and P. \"O gren. Behavior trees in robotics and AI: An introduction. CRC Press, 2018

2018
[12]

Lykov and D

A. Lykov and D. Tsetserukou. Llm-brain: Ai-driven fast generation of robot behaviour tree based on large language model. In 2024 2nd International Conference on Foundation and Large Language Models (FLLM), pages 392--397. IEEE, 2024

2024
[13]

R. A. Izzo, G. Bardaro, and M. Matteucci. Btgenbot: Behavior tree generation for robotic tasks with lightweight llms. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 9684--9690. IEEE, 2024

2024
[14]

C. R. Garrett, R. Chitnis, R. Holladay, B. Kim, T. Silver, L. P. Kaelbling, and T. Lozano-P \'e rez. Integrated task and motion planning. Annual review of control, robotics, and autonomous systems, 4 0 (1): 0 265--293, 2021

2021
[15]

Nachum, S

O. Nachum, S. S. Gu, H. Lee, and S. Levine. Data-efficient hierarchical reinforcement learning. Advances in neural information processing systems, 31, 2018

2018
[16]

Brohan, N

A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, J. Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817, 2022

2022 arXiv
[17]

Zitkovich, T

B. Zitkovich, T. Yu, S. Xu, P. Xu, T. Xiao, F. Xia, J. Wu, P. Wohlhart, S. Welker, A. Wahid, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. In Conference on Robot Learning, pages 2165--2183. PMLR, 2023

2023
[18]

Sapkota, Y

R. Sapkota, Y. Cao, K. I. Roumeliotis, and M. Karkee. Vision-language-action models: Concepts, progress, applications and challenges. arXiv preprint arXiv:2505.04769, 2025

2025
[19]

Achiam, S

J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

2023 arXiv
[20]

G. Team, R. Anil, S. Borgeaud, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023

2023 arXiv
[21]

H. Duan, J. Yang, Y. Qiao, X. Fang, L. Chen, Y. Liu, X. Dong, Y. Zang, P. Zhang, J. Wang, et al. Vlmevalkit: An open-source toolkit for evaluating large multi-modality models. In Proceedings of the 32nd ACM international conference on multimedia, pages 11198--11201, 2024

2024
[22]

S. Wu, H. Fei, L. Qu, W. Ji, and T.-S. Chua. Next-gpt: Any-to-any multimodal llm. In Forty-first International Conference on Machine Learning, 2024

2024
[23]

Ag \"u era y Arcas and P

B. Ag \"u era y Arcas and P. Norvig. Artificial general intelligence is already here. Noema Magazine, October 2023

2023
[24]

Streaming api responses

OpenAI . Streaming api responses. https://platform.openai.com/docs/guides/streaming-responses, 2025

2025
[25]

T. Bray, J. Paoli, C. M. Sperberg-McQueen, E. Maler, F. Yergeau, and J. Cowan. Extensible markup language (xml) 1.0, 2000

2000
[26]

F. Zeng, W. Gan, Y. Wang, N. Liu, and P. S. Yu. Large language models for robotics: A survey. arXiv preprint arXiv:2311.07226, 2023

2023
[27]

J. Wang, E. Shi, H. Hu, C. Ma, Y. Liu, X. Wang, Y. Yao, X. Liu, B. Ge, and S. Zhang. Large language models for robotics: Opportunities, challenges, and perspectives. Journal of Automation and Intelligence, 4 0 (1): 0 52--64, 2025

2025
[28]

Huang, P

W. Huang, P. Abbeel, D. Pathak, and I. Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International conference on machine learning, pages 9118--9147. PMLR, 2022

2022
[29]

M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakrishnan, K. Hausman, et al. Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691, 2022

2022 arXiv
[30]

Huang, F

W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang, P. Florence, A. Zeng, J. Tompson, I. Mordatch, Y. Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.05608, 2022

2022 arXiv
[31]

L. Sun, D. K. Jha, C. Hori, S. Jain, R. Corcodel, X. Zhu, M. Tomizuka, and D. Romeres. Interactive planning using large language models for partially observable robotic tasks. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 14054--14061. IEEE, 2024

2024
[32]

Yoneda, J

T. Yoneda, J. Fang, P. Li, H. Zhang, T. Jiang, S. Lin, B. Picker, D. Yunis, H. Mei, and M. R. Walter. Statler: State-maintaining language models for embodied reasoning. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 15083--15091. IEEE, 2024

2024
[33]

Huang, F

W. Huang, F. Xia, D. Shah, D. Driess, A. Zeng, Y. Lu, P. Florence, I. Mordatch, S. Levine, K. Hausman, et al. Grounded decoding: Guiding text generation with grounded models for embodied agents. Advances in Neural Information Processing Systems, 36: 0 59636--59661, 2023

2023
[34]

A. Zeng, M. Attarian, B. Ichter, K. Choromanski, A. Wong, S. Welker, F. Tombari, A. Purohit, M. Ryoo, V. Sindhwani, et al. Socratic models: Composing zero-shot multimodal reasoning with language. arXiv preprint arXiv:2204.00598, 2022

2022 arXiv
[35]

Y. Mu, Q. Zhang, M. Hu, W. Wang, M. Ding, J. Jin, B. Wang, J. Dai, Y. Qiao, and P. Luo. Embodiedgpt: Vision-language pre-training via embodied chain of thought. Advances in Neural Information Processing Systems, 36: 0 25081--25094, 2023

2023
[36]

Zheng, K

K. Zheng, K. Zhou, J. Gu, Y. Fan, J. Wang, Z. Di, X. He, and X. E. Wang. Jarvis: A neuro-symbolic commonsense reasoning framework for conversational embodied agents. arXiv preprint arXiv:2208.13266, 2022

2022 arXiv
[37]

D. Shah, B. Osi \'n ski, S. Levine, et al. Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action. In Conference on robot learning, pages 492--504. PMLR, 2023

2023
[38]

Huang, C

W. Huang, C. Wang, R. Zhang, Y. Li, J. Wu, and L. Fei-Fei. Voxposer: Composable 3d value maps for robotic manipulation with language models. arXiv preprint arXiv:2307.05973, 2023

2023 arXiv
[39]

Driess, F

D. Driess, F. Xia, M. S. M. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu, W. Huang, Y. Chebotar, P. Sermanet, D. Duckworth, S. Levine, V. Vanhoucke, K. Hausman, M. Toussaint, K. Greff, A. Zeng, I. Mordatch, and P. Florence. Palm-e: an embod...

2023
[40]

J. Gao, B. Sarkar, F. Xia, T. Xiao, J. Wu, B. Ichter, A. Majumdar, and D. Sadigh. Physically grounded vision-language models for robotic manipulation. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 12462--12469. IEEE, 2024

2024
[41]

X. Li, M. Zhang, Y. Geng, H. Geng, Y. Long, Y. Shen, R. Zhang, J. Liu, and H. Dong. Manipllm: Embodied multimodal large language model for object-centric robotic manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18061--18...

2024
[42]

Jiang, A

Y. Jiang, A. Gupta, Z. Zhang, G. Wang, Y. Dou, Y. Chen, L. Fei-Fei, A. Anandkumar, Y. Zhu, and L. Fan. Vima: General robot manipulation with multimodal prompts. arXiv preprint arXiv:2210.03094, 2 0 (3): 0 6, 2022

2022 arXiv
[43]

Nayak, A

S. Nayak, A. Morrison Orozco, M. Have, J. Zhang, V. Thirumalai, D. Chen, A. Kapoor, E. Robinson, K. Gopalakrishnan, J. Harrison, et al. Long-horizon planning for multi-agent robots in partially observable environments. Advances in Neural Information Processing Systems, 37: 0 6...

2024
[44]

G. Wang, Y. Xie, Y. Jiang, A. Mandlekar, C. Xiao, Y. Zhu, L. Fan, and A. Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291, 2023

2023 arXiv
[45]

X. Liu, A. Pesaranghader, H. Li, P. Sukcharoenchaikul, J. Kim, T. Sadhu, H. Jeon, and S. Sanner. Open-world planning via lifted regression with llm-inferred affordances for embodied agents. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguist...

2025
[46]

L. Wang, Y. Ling, Z. Yuan, M. Shridhar, C. Bao, Y. Qin, B. Wang, H. Xu, and X. Wang. Gensim: Generating robotic simulation tasks via large language models. arXiv preprint arXiv:2310.01361, 2023

2023 arXiv
[47]

C. H. Song, J. Wu, C. Washington, B. M. Sadler, W.-L. Chao, and Y. Su. Llm-planner: Few-shot grounded planning for embodied agents with large language models. In Proceedings of the IEEE/CVF international conference on computer vision, pages 2998--3009, 2023

2023
[48]

W. Yu, N. Gileadi, C. Fu, S. Kirmani, K.-H. Lee, M. G. Arenas, H.-T. L. Chiang, T. Erez, L. Hasenclever, J. Humplik, et al. Language to rewards for robotic skill synthesis. arXiv preprint arXiv:2306.08647, 2023

2023 arXiv
[49]

J. Wu, R. Antonova, A. Kan, M. Lepert, A. Zeng, S. Song, J. Bohg, S. Rusinkiewicz, and T. Funkhouser. Tidybot: Personalized robot assistance with large language models. Autonomous Robots, 47 0 (8): 0 1087--1102, 2023

2023
[50]

Y. Tang, W. Yu, J. Tan, H. Zen, A. Faust, and T. Harada. Saytap: Language to quadrupedal locomotion. arXiv preprint arXiv:2306.07580, 2023

2023 arXiv
[51]

Liang, W

J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng. Code as policies: Language model programs for embodied control. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 9493--9500. IEEE, 2023

2023
[52]

H. Ha, P. Florence, and S. Song. Scaling up and distilling down: Language-guided robot skill acquisition. In Conference on Robot Learning, pages 3766--3777. PMLR, 2023

2023
[53]

Singh, V

I. Singh, V. Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg. Progprompt: Generating situated robot task plans using large language models. arXiv preprint arXiv:2209.11302, 2022

2022 arXiv
[54]

S. H. Vemprala, R. Bonatti, A. Bucker, and A. Kapoor. Chatgpt for robotics: Design principles and model abilities. Ieee Access, 12: 0 55682--55696, 2024

2024
[55]

Dasgupta, C

I. Dasgupta, C. Kaeser-Chen, K. Marino, A. Ahuja, S. Babayan, F. Hill, and R. Fergus. Collaborating with language models for embodied reasoning. arXiv preprint arXiv:2302.00763, 2023

2023 arXiv
[56]

Mandi, S

Z. Mandi, S. Jain, and S. Song. Roco: Dialectic multi-robot collaboration with large language models. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 286--299. IEEE, 2024

2024
[57]

A. Z. Ren, A. Dixit, A. Bodrova, S. Singh, S. Tu, N. Brown, P. Xu, L. Takayama, F. Xia, J. Varley, et al. Robots that ask for help: Uncertainty alignment for large language model planners. arXiv preprint arXiv:2307.01928, 2023

2023 arXiv
[58]

P. Yuan, A. Ma, Y. Yao, H. Yao, M. Tomizuka, and M. Ding. Remac: Self-reflective and self-evolving multi-agent collaboration for long-horizon robot manipulation. arXiv preprint arXiv:2503.22122, 2025

2025 arXiv
[59]

Z. Liu, A. Bahety, and S. Song. Reflect: Summarizing robot experiences for failure explanation and correction. arXiv preprint arXiv:2306.15724, 2023

2023 arXiv
[60]

D. M. Ritchie and K. Thompson. The unix time-sharing system. Communications of the ACM, 17 0 (7): 0 365--375, 1974

1974
[61]

Zhu and Y

M. Zhu and Y. Zhou. Moss: Enabling code-driven evolution and context management for ai agents. arXiv preprint arXiv:2409.16120, 2024

2024 arXiv
[62]

Sennrich, B

R. Sennrich, B. Haddow, and A. Birch. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909, 2015

2015 arXiv
[63]

J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35: 0 24824--24837, 2022

2022
[64]

D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

2025 arXiv
[65]

Zhang, Q

K. Zhang, Q. Yao, B. Lai, J. Huang, W. Fang, D. Tao, M. Song, and S. Liu. Reasoning with reinforced functional token tuning. arXiv preprint arXiv:2502.13389, 2025

2025 arXiv
[66]

S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023

2023
[67]

W. Fan, Y. Ding, L. Ning, S. Wang, H. Li, D. Yin, T.-S. Chua, and Q. Li. A survey on rag meeting llms: Towards retrieval-augmented large language models. In Proceedings of the 30th ACM SIGKDD conference on knowledge discovery and data mining, pages 6491--6501, 2024

2024
[68]

Realtime api

OpenAI . Realtime api. https://platform.openai.com/docs/guides/realtime, 2025

2025
[69]

S. Liu, J. Chen, S. Ruan, H. Su, and Z. Yin. Exploring the robustness of decision-level through adversarial attacks on llm-based embodied models. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 8120--8128, 2024

2024
[70]

A. Liu, Y. Zhou, X. Liu, T. Zhang, S. Liang, J. Wang, Y. Pu, T. Li, J. Zhang, W. Zhou, et al. Compromising llm driven embodied agents with contextual backdoor attacks. IEEE Transactions on Information Forensics and Security, 2025

2025

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [1]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

[2] [2]

Goertzel

B. Goertzel. Artificial general intelligence: Concept, state of the art, and future prospects. Journal of Artificial General Intelligence, 5 0 (1): 0 1, 2014

work page 2014

[3] [3]

M. R. Morris, J. Sohl-Dickstein, N. Fiedel, T. Warkentin, A. Dafoe, A. Faust, C. Farabet, and S. Legg. Levels of agi for operationalizing progress on the path to agi. arXiv preprint arXiv:2311.02462, 2023

arXiv 2023

[4] [4]

Bubeck, V

S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712, 2023

arXiv 2023

[5] [5]

What is Multimodality?

L. Parcalabescu, N. Trost, and A. Frank. What is multimodality? arXiv preprint arXiv:2103.06304, 2021

work page Pith review arXiv 2021

[6] [6]

J. Lin, R. Men, A. Yang, C. Zhou, Y. Zhang, P. Wang, J. Zhou, J. Tang, and H. Yang. M6: Multi-modality-to-multi-modality multitask mega-transformer for unified pretraining. In Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining, pages 3251--3261, 2021

work page 2021

[7] [7]

T. B. Sheridan. Human--robot interaction: status and challenges. Human factors, 58 0 (4): 0 525--532, 2016

work page 2016

[8] [8]

Bartneck, T

C. Bartneck, T. Belpaeme, F. Eyssel, T. Kanda, M. Keijsers, and S. S abanovi \'c . Human-robot interaction: An introduction. Cambridge University Press, 2024

work page 2024