Pith. sign in

REVIEW 4 major objections 6 minor 29 references

Embodied LLM agents need an OS-style runtime, not a request/response loop around the model.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 07:34 UTC pith:YASY6RMC

load-bearing objection Solid early systems prototype: OS-style multi-cadence runtime for embodied LLM agents with real Go2 latency/concurrency numbers, scoped honestly as preliminary. the 4 major comments →

arxiv 2607.05482 v1 pith:YASY6RMC submitted 2026-07-06 cs.SE cs.RO

TypeGo: An OS Runtime for Embodied Agents

Guojun Chen , Alex Schott , Lin Zhong This is my paper
classification cs.SE cs.RO
keywords embodied agentsLLM planningoperating system runtimeskill kernelmulti-cadence planningspeculative skill streamingrobot concurrencynatural language prescriptions
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Large language models can turn natural language into robot plans, but sitting the model on the critical path as a request/response oracle fights real-time control and concurrent goals. This paper argues that LLM-based control of an embodied agent should instead run as a continuously executing asynchronous runtime, structured like a small operating system for the robot. TypeGo realizes that idea: multi-cadence planning loops overlap with execution, a Skill Kernel arbitrates typed body subsystems among concurrent task processes, a semantic scheduler preempts and resumes by source, and speculative skill streaming plus a fast first-action path hide model latency. Users author natural-language prescriptions that become either online goals or compiled low-latency reflexes. On a Unitree Go2 prototype task suite, the design cuts per-step delay by half versus step-by-step planning and time-to-first-action by about three-quarters versus monolithic planning, while admitting concurrent tasks and reflexes at modest overhead. The point is architectural: hide the LLM rather than wait on it, and treat the body like hardware the runtime must schedule safely.

Core claim

LLM-based control of embodied agents should not be a request/response loop with the model on the critical path; it should be a continuously executing, asynchronous OS-style runtime that overlaps multi-timescale planning with execution, arbitrates physical subsystems among concurrent processes, and hides inference latency behind ongoing motion and a fast first-action path. TypeGo instantiates that thesis and, on the authors’ current Go2 task suite, delivers roughly 50% lower per-step delay than ReAct-style planning and 73% lower time-to-first-action than plan-and-execute, while supporting concurrency at low scheduling cost.

What carries the argument

TypeGo: four concurrent agent loops (S0–S3) over a Skill Kernel that mediates typed exclusive/shared body subsystems among per-task processes, with source-dependent preemption/resumption and speculative skill streaming into a bounded step queue so planning latency is hidden behind motion.

Load-bearing premise

That gains measured on a small hand-authored skill set and a short eight-task prototype suite will still hold in richer open-world failures, denser multi-task contention, and robots with more exclusive actuators.

What would settle it

On a larger suite with denser concurrent goals and richer exclusive subsystems, measure whether TypeGo still cuts per-step delay ~50% versus re-plan-every-step and time-to-first-action ~73% versus one-shot plans without unsafe preemption or large scheduling overhead.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper argues that LLM-based control of embodied agents should be a continuously executing, multi-cadence OS-style runtime rather than a request/response loop around the model. It instantiates this in TypeGo: four asynchronous agent loops (S0–S3), a Skill Kernel that arbitrates typed exclusive/shared actuator subsystems among per-task processes, source-dependent preemption/resumption, speculative skill streaming with a bounded step queue, a fast first-action path (S1P), and natural-language prescriptions (online tasks plus offline-compiled S0 reflexes). A Unitree Go2 prototype (Kalos) is evaluated on a small task suite (T1–T8). Relative to ReAct-style step-by-step planning and monolithic plan-and-execute, TypeGo reports ~50% lower per-step delay, ~73% lower TTFA, competitive success, and low concurrency overhead on contended, parallel, and S0-reflex paths, at substantially higher token cost.

Significance. If the design holds beyond the prototype suite, the contribution is a concrete systems architecture for concurrent, interruptible, latency-hiding LLM control of physical robots—addressing three problems the LLM-robotics literature largely sidesteps: real-time reaction under slow inference, multi-task actuator arbitration, and authoring both goals and fast reactive rules in natural language. Strengths include a clear OS analogy with explicit departures (semantic scheduling, source-dependent resume/replace, bounded-interruption skills), honest failure discussion (blocked path T3, greedy local bias on T4), wall-clock metrics rather than circular fitted quantities, and measured concurrency overhead broken down by contention path (Tables 3–4). The work is early and scoped as preliminary; its value is architectural and empirical direction-setting rather than a finished agent OS.

major comments (4)
  1. §8.1 and Table 3: TTFA is a headline claim (0.87 s vs 1.46/3.27 s; 73% below PAE). TypeGo’s S1P fast path uses Groq-hosted GPT-OSS-120B while all other TypeGo layers and both baselines use GPT-5.4. The architecture intentionally tiers models, but without a same-model S1P control (or a ReAct/PAE first-action path on the same fast model) it is unclear how much of the TTFA gain is multi-cadence design versus model/host latency. A controlled ablation is load-bearing for attributing the 73% figure to the runtime.
  2. §8 (effectiveness/responsiveness/concurrency): There are no ablations that isolate speculative streaming (queue size three), S2 re-decomposition, S0 compiled reflexes, or S3 source-dependent scheduling. Table 3 therefore shows an end-to-end system win over ReAct/PAE, not which mechanism drives the 50% per-step reduction. For an OS-runtime thesis, at least streaming-on/off and S1P-on/off ablations on the same suite are needed so the central latency claims can be attributed.
  3. §8.2 and Table 4: Concurrency evidence is three scenarios, all injected onto base task T5 (user interrupt-and-return, parallel non-contending query, one-shot S0 reflex). True scheduling overhead after S1 replan is small (~0.05–0.22 s), but denser multi-process contention, multiple exclusive subsystems, and replace-without-return under competing user goals are not measured. The Skill Kernel and source-dependent policy are central design claims; the current suite under-tests them relative to the architecture’s scope.
  4. §7–§8 free parameters (S1 queue size, S0/S2/S3 rates, interruption bound t_s, S1P model choice) are fixed without sensitivity analysis. The paper’s own future-work section flags token cost and cost-aware policy; given TypeGo’s much higher normalized token use in Table 3, the responsiveness–cost tradeoff should be quantified under at least one cheaper policy (e.g., event-triggered S2, smaller queue) so the prototype’s gains are not tied only to the most expensive configuration.
minor comments (6)
  1. §6 / Listing 3: Branch conditions mix observation predicates and short NL expressions; state how NL conditions are evaluated at runtime (LLM call vs compiled predicate) and whether that path can stall the executor.
  2. Table 1 and §3: Memory is listed as an OS analogue and as future work; the prototype’s query_memory skill is easy to over-read as a full memory hierarchy. Clarify in the overview what is implemented vs aspirational.
  3. §8.1: Report absolute (not only normalized) total completion times and raw token counts alongside Table 3 so readers can interpret the responsiveness–cost tradeoff without renormalization ambiguity.
  4. §2: Related work on hierarchical control and dual-process planners is appropriate; a brief note on classical robot middleware (e.g., behavior trees, ROS action servers) would situate the Skill Kernel for systems readers outside LLM robotics.
  5. Figure 1: The diagram is dense; a short caption walk-through of one process lifecycle (create → S2/S1 stream → Skill Kernel acquire → S3 pause/resume) would help first-time readers.
  6. Typos/clarity: Abstract and §1 use “anoperating-system-style” / spacing glitches in the provided text; “hided” (§1 mechanisms) should be “hidden”; ensure consistent naming of S1P vs “fast first-action path” across §6 and §8.

Circularity Check

0 steps flagged

No significant circularity: empirical systems prototype with measured wall-clock metrics, not definitional or fitted predictions.

full rationale

TypeGo is an early systems paper whose load-bearing claims are architectural (multi-cadence S0–S3 loops, Skill Kernel arbitration, source-dependent resumption, speculative S1 queue) and empirical (Tables 3–4: TTFA, per-step delay, concurrency overhead on T1–T8). These quantities are wall-clock measurements against ReAct and plan-and-execute baselines that share the same skill library and model; they are not parameters fitted to data and then re-presented as predictions, nor quantities defined in terms of the claimed outputs. The TypeFly self-citation is used only as prior inspiration for stream-interpretation and is explicitly scoped as non-central (“TypeGo’s contribution is to embed it in a multi-cadence runtime”); it does not force the multi-cadence, Skill Kernel, or concurrency results. Classical three-layer control and OS analogies are framing, not uniqueness theorems imported to forbid alternatives. No self-definitional equations, fitted-input-as-prediction, uniqueness-from-authors, or ansatz-smuggling chain appears. Scope limits (preliminary suite, token cost, T4 greedy bias) are disclosed rather than circularly assumed away. Circularity score is therefore zero.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 3 invented entities

Load-bearing content is mostly engineering design choices and domain assumptions about interruptible skills and LLM planners, not fitted physical constants. Free parameters are implementation knobs (queue depth, loop rates, model routing). Invented entities are software abstractions, not new physical objects; independent evidence is the prototype measurements, which remain narrow.

free parameters (4)
  • S1 speculative step queue size
    Bounded look-ahead set to three; directly affects latency-hiding vs adaptivity tradeoff reported in evaluation.
  • S0 / S2 / S3 loop rates
    S0 at 100 Hz, S2 and S3 at 0.5 Hz chosen by hand; shape responsiveness and overhead numbers.
  • S1P fast-path model choice (Groq GPT-OSS-120B) vs GPT-5.4 elsewhere
    Model routing is a free systems choice that drives the sub-second TTFA claim and is not derived from theory.
  • Skill interruption bound t_s
    Bounded preemption time is required for clean scheduling but left as a skill-authoring parameter rather than measured across the library.
axioms (5)
  • domain assumption LLM inference is too slow for closed-loop real-time control if left on the critical path, so latency must be hidden by architecture rather than only by faster models.
    Stated in §1 as a fundamental latency–quality tradeoff motivating the runtime thesis.
  • domain assumption Developer-authored skills can be made interruptible within a bounded time and correctly declare exclusive subsystems.
    Skill Kernel preemption and concurrency safety in §4–§5 depend on this authoring discipline.
  • ad hoc to paper Source-dependent resume/replace (user vs reactive vs idle) is a better policy basis than numeric priorities for embodied multi-tasking.
    Core scheduling design in §5.2; not derived from classical OS theory, chosen for robot semantics.
  • domain assumption Natural-language reflexes can be offline-compiled by an LLM into correct high-frequency Python condition/action handlers.
    S0 path in §4 and §6 assumes compilation fidelity without LLM on the reaction path.
  • domain assumption Classical process/kernel/device analogies remain useful when resources are physical actuators with safety consequences.
    Guiding analogy of §3; authors note departures but still structure claims around it.
invented entities (3)
  • Skill Kernel with typed exclusive/shared subsystems no independent evidence
    purpose: Arbitrate physical actuators among concurrent processes like an OS kernel.
    Software abstraction introduced by the paper; evidence is prototype behavior, not independent external measurement.
  • Four concurrent cadence agents S0–S3 with source-dependent process lifecycle no independent evidence
    purpose: Overlap planning timescales and schedule multi-task embodied behavior.
    Architectural invention; evaluated only on the authors' small suite.
  • Natural-language prescriptions (tasks + offline-compiled reflexes) no independent evidence
    purpose: Let non-programmers author both goals and fast reactions.
    Interface abstraction; correctness depends on LLM compilation and planner routing.

pith-pipeline@v1.1.0-grok45 · 17297 in / 3504 out tokens · 32857 ms · 2026-07-11T07:34:50.429824+00:00 · methodology

0 comments
read the original abstract

Large language models (LLMs) can plan behavior for embodied agents from natural language, but treating the LLM as a request/response oracle on the critical path is fundamentally at odds with real-time control and concurrent goals. We argue for an operating-system-style runtime for embodied agents, and instantiate this idea in an early prototype, TypeGo. TypeGo structures LLM-based planning as asynchronous loops at multiple timescales that overlap with execution, and manages the agent's physical body like an OS manages hardware: the Skill Kernel arbitrates typed physical subsystems among concurrent per-task processes, a scheduler preempts them and resumes or replaces each by source, and speculative skill streaming hides LLM latency behind ongoing motion, while a fast first-action path yields visible feedback within a second. Users program behavior through natural language prescriptions that TypeGo dispatches to the LLM-based planners or compiles into low-latency interrupt handlers. Our prototype of Kalos, a Unitree Go2 quadruped, provides preliminary evidence for the design: in our current task suite, it cuts per-step delay by 50% over step-by-step planning and time-to-first-action by 73% over monolithic planning, while admitting concurrent tasks at low scheduling overhead.

Figures

Figures reproduced from arXiv: 2607.05482 by Alex Schott, Guojun Chen, Lin Zhong.

Figure 1
Figure 1. Figure 1: TypeGo runtime. A hierarchy of agents (blue) includes the global S3 (process scheduler) and S0 (reflexive layer), and the process-scoped S2 (task decomposer) and S1 (action planner). S3 manages the process lifecycle based on user input or its own decisions. S2 and S1 together turn a single task into a continuous stream of skill calls. The Skill Kernel arbitrates and schedules skill calls issued by multiple… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

29 extracted references · 11 linked inside Pith

  1. [1]

    TypeFly: Low-latency drone planning with large language models,

    G. Chen, X. Yu, N. Ling, and L. Zhong, “TypeFly: Low-latency drone planning with large language models, ”IEEE Transactions on Mobile Computing, 2025

    2025

  2. [2]

    Code as policies: Language model programs for embodied control,

    J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng, “Code as policies: Language model programs for embodied control, ” 2022, arXiv:2209.07753

    Pith/arXiv arXiv 2022

  3. [3]

    Do as I can, not as I say: Grounding language in robotic affordances,

    M. Ahnet al., “Do as I can, not as I say: Grounding language in robotic affordances, ” inConference on Robot Learning (CoRL), 2022

    2022

  4. [4]

    Voxposer: Composable 3d value maps for robotic manipulation with language models,

    W. Huang, C. Wang, R. Zhang, Y. Li, J. Wu, and L. Fei-Fei, “Voxposer: Composable 3d value maps for robotic manipulation with language models, ” inProceedings of the 7th Conference on Robot Learning (CoRL), 2023, arXiv:2307.05973

    Pith/arXiv arXiv 2023

  5. [5]

    Homerobot: Open-vocabulary mobile manipulation,

    S. Yenamandra, A. Ramachandran, K. Yadav, A. S. Wang, M. Khanna, T. Gervet, T.- Y. Yang, V. Jain, A. Clegg, J. M. Turner, Z. Kira, M. Savva, A. X. Chang, D. S. Chap- lot, D. Batra, R. Mottaghi, Y. Bisk, and C. Paxton, “Homerobot: Open-vocabulary mobile manipulation, ” inProceedings of the 7th Conference on Robot Learning (CoRL), ser. Proceedings of Machin...

    Pith/arXiv arXiv 2023

  6. [6]

    Demonstrating ok- robot: What really matters in integrating open-knowledge models for robotics,

    P. Liu, Y. Orru, J. Vakil, C. Paxton, N. Shafiullah, and L. Pinto, “Demonstrating ok- robot: What really matters in integrating open-knowledge models for robotics, ” inRobotics: Science and Systems XX, 2024

    2024

  7. [7]

    Bumble: Unifying reason- ing and acting with vision-language models for building-wide mobile manipula- tion,

    R. Shah, A. Yu, Y. Zhu, Y. Zhu, and R. Martín-Martín, “Bumble: Unifying reason- ing and acting with vision-language models for building-wide mobile manipula- tion, ” inIEEE International Conference on Robotics and Automation (ICRA), 2025, arXiv:2410.06237

    Pith/arXiv arXiv 2025

  8. [8]

    On three-layer architectures,

    E. Gat, “On three-layer architectures, ” inArtificial Intelligence and Mobile Robots: Case Studies of Successful Robot Systems, D. Kortenkamp, R. P. Bonasso, and R. Murphy, Eds. AAAI Press, 1998

    1998

  9. [9]

    Kahneman,Thinking, Fast and Slow

    D. Kahneman,Thinking, Fast and Slow. New York: Farrar, Straus and Giroux, 2011

    2011

  10. [10]

    System- 1.x: Learning to balance fast and slow planning with language models,

    S. Saha, A. Prasad, J. C.-Y. Chen, P. Hase, E. Stengel-Eskin, and M. Bansal, “System- 1.x: Learning to balance fast and slow planning with language models, ” 2025, arXiv:2407.14414

    Pith/arXiv arXiv 2025

  11. [11]

    Reasoning on a spectrum: Aligning llms to system 1 and system 2 thinking,

    A. S. Ziabari, N. Ghazizadeh, Z. Sourati, F. Karimi-Malekabadi, P. Piray, and M. Dehghani, “Reasoning on a spectrum: Aligning llms to system 1 and system 2 thinking, ” 2025, arXiv:2502.12470

    arXiv 2025

  12. [12]

    Language-conditioned robotic manipulation with fast and slow thinking,

    M. Zhu, Y. Zhu, J. Li, J. Wen, Z. Xu, Z. Che, C. Shen, Y. Peng, D. Liu, F. Feng, and J. Tang, “Language-conditioned robotic manipulation with fast and slow thinking, ” 2024, arXiv:2401.04181

    Pith/arXiv arXiv 2024

  13. [13]

    Hierarchical reasoning model,

    G. Wang, J. Li, Y. Sun, X. Chen, C. Liu, Y. Wu, M. Lu, S. Song, and Y. A. Yadkori, “Hierarchical reasoning model, ” 2025. [Online]. Available: https://arxiv.org/abs/2506.21734

    Pith/arXiv arXiv 2025

  14. [14]

    Large language models for robotics: A survey,

    F. Zeng, W. Gan, Y. Wang, N. Liu, and P. S. Yu, “Large language models for robotics: A survey, ” 2023, arXiv:2311.07226

    arXiv 2023

  15. [15]

    A survey of robot intelligence with large language models,

    H. Jeong, H. Lee, C. Kim, and S. Shin, “A survey of robot intelligence with large language models, ”Applied Sciences, vol. 14, no. 19, p. 8868, 2024

    2024

  16. [16]

    Inner monologue: Embodied reasoning through planning with language models,

    W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang, P. Florence, A. Zeng, J. Tompson, I. Mordatch, Y. Chebotar, P. Sermanet, T. Jackson, N. Brown, L. Luu, S. Levine, K. Hausman, and B. Ichter, “Inner monologue: Embodied reasoning through planning with language models, ” inConference on Robot Learning (CoRL), 2022

    2022

  17. [17]

    ReAct: Synergizing reasoning and acting in language models,

    S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao, “ReAct: Synergizing reasoning and acting in language models, ” inProc. International Conference on Learning Representations (ICLR), 2023

    2023

  18. [18]

    RT-2: Vision-language-action models transfer web knowledge to robotic control,

    A. Brohanet al., “RT-2: Vision-language-action models transfer web knowledge to robotic control, ” inConference on Robot Learning (CoRL), 2023

    2023

  19. [19]

    LLM augmented hierarchical agents,

    B. Prakash, T. Oates, and T. Mohsenin, “LLM augmented hierarchical agents, ” in CoRL 2023 Workshop on Language and Robot Learning (LangRob), 2023

    2023

  20. [20]

    Learning hierarchical domain models through environment-grounded interaction,

    C. Kienle, B. Alt, O. Arenz, and J. Peters, “Learning hierarchical domain models through environment-grounded interaction, ” 2025, arXiv:2505.13497

    arXiv 2025

  21. [21]

    Mixture-of-agents enhances large language model capabilities,

    J. Wang, J. Wang, B. Athiwaratkun, C. Zhang, and J. Zou, “Mixture-of-agents enhances large language model capabilities, ” 2024, arXiv:2406.04692

    Pith/arXiv arXiv 2024

  22. [22]

    AIOS: LLM agent operating system,

    K. Mei, Z. Li, S. Xu, R. Ye, Y. Ge, and Y. Zhang, “AIOS: LLM agent operating system, ” 2024, arXiv:2403.16971

    Pith/arXiv arXiv 2024

  23. [23]

    MemGPT: Towards LLMs as operating systems,

    C. Packer, S. Wooders, K. Lin, V. Fang, S. G. Patil, I. Stoica, and J. E. Gonzalez, “MemGPT: Towards LLMs as operating systems, ” 2023, arXiv:2310.08560

    Pith/arXiv arXiv 2023

  24. [24]

    OpenCat-Quadruped-Robot: An open source quadruped robot pet framework,

    PetoiCamp, R. Li, and contributors, “OpenCat-Quadruped-Robot: An open source quadruped robot pet framework, ” GitHub, 2021

    2021

  25. [25]

    You only look once: Unified, real-time object detection,

    J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: Unified, real-time object detection, ” inProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 779–788

    2016

  26. [26]

    OmDet: Large-scale vision-language multi-dataset pre-training with multimodal detection network,

    T. Zhao, P. Liu, and K. Lee, “OmDet: Large-scale vision-language multi-dataset pre-training with multimodal detection network, ”IET Computer Vision, 2024

    2024

  27. [27]

    OpenCLIP,

    G. Ilharco, M. Wortsman, R. Wightman, C. Gordon, N. Carlini, R. Taori, A. Dave, V. Shankar, H. Namkoong, J. Miller, H. Hajishirzi, A. Farhadi, and L. Schmidt, “OpenCLIP, ” Zenodo, Jul. 2021, doi:10.5281/zenodo.5143773

    doi:10.5281/zenodo.5143773 2021

  28. [28]

    Why reasoning fails to plan: A planning-centric analysis of long-horizon decision making in llm agents,

    Z. Wang, F. Wu, H. Wang, X. Tang, B. Li, Z. Yin, Y. Ma, Y. Li, W. Sun, X. Chen, and Y. Ye, “Why reasoning fails to plan: A planning-centric analysis of long-horizon decision making in llm agents, ” 2026, arXiv:2601.22311

    arXiv 2026

  29. [29]

    How far ahead do llms plan? uncovering the latent horizon in chain-of-thought reasoning,

    L. Xu, M. Yu, F. Meng, and J. Zhou, “How far ahead do llms plan? uncovering the latent horizon in chain-of-thought reasoning, ” 2026, arXiv:2602.02103. 7

    Pith/arXiv arXiv 2026