Pith. sign in

REVIEW 4 major objections 5 minor 28 references

Visual Language Models as Operator Agents in the Space Domain

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Pith's one-line read Vision-language models can act as spacecraft operators from GUI screenshots and, after light fine-tuning, guide a robot arm through space-hardware inspection.

desk verdict VLM-as-operator is a plausible new idea, but the claimed comparison to traditional methods rests on baselines that likely come from a different task family, so the abstract overstates what the data show. read the letter →

arxiv 2501.07802 v1 pith:MFB3GKGB submitted 2025-01-14 cs.AI physics.space-ph

classification cs.AIphysics.space-ph
keywords vision-languagemodelsautonomousspacecraftcontrolKerbalSpaceProgramLLMagentsroboticinspectionfine-tuningmultimodalreasoningoperations
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to show that a single class of model—vision-language models—can play the role of an operator across both the software and hardware sides of space missions. In the Kerbal Space Program Differential Games simulator, the authors equip a VLM with screenshots of the game's dashboard plus textual telemetry, and report that the resulting agent completes orbital approach and guard-evasion tasks with better overall task scores than most classical control baselines and text-only LLMs. On the hardware side, they fine-tune a vision-language-action model on ten episodes of camera data and report that it learns to generate position and rotation commands for inspecting a satellite-like object with a robotic arm. If these results hold, mission control and on-orbit servicing could be handled by one multimodal agent that reads the same screens and camera feeds a human operator would use, rather than by separately engineered perception and control stacks.

What carries the argument

The machinery is a multimodal agent loop in which visual frames and text share one prompt. For the software task, the KSPDG dashboard screenshot carries the navball, orientation, and mission markers; the VLM combines it with relative-position and prograde telemetry, reasons in natural language, and emits a structured function call selecting one of 27 discretized throttle combinations (left/right, forward/backward, up/down, plus no-thrust). For the hardware task, a camera frame and robot state are passed to a VLM that outputs a positional change $\Delta x$, a rotational adjustment $\Delta \theta$, and a boolean capture flag $\chi$, while a system prompt supplies the inspection goal; OpenVLA is fine-tuned as the action head using an HDF5 dataset of shuffled frame-episodes.

What would settle it

Run each agent in Tables 1 and 2 for dozens of seeded episodes and report score distributions; if the VLM scores overlap the classical and text-only baselines within error bars, the claimed advantage is not real. For the hardware claim, fine-tune with the recommended 50 episodes and measure closed-loop inspection success on unseen satellite mock-ups; if accuracy plateaus near the current validation levels, the promised real-world readiness does not follow.

Watch

Extended reading notes

Core claim

The central claim is that VLMs can act as end-to-end operator agents in the space domain by mapping visual observations directly to control actions. In the KSPDG Lady-Bandit-Guard scenarios, agents built from ChatGPT, Claude, and LLaMA received the in-game dashboard as an image alongside a few lines of telemetry and a few-shot prompt, and produced throttling actions through a discrete 27-combination action space; the best VLM agent reached a closest approach of 4.76 m and an average task score of 3,961, below the best text-only LLM score of 18,938, while classical baselines such as PPO, iLQGames, and Lambert-MPC were outperformed on the reported metrics. For hardware operations, the authors fine-tune OpenVLA, a vision-language-action model, on only 10 teleoperated episodes of satellite inspection with a 7-DoF arm, obtaining training accuracies of 70–83% and validation accuracies of 58.9–69.5% with roughly 50 mm average action deviation, and read this as evidence that a VLA model can be adapted to inspection tasks with a fraction of the recommended data.

Load-bearing premise

The results stand on a small number of simulation episodes and only ten fine-tuning examples, so the reported gaps could be run-to-run noise rather than a genuine advantage for vision-language models.

Editorial extensions

If this is right

  • A VLM spacecraft operator needs only the same GUI screenshot a human operator sees, not hand-engineered state vectors, so the same agent design can transfer across simulation interfaces.
  • Because VLMs balanced approach and guard-evasion better than text-only LLMs, visual cues such as the navball appear to carry spatial information that plain telemetry prompts lack.
  • The hardware result suggests a vision-language-action model can be repurposed for a new satellite-inspection skill with roughly one-fifth of the recommended training episodes, lowering the data cost of deploying such agents.
  • Latency, not decision quality, is the current bottleneck in simulated operation; faster inference or output caching would be the lever that makes these agents usable in real-time missions.
  • The same end-to-end framework covers both software control and hardware inspection, so improvements in VLM visual reasoning should improve both paradigms together.
  • Beyond the paper's own claims, the same screenshot-plus-telemetry prompt should carry over to any graphical mission-control console, meaning a VLM could supervise a legacy ground station without changing the interface.
  • The ten-episode fine-tuning result is a proof of concept, not a deployment result; the natural next test is scaling to the recommended 50 episodes and measuring closed-loop inspection success, which would show whether the reported accuracy growth continues.
  • Because the fastest simulator scores came with high latency, an implication the authors leave implicit is that distilling the VLM into a smaller, faster model may be the practical route to real-time operation even if per-step accuracy drops.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper describes two empirical deployments of Vision-Language Models (VLMs): (1) using GPT-4o, Claude 3.5, and LLaMA 3.2 as operator agents that read GUI screenshots and telemetry in the Kerbal Space Program Differential Games (KSPDG) environment, and (2) fine-tuning OpenVLA on ten teleoperated episodes to control an xArm 7 robot for satellite inspection. The central claim, stated in the abstract, is that VLMs can compete with traditional methods and non-multimodal LLMs in simulation, and show promise in real-world applications. The paper includes comparative tables of distances and scores, latency measurements, and fine-tuning loss/accuracy metrics, and frames the work as preliminary.

Significance. If the central comparison were valid, the paper would be a useful demonstration that a single VLM can perform closed-loop control of a simulated spacecraft from raw GUI screenshots and can, after light fine-tuning, propose inspection actions for a robot arm. The authors are transparent about the preliminary nature of the hardware experiments and about the latency bottleneck, and they correctly cite the BALROG benchmark's finding that LLMs currently outperform VLMs on many reasoning tasks. However, the load-bearing quantitative comparison to traditional methods rests on Table 1, which is internally inconsistent and does not currently support the abstract's claim.

major comments (4)
  1. [Section III.C, Table 1] The comparison with traditional methods is not established as reported. The rows for Naive, PPO, iLQGames, and Lambert-MPC are labeled as extracted from the SpaceGym paper, but they report only Best/Avg distance and leave the Avg. to Guard and Avg. Score columns empty, whereas the seven LLM/VLM rows report all four columns and are described by the LBG score from Eq. (1). This strongly suggests the traditional baselines were evaluated in the Pursuer-Evader scenario family rather than the Lady-Bandit-Guard scenarios (lg0-lg3, i1-i2) used for the model rows. If these numbers were not produced on the same scenario identifiers with the same scoring function, the statement in Section III.C that the agents 'outperform most traditional methods' does not follow. The authors should either rerun the traditional methods on the identical LBG scenarios or clearly restrict the claim to a scenario-matched subset.
  2. [Table 2] Table 2 contains an arithmetical inconsistency: for Claude VLM the average latency (9,373 ms) exceeds the stated worst latency (7,409 ms), and for LLaMA VLM the average (11,317 ms) exceeds the stated worst (9,664 ms). Since an average cannot exceed the maximum of the sample, at least one set of numbers is erroneous. Additionally, the LLaMA VLM best latency is listed as '3', which is likely a typo (possibly 7,139). The authors should correct the table and report the number of runs, seeds, and full latency distributions.
  3. [Section III.C and IV.B] No run counts, seeds, or standard deviations are reported for Tables 1-4, so the point estimates could be noise. For instance, the ChatGPT VLM and Claude VLM rows in Table 1 show average distances to the Lady of 30.42 m and 52.07 m but scores of 5,626 and 21,488; without multiple episodes per scenario, a reader cannot assess whether these differences are meaningful. The same applies to the fine-tuning results in Section IV.B, where ten episodes are explicitly acknowledged to be far below the recommended fifty. The authors should report episode counts per scenario, repeated runs, and variance, or explicitly characterize the results as illustrative rather than comparative.
  4. [Abstract and Section VI] The abstract claims that VLMs show 'promise in real-world applications', but the only real-world evidence is the ten-episode OpenVLA fine-tuning in Section IV.B, whose validation accuracy is 58.9-69.5% and whose average L1 deviation is about 50 mm, with no baseline comparison and no evaluation on an unseen task. The paper's own text repeatedly labels these results as preliminary, so the abstract's claim is stronger than the evidence. The authors should soften the claim or add a quantitative evaluation that supports it.
minor comments (5)
  1. [Section III.A] The text first says the action space is discretized into 9 distinct actions, then states that each of three axes has three options for a total of 3×3×3=27 permutations; these numbers are inconsistent and should be reconciled.
  2. [Section II.C, Eq. (1)] The displayed formula appears garbled ('dm_lb2+ 𝑎 / dm_bg+𝑏'); the intended expression (dm_lb^2 + a/dm_bg + b) should be typeset correctly.
  3. [Table 2] The best-latency column for LLaMA VLM reads '3' and should be corrected to a plausible value; also consider using consistent decimal places across rows.
  4. [Figure 1 caption] The caption refers to the system as an 'LLM (e.g., ChatGPT)' even though the paper's contribution is the VLM extension; update the caption to reflect the actual agent architecture.
  5. [References] Reference [26] cites a paper on emotional musical prosody, which appears unrelated to the UFactory xArm 7 hardware; please replace it with an appropriate hardware reference.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the performance claims are directly measured in simulation and via train/validation splits, not derived from fitted parameters or self-citations.

full rationale

This is an empirical deployment study rather than a derivation chain. The KSPDG results in Tables 1 and 2 are direct measurements of off-the-shelf and fine-tuned LLM/VLM agents acting in the SpaceGym/KSPDG environment, and the score in Eq. (1) is the challenge-defined evaluation metric, not a parameter fitted to make the claim true. The OpenVLA results in Tables 3 and 4 are training and validation losses/accuracies from a 90-10 split, so the 'promise' claim is grounded in held-out validation numbers rather than by construction. The only self-referential citations are refs. [12]-[14], the authors' prior LLM-agent work; they are cited as design context and do not supply the Table 1-4 numbers. The comparison to traditional methods is attributed to the external SpaceGym paper [23], so even if the baseline scenario matching is questionable, it is not a self-citation loop. I find no equation that reduces to its own input and no fitted quantity renamed as a prediction. Table 2's average-latency values exceeding worst-latency values, and the lack of scenario IDs/seeds for Table 1, are data-quality and reporting concerns, not circularity.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new entities and fits no numerical parameters. Its central claim depends on domain assumptions about VLM visual reliability, the KSPDG scoring function, and the sufficiency of a 10-episode fine-tuning dataset.

assumptions (4)
  • domain assumption VLM visual interpretation of screenshots is reliable enough for control decisions.
    The agent's action is derived from in-game screenshots plus telemetry (Section III.B); if the VLM cannot localize the navball prograde marker, the measured performance is not attributable to visual reasoning.
  • domain assumption The KSPDG scoring function (Eq. 1) is the correct mission objective.
    Scores in Table 1 are computed from closest-approach distances using this environment-provided equation; the paper adopts it without validation against mission success.
  • ad hoc to paper Ten teleoperation episodes are a sufficient basis to conclude the fine-tuned OpenVLA model generalizes.
    Section IV.B explicitly notes that 10 episodes is one-fifth of the recommended 50, yet uses validation accuracy of 69.5 percent to claim the model is not overfitting and is promising.
  • domain assumption The selected models (Claude 3.5, GPT-4o, LLaMA 3.2, OpenVLA) are representative of current VLMs.
    The paper generalizes to 'VLMs' from a convenience sample of API-accessible models (Sections III.C and IV.B).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Visual Language Models as Operator Agents in the Space Domain." pith.science (2026). https://pith.science/paper/MFB3GKGB

@misc{pith2026250107802,
  author       = {Pith},
  title        = {Pith review of: Visual Language Models as Operator Agents in the Space Domain},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MFB3GKGB}},
  note         = {Machine review of arXiv:2501.07802}
}
read the original abstract

This paper explores the application of Vision-Language Models (VLMs) as operator agents in the space domain, focusing on both software and hardware operational paradigms. Building on advances in Large Language Models (LLMs) and their multimodal extensions, we investigate how VLMs can enhance autonomous control and decision-making in space missions. In the software context, we employ VLMs within the Kerbal Space Program Differential Games (KSPDG) simulation environment, enabling the agent to interpret visual screenshots of the graphical user interface to perform complex orbital maneuvers. In the hardware context, we integrate VLMs with robotic systems equipped with cameras to inspect and diagnose physical space objects, such as satellites. Our results demonstrate that VLMs can effectively process visual and textual data to generate contextually appropriate actions, competing with traditional methods and non-multimodal LLMs in simulation tasks, and showing promise in real-world applications.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

28 extracted references · 9 canonical work pages

  1. [1]

    Introducing ChatGPT,

    OpenAI, “Introducing ChatGPT,”https://openai.com/blog/chatgpt, 2022. (Accessed: 11-27-2024)

  2. [2]

    Hello, GPT-4o!

    OpenAI, “Hello, GPT-4o!” , May 2024. URLhttps://openai.com/index/hello-gpt-4o/, accessed: 11-28-2024

  3. [3]

    Llama 3.2: Vision and Edge for Mobile Devices,

    AI, M., “Llama 3.2: Vision and Edge for Mobile Devices,” , September 2024. URLhttps://ai.meta.com/blog/llama- 3-2-connect-2024-vision-edge-mobile-devices/ , accessed: 2024-11-28

  4. [4]

    Introducing Claude 3.5: Faster, safer, smarter, and with a sense of humor,

    Anthropic, “Introducing Claude 3.5: Faster, safer, smarter, and with a sense of humor,” , June 2024. URLhttps: //www.anthropic.com/news/claude-3-5-sonnet, accessed: 2024-12-02

  5. [5]

    OpenVLA: An Open-Source Vision-Language-Action Model,

    Kim, M. J., Pertsch, K., Karamcheti, S., Xiao, T., Balakrishna, A., Nair, S., Rafailov, R., Foster, E., Lam, G., Sanketi, P., Vuong, Q., Kollar, T., Burchfiel, B., Tedrake, R., Sadigh, D., Levine, S., Liang, P., and Finn, C., “OpenVLA: An Open-Source Vision-Language-Action Model,” , 2024. URLhttps://arxiv.org/abs/2406.09246

  6. [6]

    Vision-Language Foundation Models as Effective Robot Imitators,

    Li, X., Liu, M., Zhang, H., Yu, C., Xu, J., Wu, H., Cheang, C., Jing, Y., Zhang, W., Liu, H., Li, H., and Kong, T., “Vision-Language Foundation Models as Effective Robot Imitators,” , 2024. URLhttps://arxiv.org/abs/2311.01378

  7. [7]

    The Rise and Potential of Large Language Model Based Agents: A Survey,

    Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., Zheng, R., Fan, X., Wang, X., Xiong, L., Zhou, Y., Wang, W., Jiang, C., Zou, Y., Liu, X., Yin, Z., Dou, S., Weng, R., Cheng, W., Zhang, Q., Qin, W., Zheng, Y., Qiu, X., Huang, X., and Gui, T., “The Rise and Potential of Large Language Model Based Agents: A Surve...

  8. [8]

    JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Models,

    Wang, Z., Cai, S., Liu, A., Jin, Y., Hou, J., Zhang, B., Lin, H., He, Z., Zheng, Z., Yang, Y., Ma, X., and Liang, Y., “JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Models,” , 2023. URL https://arxiv.org/abs/2311.05997

Show all 28 references
  1. [9]

    Cradle: Empowering Foundation Agents Towards General Computer Control,

    Tan, W., Zhang, W., Xu, X., Xia, H., Ding, Z., Li, B., Zhou, B., Yue, J., Jiang, J., Li, Y., An, R., Qin, M., Zong, C., Zheng, L., Wu, Y., Chai, X., Bi, Y., Xie, T., Gu, P., Li, X., Zhang, C., Tian, L., Wang, C., Wang, X., Karlsson, B. F., An, B., Yan, S., and Lu, Z., “Cradle:...

  2. [10]

    Project Sid: Many-agent simulations toward AI civilization,

    AL, A., Ahn, A., Becker, N., Carroll, S., Christie, N., Cortes, M., Demirci, A., Du, M., Li, F., Luo, S., Wang, P. Y., Willows, M., Yang, F., and Yang, G. R., “Project Sid: Many-agent simulations toward AI civilization,” , 2024. URL https://arxiv.org/abs/2411.00114

  3. [11]

    BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games,

    Paglieri, D., Cupiał, B., Coward, S., Piterbarg, U., Wolczyk, M., Khan, A., Pignatelli, E., Łukasz Kuciński, Pinto, L., Fergus, R., Foerster, J. N., Parker-Holder, J., and Rocktäschel, T., “BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games,” , 2024. URLhttps://arxiv....

  4. [12]

    Language Models are Spacecraft Operators,

    Rodriguez-Fernandez, V., Carrasco, A., Cheng, J., Scharf, E., Siew, P. M., and Linares, R., “Language Models are Spacecraft Operators,” , 2024. URLhttps://arxiv.org/abs/2404.00413

  5. [13]

    Fine-tuning LLMs for Autonomous Spacecraft Control: A Case Study UsingKerbalSpaceProgram,

    Carrasco, A., Rodriguez-Fernandez, V., and Linares, R., “Fine-tuning LLMs for Autonomous Spacecraft Control: A Case Study UsingKerbalSpaceProgram,” ProceedingsofSPAICE2024: TheFirstJointEuropeanSpaceAgency/IAAConferenceonAI inandforSpace,editedbyD.Dold, A.Hadjiivanov, andD.Izz...

  6. [14]

    Fine-Tuned Language Models as Space Systems Controllers,

    Zucchelli, E. M., Wu, D., Briden, J., Hofmann, C., Rodriguez-Fernandez, V., and Linares, R., “Fine-Tuned Language Models as Space Systems Controllers,”Proceedingsof theAAS/AIAAAstrodynamicsSpecialistConference, Broomfield, CO, 2024

  7. [15]

    Adapting a Foundation Model for Space-based Tasks,

    Foutter, M., Bhoj, P., Sinha, R., Elhafsi, A., Banerjee, S., Agia, C., Kruger, J., Guffanti, T., Gammelli, D., D’Amico, S., and Pavone, M., “Adapting a Foundation Model for Space-based Tasks,” , 2024. URL https://arxiv.org/abs/2408.05924

  8. [16]

    Chain-of-Thought Prompting Elicits Reasoning in Large Language Models,

    Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., and Zhou, D., “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models,” , 2023. URL https://arxiv.org/abs/2201.11903. 12

  9. [17]

    ReAct: Synergizing Reasoning and Acting in Language Models,

    Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y., “ReAct: Synergizing Reasoning and Acting in Language Models,” , 2023. URL https://arxiv.org/abs/2210.03629

  10. [18]

    Learning Transferable Visual Models From Natural Language Supervision,

    Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I., “Learning Transferable Visual Models From Natural Language Supervision,” , 2021. URL https://arxiv.org/abs/2103.00020

  11. [19]

    Flamingo: a Visual Language Model for Few-Shot Learning,

    Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., Ring, R., Rutherford, E., Cabi, S., Han, T., Gong, Z., Samangooei, S., Monteiro, M., Menick, J., Borgeaud, S., Brock, A., Nematzadeh, A., Sharifzadeh, S., B...

  12. [20]

    BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation,

    Li, J., Li, D., Xiong, C., and Hoi, S., “BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation,” , 2022. URL https://arxiv.org/abs/2201.12086

  13. [21]

    Visual Instruction Tuning,

    Liu, H., Li, C., Wu, Q., and Lee, Y. J., “Visual Instruction Tuning,” , 2023. URL https://arxiv.org/abs/2304.08485

  14. [22]

    Solving Robotics Problems in Zero-Shot with Vision-Language Models,

    Wang, Z., Shen, R., and Stadie, B. C., “Solving Robotics Problems in Zero-Shot with Vision-Language Models,” , 2024. URL https://openreview.net/forum?id=RQDuFF1rOn

  15. [23]

    SpaceGym: Discrete and Differential Games in Non-Cooperative Space Operations,

    Allen, R. E., Rachlin, Y., Ruprecht, J., Loughran, S., Varey, J., and Viggh, H., “SpaceGym: Discrete and Differential Games in Non-Cooperative Space Operations,”2023IEEE AerospaceConference, IEEE, 2023, pp. 1–12

  16. [24]

    Exploring the Numerical Reasoning Capabilities of Language Models: A Comprehensive Analysis on Tabular Data,

    Akhtar, M., Shankarampeta, A., Gupta, V., Patil, A., Cocarascu, O., and Simperl, E., “Exploring the Numerical Reasoning Capabilities of Language Models: A Comprehensive Analysis on Tabular Data,” , 2023. URL https://arxiv.org/abs/2311.02216

  17. [25]

    O1: High-Precision Numerical Reasoning Model,

    OpenAI, “O1: High-Precision Numerical Reasoning Model,” , September 2024. URL https://openai.com/o1/, accessed: 2024-11-29

  18. [26]

    Emotion Musical Prosody for Robotic Groups and Entitativity,

    Savery, R., Rogel, A., and Weinberg, G., “Emotion Musical Prosody for Robotic Groups and Entitativity,” 2021, pp. 440–446. https://doi.org/10.1109/RO-MAN50785.2021.9515314

  19. [27]

    Intel RealSense Stereoscopic Depth Cameras,

    Keselman, L., Woodfill, J. I., Grunnet-Jepsen, A., and Bhowmik, A., “Intel RealSense Stereoscopic Depth Cameras,” , 2017. URL https://arxiv.org/abs/1705.05548

  20. [28]

    FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness,

    Dao, T., Fu, D. Y., Ermon, S., Rudra, A., and Ré, C., “FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness,” , 2022. URL https://arxiv.org/abs/2205.14135. 13

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.