REVIEW 4 major objections 5 minor 28 references
Visual Language Models as Operator Agents in the Space Domain
T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read Vision-language models can act as spacecraft operators from GUI screenshots and, after light fine-tuning, guide a robot arm through space-hardware inspection.
desk verdict VLM-as-operator is a plausible new idea, but the claimed comparison to traditional methods rests on baselines that likely come from a different task family, so the abstract overstates what the data show. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is a multimodal agent loop in which visual frames and text share one prompt. For the software task, the KSPDG dashboard screenshot carries the navball, orientation, and mission markers; the VLM combines it with relative-position and prograde telemetry, reasons in natural language, and emits a structured function call selecting one of 27 discretized throttle combinations (left/right, forward/backward, up/down, plus no-thrust). For the hardware task, a camera frame and robot state are passed to a VLM that outputs a positional change $\Delta x$, a rotational adjustment $\Delta \theta$, and a boolean capture flag $\chi$, while a system prompt supplies the inspection goal; OpenVLA is fine-tuned as the action head using an HDF5 dataset of shuffled frame-episodes.
What would settle it
Run each agent in Tables 1 and 2 for dozens of seeded episodes and report score distributions; if the VLM scores overlap the classical and text-only baselines within error bars, the claimed advantage is not real. For the hardware claim, fine-tune with the recommended 50 episodes and measure closed-loop inspection success on unseen satellite mock-ups; if accuracy plateaus near the current validation levels, the promised real-world readiness does not follow.
Extended reading notes
Core claim
The central claim is that VLMs can act as end-to-end operator agents in the space domain by mapping visual observations directly to control actions. In the KSPDG Lady-Bandit-Guard scenarios, agents built from ChatGPT, Claude, and LLaMA received the in-game dashboard as an image alongside a few lines of telemetry and a few-shot prompt, and produced throttling actions through a discrete 27-combination action space; the best VLM agent reached a closest approach of 4.76 m and an average task score of 3,961, below the best text-only LLM score of 18,938, while classical baselines such as PPO, iLQGames, and Lambert-MPC were outperformed on the reported metrics. For hardware operations, the authors fine-tune OpenVLA, a vision-language-action model, on only 10 teleoperated episodes of satellite inspection with a 7-DoF arm, obtaining training accuracies of 70–83% and validation accuracies of 58.9–69.5% with roughly 50 mm average action deviation, and read this as evidence that a VLA model can be adapted to inspection tasks with a fraction of the recommended data.
Load-bearing premise
The results stand on a small number of simulation episodes and only ten fine-tuning examples, so the reported gaps could be run-to-run noise rather than a genuine advantage for vision-language models.
Editorial extensions
If this is right
- A VLM spacecraft operator needs only the same GUI screenshot a human operator sees, not hand-engineered state vectors, so the same agent design can transfer across simulation interfaces.
- Because VLMs balanced approach and guard-evasion better than text-only LLMs, visual cues such as the navball appear to carry spatial information that plain telemetry prompts lack.
- The hardware result suggests a vision-language-action model can be repurposed for a new satellite-inspection skill with roughly one-fifth of the recommended training episodes, lowering the data cost of deploying such agents.
- Latency, not decision quality, is the current bottleneck in simulated operation; faster inference or output caching would be the lever that makes these agents usable in real-time missions.
- The same end-to-end framework covers both software control and hardware inspection, so improvements in VLM visual reasoning should improve both paradigms together.
- Beyond the paper's own claims, the same screenshot-plus-telemetry prompt should carry over to any graphical mission-control console, meaning a VLM could supervise a legacy ground station without changing the interface.
- The ten-episode fine-tuning result is a proof of concept, not a deployment result; the natural next test is scaling to the recommended 50 episodes and measuring closed-loop inspection success, which would show whether the reported accuracy growth continues.
- Because the fastest simulator scores came with high latency, an implication the authors leave implicit is that distilling the VLM into a smaller, faster model may be the practical route to real-time operation even if per-step accuracy drops.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper describes two empirical deployments of Vision-Language Models (VLMs): (1) using GPT-4o, Claude 3.5, and LLaMA 3.2 as operator agents that read GUI screenshots and telemetry in the Kerbal Space Program Differential Games (KSPDG) environment, and (2) fine-tuning OpenVLA on ten teleoperated episodes to control an xArm 7 robot for satellite inspection. The central claim, stated in the abstract, is that VLMs can compete with traditional methods and non-multimodal LLMs in simulation, and show promise in real-world applications. The paper includes comparative tables of distances and scores, latency measurements, and fine-tuning loss/accuracy metrics, and frames the work as preliminary.
Significance. If the central comparison were valid, the paper would be a useful demonstration that a single VLM can perform closed-loop control of a simulated spacecraft from raw GUI screenshots and can, after light fine-tuning, propose inspection actions for a robot arm. The authors are transparent about the preliminary nature of the hardware experiments and about the latency bottleneck, and they correctly cite the BALROG benchmark's finding that LLMs currently outperform VLMs on many reasoning tasks. However, the load-bearing quantitative comparison to traditional methods rests on Table 1, which is internally inconsistent and does not currently support the abstract's claim.
major comments (4)
- [Section III.C, Table 1] The comparison with traditional methods is not established as reported. The rows for Naive, PPO, iLQGames, and Lambert-MPC are labeled as extracted from the SpaceGym paper, but they report only Best/Avg distance and leave the Avg. to Guard and Avg. Score columns empty, whereas the seven LLM/VLM rows report all four columns and are described by the LBG score from Eq. (1). This strongly suggests the traditional baselines were evaluated in the Pursuer-Evader scenario family rather than the Lady-Bandit-Guard scenarios (lg0-lg3, i1-i2) used for the model rows. If these numbers were not produced on the same scenario identifiers with the same scoring function, the statement in Section III.C that the agents 'outperform most traditional methods' does not follow. The authors should either rerun the traditional methods on the identical LBG scenarios or clearly restrict the claim to a scenario-matched subset.
- [Table 2] Table 2 contains an arithmetical inconsistency: for Claude VLM the average latency (9,373 ms) exceeds the stated worst latency (7,409 ms), and for LLaMA VLM the average (11,317 ms) exceeds the stated worst (9,664 ms). Since an average cannot exceed the maximum of the sample, at least one set of numbers is erroneous. Additionally, the LLaMA VLM best latency is listed as '3', which is likely a typo (possibly 7,139). The authors should correct the table and report the number of runs, seeds, and full latency distributions.
- [Section III.C and IV.B] No run counts, seeds, or standard deviations are reported for Tables 1-4, so the point estimates could be noise. For instance, the ChatGPT VLM and Claude VLM rows in Table 1 show average distances to the Lady of 30.42 m and 52.07 m but scores of 5,626 and 21,488; without multiple episodes per scenario, a reader cannot assess whether these differences are meaningful. The same applies to the fine-tuning results in Section IV.B, where ten episodes are explicitly acknowledged to be far below the recommended fifty. The authors should report episode counts per scenario, repeated runs, and variance, or explicitly characterize the results as illustrative rather than comparative.
- [Abstract and Section VI] The abstract claims that VLMs show 'promise in real-world applications', but the only real-world evidence is the ten-episode OpenVLA fine-tuning in Section IV.B, whose validation accuracy is 58.9-69.5% and whose average L1 deviation is about 50 mm, with no baseline comparison and no evaluation on an unseen task. The paper's own text repeatedly labels these results as preliminary, so the abstract's claim is stronger than the evidence. The authors should soften the claim or add a quantitative evaluation that supports it.
minor comments (5)
- [Section III.A] The text first says the action space is discretized into 9 distinct actions, then states that each of three axes has three options for a total of 3×3×3=27 permutations; these numbers are inconsistent and should be reconciled.
- [Section II.C, Eq. (1)] The displayed formula appears garbled ('dm_lb2+ 𝑎 / dm_bg+𝑏'); the intended expression (dm_lb^2 + a/dm_bg + b) should be typeset correctly.
- [Table 2] The best-latency column for LLaMA VLM reads '3' and should be corrected to a plausible value; also consider using consistent decimal places across rows.
- [Figure 1 caption] The caption refers to the system as an 'LLM (e.g., ChatGPT)' even though the paper's contribution is the VLM extension; update the caption to reflect the actual agent architecture.
- [References] Reference [26] cites a paper on emotional musical prosody, which appears unrelated to the UFactory xArm 7 hardware; please replace it with an appropriate hardware reference.
Circularity Check
No significant circularity: the performance claims are directly measured in simulation and via train/validation splits, not derived from fitted parameters or self-citations.
full rationale
This is an empirical deployment study rather than a derivation chain. The KSPDG results in Tables 1 and 2 are direct measurements of off-the-shelf and fine-tuned LLM/VLM agents acting in the SpaceGym/KSPDG environment, and the score in Eq. (1) is the challenge-defined evaluation metric, not a parameter fitted to make the claim true. The OpenVLA results in Tables 3 and 4 are training and validation losses/accuracies from a 90-10 split, so the 'promise' claim is grounded in held-out validation numbers rather than by construction. The only self-referential citations are refs. [12]-[14], the authors' prior LLM-agent work; they are cited as design context and do not supply the Table 1-4 numbers. The comparison to traditional methods is attributed to the external SpaceGym paper [23], so even if the baseline scenario matching is questionable, it is not a self-citation loop. I find no equation that reduces to its own input and no fitted quantity renamed as a prediction. Table 2's average-latency values exceeding worst-latency values, and the lack of scenario IDs/seeds for Table 1, are data-quality and reporting concerns, not circularity.
Assumptions & free parameters
assumptions (4)
- domain assumption VLM visual interpretation of screenshots is reliable enough for control decisions.
- domain assumption The KSPDG scoring function (Eq. 1) is the correct mission objective.
- ad hoc to paper Ten teleoperation episodes are a sufficient basis to conclude the fine-tuned OpenVLA model generalizes.
- domain assumption The selected models (Claude 3.5, GPT-4o, LLaMA 3.2, OpenVLA) are representative of current VLMs.
Cite this review
Pith. "Pith review of Visual Language Models as Operator Agents in the Space Domain." pith.science (2026). https://pith.science/paper/MFB3GKGB
@misc{pith2026250107802,
author = {Pith},
title = {Pith review of: Visual Language Models as Operator Agents in the Space Domain},
year = {2026},
howpublished = {\url{https://pith.science/paper/MFB3GKGB}},
note = {Machine review of arXiv:2501.07802}
}
read the original abstract
This paper explores the application of Vision-Language Models (VLMs) as operator agents in the space domain, focusing on both software and hardware operational paradigms. Building on advances in Large Language Models (LLMs) and their multimodal extensions, we investigate how VLMs can enhance autonomous control and decision-making in space missions. In the software context, we employ VLMs within the Kerbal Space Program Differential Games (KSPDG) simulation environment, enabling the agent to interpret visual screenshots of the graphical user interface to perform complex orbital maneuvers. In the hardware context, we integrate VLMs with robotic systems equipped with cameras to inspect and diagnose physical space objects, such as satellites. Our results demonstrate that VLMs can effectively process visual and textual data to generate contextually appropriate actions, competing with traditional methods and non-multimodal LLMs in simulation tasks, and showing promise in real-world applications.
Reference graph
Works this paper leans on
-
[1]
OpenAI, “Introducing ChatGPT,”https://openai.com/blog/chatgpt, 2022. (Accessed: 11-27-2024)
work page 2022
-
[2]
OpenAI, “Hello, GPT-4o!” , May 2024. URLhttps://openai.com/index/hello-gpt-4o/, accessed: 11-28-2024
work page 2024
-
[3]
Llama 3.2: Vision and Edge for Mobile Devices,
AI, M., “Llama 3.2: Vision and Edge for Mobile Devices,” , September 2024. URLhttps://ai.meta.com/blog/llama- 3-2-connect-2024-vision-edge-mobile-devices/ , accessed: 2024-11-28
work page 2024
-
[4]
Introducing Claude 3.5: Faster, safer, smarter, and with a sense of humor,
Anthropic, “Introducing Claude 3.5: Faster, safer, smarter, and with a sense of humor,” , June 2024. URLhttps: //www.anthropic.com/news/claude-3-5-sonnet, accessed: 2024-12-02
work page 2024
-
[5]
OpenVLA: An Open-Source Vision-Language-Action Model,
Kim, M. J., Pertsch, K., Karamcheti, S., Xiao, T., Balakrishna, A., Nair, S., Rafailov, R., Foster, E., Lam, G., Sanketi, P., Vuong, Q., Kollar, T., Burchfiel, B., Tedrake, R., Sadigh, D., Levine, S., Liang, P., and Finn, C., “OpenVLA: An Open-Source Vision-Language-Action Model,” , 2024. URLhttps://arxiv.org/abs/2406.09246
arXiv 2024
-
[6]
Vision-Language Foundation Models as Effective Robot Imitators,
Li, X., Liu, M., Zhang, H., Yu, C., Xu, J., Wu, H., Cheang, C., Jing, Y., Zhang, W., Liu, H., Li, H., and Kong, T., “Vision-Language Foundation Models as Effective Robot Imitators,” , 2024. URLhttps://arxiv.org/abs/2311.01378
arXiv 2024
-
[7]
The Rise and Potential of Large Language Model Based Agents: A Survey,
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., Zheng, R., Fan, X., Wang, X., Xiong, L., Zhou, Y., Wang, W., Jiang, C., Zou, Y., Liu, X., Yin, Z., Dou, S., Weng, R., Cheng, W., Zhang, Q., Qin, W., Zheng, Y., Qiu, X., Huang, X., and Gui, T., “The Rise and Potential of Large Language Model Based Agents: A Surve...
arXiv 2023
-
[8]
JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Models,
Wang, Z., Cai, S., Liu, A., Jin, Y., Hou, J., Zhang, B., Lin, H., He, Z., Zheng, Z., Yang, Y., Ma, X., and Liang, Y., “JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Models,” , 2023. URL https://arxiv.org/abs/2311.05997
arXiv 2023
Show all 28 references
-
[9]
Cradle: Empowering Foundation Agents Towards General Computer Control,
Tan, W., Zhang, W., Xu, X., Xia, H., Ding, Z., Li, B., Zhou, B., Yue, J., Jiang, J., Li, Y., An, R., Qin, M., Zong, C., Zheng, L., Wu, Y., Chai, X., Bi, Y., Xie, T., Gu, P., Li, X., Zhang, C., Tian, L., Wang, C., Wang, X., Karlsson, B. F., An, B., Yan, S., and Lu, Z., “Cradle:...
2024 arXiv
-
[10]
Project Sid: Many-agent simulations toward AI civilization,
AL, A., Ahn, A., Becker, N., Carroll, S., Christie, N., Cortes, M., Demirci, A., Du, M., Li, F., Luo, S., Wang, P. Y., Willows, M., Yang, F., and Yang, G. R., “Project Sid: Many-agent simulations toward AI civilization,” , 2024. URL https://arxiv.org/abs/2411.00114
2024 arXiv
-
[11]
BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games,
Paglieri, D., Cupiał, B., Coward, S., Piterbarg, U., Wolczyk, M., Khan, A., Pignatelli, E., Łukasz Kuciński, Pinto, L., Fergus, R., Foerster, J. N., Parker-Holder, J., and Rocktäschel, T., “BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games,” , 2024. URLhttps://arxiv....
2024 arXiv
-
[12]
Language Models are Spacecraft Operators,
Rodriguez-Fernandez, V., Carrasco, A., Cheng, J., Scharf, E., Siew, P. M., and Linares, R., “Language Models are Spacecraft Operators,” , 2024. URLhttps://arxiv.org/abs/2404.00413
2024 arXiv
-
[13]
Fine-tuning LLMs for Autonomous Spacecraft Control: A Case Study UsingKerbalSpaceProgram,
Carrasco, A., Rodriguez-Fernandez, V., and Linares, R., “Fine-tuning LLMs for Autonomous Spacecraft Control: A Case Study UsingKerbalSpaceProgram,” ProceedingsofSPAICE2024: TheFirstJointEuropeanSpaceAgency/IAAConferenceonAI inandforSpace,editedbyD.Dold, A.Hadjiivanov, andD.Izz...
2024 doi
-
[14]
Fine-Tuned Language Models as Space Systems Controllers,
Zucchelli, E. M., Wu, D., Briden, J., Hofmann, C., Rodriguez-Fernandez, V., and Linares, R., “Fine-Tuned Language Models as Space Systems Controllers,”Proceedingsof theAAS/AIAAAstrodynamicsSpecialistConference, Broomfield, CO, 2024
2024
-
[15]
Adapting a Foundation Model for Space-based Tasks,
Foutter, M., Bhoj, P., Sinha, R., Elhafsi, A., Banerjee, S., Agia, C., Kruger, J., Guffanti, T., Gammelli, D., D’Amico, S., and Pavone, M., “Adapting a Foundation Model for Space-based Tasks,” , 2024. URL https://arxiv.org/abs/2408.05924
2024 arXiv
-
[16]
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models,
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., and Zhou, D., “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models,” , 2023. URL https://arxiv.org/abs/2201.11903. 12
2023 arXiv
-
[17]
ReAct: Synergizing Reasoning and Acting in Language Models,
Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y., “ReAct: Synergizing Reasoning and Acting in Language Models,” , 2023. URL https://arxiv.org/abs/2210.03629
2023 arXiv
-
[18]
Learning Transferable Visual Models From Natural Language Supervision,
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I., “Learning Transferable Visual Models From Natural Language Supervision,” , 2021. URL https://arxiv.org/abs/2103.00020
2021 arXiv
-
[19]
Flamingo: a Visual Language Model for Few-Shot Learning,
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., Ring, R., Rutherford, E., Cabi, S., Han, T., Gong, Z., Samangooei, S., Monteiro, M., Menick, J., Borgeaud, S., Brock, A., Nematzadeh, A., Sharifzadeh, S., B...
2022 arXiv
-
[20]
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation,
Li, J., Li, D., Xiong, C., and Hoi, S., “BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation,” , 2022. URL https://arxiv.org/abs/2201.12086
2022 arXiv
-
[21]
Visual Instruction Tuning,
Liu, H., Li, C., Wu, Q., and Lee, Y. J., “Visual Instruction Tuning,” , 2023. URL https://arxiv.org/abs/2304.08485
2023 arXiv
-
[22]
Solving Robotics Problems in Zero-Shot with Vision-Language Models,
Wang, Z., Shen, R., and Stadie, B. C., “Solving Robotics Problems in Zero-Shot with Vision-Language Models,” , 2024. URL https://openreview.net/forum?id=RQDuFF1rOn
2024
-
[23]
SpaceGym: Discrete and Differential Games in Non-Cooperative Space Operations,
Allen, R. E., Rachlin, Y., Ruprecht, J., Loughran, S., Varey, J., and Viggh, H., “SpaceGym: Discrete and Differential Games in Non-Cooperative Space Operations,”2023IEEE AerospaceConference, IEEE, 2023, pp. 1–12
2023
-
[24]
Exploring the Numerical Reasoning Capabilities of Language Models: A Comprehensive Analysis on Tabular Data,
Akhtar, M., Shankarampeta, A., Gupta, V., Patil, A., Cocarascu, O., and Simperl, E., “Exploring the Numerical Reasoning Capabilities of Language Models: A Comprehensive Analysis on Tabular Data,” , 2023. URL https://arxiv.org/abs/2311.02216
2023 arXiv
-
[25]
O1: High-Precision Numerical Reasoning Model,
OpenAI, “O1: High-Precision Numerical Reasoning Model,” , September 2024. URL https://openai.com/o1/, accessed: 2024-11-29
2024
-
[26]
Emotion Musical Prosody for Robotic Groups and Entitativity,
Savery, R., Rogel, A., and Weinberg, G., “Emotion Musical Prosody for Robotic Groups and Entitativity,” 2021, pp. 440–446. https://doi.org/10.1109/RO-MAN50785.2021.9515314
2021
-
[27]
Intel RealSense Stereoscopic Depth Cameras,
Keselman, L., Woodfill, J. I., Grunnet-Jepsen, A., and Bhowmik, A., “Intel RealSense Stereoscopic Depth Cameras,” , 2017. URL https://arxiv.org/abs/1705.05548
2017 arXiv
-
[28]
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness,
Dao, T., Fu, D. Y., Ermon, S., Rudra, A., and Ré, C., “FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness,” , 2022. URL https://arxiv.org/abs/2205.14135. 13
2022 arXiv
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.