Autonomous Computer Vision Development with Agentic AI

Jin Kim; John M. Hoffman; Matthew S. Brown; Muhammad Wahi-Anwa; Sangyun Park; Shawn Shin

REVIEW 3 major objections 4 minor 17 references

Autonomous Computer Vision Development with Agentic AI

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An LLM-based agent can autonomously plan, configure, train, and run a chest X-ray segmentation system from a single natural-language prompt, reaching 0.96 Dice on lungs.

desk verdict Honest proof of concept that an LLM agent can assemble and execute a SimpleMind segmentation pipeline, but the demonstration mostly validates template retrieval, not autonomous planning. read the letter →

arxiv 2506.11140 v3 pith:QKFJKN2R submitted 2025-06-11 cs.CV cs.AIcs.MA

Jin Kim , Muhammad Wahi-Anwa , Sangyun Park , Shawn Shin , John M. Hoffman , Matthew S. Brown This is my paper · ORCID

classification cs.CVcs.AIcs.MA

keywords AgenticAILargeLanguageModelsAutonomousagentsSimpleMindKnowledgegraphsMedicalimageanalysisSegmentationPlanning

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper reports a proof-of-concept that an LLM-based agent can take a plain-language request for a medical-image-analysis task, plan the whole workflow itself, write the configuration file, train the neural networks, and run inference without a human data scientist in the loop. On a request to build a lungs, heart, and ribs segmenter for chest X-rays, the agent produced a SimpleMind knowledge graph in YAML, iteratively fixed its own errors using a verifier, then executed the training and inference scripts. The finished system scored mean Dice coefficients of 0.963 for lungs, 0.824 for heart, and 0.830 for ribs on held-out test sets. The authors are explicit that the task closely matched template examples in the agent's instructions, so the demonstration shows template-guided autonomy rather than open-ended design.

What carries the argument

The mechanism that carries the argument is the SimpleMind knowledge graph: a YAML file specifying a directed workflow of configurable tools (image loading, preprocessing, neural-network segmentation, mask post-processing, and saving), organized into chunks and supernodes, with a blackboard passing messages between tools. The LLM agent plans this graph in JSON to avoid formatting errors, a Verifier checks the YAML against tool input and output definitions and returns error messages, and the agent iterates until the configuration is valid. That representation turns an open-ended natural-language request into a checkable, executable artifact, which is what lets an LLM both design and run the vision system.

What would settle it

Remove the complete trachea template and the partial lungs and heart examples from the system prompt, leave the tool inventory in place, and run the exact same lungs-heart-ribs request; if the agent cannot produce a valid, executable YAML configuration, the claimed autonomy is template-copying rather than planning.

Watch

Extended reading notes

Core claim

The central claim is that a specialized computer vision system can be built autonomously from a natural language prompt using Agentic AI methods. The paper shows this by extending SimpleMind, a configurable cognitive-AI environment for medical image analysis, so that an OpenManus LLM agent can decompose the requested task into a planning step, convert that plan into a validated YAML knowledge graph that configures SimpleMind tools, and then run SM-Learn for training and SM-Think for inference. The agent's final plan defined separate supernodes for lungs, heart, and ribs, each containing image preprocessing, a deep-learning segmentation network, and mask post-processing, wired together by SimpleMind's blackboard. After five iterations of verification and correction, the configuration executed successfully, and the resulting segmentations matched the reference masks at the reported Dice scores. The paper argues this demonstrates the potential to automate the configuration and execution that a data scientist traditionally performs when building computer vision applications.

Load-bearing premise

The agent's planning works only because the requested task is a near-copy of the template examples and documentation bundled in its system prompt; if the task demanded a genuinely new tool arrangement, the demonstration gives no evidence the agent would succeed.

Editorial extensions

If this is right

A user can move from a natural-language description of a segmentation task to trained, tested models without hand-writing configuration, as long as the target task resembles the documented examples.
The verify-and-refine loop offers a general recipe for making LLM-generated configuration reliable: let an automatic checker catch format and input-output type errors and feed them back as text.
Because the agent can execute terminal commands, the same architecture can close the loop from dataset retrieval to training to inference when combined with data-fetching agents and parameter-optimization methods, as the authors suggest.
The approach transfers to other medical-image tasks and modalities in proportion to the quality of the documentation and examples supplied in the prompt; the paper lists richer tool documentation as a prerequisite.
With SimpleMind, the planning agent is complemented by spatial and logical reasoning agents, so the resulting system retains symbolic constraints on top of deep learning rather than being a pure black-box pipeline.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural next experiment is to remove the template examples from the system prompt and re-run the identical lungs-heart-ribs request; if the agent still produces a valid plan, the claim of autonomous planning is stronger, but if it fails, the contribution is best described as template-guided configuration.
The verifier pattern could extend beyond SimpleMind to any declarative pipeline language: LLM agents write the configuration and a static analyzer polices it, which would be a testable way to harden agentic workflows in other domains.
The reported Dice scores have no direct comparison against a human data scientist configuring the same tools, so an apples-to-apples test would separate the cost of autonomy from the cost of the pipeline itself.
The paper implicitly assumes that the natural-language prompt plus the system-prompt documentation fully specify the task; a more complex description that requires combining findings across images or inventing a novel tool arrangement would test whether the agent reasons beyond its examples.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a legitimate, honestly described proof of concept. The authors show that an OpenManus agent, given a natural language prompt plus a system prompt containing a full trachea config and partial lungs/heart configs, can generate a SimpleMind knowledge graph for lungs/heart/ribs segmentation, run the training and inference scripts, and get acceptable dice on small test sets. That is real; the execution trace in Table 1 is detailed and the authors are straight about the main limitation: the prompt was limited and the agent likely relied heavily on the provided examples.

What's new: I don't know of another paper that applies an LLM agent to generate a SimpleMind knowledge graph and executes it end-to-end. It's a useful integration of existing pieces, and it's framed as a work in progress rather than a finished capability. The architecture (LLM planner + verifier + symbolic execution environment) is sensible, and the authors' willingness to show the iterative refinement loop adds credibility.

The soft spots are real, and the main one is the one the authors themselves name: the target task is nearly a copy of the trachea template plus partial lungs/heart configs. The system prompt in Appendix A includes those examples, so the 'planning' step is closer to retrieval and light adaptation. The ribs config still shows a placeholder weights_url '1ribs_id' that would not resolve to a valid file, which makes me wonder how much of the rib pipeline was actually executed as shown. The dice scores are reported without error bars or significance tests, and the heart labels are private, so the 0.824 result can't be independently replicated. The code link points to a GitHub repo for OpenManus-SimpleMind, but the full agent artifacts and prompts beyond the appendix aren't all there.

None of this kills the paper. The narrow claim—that an agent can generate a valid config and execute the SimpleMind workflow with reasonable results—is supported. The broad 'autonomous development' claim in the abstract is not, and it should be tempered. A revision with a non-agentic or human baseline, a task that isn't template-adjacent, and proper error bars would strengthen it considerably.

Who it's for: researchers working on LLM agents for medical imaging pipelines, and anyone building symbolic-plus-LLM integration. It won't change the field, but it's a fair data point. I'd send it to review—the execution trace and honest limitation section give referees something concrete to work with. My recommendation is major revision, not desk reject.

Referee Report

3 major / 4 minor

Summary. The paper reports a proof-of-concept in which an OpenManus-based LLM agent, given a natural-language prompt requesting a SimpleMind configuration for lungs, heart, and ribs segmentation on chest X-rays, generates a YAML knowledge-graph plan, verifies it, runs SM-Learn for training, and runs SM-Think for inference. The agent's execution trace is documented in Table 1, and mean Dice scores of 0.963 (lungs, 50 cases), 0.824 (heart, 49 cases), and 0.830 (ribs, 50 cases) are reported. The authors frame this as demonstrating that a specialized computer vision system can be built autonomously from a natural-language prompt, while also acknowledging in the Discussion that the prompting was limited and that the LLM likely relied on provided template examples.

Significance. If the result holds as stated, this is a modest but useful proof of concept for LLM-driven configuration of medical-image-analysis pipelines: the agent did not merely emit plausible YAML, it interacted with a verifier, revised its plan, and executed training and inference in a real environment. The paper has genuine strengths: Section 3.4's execution trace is transparent about iterative failures and corrections, and Section 4 is unusually candid about the role of the provided templates. However, the significance is limited by the fact that the evaluation target is nearly a remix of configurations already present in the system prompt, so the demonstration does not yet establish autonomous planning for novel tasks. The quantitative results are also presented without error bars or public ground truth for the heart labels, which limits their independent value.

major comments (3)

[Section 3.3 / Appendix A / Section 4] The central claim that a computer vision system can be built 'autonomously from a natural language prompt' is not fully supported by the experimental design. The system prompt in Appendix A includes a complete trachea segmentation knowledge graph, partial lungs and heart configurations, and a detailed tool dictionary; the user prompt asks for lungs, heart, and ribs segmentation on the same modality. The generated plan is therefore close to a template-following exercise, with the ribs configuration being the only genuinely new component. The authors acknowledge this in Section 4, but the abstract and introduction present the result without this qualification. Please soften the claim to 'template-guided autonomous configuration' or add an experiment with a target task not covered by the provided examples to support generalization.
[Section 3.4 (ribs_chest_xr YAML excerpt)] The generated ribs configuration contains the field 'weights_url: https://drive.google.com/file/d/1ribs_id/view?usp=sharing'. The string '1ribs_id' appears to be a placeholder rather than a valid file identifier, so this URL would not resolve to a real file. If this configuration was executed exactly as shown, please clarify how SM-Learn or SM-Think handled this URL, or correct the excerpt. If the configuration shown is an abbreviated or illustrative version, please state that explicitly and provide the exact YAML that was actually executed, together with the training/inference logs, so that the reader can verify that the reported Dice scores came from the agent-generated plan.
[Section 3.5] The mean Dice scores are reported as single-point numbers without confidence intervals, standard deviations, or statistical comparison to any baseline, and the heart ground truth is private. Because the core claim is that the agent autonomously produced a working pipeline, the absence of these statistics is not fatal, but it does weaken the quantitative component of the demonstration. At minimum, report per-case variability and the number of test images per structure in the results table, and make the heart labels available or specify access conditions. Without this, the heart result of 0.824 cannot be independently reproduced or assessed.

minor comments (4)

[Throughout] The manuscript typesets 'YAML' as 'Y AML' in many places, including the abstract and Section 3.4. Please correct this to 'YAML' for consistency with standard usage.
[Table 1] The execution trace contains stray 'nosep' tokens, for example 'nosep JSON extraction failures' and 'nosep Verifier: Invalid input linking'. These appear to be LaTeX artifacts and should be removed.
[References] Reference [14] is incomplete: it lists only 'Inc, G.' as the author. Please provide the full citation for the COVID-19 chest X-ray segmentation dataset.
[Section 3.2] The train/test split is described as taking the first 100 image-label pairs for training and the next 50 for testing. Please clarify whether these datasets were in a randomized order or in their original file order, since a sequential split may not be representative of the data distribution.

Circularity Check

1 steps flagged · score 6.0 of 10

Template examples in the system prompt make the 'autonomous planning' demonstration largely a retrieval or remix task; the authors explicitly acknowledge this, yielding partial circularity.

other [Abstract; Section 3.3 User Prompt; Section 4 Discussion; Appendix A.1 System Prompts]
"Abstract: 'We demonstrate that a specialized computer vision system can be built autonomously from a natural language prompt.' Discussion: 'the prompting used was quite limited and the LLM likely based its planning largely on provided template examples... The online example includes a tool configuration for trachea segmentation on chest x-ray and partial configuration examples for lungs and heart (ribs are not included).' System prompt: 'Follow the similar pipeline as the example.'"

The claimed autonomous 'planning' is evaluated on a task whose answer is embedded in the input. The system prompt contains a complete trachea segmentation knowledge graph and partial lungs and heart configurations, while the user prompt asks for lungs, heart, and ribs. The generated configuration therefore reduces to retrieving and remixing those examples: the lungs and heart plans are near-completions of the partial examples, and the ribs plan closely follows the trachea template, even retaining a placeholder '1ribs_id' in its weights_url.

full rationale

The paper is a proof of concept that an agent can generate a SimpleMind configuration, run training, and run inference. The execution steps (SM-Learn and SM-Think) are real and not circular. However, the strongest claim that a specialized computer vision system can be 'built autonomously from a natural language prompt' is weakened because the system prompt already includes a full trachea example and partial lungs and heart examples, and the task asks for lungs, heart, and ribs. The authors themselves state that the LLM 'likely based its planning largely on provided template examples,' so the planning contribution reduces largely to template following. Because the limitation is disclosed and the execution is genuine, the circularity is partial rather than total; the score reflects that the 'prediction' of the plan is partly an input-derived remix, not an independent planning result.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new theoretical entities. Its central claim rests on the reliability of the SimpleMind framework, the accuracy of the reference labels, and the faithfulness of the reported agent trace. The main free choices are configuration parameters the agent inherited from prompt examples.

free parameters (2)

resize target_shape = [512, 512]
Chosen by the agent and inherited from the template example in the system prompt. Not fitted to data, but a hand-selected configuration parameter.
prediction_threshold = 0.5
Set in the generated YAML for the tf2_segmentation agent. Selected by the agent from the example; not tuned on a validation set.

assumptions (3)

domain assumption SimpleMind's tools and knowledge graph format work as documented.
The paper relies on SimpleMind (reference [4]) and the agent's generated YAML being executable; the Verifier is used, but the framework's behavior is taken as given.
domain assumption The reference segmentation masks in the three datasets are accurate.
Dice scores are computed against these labels. The heart labels were generated by UCLA analysts and are not public, so their quality cannot be independently checked.
domain assumption The agent execution trace in Table 1 accurately reflects the agent's behavior.
The paper presents the agent's tool calls and outputs as evidence of autonomy; there is no shipped log or code to verify the trace.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Autonomous Computer Vision Development with Agentic AI." pith.science (2026). https://pith.science/paper/QKFJKN2R

@misc{pith2026250611140,
  author       = {Pith},
  title        = {Pith review of: Autonomous Computer Vision Development with Agentic AI},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QKFJKN2R}},
  note         = {Machine review of arXiv:2506.11140}
}

read the original abstract

Agentic Artificial Intelligence (AI) systems leveraging Large Language Models (LLMs) exhibit significant potential for complex reasoning, planning, and tool utilization. We demonstrate that a specialized computer vision system can be built autonomously from a natural language prompt using Agentic AI methods. This involved extending SimpleMind (SM), an open-source Cognitive AI environment with configurable tools for medical image analysis, with an LLM-based agent, implemented using OpenManus, to automate the planning (tool configuration) for a particular computer vision task. We provide a proof-of-concept demonstration that an agentic system can interpret a computer vision task prompt, plan a corresponding SimpleMind workflow by decomposing the task and configuring appropriate tools. From the user input prompt, "provide sm (SimpleMind) config for lungs, heart, and ribs segmentation for cxr (chest x-ray)"), the agent LLM was able to generate the plan (tool configuration file in YAML format), and execute SM-Learn (training) and SM-Think (inference) scripts autonomously. The computer vision agent automatically configured, trained, and tested itself on 50 chest x-ray images, achieving mean dice scores of 0.96, 0.82, 0.83, for lungs, heart, and ribs, respectively. This work shows the potential for autonomous planning and tool configuration that has traditionally been performed by a data scientist in the development of computer vision applications.

Figures

Figures reproduced from arXiv: 2506.11140 by the authors.

**Figure 2.** Detailed system architecture illustrating the interaction between the OpenManus agentic AI framework and [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** The Agentic AI approach (a) involves an LLM agent (OpenManus) generating a plan (Knowledge Graph), [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: SimpleMind segmentation result examples for lungs, heart, and ribs in the test set. Red indicates pixels [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 11 canonical work pages

[1]

& Chao, A

Masterman, T., Besen, S., Sawtell, M. & Chao, A. The landscape of emerging ai agent architectures for reasoning, planning, and tool calling: A survey.ArXiv Preprint ArXiv:2404.11584. (2024)

arXiv 2024
[2]

Towards Autonomous Agents: Adaptive-planning, Reasoning, and Acting in Language Models

Dutta, A. & Hsiao, Y . Towards Autonomous Agents: Adaptive-planning, Reasoning, and Acting in Language Models.ArXiv Preprint ArXiv:2408.06458. (2024) 9 APREPRINT- SEPTEMBER12, 2025

work page Pith review arXiv 2024
[3]

& Wen, J

Jiang, J., Zhou, K., Zhao, W., Song, Y ., Zhu, C., Zhu, H. & Wen, J. Kg-agent: An efficient autonomous agent framework for complex reasoning over knowledge graph.ArXiv Preprint ArXiv:2402.11163. (2024)

arXiv 2024
[4]

& Brown, M

Choi, Y ., Wahi-Anwar, M. & Brown, M. SimpleMind: an open-source software environment that adds thinking to deep neural networks.Plos One.18, e0283587 (2023)

work page 2023
[5]

& Tang, X

Liang, X., Xiang, J., Yu, Z., Zhang, J., Hong, S., Fan, S. & Tang, X. OpenManus: An open-source framework for building general AI agents. (Zenodo,2025), https://doi.org/10.5281/zenodo.15186407

doi:10.5281/zenodo.15186407 2025
[6]

& Others A survey on large language model based autonomous agents.Frontiers Of Computer Science.18, 186345 (2024)

Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., Chen, Z., Tang, J., Chen, X., Lin, Y . & Others A survey on large language model based autonomous agents.Frontiers Of Computer Science.18, 186345 (2024)

work page 2024
[7]

& Cao, Y

Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K. & Cao, Y . React: Synergizing reasoning and acting in language models.International Conference On Learning Representations (ICLR). (2023)

work page 2023
[8]

& Yao, S

Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K. & Yao, S. Reflexion: Language agents with verbal reinforcement learning.Advances In Neural Information Processing Systems.36pp. 8634-8652 (2023)

work page 2023

Show all 17 references

[9]

& Chen, E

Huang, X., Liu, W., Chen, X., Wang, X., Wang, H., Lian, D., Wang, Y ., Tang, R. & Chen, E. Understanding the planning of LLM agents: A survey.ArXiv Preprint ArXiv:2402.02716. (2024)

2024 arXiv
[10]

Hao, S., Gu, Y ., Ma, H., Hong, J., Wang, Z., Wang, D. & Hu, Z. Reasoning with language model is planning with world model.ArXiv Preprint ArXiv:2305.14992. (2023)

2023 arXiv
[11]

& Stoica, I

Kwon, W., Li, Z., Zhuang, S., Sheng, Y ., Zheng, L., Yu, C., Gonzalez, J., Zhang, H. & Stoica, I. Efficient Memory Management for Large Language Model Serving with PagedAttention.Proceedings Of The ACM SIGOPS 29th Symposium On Operating Systems Principles. (2023)

2023
[12]

& Nguyen, H

Nguyen, H., Le, T., Pham, H. & Nguyen, H. VinDr-RibCXR: A benchmark dataset for automatic segmentation and labeling of individual ribs on chest X-rays.ArXiv Preprint ArXiv:2107.01327. (2021)

2021 arXiv
[13]

& McDonald, C

Candemir, S., Jaeger, S., Palaniappan, K., Musco, J., Singh, R., Xue, Z., Karargyris, A., Antani, S., Thoma, G. & McDonald, C. Lung segmentation in chest radiographs using anatomical atlases with nonrigid registration.IEEE Transactions On Medical Imaging.33, 577-590 (2013)

2013
[14]

Covid-19 chest x-ray segmentations dataset

Inc, G. Covid-19 chest x-ray segmentations dataset. (2020)

2020
[15]

Hayes-Roth, B. A blackboard architecture for control.Artificial Intelligence.26, 251-321 (1985) 10 APREPRINT- SEPTEMBER12, 2025 A System Prompts A.1 Agent (System) Prompts System prompts were supplied to the plan-generating LLM as a preface to the user command. The system prom...

1985
[16]

chunks"‘. The value of ‘

**Hierarchy:** The JSON object has a top-level key ‘"chunks"‘. The value of ‘"chunks"‘ is an object containing one or more *supernode* objects. Each *supernode* object is keyed by its unique ‘supernode_name‘ (string). * Each *supernode* object contains one or more *chunk* obje...
[17]

supernode_output

and potentially its input sources (see Rule 6) or special flags (‘"supernode_output"‘, ‘"chunk_output"‘). The online documentation also included several example plans, including a complete plan for trachea segmentation that configures tools for preprocessing, neural network, p...

2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

& Chao, A

Masterman, T., Besen, S., Sawtell, M. & Chao, A. The landscape of emerging ai agent architectures for reasoning, planning, and tool calling: A survey.ArXiv Preprint ArXiv:2404.11584. (2024)

arXiv 2024

[2] [2]

Towards Autonomous Agents: Adaptive-planning, Reasoning, and Acting in Language Models

Dutta, A. & Hsiao, Y . Towards Autonomous Agents: Adaptive-planning, Reasoning, and Acting in Language Models.ArXiv Preprint ArXiv:2408.06458. (2024) 9 APREPRINT- SEPTEMBER12, 2025

work page Pith review arXiv 2024

[3] [3]

& Wen, J

Jiang, J., Zhou, K., Zhao, W., Song, Y ., Zhu, C., Zhu, H. & Wen, J. Kg-agent: An efficient autonomous agent framework for complex reasoning over knowledge graph.ArXiv Preprint ArXiv:2402.11163. (2024)

arXiv 2024

[4] [4]

& Brown, M

Choi, Y ., Wahi-Anwar, M. & Brown, M. SimpleMind: an open-source software environment that adds thinking to deep neural networks.Plos One.18, e0283587 (2023)

work page 2023

[5] [5]

& Tang, X

Liang, X., Xiang, J., Yu, Z., Zhang, J., Hong, S., Fan, S. & Tang, X. OpenManus: An open-source framework for building general AI agents. (Zenodo,2025), https://doi.org/10.5281/zenodo.15186407

doi:10.5281/zenodo.15186407 2025

[6] [6]

& Others A survey on large language model based autonomous agents.Frontiers Of Computer Science.18, 186345 (2024)

Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., Chen, Z., Tang, J., Chen, X., Lin, Y . & Others A survey on large language model based autonomous agents.Frontiers Of Computer Science.18, 186345 (2024)

work page 2024

[7] [7]

& Cao, Y

Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K. & Cao, Y . React: Synergizing reasoning and acting in language models.International Conference On Learning Representations (ICLR). (2023)

work page 2023

[8] [8]

& Yao, S

Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K. & Yao, S. Reflexion: Language agents with verbal reinforcement learning.Advances In Neural Information Processing Systems.36pp. 8634-8652 (2023)

work page 2023