Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

SimuGen: Multi-modal Agentic Framework for Constructing Block Diagram-Based Simulation Models

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A multi-agent pipeline reads Simulink diagram images and produces executable simulation code, reporting 94.5% average reproduction accuracy.

desk verdict SimuGen is a plausible engineering system with a sensible agent decomposition, but the recall-only metric and thin evaluation do not support the 94.5% reproduction accuracy claim. read the letter →

arxiv 2506.15695 v2 pith:EKQGBPD3 submitted 2025-05-28 cs.LG

Xinxing Ren , Qianbo Zang , Zekun Guo This is my paper · ORCID
classification cs.LG
keywords SimuGenSimulinkmodelgenerationmulti-agentLLMframeworkblockdiagramunderstandingmultimodalsimulationcodeunittestreviewerdebuglocator
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SimuGen is an attempt to close a gap the authors identify: large language models can write text code but fail to produce reliable Simulink simulation code from text prompts alone, because Simulink's two-dimensional block structure and its engineering semantics are underrepresented in pretraining. The paper proposes a multi-agent, multimodal framework that takes a Simulink diagram image as input, reads the blocks and connections visually, retrieves structured descriptions of 50 block types from a domain database, and generates executable Simulink construction code through specialized agents. On nine hand-selected simulation tasks spanning general, physical, electrical, and automotive modeling, the framework reports an average reproduction accuracy of 94.5%, with the Unit Test Reviewer and Debug Locator contributing a combined accuracy gain that exceeds either alone. If this result holds, it would make diagram-to-simulation conversion, and the documentation that goes with it, automatable rather than a manual reverse-engineering chore.

What carries the argument

The load-bearing mechanism is the feedback loop between the Investigator and the Unit Test Reviewer, backed by a retrieval database of 50 block definitions that supply library paths, port names, and parameter rules. The Reviewer checks eight conformance criteria, including duplicate connections, disciplined input/output ports, formatting, and complete port coverage, and sends failures back to the Investigator; the Debug Locator then parses runtime errors and routes blame either to the Block Builder's code or back to the Investigator's connection description. The reported accuracy is the average of two recall ratios, block matches over ground-truth blocks and connection matches over ground-truth connections, so the loop's job is to make the generated model contain the ground-truth elements.

What would settle it

Re-run the nine tasks scoring both missing and extra elements (for instance with F1) or feed the pipeline diagrams it has not seen, including diagrams with subsystems; if the score drops well below 94.5%, the headline number was an artifact of counting only ground-truth elements.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that a diagram image plus structured block knowledge is enough for a multimodal LLM to reconstruct an executable Simulink model: the investigator agent extracts blocks and connections from the image, the unit test reviewer enforces eight standards-conformance checks on the proposed wiring, the block builder emits construction code using only add_block and add_line, the executor runs it, and the debug locator decides whether a failure is a code bug or a flaw in the investigator's description before looping back. With the multimodal investigator o4-mini and the reasoning agents o3-mini, the system reports 94.5% average reproduction accuracy over nine ground-truth models, versus 83.2% with GPT-4.1 as investigator. The ablation results argue that the reviewer and debugger are not interchangeable conveniences: removing both drops accuracy to 51.7%, and the reviewer is useful even though it cannot directly verify whether a connection is semantically correct.

Load-bearing premise

The reported score counts only what is missing, never what is extra: a model that adds spurious blocks and wires can still score perfectly.

Editorial extensions

If this is right

  • If the reported accuracy holds, an engineer can hand SimuGen a screenshot of an existing Simulink diagram and receive runnable construction code plus a four-part simulation report.
  • The system should be applied where diagrams are small to medium in size: the paper observes accuracy near 100% for models with fewer than about 10 connections and degradation beyond that.
  • The ablation implies that any practical deployment needs both a standards checker and a runtime debugger; removing both makes most complex tasks fail.
  • At roughly $0.17 and 275 seconds per model, the pipeline is cheap enough to regenerate models repeatedly during debugging.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the accuracy metric divides only by ground-truth block and connection counts, a generated model that adds spurious elements can still score 100%; the 94.5% figure would likely move if the metric penalized false positives or if the system faced held-out diagrams.
  • The nine-task benchmark is hand-selected and each diagram is paired with a textual explanation, so the natural next stress test is a larger, uncurated set of diagrams, especially ones with subsystems, which the investigator prompt explicitly forbids.
  • The reviewer's eight rules could be extracted into a standalone, LLM-free linter for Simulink connection descriptions, giving the same standards enforcement without per-task agent cost.
  • One could also reverse the pipeline: feed SimuGen's generated reports back as training data to teach smaller models the mapping from diagram layouts to wiring, addressing the data scarcity the paper cites.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. SimuGen is a multi-agent framework that takes a Simulink block-diagram image as input and produces executable MATLAB/Simulink code via six specialized agents: Investigator, Unit Test Reviewer, Block Builder, Executor, Debug Locator, and Report Writer, supported by a database of 50 Simulink block descriptions. The paper evaluates the system on nine hand-selected Simulink models spanning four application domains. With o4-mini as the visual Investigator and o3-mini for the remaining agents, the authors report an average reproduction accuracy of 94.5%, defined by Eq. (1) as the unweighted average of ground-truth block recall and ground-truth connection recall. Ablation results removing the Unit Test Reviewer and/or Debug Locator are presented in Table 4, and a detailed Bipolar Transistor case study is provided in Appendix E.

Significance. If the reported numbers were reliable, SimuGen would be a useful step toward automating Simulink model construction from diagrams, an area with little prior work. The modular agentic design, the inclusion of full prompts in Appendix C, the detailed Bipolar Transistor trace in Appendix E, and the public code repository are genuine strengths. However, the evaluation as written does not establish the headline accuracy: the metric ignores spurious blocks and connections, the ablation table is internally inconsistent, the results come from single runs on a small self-selected set, and the case study shows the pipeline adding connections that were not extracted from the diagram. The framework is promising, but the quantitative claims need re-evaluation before the paper can be accepted.

major comments (4)
  1. [§4.1, Eq. (1), and Appendix E] The accuracy metric in Eq. (1) is recall-only for both blocks and connections: the denominators are |B_GT| and |C_GT|, so extra or spurious elements in the generated model are never penalized. The Executor (§3.3) only checks that the code runs, not that the resulting model matches the diagram. Appendix E shows this is not hypothetical: the Block Builder's final Bipolar Transistor code contains connections that are absent from the Investigator's extracted connection list, including 'Voltage-Controlled Voltage source/RConn2' and 'Current-Controlled Current source/RConn2' to 'Electrical Reference/LConn1', and 'RLoad/LConn1' to both controlled-source LConn2 ports. Whether or not these additions happen to match the ground truth, the reported accuracy cannot distinguish faithful reconstruction from over-generation. I recommend reporting precision or F1 per block and per connection, along with Bmatch, Cmatch, and false-positive counts for each task.
  2. [§4.3, Table 4] The ablation table is internally inconsistent. Summing the 'w/o Unit Test Reviewer' column gives (100 + 73.3 + 95.5 + 100 + 92.8 + 61.65 + 95.3 + 96.15 + 91.25) / 9 = 89.55%, not the reported 86.21%. The 'w/o Debug Locator' average of 83.9% and the 'w/o All' average of 51.7% are recovered only if the '–' entries are counted as 0, but the table and text never state that a dash denotes a failed run with accuracy 0. These arithmetic and reporting issues directly affect the claim in §4.3 that the Unit Test Reviewer and Debug Locator have a synergistic effect; the claim needs corrected arithmetic and an explicit policy for failed runs.
  3. [§4.1–§4.2, Tables 1–2] All quantitative results come from a single run per task on nine hand-selected examples. There are no error bars, no repeated trials, and no held-out diagrams; Table 2 reports exactly one number per task. Because the selection of tasks and the development of the prompts and database are not described in a way that permits external validation, the 94.5% average is a point estimate with unknown run-to-run variability and unknown generalization to new diagrams. I recommend at least three to five independent runs per task with mean and standard deviation, and an evaluation on diagrams not used during development.
  4. [§3.3, §3.4, and Appendix E] The paper's own contribution statement says the Unit Test Reviewer cannot directly assess the correctness of connections, and the Executor only verifies that code runs. The Debug Locator in Appendix E compounds this problem: it reports that the generated code 'follows Implementation Details exactly' and sets Investigator_error=true because two lines target the same LConn2 port. However, the code at that point contains RLoad/LConn1-to-CCCS/LConn2 and RLoad/LConn1-to-VCVS/LConn2 connections that are not in the Investigator's connection list, so the alleged discrepancy was introduced by the Block Builder, not the Investigator. This breaks the modular error attribution that the framework relies on and means execution success is not evidence of reproduction fidelity.
minor comments (5)
  1. [§4.3] The heading 'Alblation' should be 'Ablation', and the text contains typos such as 'MEanwhile' and 'SimGen' (in Appendix E) that should be corrected.
  2. [§2.2] The text says 'As illustrated in Figure 2' when describing common LLM errors, but Figure 2 is the architecture overview; the errors appear to be shown in Figure 1.
  3. [§4.3] The sentence 'Overall, from Table 3, we observe...' should refer to Table 4, which is where the ablation results are presented.
  4. [Table 4] The dash '–' is used without a legend; the table should explicitly state whether a dash indicates a failed run (accuracy 0), a not-run condition, or a missing measurement.
  5. [Appendix E] The anonymous link to the full report should be replaced with a stable reference or removed, since the report is part of the review artifact and the link may not persist.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the central result is an externally evaluated empirical pipeline, and the only self-citation is non-load-bearing.

full rationale

SimuGen's central claim is an empirical evaluation: the framework converts diagram images, with a 50-block database and agent prompts, into executable Simulink code, and Table 2 reports measured accuracy against ground-truth models using the recall-based metric in Eq. (1). The derivation chain is not circular: no parameter is fitted to a subset of data and then reported as a prediction; the Unit Test Reviewer and Debug Locator are validated by ablation against the same ground-truth models; and no uniqueness theorem or ansatz is imported from the authors' prior work. The only co-author self-citation is [15] (AutoKaggle, with co-author Qianbo Zang), used in related work as an example of multi-agent data-science frameworks; it is not load-bearing for any claim. The recall-only form of Eq. (1), with denominators |B_GT| and |C_GT| ignoring false-positive blocks and connections, is a legitimate evaluation-validity and generalizability concern because the nine tasks are self-selected, but it is not a circular-equivalence in the derivation chain: the paper defines its accuracy metric, measures it externally against ground truth, and does not derive the framework's behavior from that metric. Hence no circularity step can be exhibited.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The ledger contains no invented physical entities. The free parameters are evaluation and database design choices. The central claim rests on the assumption that diagram images, the 50-block database, successful code execution, and the recall-only metric jointly measure what the paper calls accuracy.

free parameters (2)
  • Accuracy metric equal weighting = 0.5 block recall + 0.5 connection recall
    Eq. (1) defines accuracy as the unweighted mean of two recall terms; this hand-chosen weighting drives every reported accuracy number.
  • Simulink block database = 50 blocks, selected by authors
    The 50-block database in Section 3.2 was constructed by the authors and is used for all tasks; its coverage directly affects the results.
assumptions (4)
  • domain assumption Simulink diagram images contain enough visual information for an LLM to identify all blocks and connections.
    The whole pipeline assumes the Investigator can extract correct block types and port connections from the input image (Section 3.3 and prompts in Appendix C).
  • domain assumption Successful execution of the generated MATLAB code implies a correct simulation model.
    The Executor only runs code that builds the model; the paper does not compare simulation outputs numerically, so 'successful execution' is a proxy for correctness.
  • domain assumption The ground truth models used for evaluation are correct and complete.
    Accuracy is measured against GT models in Table 1; if any GT model is wrong or simplified, the metric is invalid.
  • ad hoc to paper The eight reviewer rules are sufficient to validate connection integrity.
    The Unit Test Reviewer's checks are hand-defined for this work (Section 3.3, Appendix C) and the paper itself concedes the reviewer cannot directly assess correctness of block connections.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SimuGen: Multi-modal Agentic Framework for Constructing Block Diagram-Based Simulation Models." pith.science (2026). https://pith.science/paper/EKQGBPD3

@misc{pith2026250615695,
  author       = {Pith},
  title        = {Pith review of: SimuGen: Multi-modal Agentic Framework for Constructing Block Diagram-Based Simulation Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EKQGBPD3}},
  note         = {Machine review of arXiv:2506.15695}
}
read the original abstract

Recent advances in large language models (LLMs) have shown impressive performance in mathematical reasoning and code generation. However, LLMs still struggle in the simulation domain, particularly in generating Simulink models, which are essential tools in engineering and scientific research. Our preliminary experiments indicate that LLM agents often fail to produce reliable and complete Simulink simulation code from text-only inputs, likely due to the lack of Simulink-specific data in their pretraining. To address this challenge, we propose SimuGen, a multimodal agent-based framework that automatically generates accurate Simulink simulation code by leveraging both the visual Simulink diagram and domain knowledge. SimuGen coordinates several specialized agents, including an investigator, unit test reviewer, code generator, executor, debug locator, and report writer, supported by a domain-specific knowledge base. This collaborative and modular design enables interpretable, robust, and reproducible Simulink simulation generation. Our source code is publicly available at https://github.com/renxinxing123/SimuGen_beta.

Figures

Figures reproduced from arXiv: 2506.15695 by the authors.

Figure 1
Figure 1. Common errors when users want LLM to generate MATLAB code for constructing Simulink [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Overview of the agent-based architecture used in the SimuGen framework. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Simulation Code Generation for Fluid Systems using Large Language Models: Benchmarking Models and Prompting Strategies

    cs.LG 2026-07 conditional novelty 5.0 of 10

    Across 10 LLMs and 5 prompting strategies, curated context yields the best syntactic quality, yet simulation fidelity stays near zero for nearly all configurations.

Reference graph

Works this paper leans on

59 extracted references · 47 canonical work pages · cited by 1 Pith paper

  1. [1]

    Generative artificial intelligence for model-based graphical programming in automotive function development

    Abdelrahman Abdalla, Harsh Pandey, Behzad Shomali, Joschka Schaub, Arne Müller, Markus Eisenbarth, and Jakob Andert. Generative artificial intelligence for model-based graphical programming in automotive function development. Available at SSRN 5153452, 2024

    work page 2024

  2. [2]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  3. [3]

    Evaluating large language models trained on code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021

    arXiv 2021

  4. [4]

    Extending context window of large language models via semantic compression

    Weizhi Fei, Xueyan Niu, Pingyi Zhou, Lu Hou, Bo Bai, Lei Deng, and Wei Han. Extending context window of large language models via semantic compression. In Findings of the Association for Computational Linguistics ACL 2024 , pages 5169–5181, 2024. 9

    work page 2024

  5. [5]

    Budgetmlagent: A cost-effective llm multi-agent system for automating machine learning tasks

    Shubham Gandhi, Manasi Patwardhan, Lovekesh Vig, and Gautam Shroff. Budgetmlagent: A cost-effective llm multi-agent system for automating machine learning tasks. In Proceedings of the 4th International Conference on AI-ML Systems , pages 1–9, 2024

    work page 2024

  6. [6]

    CRITIC: Large language models can self-correct with tool-interactive critiquing

    Zhibin Gou, Zhihong Shao, Yeyun Gong, yelong shen, Yujiu Yang, Nan Duan, and Weizhu Chen. CRITIC: Large language models can self-correct with tool-interactive critiquing. The Twelfth International Conference on Learning Representations, 2024

    work page 2024

  7. [7]

    DS-agent: Automated data science by empowering large language models with case-based reasoning

    Siyuan Guo, Cheng Deng, Ying Wen, Hechang Chen, Yi Chang, and Jun Wang. DS-agent: Automated data science by empowering large language models with case-based reasoning. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference on Ma...

    work page 2024

  8. [8]

    Data interpreter: An llm agent for data science

    Sirui Hong, Yizhang Lin, Bang Liu, Bangbang Liu, Binhao Wu, Ceyao Zhang, Chenxing Wei, Danyang Li, Jiaqi Chen, Jiayi Zhang, et al. Data interpreter: An llm agent for data science. arXiv preprint arXiv:2402.18679, 2024

    arXiv 2024

Show all 59 references
  1. [9]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  2. [10]

    Swe-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations , 2024

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. Swe-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations , 2024

    2024

  3. [11]

    DSBench: How far are data science agents from becoming data science experts? The Thirteenth International Conference on Learning Representations , 2025

    Liqiang Jing, Zhehui Huang, Xiaoyang Wang, Wenlin Yao, Wenhao Yu, Kaixin Ma, Hongming Zhang, Xinya Du, and Dong Yu. DSBench: How far are data science agents from becoming data science experts? The Thirteenth International Conference on Learning Representations , 2025

    2025

  4. [12]

    Camel: Communicative agents for" mind" exploration of large language model society

    Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for" mind" exploration of large language model society. Advances in Neural Information Processing Systems, 36:51991–52008, 2023

    2023

  5. [13]

    Long-context llms struggle with long in-context learning

    Tianle Li, Ge Zhang, Quy Duc Do, Xiang Yue, and Wenhu Chen. Long-context llms struggle with long in-context learning. Transactions on Machine Learning Research, 2024

    2024

  6. [14]

    Competition-level code generation with alphacode

    Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with alphacode. Science, 378(6624):1092–1097, 2022

    2022

  7. [15]

    Autokaggle: A multi-agent framework for autonomous data science competitions

    Ziming Li, Qianbo Zang, David Ma, Jiawei Guo, Tianyu Zheng, Minghao Liu, Xinyao Niu, Yue Wang, Jian Yang, Jiaheng Liu, Wanjun Zhong, Wangchunshu Zhou, Stephen Huang, and Ge Zhang. Autokaggle: A multi-agent framework for autonomous data science competitions. ICLR 2025 Third Wor...

    2025

  8. [16]

    Hm- rag: Hierarchical multi-agent multimodal retrieval augmented generation

    Pei Liu, Xin Liu, Ruoyu Yao, Junming Liu, Siyuan Meng, Ding Wang, and Jun Ma. Hm- rag: Hierarchical multi-agent multimodal retrieval augmented generation. arXiv preprint arXiv:2504.12330, 2025

    2025 arXiv

  9. [17]

    Toward automated simulation research workflow through llm prompt engineering design

    Zhihan Liu, Yubo Chai, and Jianfeng Li. Toward automated simulation research workflow through llm prompt engineering design. Journal of Chemical Information and Modeling , 65(1):114–124, 2024

    2024

  10. [18]

    Towards fully autonomous research powered by llms: Case study on simulations

    Zhihan Liu, Yubo Chai, and Jianfeng Li. Towards fully autonomous research powered by llms: Case study on simulations. arXiv preprint arXiv:2408.15512, 2024

    2024 arXiv

  11. [19]

    Requirements-driven slicing of simulink models using llms

    Dipeeka Luitel, Shiva Nejati, and Mehrdad Sabetzadeh. Requirements-driven slicing of simulink models using llms. In 2024 IEEE 32nd International Requirements Engineering Conference Workshops (REW), pages 72–82. IEEE, 2024

    2024

  12. [20]

    Simulink - simulation and model-based design, 2025

    MathWorks. Simulink - simulation and model-based design, 2025. 10

    2025

  13. [21]

    Codegen: An open large language model for code with multi-turn program synthesis

    Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis. In The Eleventh International Conference on Learning Representations , 2023

    2023

  14. [22]

    ChatDev: Communicative agents for software development

    Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, Juyuan Xu, Dahai Li, Zhiyuan Liu, and Maosong Sun. ChatDev: Communicative agents for software development. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editor...

    2024

  15. [23]

    Slgpt: Using transfer learning to directly generate simulink model files and find bugs in the simulink toolchain

    Sohil Lal Shrestha and Christoph Csallner. Slgpt: Using transfer learning to directly generate simulink model files and find bugs in the simulink toolchain. In Proceedings of the 25th International Conference on Evaluation and Assessment in Software Engineering , pages 260– 265, 2021

    2021

  16. [24]

    Bissyandé

    Xunzhu Tang, Kisub Kim, Yewei Song, Cedric Lothritz, Bei Li, Saad Ezzini, Haoye Tian, Jacques Klein, and Tegawendé F. Bissyandé. CodeAgent: Autonomous communicative agents for code review. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors,Proceedings of the 2024 Co...

    2024

  17. [25]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems , 35:24824–24837, 2022

    2022

  18. [26]

    Swe-agent: Agent-computer interfaces enable automated software engineering

    John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems , 37:50528–50652, 2024

    2024

  19. [27]

    If llm is the wizard, then code is the wand: A survey on how code empowers large language models to serve as intelligent agents

    Ke Yang, Jiateng Liu, John Wu, Chaoqi Yang, Yi R Fung, Sha Li, Zixuan Huang, Xu Cao, Xingyao Wang, Yiquan Wang, et al. If llm is the wizard, then code is the wand: A survey on how code empowers large language models to serve as intelligent agents. arXiv preprint arXiv:2401.00812, 2024

    2024 arXiv

  20. [28]

    Chain of agents: Large language models collaborating on long-context tasks

    Yusen Zhang, Ruoxi Sun, Yanfei Chen, Tomas Pfister, Rui Zhang, and Sercan Arik. Chain of agents: Large language models collaborating on long-context tasks. Advances in Neural Information Processing Systems, 37:132208–132237, 2024

    2024

  21. [29]

    A survey of large language models for code: Evolution, benchmarking, and future trends

    Zibin Zheng, Kaiwen Ning, Yanlin Wang, Jingwen Zhang, Dewu Zheng, Mingxi Ye, and Jiachi Chen. A survey of large language models for code: Evolution, benchmarking, and future trends. arXiv preprint arXiv:2311.10372, 2023. 11 A Appendix: Challenges To illustrate common conceptua...

    2023 arXiv

  22. [30]

    Provide the blocks needed and their connections (there should be a Scope to observe the result)

  23. [31]

    request_blocks

    Generate the simulation code using matlab.engine. Note: Only generate blocks and connections; there is no need to set parameters. Blocks and Library Paths Table A.1: Blocks used in the generated RC circuit model Block Name Library Path Solver Configuration Simscape > Commonly ...

  24. [32]

    **The blocks used in the simulation:** A list of blocks utilized in the simulation

  25. [33]

    PortX and PortY are the real internal port names, not the visual labels; BlockA and BlockB are the block names shown in the diagram

    **Connections Description:** This details how the blocks are connected, with each connection formatted as: ‘‘‘ BlockA (BlockA’s block type) PortX (**related parameter setting to match the port number if necessary**)<-> BlockB (BlockB’s block type) PortY (** related parameter s...

  26. [34]

    **Identify the exist of block list:** - Verify if the identified block are listed first, such as: Swing bus 1pu (Load Flow Source) Electrical Reference 1 (Electrical Reference) Load 90 MW 30 Mvar (Wye-Connected Load)

  27. [35]

    The blocks used in the simulation

    **Identify any Extra Blocks:** - Verify if there are blocks mentioned in "The blocks used in the simulation" (** A specific block, not the real block name**) that do not appear in the Connections Description

  28. [36]

    **Formatting of block name:** - Make sure the block names are NEVER included the special symbol ’/’

  29. [37]

    **Formatting of connection description** - Make sure the formatting of connection is strictily formmated as: BlockA (BlockA’s block type) PortX (**related parameter setting to match the port number if necessary**) For example: Current-Controlled Current Source (-h_feIb) (Curre...

  30. [38]

    **Validate Parameter Settings in Connections:** - Check that the (**related parameter setting to match the port number if necessary**) for each connection is correctly provided and matches the expected configuration from the **simulation blocks description**

  31. [39]

    **Detect Duplicate Connections:** 20 - Check if there are any duplicate connections where the **same block’s Input port (not Output port)** is connected more than once (with **ONLY** the exception of Electrical Reference and Solver Configuration). For example: ’’’ Discrete-Tim...

  32. [40]

    **Validate Block Connection Types:** - Ensure that no block is connected to another block that only has dedicated output port, for example: ’’’ Gain 1 (Gain) 1 <-> Constant 1 (Constant) 1 ’’’ is not allowed, since Constant is a block only has dedicated output - Ensure that no ...

  33. [41]

    The blocks used in the simulation

    **Verify Complete Port Connections:** - Under the premise of the already set parameter settings, check whether every input and output port of each block has a connection. For example: Discrete-Time Integrator (Discrete-Time Integrator) 1 (‘ExternalReset‘ = ‘none‘, ‘InitialCond...

  34. [42]

    - Review the detailed **blocks description**, which lists block types, paths, and port description

    **Analyze and Understand:** - Read the provided implementation code information extracted by the Investigator. - Review the detailed **blocks description**, which lists block types, paths, and port description. - Examine the set of **functions** that you are permitted to use

  35. [43]

    - Keep in mind that the error message might be triggered by issues originating in earlier code

    **Locate Error Source:** - Analyze the execution code and error message to determine the 5-10 lines of code that are most likely causing the error, as well as an additional 5-10 lines of code that are related to the error. - Keep in mind that the error message might be trigger...

  36. [44]

    This means verifying that the block types and the corresponding port names in the code are identical to those specified

    **Assess Connection and Parameter Integrity to Determine Error Origin or Provide Fix Recommendations:** - First, check if the code’s connection configuration exactly matches the Investigator’s Implementation Details. This means verifying that the block types and the correspond...

  37. [45]

    What real -world system or process does it aim to represent or replicate?

    **What is the simulation about?** Describe the purpose, context, and overall objective of the simulation. What real -world system or process does it aim to represent or replicate?

  38. [46]

    Clearly outline the step-by-step process of how the simulation is structured

    **What are the main simulation steps?** Break down the simulation **(not code implementation)** into distinct stages or functional modules. Clearly outline the step-by-step process of how the simulation is structured

  39. [47]

    **What theoretical knowledge and mathematical modelling are involved in each step ?** For **above** every simulation step, explain the relevant theoretical foundations and mathematical models involved (e.g., control theory, physical modelling, system dynamics, signal flow, etc.)

  40. [48]

    request_blocks

    **How is it implemented in code?** Provide and explain the corresponding code for **each above step** (you may reorganize the code to match the stepwise structure). Highlight how the code reflects both theoretical concepts and the simulation block diagram. Your final report mu...

  41. [49]

    The blocks used in the simulation

    Block List Existence: - The provided "The blocks used in the simulation" list is consistent with the request_blocks from the JSON. Although multiple resistor-instances (R1, RBias, R2, RLoad) appear in the connections, they all use the "Resistor" type from the list

  42. [50]

    All block names in the provided simulation list appear in the connections description

    Extra Blocks: - No extra blocks are present. All block names in the provided simulation list appear in the connections description

  43. [51]

    Formatting of Block Name: - All block names are formatted correctly with no "/" symbols

  44. [52]

    - There is no unnecessary parameter explanation text; the formatting rules are followed

    Formatting of Connection Description: - Each connection is shown in the required format: BlockName (BlockType) PortName <-> BlockName (BlockType) PortName. - There is no unnecessary parameter explanation text; the formatting rules are followed

  45. [53]

    This is acceptable based on the simulation blocks description

    Parameter Settings in Connections: - No parameter settings (parenthesized settings) were expected or provided. This is acceptable based on the simulation blocks description

  46. [54]

    Input Port

    Duplicate Connections: - There are several connections to Electrical Reference’s LConn1. However, since Electrical Reference (and Solver Configuration) is allowed to have duplicate input connections, no duplicate-connection error is flagged. - No other dedicated input port (as...

  47. [55]

    electrical conserving

    Block Connection Types: - All connections are made between ports of similar "electrical conserving" type (or between the dedicated ports in PS-Simulink Converter, Electrical Reference, Solver Configuration, and Scope). No block that is only a dedicated output is connected to a...

  48. [56]

    Investigator_unit_test_pass

    Complete Port Connections: - For the Capacitor, AC Voltage Source, Resistors, Electrical Reference, Voltage Sensor, PS-Simulink Converter, Solver Configuration, and Scope, every defined port has a connection. - However, for the Voltage-Controlled Voltage Source (which has four...

  49. [57]

    Analysis and Understanding: - The code adds blocks and makes connections exactly as specified by the Investigato r’s block list and port names. Each add_block call uses the proper library path and each add_line call refers to the designated port (for example LConn2 on both Vol...

  50. [58]

    The second port already has a line connection

    Error Location: - The error message "The second port already has a line connection" is thrown at the call: --eng.add_line(model_name, ’RLoad/LConn1’, ’Current-Controlled Current source/LConn2 ’, nargout=0) -- The related lines also include the earlier connection made with: --e...

  51. [59]

    Investigator_error

    Connection and Parameter Integrity: - A careful check reveals that the block types, port names, and connections match the Investigator’s Implementation Details. - The error is not caused by erroneous setting of the block’s path or wrong parameters, but it results from the fact...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.