Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

JARVIS: A Multi-Agent Code Assistant for High-Quality EDA Script Generation

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read JARVIS claims that coupling a domain-adapted LLM with an AST-based compiler and rule enforcement in a multi-agent loop reaches 92/93/81 pass@1 on EDA script benchmarks.

desk verdict Credible industrial multi-agent code generation paper with clean ablations, but the headline pass@1 numbers are likely overstated until guardrail refusals and benchmark provenance are reported. read the letter →

arxiv 2505.14978 v2 pith:PS44P7YE submitted 2025-05-20 cs.SE cs.AIcs.LG

Ghasem Pasandi , Kishor Kunal , Varun Tej , Kunjal Shah , Hanfei Sun , Sumit Jain , Chunhui Li , Chenhui Deng
show 3 more authors
Teodor-Dumitru Ene Haoxing Ren Sreedhar Pratty
This is my paper
classification cs.SEcs.AIcs.LG
keywords EDAscriptgenerationmulti-agentLLMsyntheticdataAST-basedcompilerretrieval-augmenteddomainadaptationchipdesignautomationcode
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that a domain-specific code assistant for a custom VLSI EDA tool can be built by combining a language model fine-tuned on synthetic data with a set of specialized tools: an AST-based compiler that checks object-attribute relationships, a rule-enforcement module, retrieval, a code-fixing agent, and a guardrail, arranged in a multi-agent refinement loop. If correct, the framework would overcome the data scarcity and hallucination problems that make general LLMs near-useless on proprietary EDA APIs. The central evidence is a set of three internal benchmarks on which the full flow scores 92%, 93%, and 81% pass@1, versus 67%, 62%, and 43% for the best single model without the loop. The authors argue this shows LLMs can be made reliable in specialized engineering domains through tool-assisted self-correction.

What carries the argument

The load-bearing mechanism is the AST-based custom compiler and the API graph built from the tool's man page. The compiler converts generated code into an abstract syntax tree, walks the graph checking every object-attribute relationship against the known tool API, and, when a hallucinated attribute appears, returns the list of valid attributes or the shortest path from the object to the desired attribute. This structured feedback is fed into a ReAct-style top agent that iterates over episodes; each episode ends with a progress summary that seeds the next, which prevents the infinite-loop failure mode of naive agent loops. A second mechanism is the synthetic data pipeline: a random code generator traverses the same API graph to produce syntactically valid snippets, and an LLM adds line-by-line comments and generates questions, producing 35,000 extra fine-tuning examples.

What would settle it

Independently re-running the three benchmarks with a blinded scoring rubric and counting guardrail refusals in the denominator would either reproduce the 92/93/81 figures or reveal that selective submission and permissive scoring drive them.

Watch

Extended reading notes

Core claim

The paper's central claim is that EDA script generation for a proprietary VLSI tool becomes reliable only when a domain-adapted LLM is embedded in a multi-agent refinement loop with tool feedback. The authors report that the full JARVIS flow achieves 92% pass@1 on B-easy, 93% on B-medium, and 81% on B-hard, compared with 67%, 62%, and 43% for the best-performing single model (domain-adapted with synthetic data but no agent loop), and near 0% for off-the-shelf models. The improvement is attributed to each component: synthetic fine-tuning data from an AST-based random code generator plus LLM comments, a custom compiler that simulates code against a tool API graph and returns valid attribute paths, rule enforcement that encodes domain recipes, retrieval from tool documentation, a code-fixing agent that supplies example snippets, and a guardrail that filters final output. The ablations show removing any component lowers accuracy, with RuleEnforce contributing the largest gain on hard questions.

Load-bearing premise

The reported pass rates assume that the authors' self-constructed benchmarks and their human evaluation of medium and hard outputs are an unbiased measure of script-generation quality.

Editorial extensions

If this is right

  • If the results hold, any proprietary engineering tool with parseable documentation can get a similar assistant: parse the man page into an API graph, generate synthetic snippets, fine-tune a domain model, and wrap it in the agent loop.
  • The 25-point gain on B-hard from RuleEnforce suggests that for complex, multi-step scripts, explicit domain rules matter more than additional retrieval or examples.
  • The near-zero scores of off-the-shelf models imply that general LLM coding assistants should not be expected to handle custom tool APIs without domain adaptation and verification tools.
  • The multi-episode termination summary, by preventing repeated tool calls, makes agent-based code generation more predictable for production use.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The authors do not report how often the Guardrail Agent refuses to answer; since the pass@1 denominator presumably excludes refusals, the 81% B-hard figure may overstate raw generation ability. Publishing refusal counts would make the headline number interpretable.
  • The B-medium and B-hard scores rely on the authors' own human evaluation with no published rubric or inter-rater statistics; an independent blind re-scoring could shift the numbers.
  • The same framework should transfer to other data-scarce scripting domains, such as FPGA tooling or EDA physical-design flows, provided an API graph can be extracted from documentation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents JARVIS, a multi-agent framework for generating Python scripts for an internal, custom VLSI EDA tool. The authors construct a synthetic training set of 35,000 code snippets by randomly traversing the tool's API graph and using an LLM to add comments and questions; they then fine-tune ChipNeMo with this data (DAPT+DSFT). The inference pipeline is a multi-agent ReAct flow that combines a code generator, a RuleEnforce tool, an AST-based code compiler that simulates code and returns corrective feedback, RAG over tool documentation, a code-fixing agent, and a guardrail agent that filters the final output. The system is evaluated on three self-constructed benchmarks (B-easy: 150 questions, B-medium: 30, B-hard: 20) using pass@1 with temperature 0. The full flow achieves 92%, 93%, and 81% on B-easy, B-medium, and B-hard, respectively, compared to best single-model rates of 67%, 62%, and 43% (Tables I and II).

Significance. If the reported results hold, the paper makes a useful engineering contribution: it demonstrates a concrete recipe for adapting LLMs to a data-scarce, domain-specific code-generation setting by combining synthetic data derived from tool documentation with a multi-agent feedback loop that provides structured, compiler- and rule-based corrections. The experimental design has notable strengths: deterministic decoding (temperature 0), pass@1 as the metric, and ablation studies that isolate the contributions of RuleEnforce, RAG, the code-fixing agent, and the initial model. However, the validity of the central claim — that the full JARVIS flow delivers 92/93/81 pass@1 and that these gains represent general improvement rather than measurement artifacts — is currently undermined by three load-bearing evaluation issues: unreported guardrail refusals, benchmark construction that overlaps with the training-data pipeline, and human evaluation without a published rubric or inter-rater statistics. These issues are fixable with additional reporting and analysis, but they must be addressed before the quantitative claims can be accepted.

major comments (4)
  1. [Section IV-6 and Algorithm 2 (line 17)] The Guardrail Agent can decline to answer a query by emitting a standardized message ('Sorry, I don't have enough knowledge...') instead of code. The paper never reports how often this refusal path is taken on each benchmark. If the reported pass@1 figures are computed only over outputs that pass the guardrail, they are not comparable to the single-model baselines in Table I, which always emit a code answer, and they would overstate task-level accuracy. The authors should report the refusal counts per benchmark, and compute pass@1 with refusals counted as failures (or report both conditional and unconditional pass@1).
  2. [Sections III and V] The evaluation benchmarks appear to be constructed from the same tool API graph (man-page/API documentation) that was used to generate the 35,000 synthetic training snippets in Section III. In particular, B-easy is scored by string-matching 'desired APIs', which may reward recall of the training distribution rather than genuine script-generation ability. This is a circularity risk that affects the central claim. The authors should either provide evidence that the evaluation questions were authored independently of the SDG pipeline, or quantify the overlap between the training snippets and the benchmark questions (e.g., API-ngram overlap, question semantic similarity), and report performance on a held-out set of questions that are not derived from the same graph.
  3. [Section V (B-medium and B-hard)] The B-medium and B-hard results (93% and 81%) rest entirely on human evaluation, but the paper gives no scoring rubric, no description of how many annotators evaluated each answer, and no inter-rater agreement statistics. These numbers are central to the claimed improvement and are not independently auditable. Please provide the evaluation rubric (e.g., what counts as a correct script, how partial credit is handled), the number of independent annotators, and inter-rater agreement (e.g., Cohen's kappa) for both benchmarks.
  4. [Abstract and Section II/V] The abstract claims 'significant improvements over state-of-the-art domain-specific models,' but the experimental comparison set is limited to GPT-4o, LLaMa 3.1, and three in-house ChipNeMo variants, all evaluated on the authors' own benchmarks. The related work discusses TOOLLLM and CodeAgent, but these existing domain-specific or agent-based approaches are not compared against. Either add comparisons to prior EDA LLM agents that can run on this task, or revise the 'state-of-the-art' claim to be scoped to the systems actually compared.
minor comments (5)
  1. [Algorithm 2] The configuration values for TimeLimit, ItrLimit, and the number of multi-episode iterations are not reported; please provide these hyperparameters for reproducibility.
  2. [Listing 2 caption] The caption contains a typo: 'Linsting 1' should be 'Listing 1'.
  3. [Section V-A] The benchmark sizes are small (B-easy: 150, B-medium: 30, B-hard: 20) and the reported differences between configurations are often a few percentage points; the paper should report exact counts or confidence intervals so that the magnitude of the gains can be assessed.
  4. [Section V] The description of the B-easy auto-evaluation is ambiguous: please specify whether 'string match for the desired APIs' means exact API-name matching, whether the order of API calls matters, and how partial matches are scored.
  5. [References] Several references have formatting or typographical issues, including [4] with an extra space, [12] with a malformed author entry, and [14] with an incomplete author list; a careful pass over the bibliography is needed.

Circularity Check

0 steps flagged · score 0.0 of 10

No construction-level circularity; empirical validity threats are noted but do not reduce to a fit or self-citation chain.

full rationale

The paper's central claim is an empirical performance comparison, not a derivation, and I find no step in which a predicted quantity is defined in terms of a fitted quantity or in which a load-bearing conclusion rests on an unverified self-citation. The contribution of each component is supported by explicit ablations (Table II), and the base ChipNeMo model is evaluated in this paper rather than merely assumed, so citing prior work for ChipNeMo is not a load-bearing circularity. The evaluation does carry validity concerns: the SDG training data and the benchmarks are both tied to the same internal tool API graph (Sections III and V), B-medium and B-hard rely on unaudited human evaluation, and the Guardrail's refusal path (Algorithm 2 line 17, Section IV-6) is not reported, leaving the pass@1 denominator unclear. These are measurement and contamination threats, not construction-level equivalence: the paper does not state that benchmark questions were generated by the SDG pipeline, nor that pass@1 is computed only over accepted outputs, so I cannot exhibit the specific reduction required to declare the result circular.

Assumptions & free parameters 4 free parameters · 3 assumptions · 2 invented entities

Everything the central claim rests on that the reader did not pay for upstream: the fine-tuned ChipNeMo-jarvis weights (set by the SFT hyperparameters and the 35,000 synthetic samples), the completeness of the tool's man-page/API graph used by the random code generator and the compiler, the validity of LLM-generated comments as intent labels, and an evaluation designed by the same team from the same documentation. No new physical entities are proposed; the JARVIS components are software artifacts whose utility is evidenced only within this paper's internal benchmarks.

free parameters (4)
  • SFT hyperparameters (learning rate, batch size, epochs) = 5e-6, 128, 1 epoch
    These training settings produce the ChipNeMo-jarvis weights that generate the initial code for all experiments (Section III).
  • Synthetic dataset size = 35,000 examples
    The number of random-AST and LLM-labeled snippets created for DSFT; the paper reports only a with/without ablation, not a scaling analysis (Section III).
  • RuleEnforce retrieved rule count = top 10 rules
    The number of auto-extracted rules appended for each query; ablations show RuleEnforce moves B-hard accuracy by about 25 percentage points (Section IV-2, Table II).
  • Multi-episode count and iteration limits = not reported
    TimeLimit and ItrLimit in Algorithm 2 define how many refinement rounds the agent runs; the paper does not report their values, leaving a reproducibility gap.
assumptions (3)
  • domain assumption The internal EDA tool's man page and API graph are a complete enough specification to generate valid code with the random AST generator and to verify candidate scripts with the compiler.
    The entire SDG (Algorithm 1) and compiler (Section IV-3) rely on this graph being faithful; an incomplete graph would distort both training data and acceptance decisions.
  • domain assumption LLM-generated comments and questions (via Nemotron-340B) preserve the intent of the randomly generated code.
    SFT question/answer pairs and the Code Fixing Agent's intent inference (Section IV-5) both depend on comments accurately reflecting code behavior.
  • ad hoc to paper The B-easy/medium/hard benchmarks, hand-built by the authors, measure real-world EDA script quality rather than in-distribution retrieval.
    Benchmark construction is described only in Section V; no external or held-out evaluation is provided.
invented entities (2)
  • JARVIS multi-agent flow (Top Agent, Code Fixing Agent, Guardrail Agent, RuleEnforce, AST-based Code Compiler)
    purpose: Coordinates code generation, rule insertion, simulation feedback, and final quality gating of generated EDA scripts.
    The value of each software component is demonstrated only inside the paper's internal benchmarks; no external validation or released artifact provides an independent check.
  • ChipNeMo-jarvis, the DSFT-trained code generator
    purpose: Generates initial and code-fixing snippets for the target EDA tool using SFT on 35,000 synthetic examples.
    Its performance is measured only on the authors' self-constructed benchmarks, not on an external held-out set.

how reviews work

0 comments
Cite this review

Pith. "Pith review of JARVIS: A Multi-Agent Code Assistant for High-Quality EDA Script Generation." pith.science (2026). https://pith.science/paper/PS44P7YE

@misc{pith2026250514978,
  author       = {Pith},
  title        = {Pith review of: JARVIS: A Multi-Agent Code Assistant for High-Quality EDA Script Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PS44P7YE}},
  note         = {Machine review of arXiv:2505.14978}
}
read the original abstract

This paper presents JARVIS, a novel multi-agent framework that leverages Large Language Models (LLMs) and domain expertise to generate high-quality scripts for specialized Electronic Design Automation (EDA) tasks. By combining a domain-specific LLM trained with synthetically generated data, a custom compiler for structural verification, rule enforcement, code fixing capabilities, and advanced retrieval mechanisms, our approach achieves significant improvements over state-of-the-art domain-specific models. Our framework addresses the challenges of data scarcity and hallucination errors in LLMs, demonstrating the potential of LLMs in specialized engineering domains. We evaluate our framework on multiple benchmarks and show that it outperforms existing models in terms of accuracy and reliability. Our work sets a new precedent for the application of LLMs in EDA and paves the way for future innovations in this field.

Figures

Figures reproduced from arXiv: 2505.14978 by the authors.

Figure 1
Figure 1. Overview of the JARVIS framework, illustrating dif [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Example for a tool API graph constructed from man [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Overview of the multi-agent flow, listing the vari [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 3
Figure 3. Figure 3: Next, in case of hallucination, our compiler returns the [PITH_FULL_IMAGE:figures/full_fig_p005_3.png]
Figure 4
Figure 4. Figure 4: This demonstrates the great potential of our multi-agent [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. AgenticECO: An Agentic Framework for ECO on 3D Integrated Circuits

    cs.AI 2026-08 conditional novelty 6.0 of 10

    An agentic framework with a minimal-disturbance router and independent verifier clears post-route hybrid-bond spacing defects in 3D-IC designs (7/9 on one backbone, 9/9 on another) with low disturbance and zero clock-...

Reference graph

Works this paper leans on

25 extracted references · 10 canonical work pages · cited by 1 Pith paper

  1. [1]

    M. AI. Codellama: A large language model for code generation, 2024

    work page 2024

  2. [2]

    S. AI. Stablecode: A large language model for code generation, 2024

    work page 2024

  3. [3]

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. Pond ´e, et al. Evaluating large language models trained on code. ArXiv/2107.03374, 2021

    arXiv 2021

  4. [4]

    Fakhoury, A

    S. Fakhoury, A. Naik, G. Sakkas, S. Chakraborty, and S. K. Lahiri. Llm- based test-driven interactive code generation: User study and empirical evaluation. arXiv/2404.10100, 2024

    arXiv 2024

  5. [5]

    Y . Ge, W. Hua, K. Mei, j. ji, J. Tan, S. Xu, Z. Li, and Y . Zhang. Openagi: When llm meets domain experts. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, pages 5539–5568, 2023

    work page 2023

  6. [6]

    L. Gong, M. Elhoushi, and A. Cheung. Ast-t5: Structure-aware pretrain- ing for code generation and understanding. arXiv/2401.03003, 2024

    arXiv 2024

  7. [7]

    Q. Gu. Llm-based code generation method for golang compiler testing. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, page 2201–2203, 2023

    work page 2023

  8. [8]

    Guo and Y

    X. Guo and Y . Chen. Generative ai for synthetic data generation: Methods, challenges and the future. arXiv/2403.04190, 2024

    arXiv 2024

Show all 25 references
  1. [9]

    N. Jain, T. Zhang, W.-L. Chiang, J. E. Gonzalez, K. Sen, and I. Sto- ica. Llm-assisted code cleaning for training accurate code generators. arXiv/2311.14904, 2023

    2023 arXiv

  2. [10]

    S. Jain, A. Dora, K. S. Sam, and P. Singh. Llm agents improve semantic code search. arXiv/2408.11058, 2024

    2024 arXiv

  3. [11]

    Koziolek, S

    H. Koziolek, S. Gr ¨uner, R. Hark, V . Ashiwal, S. Linsbauer, and N. Es- kandani. Llm-based and retrieval-augmented control code generation. In Proceedings of the 1st International Workshop on Large Language Models for Code, page 22–29, 2024

    2024

  4. [12]

    F. Lin, D. J. Kim, Tse-Husn, and Chen. When llm-based code generation meets the software development process. arXiv/2403.15852, 2024

    2024 arXiv

  5. [13]

    Y .-C. Lin, A. Kumar, N. Chang, W. Zhang, M. Zakir, R. Apte, H. He, C. Wang, and J.-S. R. Jang. Novel preprocessing technique for data embedding in engineering code generation using large language model. arXiv/2311.16267, 2024

    2024 arXiv

  6. [14]

    Liu, T.-D

    M. Liu, T.-D. Ene, R. Kirby, C. Cheng, N. Pinckney, R. Liang, J. Alben, H. Anand, K. Kunal, Ismet, et al. Chipnemo: Domain-adapted llms for chip design. arXiv/2311.00176, 2023

    2023 arXiv

  7. [15]

    Z. Liu, W. Ping, R. Roy, P. Xu, C. Lee, M. Shoeybi, and B. Catan- zaro. Chatqa: Surpassing gpt-4 on conversational qa and rag. arXiv/2401.10225, 2024

    2024 arXiv

  8. [16]

    L. Long, R. Wang, R. Xiao, J. Zhao, X. Ding, G. Chen, and H. Wang. On llms-driven synthetic data generation, curation, and evaluation: A survey. arXiv/2406.15126, 2024

    2024 arXiv

  9. [17]

    D. Nam, A. Macvean, V . Hellendoorn, B. Vasilescu, and B. Myers. Using an llm to help with code understanding. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, 2024

    2024

  10. [18]

    A. Ni, S. Iyer, D. Radev, V . Stoyanov, W.-T. Yih, S. Wang, and X. V . Lin. LEVER: Learning to verify language-to-code generation with execution. In Proceedings of the 40th International Conference on Machine Learning, volume 202, pages 26106–26128, 2023

    2023

  11. [19]

    Adler, N

    Nvidia, :, B. Adler, N. Agarwal, A. Aithal, D. H. Anh, P. Bhattacharya, A. Brundyn, J. Casper, et al. Nemotron-4 340b technical report. arXiv/2406.11704, 2024

    2024 arXiv

  12. [20]

    B. Project. Starcoder: A large language model for code generation, 2024

    2024

  13. [21]

    Y . Qin, S. Liang, Y . Ye, K. Zhu, L. Yan, Y .-T. Lu, Y . Lin, X. Cong, X. Tang, B. Qian, S. Zhao, R. Tian, R. Xie, J. Zhou, M. H. Gerstein, D. Li, Z. Liu, and M. Sun. Toolllm: Facilitating large language models to master 16000+ real-world apis. arXiv 2307.16789, 2023

    2023 arXiv

  14. [22]

    Rafailov, A

    R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn. Direct preference optimization: Your language model is secretly a reward model. arXiv/2305.18290, 2024

    2024 arXiv

  15. [23]

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, and D. Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023

    2023

  16. [24]

    S. Yao, D. Yu, J. Zhao, I. Shafran, K. Narasimhan, and Y . Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023

    2023

  17. [25]

    Zhang, J

    K. Zhang, J. Li, G. Li, X. Shi, and Z. Jin. Codeagent: Enhancing code generation with tool-integrated agent systems for real-world repo-level coding challenges. arXiv/2401.07339, 2024

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.