Pith. sign in

REVIEW 4 major objections 5 minor 37 references

VeriTrace: Human-Like Temporal Exploration Completes Agentic Action Space

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read An agent that chooses its own waveform queries reaches 100% Pass@1 on the 156-problem VerilogEval-V2 benchmark.

desk verdict A plausible and genuinely new agentic-debugging idea that overreaches on causal attribution; the 100% claim needs a defined Pass@1 protocol and an ablation that isolates action-space control from mere waveform access. read the letter →

arxiv 2608.02878 v1 pith:7VGW5I6M submitted 2026-08-03 cs.AI

Yu-Tung Liu , Cunxi Yu This is my paper · ORCID
classification cs.AI
keywords VerilogRTLgenerationmulti-agentLLMsystemsdebuggingactionspaceAgenticTemporalExplorationwaveformtracingfunctionalcorrectnessEval-V2finitestatemachine
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Prior multi-agent systems for LLM-based Verilog generation plateau at roughly 95% Pass@1 on VerilogEval-V2 because their debugging agents can only look at a predetermined set of signals at a predetermined time. VeriTrace removes that restriction: a dedicated Inspector agent chooses which signals to query, which time window to inspect, and how many exploration rounds to run, using waveform snapshots to test and refine hypotheses about the root cause. The paper claims that with this complete debugging action space, the system reaches 100% Pass@1 on all 156 problems, the first perfect score on this benchmark, and that on the same LLM backbone it beats a reproduced prior system by 5.1 points. The accompanying ablation shows the effect is concentrated on FSM problems: adding the Inspector lifts the hardest three from 0–33% to 100% Pass@1. The underlying claim is that the final accuracy gap in LLM-based RTL generation is a debugging-action-space gap rather than a model-capability gap.

What carries the argument

The load-bearing mechanism is the inspector's query primitive $\mathrm{TRACE}(W, \sigma, t_{\text{start}}, t_{\text{end}})$ together with the deliberation loop $\tau_i, a_i = \mathrm{THINK}(S, C, t_{\text{err}}, \omega_1, \dots, \omega_i)$, where $a_i \in \{\mathrm{QUERY}, \mathrm{FINISH}\}$. The Testbench Agent's decision to dump the full waveform $W$ makes every signal available, so the Inspector's signal choices and time-window choices are never restricted by what a static analysis tool happened to expose. This joint control over signal selection, temporal bounds, and iteration depth is what the paper names Agentic Temporal Exploration; it converts debugging from pattern matching on a fixed snapshot into hypothesis-driven root-cause analysis.

What would settle it

A reader could rerun VeriTrace on VerilogEval-V2 with the Inspector still active but with signal selection fixed to top-level I/O signals and time windows frozen at the first mismatch; if Pass@1 stays at 100% and the FSM problems do not regress, then the claimed causal role of temporal exploration is not confirmed.

Watch

Extended reading notes

Core claim

The paper's central claim is that the persistent roughly 5% failure rate of state-of-the-art LLM multi-agent systems on VerilogEval-V2 is caused by an incomplete debugging action space, not by insufficient model capability. VeriTrace's Inspector agent operates over the full space: at iteration $i$ it chooses a signal set $\sigma_i$ and a time window $[t_{\text{start}}^i, t_{\text{end}}^i]$, obtains a natural-language waveform snapshot $\omega_i = \mathrm{TRACE}(W, \sigma_i, t_{\text{start}}^i, t_{\text{end}}^i)$ from the complete dumped waveform $W$, reasons over the accumulated evidence, and either queries again or emits a diagnostic report. The Debugger then makes targeted edits, and an edit is kept only if the mismatch count strictly decreases. On the full 156-problem benchmark this architecture achieves 100% Pass@1 with a frontier LLM, and on a controlled same-model comparison it outperforms the strongest reproduced open-source baseline by +5.1% (97.4% vs 92.3%). The authors attribute this entire gain to debugging agency, and the per-problem ablation supports that attribution on FSM problems, raising Probs 137, 149, and 155 from 0–33% to 100% Pass@1.

Load-bearing premise

The load-bearing premise is that dumping every signal's waveform and letting the Inspector query any signal in any time window exposes the root cause of any functional mismatch; the paper itself weakens this in the limitations by noting that industrial-scale designs would need adaptive waveform dumping or on-demand re-simulation.

Editorial extensions

If this is right

  • On the 156 single-module VerilogEval-V2 problems, an LLM with full joint control over signal selection, time-window bounds, and iteration depth can reach 100% Pass@1, the first perfect score reported on this benchmark.
  • The hardest failure cases, FSM designs with feedback, are precisely where the Inspector matters: the three most debug-intensive FSM problems go from 0–33% to 100% Pass@1 when the Inspector is present.
  • Because the Inspector consumes only targeted waveform snapshots, debugging tokens fall by about 18% overall and by more than 5x on one FSM problem (Prob 155), so more diagnostic freedom can be cheaper than blind retry.
  • The same architecture reaches 97.4% with an older LLM backbone and 100% with a newer one, so action-space design complements rather than replaces model improvement.
  • As presented, the approach presumes a golden testbench and a fully dumped waveform; scaling to multi-module industrial designs would require adaptive waveform dumping or on-demand re-simulation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the action-space argument should generalize beyond Verilog to other LLM debugging loops where the agent's observations are restricted in advance, such as timing-violation analysis, protocol-trace debugging, or hardware/software co-simulation; the broader principle may be to let the agent choose its own evidence.
  • Editorial inference: the per-problem ablation could be turned into a controlled instrument by degrading only one dimension at a time, disabling signal selection, then freezing the time window, then capping iterations, which would map the Pass@1 drop per dimension and identify exactly where the 5% gap enters.
  • Editorial inference: if debugging agency is truly the bottleneck, then rerunning the same architecture with different LLM families should preserve the ordering over baselines even when absolute accuracy varies, since the action-space advantage would be model-agnostic; this is a direct consequence the paper does not test.
  • Editorial inference: the 18% token saving suggests an economic corollary, namely that giving agents more freedom over what they observe can reduce total compute because targeted queries replace full-context dumps; a testable extension is to measure cost-per-pass rather than Pass@1 alone on larger multi-module benchmarks.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. VeriTrace is a multi-agent Verilog RTL generation system proposed by Liu and Yu. The paper argues that existing multi-agent LLM systems plateau near 95% on VerilogEval-V2 because their debugging agents cannot choose which signals to inspect, which time windows to query, or how many iterations of inspection to run. VeriTrace adds an Inspector agent that repeatedly calls TRACE(W, sigma_i, t_start_i, t_end_i) over the full dumped waveform, separates diagnosis from code repair, and accepts only debugger edits that reduce mismatch count. The paper reports 100% Pass@1 on VerilogEval-V2 with Claude Sonnet 4.5, 97.4% with Claude Sonnet 4.0, an ablation in which removing the Inspector lowers overall Pass@1 from 100% to 98.29% and FSM problem pass rates from 100% to 0-33%, and approximately 18% lower aggregate token consumption. It concludes that debugging agency, rather than merely richer waveform feedback, is the key to closing the gap.

Significance. The result is potentially significant: if the 100% Pass@1 score is reproducible and the mechanism claim is supported, VeriTrace would be the first open system to solve all VerilogEval-V2 problems, and the action-space perspective would be a useful design principle for hardware LLM agents. Strengths of the paper are its use of an external benchmark with golden testbenches, a clearly described ReAct-style formalization of the Inspector, a deterministic control flow, and per-problem token and call measurements for debugging-intensive cases. These design choices make the main score a falsifiable empirical claim rather than a definitional consequence. The paper is also honest in Section V about scalability limits for industrial designs. The weakness is that the experiments do not yet isolate the paper's central explanatory variable (action-space completeness) from the simpler hypothesis that any signal-level waveform feedback helps; the benchmark adjustment and the unspecified Pass@1 estimator further qualify the headline.

major comments (4)
  1. [§IV-C and §III-B, Eq. (1)] The ablation does not isolate the 'complete debugging action space' that is the paper's central mechanism. The 'w/o Inspector' condition has no access to TRACE(W, sigma_i, t_start_i, t_end_i) at all, so removing the Inspector changes two variables at once: whether the debugger receives signal-level waveform feedback, and whether an agent can choose signals and time windows. The FSM improvements in Table IV (Probs 137, 149, 155: 0-33% to 100%) therefore establish that waveform information helps, but they do not establish that joint control over signal selection and time windows, rather than richer feedback alone, is what closes the gap. Add an intermediate ablation in which the Inspector receives the same full waveform access but is restricted to a fixed signal set and a fixed one-cycle window (or the MAGE-style first-error checkpoint) on the same backbone; without such a condition, the central claim in §VI that 'debugging agency, not just richer feedback, is the key' is unsupported.
  2. [§IV-A and Table II] The Pass@1 estimator is not specified precisely enough to evaluate the 100% result. The text says 'we report with n=3' and 'highest PASS@1 accuracies from the literature,' but it does not state whether a problem is counted as passed when at least one of the three runs passes, whether the reported numbers are the mean single-attempt probability with the standard pass@k correction, or whether the best run per problem is selected. Because the central claim depends on a single number, specify the estimator and, ideally, report the per-run outcome counts (e.g., 156/156 problems passed in all three runs).
  3. [§IV-A, footnote 1] The 100% result is on an adjusted benchmark: one problem's mismatched testbench and prompt were changed after consultation with the authors of [7]. As reported, the reader cannot tell whether the adjusted problem would also have been solvable by baselines under the same adjustment, or how the adjustment affects comparability with the literature numbers in Table II. Provide the problem identifier, the exact modification, and re-run the closest baseline (at least MAGE on the same backbone) on the adjusted version. Without this, the headline 'perfect functional correctness on VerilogEval-V2' is conditional on a benchmark alteration that may help VeriTrace differentially.
  4. [§IV-B] The sentence 'This gain is attributable entirely to VeriTrace's debugging agency, as both systems share the same underlying LLM' overstates what the comparison controls. Same backbone does not make the comparison controlled: VeriTrace and MAGE differ in agent decomposition, prompt design, testbench generation, waveform-dumping strategy, and acceptance/rollback logic, any of which could explain part of the +5.1%. Either add an ablation that toggles only the Inspector/tool interface within the same codebase, or rephrase the attribution to say the gain is associated with the full VeriTrace system rather than the debugging agency alone.
minor comments (5)
  1. [Table II and §VI] The 'Open Source' checkmark for Ours in Table II is not supported by the text, since Section VI only promises future public availability; provide an artifact identifier or relabel the column.
  2. [§IV-C, Tables III-IV] The 18% token-reduction claim should be labeled as a benchmark-average aggregate; Table IV shows per-problem variation, for example Prob137 consumes 383.6k tokens with the Inspector versus 348.2k without it.
  3. [§III-B] The statement that the testbench is not exposed to the Inspector is in tension with the fact that the waveform W is produced by the modified testbench; clarify whether stimulus values and internal testbench signals are visible to the Inspector in the queried trace.
  4. [§IV-A] Reporting the 'highest PASS@1 accuracies from the literature' is vague and mixes metrics, since ACE-RTL's APR is a different quantity from Pass@1; list each literature value with its original metric and sampling protocol.
  5. [§IV-B and §IV-C] A brief sensitivity study for N, M, and temperature would strengthen the claim that the 100% score is not brittle; the current fixed values N=5, M=10, and t=0.85 are reported without any perturbation or robustness check.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the 100% Pass@1 result is an externally evaluated empirical outcome against golden testbenches, with no fitted parameter or self-citation forcing the result.

full rationale

The central result is measured on VerilogEval-V2 using provided golden testbenches, so correctness is defined by external simulation, not by VeriTrace's own equations or definitions. The Inspector's TRACE operation (Eq. 1) is a tool interface, and the ReAct-style loop (Eq. 2) is a generic control-flow pattern; neither encodes the benchmark answers. Hyperparameters N=5 and M=10 are stated without sensitivity analysis, but they are not fitted to benchmark outputs in a way that makes the 100% score a definitional consequence. The paper contains no load-bearing self-citations: references [3], [7], [10], [11], and [21] are authored by disjoint groups with no overlap with Liu and Yu, and the footnote about consulting the benchmark authors is a data-correction note, not an argumentative premise. The strongest caveat is experimental rather than circular: the ablation in Tables III-IV contrasts full waveform access plus agentic query selection against a configuration with no waveform access at all, so the causal claim that debugging agency rather than richer feedback is the key is not cleanly isolated. That is a confound in attribution, not a reduction of the claimed result to its own inputs. Similarly, the 18% token reduction is a reported measurement. Under the quoted-evidence standard, no circular step was found.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The system adds no physical entities; it relies on standard LLM agents, simulation, and waveform data. The central numbers rest on three tuned hyperparameters (N, M, and temperature), a three-run evaluation protocol, and the operational assumption that waveform queries expose all relevant internal state.

free parameters (4)
  • N (max Inspector iterations) = 5
    Maximum number of waveform queries per debug cycle; no sensitivity analysis is reported (Section IV.A, III-B).
  • M (max debug loop iterations) = 10
    Maximum number of Inspector/Debugger cycles; no sensitivity analysis is reported (Section IV.A).
  • LLM temperature = 0.85
    Sampling temperature used during RTL generation; no ablation is reported (Section IV-C).
  • n (number of evaluation runs) = 3
    Pass@1 is reported over n=3 runs, but the paper does not define whether this is an average or a best-of-3 protocol (Section IV-A).
assumptions (3)
  • domain assumption Frontier LLMs (Claude Sonnet 4.0 and 4.5) can form correct root-cause hypotheses from natural-language waveform snapshots.
    The Inspector's THINK and QUERY loop only helps if the LLM's hypothesis and signal and time-window selections reliably point to the true bug; this is assumed in Section III-B and not independently verified.
  • domain assumption The golden testbench plus simulator defines functional correctness, and dumping all waveforms gives full observability of internal state.
    The evaluation assumes the testbench is ground truth and that the full VCD waveform exposes the root cause; one problem's testbench and prompt were adjusted after consulting the benchmark authors (Section IV-A footnote).
  • domain assumption Monotonic decrease in mismatch count guarantees convergence to correct code.
    The Debugger accepts edits only when m(C') < m(C); this assumes the mismatch count is a faithful progress metric, which can fail when a fix trades one mismatch for another with equal counts or when mismatch count is not monotone in semantic distance (Section III-C).

how reviews work

0 comments
Cite this review

Pith. "Pith review of VeriTrace: Human-Like Temporal Exploration Completes Agentic Action Space." pith.science (2026). https://pith.science/paper/7VGW5I6M

@misc{pith2026260802878,
  author       = {Pith},
  title        = {Pith review of: VeriTrace: Human-Like Temporal Exploration Completes Agentic Action Space},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7VGW5I6M}},
  note         = {Machine review of arXiv:2608.02878}
}
read the original abstract

Large language models have shown promise for automated Verilog RTL generation, yet state-of-the-art multi-agent systems plateau at ~95% accuracy on standard benchmarks. We trace this ceiling to an incomplete debugging action space: existing systems restrict which signals the agent can inspect, which time windows it can query, or both, reducing debugging to pattern matching on a narrow, predetermined view of circuit behavior rather than hypothesis-driven root-cause analysis. We present VeriTrace, a multi-agent system whose Inspector agent operates over a complete debugging action space, with independent control over signal selection, time-window bounds, and iteration depth. This capability, which we term Agentic Temporal Exploration, enables the agent to form hypotheses about failure causes, query the waveform for evidence, and refine its understanding iteratively, mirroring the exploratory process of human verification engineers. VeriTrace achieves 100\% Pass@1 on VerilogEval-V2, the first system to attain perfect functional correctness on this benchmark. On a shared Claude Sonnet 4.0 backbone, VeriTrace outperforms the strongest reproduced baseline by +5.1%, demonstrating that debugging agency closes the final accuracy gap.

Figures

Figures reproduced from arXiv: 2608.02878 by the authors.

Figure 1
Figure 1. Overview of VeriTrace • Rather than ingesting full VCD waveform dumps, the In￾spector queries only the signals and time windows relevant to its current hypothesis, reducing token consumption by 18% and enabling a scalable path toward larger designs. • VeriTrace achieves 100% Pass@1 on VerilogEval-V2, to the best of our knowledge the first open source system to attain perfect functional correctness on this benchmark.… view at source ↗
Figure 2
Figure 2. An example of Inspector Agentic Temporal Exploration for Problem 155. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

37 extracted references · 21 canonical work pages

  1. [7]

    Revisiting verilogeval: A year of improvements in large-language models for hardware code generation,

    N. Pinckney, C. Batten, M. Liu, H. Ren, and B. Khailany, “Revisiting verilogeval: A year of improvements in large-language models for hardware code generation,”ACM Transactions on Design Automation of Electronic Systems, 2025

    work page 2025

  2. [1]

    From software threads to parallel hardware in high-level synthesis for fpgas,

    J. Choi, S. Brown, and J. Anderson, “From software threads to parallel hardware in high-level synthesis for fpgas,” in2013 International Conference on Field-Programmable Technology (FPT), 2013, pp. 270– 277

    work page 2013

  3. [2]

    Challenges designing for fpgas using high-level synthesis,

    C. J. Faber, S. D. Harris, Z. Xiac, R. D. Chamberlain, and A. M. Cabrera, “Challenges designing for fpgas using high-level synthesis,” in2022 IEEE High Performance Extreme Computing Conference (HPEC), 2022, pp. 1–7

    work page 2022

  4. [3]

    Verilogeval: Evaluating large language models for verilog code generation,

    M. Liu, N. Pinckney, B. Khailany, and H. Ren, “Verilogeval: Evaluating large language models for verilog code generation,” in2023 IEEE/ACM International Conference on Computer Aided Design (ICCAD). IEEE, 2023, pp. 1–8

    2023

  5. [4]

    A survey of research in large language models for electronic design automation,

    J. Pan, G. Zhou, C.-C. Chang, I. Jacobson, J. Hu, and Y . Chen, “A survey of research in large language models for electronic design automation,” ACM Transactions on Design Automation of Electronic Systems, vol. 30, no. 3, pp. 1–21, 2025

    2025

  6. [5]

    Rtllm: An open-source benchmark for design rtl generation with large language model,

    Y . Lu, S. Liu, Q. Zhang, and Z. Xie, “Rtllm: An open-source benchmark for design rtl generation with large language model,” in2024 29th Asia and South Pacific Design Automation Conference (ASP-DAC). IEEE, 2024, pp. 722–727

    2024

  7. [6]

    Benchmarking large language models for automated verilog rtl code generation,

    S. Thakur, B. Ahmad, Z. Fan, H. Pearce, B. Tan, R. Karri, B. Dolan- Gavitt, and S. Garg, “Benchmarking large language models for automated verilog rtl code generation,” 2022

    work page 2022

  8. [8]

    Betterv: Controlled verilog generation with discriminative guidance,

    P. Zehua, H. Zhen, M. Yuan, Y . Huang, and B. Yu, “Betterv: Controlled verilog generation with discriminative guidance,” inForty-first International Conference on Machine Learning, 2024

    work page 2024

Show all 37 references
  1. [9]

    Rtlfixer: Automatically fixing rtl syntax errors with large language models,

    Y .-D. Tsai, M. Liu, and H. Ren, “Rtlfixer: Automatically fixing rtl syntax errors with large language models,” 2024

    2024

  2. [10]

    Verilogcoder: Autonomous verilog coding agents with graph-based planning and abstract syntax tree (ast)- based waveform tracing tool,

    C.-T. Ho, H. Ren, and B. Khailany, “Verilogcoder: Autonomous verilog coding agents with graph-based planning and abstract syntax tree (ast)- based waveform tracing tool,” inProceedings of the AAAI Conference on Artificial Intelligence, 2025, vol. 39, pp. 300–307

    2025

  3. [11]

    Mage: A multi-agent engine for automated rtl code generation,

    Y . Zhao, H. Zhang, H. Huang, Z. Yu, and J. Zhao, “Mage: A multi-agent engine for automated rtl code generation,” in2025 62nd ACM/IEEE Design Automation Conference (DAC). IEEE, 2025, pp. 1–7

    2025

  4. [12]

    Autogen: Enabling next-gen llm applications via multi-agent conversations,

    Q. Wu, G. Bansal, J. Zhang, Y . Wu, B. Li, E. Zhu, L. Jiang, X. Zhang, S. Zhang, J. Liu, et al., “Autogen: Enabling next-gen llm applications via multi-agent conversations,” inFirst conference on language modeling, 2024

    2024

  5. [13]

    Teaching large language models to self-debug,

    X. Chen, M. Lin, N. Sch ¨arli, and D. Zhou, “Teaching large language models to self-debug,”arXiv preprint arXiv:2304.05128, 2023

    2023 arXiv

  6. [14]

    Rtlcoder: Fully open-source and efficient llm-assisted rtl code generation technique,

    S. Liu, W. Fang, Y . Lu, J. Wang, Q. Zhang, H. Zhang, and Z. Xie, “Rtlcoder: Fully open-source and efficient llm-assisted rtl code generation technique,”IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, vol. 44, no. 4, pp. 1448–1461, 2024

    2024

  7. [15]

    Scalertl: Scaling llms with reasoning data and test-time compute for accurate rtl code generation,

    C. Deng, Y .-D. Tsai, G.-T. Liu, Z. Yu, and H. Ren, “Scalertl: Scaling llms with reasoning data and test-time compute for accurate rtl code generation,” in2025 ACM/IEEE 7th Symposium on Machine Learning for CAD (MLCAD). IEEE, 2025, pp. 1–9

    2025

  8. [16]

    Evaluating large language models trained on code,

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockman, et al., “Evaluating large language models trained on code,”arXiv preprint arXiv:2107.03374, 2021

    2021 arXiv

  9. [17]

    Chipnemo: Domain- adapted llms for chip design,

    M. Liu, T.-D. Ene, R. Kirby, C. Cheng, N. Pinckney, R. Liang, J. Alben, H. Anand, S. Banerjee, I. Bayraktaroglu, et al., “Chipnemo: Domain- adapted llms for chip design,”arXiv preprint arXiv:2311.00176, 2023

    2023 arXiv

  10. [18]

    Origen: Enhancing rtl code generation with code-to-code augmentation and self-reflection,

    F. Cui, C. Yin, K. Zhou, Y . Xiao, G. Sun, Q. Xu, Q. Guo, Y . Liang, X. Zhang, D. Song, et al., “Origen: Enhancing rtl code generation with code-to-code augmentation and self-reflection,” inProceedings of the 43rd IEEE/ACM International Conference on Computer-Aided Design, 202...

    2024

  11. [19]

    Craftrtl: High-quality synthetic data generation for verilog code models with correct-by-construction non-textual representations and targeted code repair,

    M. Liu, Y .-D. Tsai, W. Zhou, and H. Ren, “Craftrtl: High-quality synthetic data generation for verilog code models with correct-by-construction non-textual representations and targeted code repair,”arXiv preprint arXiv:2409.12993, 2024

    2024 arXiv

  12. [20]

    Survey of machine learning for software-assisted hardware design verification: Past, present, and prospect,

    N. Wu, Y . Li, H. Yang, H. Chen, S. Dai, C. Hao, C. Yu, and Y . Xie, “Survey of machine learning for software-assisted hardware design verification: Past, present, and prospect,”ACM Transactions on Design Automation of Electronic Systems, vol. 29, no. 4, pp. 1–42, 2024

    2024

  13. [21]

    Ace-rtl: When agentic context evolution meets rtl-specialized llms,

    C. Deng, Z. Yu, G.-T. Liu, N. Pinckney, and H. Ren, “Ace-rtl: When agentic context evolution meets rtl-specialized llms,”arXiv preprint arXiv:2602.10218, 2026

    2026

  14. [22]

    Metagpt: Meta programming for a multi-agent collaborative framework,

    S. Hong, M. Zhuge, J. Chen, X. Zheng, Y . Cheng, J. Wang, C. Zhang, Z. Wang, S. K. S. Yau, Z. Lin, et al., “Metagpt: Meta programming for a multi-agent collaborative framework,” inThe twelfth international conference on learning representations, 2023

    2023

  15. [23]

    Chateda: A large language model powered autonomous agent for eda,

    H. Wu, Z. He, X. Zhang, X. Yao, S. Zheng, H. Zheng, and B. Yu, “Chateda: A large language model powered autonomous agent for eda,” IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, vol. 43, no. 10, pp. 3184–3197, 2024

    2024

  16. [24]

    React: Synergizing reasoning and acting in language models,

    S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y . Cao, “React: Synergizing reasoning and acting in language models,”arXiv preprint arXiv:2210.03629, 2022

    2022 arXiv

  17. [25]

    Toolformer: Language models can teach themselves to use tools,

    T. Schick, J. Dwivedi-Yu, R. Dess `ı, R. Raileanu, M. Lomeli, E. Hambro, L. Zettlemoyer, N. Cancedda, and T. Scialom, “Toolformer: Language models can teach themselves to use tools,”Advances in neural information processing systems, vol. 36, pp. 68539–68551, 2023

    2023

  18. [26]

    Swe-agent: Agent-computer interfaces enable automated software engineering,

    J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press, “Swe-agent: Agent-computer interfaces enable automated software engineering,”Advances in Neural Information Processing Systems, vol. 37, pp. 50528–50652, 2024

    2024

  19. [27]

    Re- flexion: Language agents with verbal reinforcement learning,

    N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao, “Re- flexion: Language agents with verbal reinforcement learning,”Advances in neural information processing systems, vol. 36, pp. 8634–8652, 2023

    2023

  20. [28]

    Openhands: An open platform for ai software developers as generalist agents,

    X. Wang, B. Li, Y . Song, F. F. Xu, X. Tang, M. Zhuge, J. Pan, Y . Song, B. Li, J. Singh, et al., “Openhands: An open platform for ai software developers as generalist agents,”arXiv preprint arXiv:2407.16741, 2024

    2024 arXiv

  21. [29]

    Self-refine: Iterative refinement with self-feedback,

    A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y . Yang, et al., “Self-refine: Iterative refinement with self-feedback,”Advances in neural information processing systems, vol. 36, pp. 46534–46594, 2023

    2023

  22. [30]

    Executable code actions elicit better llm agents,

    X. Wang, Y . Chen, L. Yuan, Y . Zhang, Y . Li, H. Peng, and H. Ji, “Executable code actions elicit better llm agents,” inForty-first International Conference on Machine Learning, 2024

    2024

  23. [31]

    V oyager: An open-ended embodied agent with large language models,

    G. Wang, Y . Xie, Y . Jiang, A. Mandlekar, C. Xiao, Y . Zhu, L. Fan, and A. Anandkumar, “V oyager: An open-ended embodied agent with large language models,”arXiv preprint arXiv:2305.16291, 2023

    2023 arXiv

  24. [32]

    Aivril: Ai-driven rtl generation with verification in-the-loop,

    M. ul Islam, H. Sami, P.-E. Gaillardon, V . Tenace, et al., “Aivril: Ai-driven rtl generation with verification in-the-loop,”arXiv preprint arXiv:2409.11411, 2024

    2024 arXiv

  25. [33]

    Verdi debug: Streamlining verification engineers’ workflow,

    Synopsys, “Verdi debug: Streamlining verification engineers’ workflow,” 2024, Synopsys Blog

    2024

  26. [34]

    Verdi automated debug system,

    Synopsys, Inc., “Verdi automated debug system,” https://www.synopsys. com/verification/debug/verdi.html, 2024, Accessed: 2025-02-05

    2024

  27. [35]

    System card: Claude sonnet 4.5,

    Anthropic, “System card: Claude sonnet 4.5,” Tech. Rep., Anthropic, September 2025

    2025

  28. [36]

    Comprehensive verilog design problems: A next-generation benchmark dataset for evaluating large language models and agents on rtl design and verification,

    N. Pinckney, C. Deng, C.-T. Ho, Y .-D. Tsai, M. Liu, W. Zhou, B. Khailany, and H. Ren, “Comprehensive verilog design problems: A next-generation benchmark dataset for evaluating large language models and agents on rtl design and verification,” 2025

    2025

  29. [37]

    Rtl-repo: A benchmark for evaluating llms on large-scale rtl design projects,

    A. Allam and M. Shalan, “Rtl-repo: A benchmark for evaluating llms on large-scale rtl design projects,” in2024 IEEE LLM Aided Design Workshop (LAD). IEEE, 2024, pp. 1–5

    2024

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.