REVIEW 8 cited by
VerilogEval: Evaluating Large Language Models for Verilog Code Generation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The increasing popularity of large language models (LLMs) has paved the way for their application in diverse domains. This paper proposes a benchmarking framework tailored specifically for evaluating LLM performance in the context of Verilog code generation for hardware design and verification. We present a comprehensive evaluation dataset consisting of 156 problems from the Verilog instructional website HDLBits. The evaluation set consists of a diverse set of Verilog code generation tasks, ranging from simple combinational circuits to complex finite state machines. The Verilog code completions can be automatically tested for functional correctness by comparing the transient simulation outputs of the generated design with a golden solution. We also demonstrate that the Verilog code generation capability of pretrained language models could be improved with supervised fine-tuning by bootstrapping with LLM generated synthetic problem-code pairs.
Forward citations
Cited by 8 Pith papers
-
WaveformQA: Benchmarking LLM Temporal Reasoning on Digital Waveforms
LLMs answer simple waveform queries well but fail on multi-signal temporal reasoning, and event-time JSON waveforms beat standard VCD by 37–53% in accuracy, according to a new 360-question benchmark.
-
ArchEval: Measuring AI Agents as Computer Architects
LLM agents beat architecture baselines with full simulator harnesses, but only one configuration stays above baseline without feedback, and performance modeling remains weak.
-
Open-Source LLM-Driven Formal Verification: A Multi-Agent Pipeline for RTL Repair
An open-source LLM-based pipeline repaired a simple ALU bug with k-induction proof, but succeeded on only 1 of 6 benchmarks.
-
Iceberg: Enhancing HLS Modeling with Synthetic Data
Pretraining on LLM-generated HLS programs plus GNN weak labels reduces few-shot latency prediction error by 86% on six real-world applications.
-
Spec2RTL-Agent: Automated Hardware Code Generation from Complex Specifications Using LLM Agent Systems
On three NIST crypto standards (AES, DSS, HMAC), Spec2RTL-Agent generates RTL via a multi-agent pipeline from pseudocode to Python to synthesizable C++, reporting 3/3 correct designs with about 4.3 human interventions...
-
hdl2v: A Code Translation Dataset for Enhanced LLM Verilog Generation
A new dataset of VHDL, Chisel, and PyMTL3 code paired with compiled Verilog improves LLM Verilog generation by up to 23% on VerilogEvalV2 after fine-tuning.
-
VeriLoC: Line-of-Code Level Prediction of Hardware Design Quality from Verilog Code
A method that predicts line-level timing and congestion issues directly from Verilog code using CL-Verilog embeddings and gradient-boosted classifiers.
-
VerilogDB: The Largest, Highest-Quality Dataset with a Preprocessing Framework for LLM-based RTL Generation
A new pipeline and dataset of 20,392 synthesis-checked Verilog modules for LLM fine-tuning is presented, claimed to be the largest high-quality dataset of its kind.
Discussion (0). Sign in to comment.