Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

CorrectBench: Automatic Testbench Generation with Functional Self-Correction using LLMs for HDL Design

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read CorrectBench claims that LLM-generated hardware testbenches can be validated and corrected using only a natural-language specification, reaching a 70.13% pass ratio.

desk verdict A genuine step forward in LLM-based testbench generation with a clever pseudo-oracle validator; the evaluation is solid but the core independence assumption is untested and the thresholds are fitted to the same data. read the letter →

arxiv 2411.08510 v1 pith:M7ZUG4SU submitted 2024-11-13 cs.SE

Ruidi Qiu , Grace Li Zhang , Rolf Drechsler , Ulf Schlichtmann , Bing Li This is my paper · ORCID
classification cs.SE
keywords LLM-basedtestbenchgenerationHDLverificationfunctionalself-correctionRS-matrixvalidationsequentialcircuitsRTLsimulationhardwaredesignautomation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CorrectBench addresses the failure of LLM-generated hardware testbenches by adding functional self-validation and self-correction. Its central claim is that a testbench can be judged correct or wrong using only the RTL specification plus a panel of 20 deliberately imperfect LLM-generated RTL designs, reaching 88.85% validation accuracy. When a scenario is flagged, a conversational corrector uses the bug report to repair the Python checker, lifting the overall Eval2 pass ratio to 70.13% across 156 tasks, compared with 52.18% for AutoBench and 33.33% for direct LLM generation. The largest gain is in sequential circuits, where the pass ratio rises from 11.47% for direct generation to 54.93%.

What carries the argument

The RS (RTL-Scenario) matrix is an $N_R \times N_S$ boolean matrix whose rows are 20 LLM-generated RTL designs and whose columns are testbench scenarios; each entry records whether the RTL's simulation output agrees with the testbench for that scenario. The 70%-wrong criterion marks a scenario as wrong when at least 70% of the RTL designs disagree, and the more-than-25%-green-rows rule accepts a testbench outright when many RTL designs match it across all scenarios. This matrix supplies both the go or no-go decision and the scenario-level bug reports that the two-stage corrector consumes.

What would settle it

Construct a testbench whose Python checker shares the LLM's systematic misreading, such as both treating an active-low reset as active-high, run the validator's 20 generated RTLs, and check whether the RS matrix still shows more than 25% fully green rows; if the validator reports correct for a testbench that fails on mutant RTLs, the independence assumption is broken.

Watch

Extended reading notes

Core claim

CorrectBench's central claim is that an LLM-generated testbench can be judged without any golden RTL. The validator asks the LLM to produce 20 imperfect RTL designs from the same specification, simulates each against every test scenario in the testbench, and arranges the pass or fail outcomes into an RTL-scenario matrix. A scenario is flagged as wrong when at least 70% of the RTL designs disagree with the testbench's expected output, unless more than 25% of the RTL rows are entirely green, in which case the testbench is accepted. The reported validation accuracy is 88.85%, and feeding the flagged scenarios to a conversational corrector raises the Eval2 pass ratio on 156 HDL tasks to 70.13%, with sequential circuits improving from 11.47% for direct generation to 54.93%.

Load-bearing premise

The load-bearing premise is that the LLM's mistakes in the 20 generated RTL designs are random and mutually independent, so a scenario that most RTLs fail must be a testbench bug; if the LLM systematically misreads the specification the same way in both the RTLs and the checker, a wrong testbench could be certified as correct.

Editorial extensions

If this is right

  • Testbench validation no longer requires a golden RTL; a set of known-imperfect LLM-generated RTLs plus the natural-language spec is enough to reach 88.85% validator accuracy.
  • Sequential circuits, the hardest category, benefit most: the Eval2 pass ratio rises from 11.47% for direct generation and 33.87% for AutoBench to 54.93%.
  • Scenario-level bug reports let the corrector fix specific wrong scenarios; 9.2 of the 26.8 validator-enabled passes are credited to correction rather than regeneration.
  • A stricter validator threshold reduces false accepts but risks non-convergence and higher token cost, so the 70% threshold is a deliberate balance between these effects.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same validate-by-a-panel-of-imperfect-instances idea could transfer to other LLM-generated artifacts with no ground truth, such as RTL designs or formal properties, where a majority-vote matrix could flag suspicious outputs.
  • The independence assumption is testable: generating the 20 RTLs with several different LLM families, or with prompts that share a deliberate misinterpretation, should change validator accuracy and reveal how much of the 88.85% depends on uncorrelated errors.
  • The 70% and 25% thresholds were selected on the same labeled data used to report accuracy, so a sensitivity sweep on unseen tasks would quantify any overfitting and could replace the heuristic with a data-driven choice.
  • The gap between validator accuracy and final Eval2 pass ratio suggests that coverage-based validation, named as future work in the paper, is the natural next step for catching testbenches that pass the matrix but still miss design behaviour.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. CorrectBench is an LLM-based framework for automatic Verilog testbench generation. It uses AutoBench to produce a hybrid testbench (Verilog driver plus Python checker) from a natural-language hardware specification, then validates the testbench by simulating 20 LLM-generated 'imperfect' RTL designs and building an RTL-scenario (RS) matrix. A testbench is declared wrong if at least 70% of the RTLs disagree with it on some scenario, unless more than 25% of RTL rows are entirely green, in which case the testbench is accepted. When validation fails, a two-stage conversational corrector uses scenario-level bug information; after three correction attempts the loop reboots, up to ten times. On 156 HDLBits tasks with gpt-4o-2024-08-06, CorrectBench reports an Eval2 pass ratio of 70.13%, versus 52.18% for AutoBench and 33.33% for direct LLM generation, and a validator accuracy of 88.85%. Additional experiments with gpt-4o-mini and Claude-3.5-Sonnet show similar qualitative trends.

Significance. The contribution is potentially significant: a fully automatic, specification-only pipeline with functional self-validation and self-correction is a clear step beyond AutoBench's syntax-only checking, and the sequential-circuit improvement (54.93% vs. 33.87% in Table I) is practically meaningful. The final Eval2 claim is evaluated with external mutant RTLs, so the central pass-ratio comparison is not circular. The open-sourced code and results are also a strength. However, the validator's core independence assumption is not empirically supported, the reported 88.85% validation accuracy is a threshold-selected number computed on the same labeled data used to choose the threshold, and the main comparative results lack any measure of run-to-run variability. These issues are load-bearing because a false validator acceptance terminates the loop and is counted as a pass in the headline Eval2 numbers.

major comments (4)
  1. [III-B1] The validator's correctness rests on the assertion that errors in LLM-generated RTL designs are 'randomly distributed' and therefore unlikely to coincide with errors in the LLM-generated Python checker. Since both the validator RTLs and the AutoBench-generated checker are produced by the same model from the same natural-language SPEC, common HDL misunderstandings (e.g., reset polarity, shift direction, off-by-one indexing) can appear in both artifacts. In that case the RS matrix shows few red columns, and the >25%-green-row rule in Section III-B2 can force a 'correct' verdict for a systematically wrong testbench. The paper provides no experiment with adversarially correlated checker/RTL errors and no false-acceptance rate for the validator. Please add an explicit test that induces correlated checker and RTL errors, and report how often the validator accepts a wrong testbench under that scenario.
  2. [IV-C, Fig. 6(a)] The 70%-wrong threshold and the 25% row rule are selected after inspecting the same 1560 labeled testbenches on which the 88.85% validation accuracy is reported. This is model selection on the test set: the accuracy is a fitted quantity and may not generalize to other tasks, other LLMs, or the different distribution of testbenches produced inside the CorrectBench loop with correction and rebooting. The paper acknowledges that the criterion 'may not be optimal' but does not address the selection bias. Please report held-out or cross-validated accuracy, or otherwise fix the thresholds independently of the labeled set.
  3. [Table I, Section IV-B1] The main results are averages of five repeated runs, but Table I reports only the means, with no standard deviation, confidence interval, or significance test. The headline differences (70.13% vs. 52.18% vs. 33.33%) are large, but without run-level data a reader cannot assess stability, and a claimed improvement of 26.8/28.0 tasks in Table III may be within run-to-run noise. Please report per-run results and a variance or confidence measure for at least the primary Eval2 comparison, and clarify whether the parenthetical percentages in Table I are percentage-point improvements over the baseline while the text uses relative percentages.
  4. [III-B2] The rule that a testbench is 'directly considered correct' if more than 25% of RTL rows are entirely green is an additional free parameter with no sensitivity analysis. Combined with the 70% column threshold, it can override scenario-level evidence of errors and force a pass verdict even when several scenario columns are red for most RTLs. Since the downstream Eval2 pass ratio is gated by this validator verdict, the choice of 25% (and also the choices N_R=20, I_max_C=3, I_max_R=10) should be justified with a sensitivity study or shown not to affect the main conclusions.
minor comments (5)
  1. [Abstract, Section IV-B] The abstract states that CorrectBench is '62.18% higher' than AutoBench on sequential tasks, while Section IV-B says '66.18% higher'; the ratio 54.93/33.87 - 1 is 62.18%, so the two numbers should be reconciled.
  2. [Table I] The parenthetical improvements in Table I are labeled 'improvement of the method compared with the baseline,' but the main text interprets them relative to AutoBench; clarify whether they are absolute percentage-point differences or relative improvements.
  3. [Figure 4] The caption says 'The two matrices on the left represent the correct TBs, whereas the matrix on the right indicates errors,' but the figure appears to contain three matrices; please specify which panel corresponds to which condition.
  4. [IV-A3] The mutant RTL dataset is described only as an extension of VerilogEval-Human; the process by which mutants were generated, how many mutants exist per task, and how they were selected for Eval2 should be documented for reproducibility.
  5. [Algorithm 1, line 15] If the maximum reboot iteration is exceeded, the action becomes 'Pass' even though the testbench may still be wrong; this is semantically odd, and the manuscript should explain that this is a stop condition rather than a correctness verdict.

Circularity Check

1 steps flagged · score 6.0 of 10

Validator accuracy is an in-sample threshold fit; the Eval2 pass-ratio claim is externally evaluated and not circular.

  1. fitted input called prediction [Section IV-C, Fig. 6(a); threshold defined in Section III-B2]
    "we collected 1560 testbenches from the results of [15] and ran the validators with different criteria ( 100%-wrong, 70%-wrong and 50%-wrong) on them. ... Among the three criteria, 70%-wrong achieves the highest global validation accuracy at 88.85%, which is the criterion employed in our study."

    The 70%-wrong threshold is selected by comparing three thresholds on the same 1560 labeled testbenches and choosing the one with the highest validation accuracy. The reported 88.85% is therefore the in-sample maximum of that threshold sweep, not an independent estimate of the validator's success rate. The abstract then presents this fitted maximum as the method's fixed property: 'the proposed approach can validate the correctness of the generated testbenches with a success rate of 88.85%.' Thus the headline validation statistic reduces to the threshold-selection criterion by construction.

full rationale

The central Eval2 result (70.13% vs. 52.18% vs. 33.33%) is not circular: final testbenches are evaluated externally by AutoEval's Eval2, which uses 10 mutant RTLs and requires agreement with a golden testbench, independent of CorrectBench's own validator verdict. The use of the same authors' AutoBench [15] as generator and AutoEval [15] as evaluator is self-citation, but the same mutant-based criterion is applied uniformly to all methods, so it does not reduce the main comparison to an input of the method. The load-bearing circular step is confined to the validator's headline accuracy: Section IV-C tunes the 70%-wrong criterion on the same labeled 1560 testbenches used to measure accuracy, so the 88.85% figure is a fitted in-sample maximum presented as a predicted success rate. Separately, the validator's soundness relies on the untested Section III-B1 assumption that LLM-generated RTL errors are 'randomly distributed' and therefore unlikely to coincide with checker errors; if the same LLM systematically misinterprets an ambiguous SPEC in both the RTLs and the Python checker, a wrong testbench can pass. That is an empirical soundness risk, not a demonstrated circular reduction, so it does not further raise the circularity score.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The validator's thresholds and loop limits are chosen by hand or from the same labeled data used to report accuracy, and the core independence-of-errors assumption is not validated. No new physical entities are introduced.

free parameters (5)
  • Validation threshold (70%-wrong) = 70% scenario disagreement
    Selected from comparing 50%, 70%, and 100% thresholds on the 1560 labeled testbenches; the chosen criterion reports 88.85% validation accuracy on the same set.
  • Row-rule threshold (>25% green rows) = 25%
    A hand-chosen rule to prevent false-positive classification of correct testbenches; not independently justified.
  • Number of generated RTL designs N_R = 20
    Set to 20 'in this work'; no sensitivity analysis.
  • Max correction iterations I_max_C = 3
    Algorithm 1; set experimentally, no sensitivity analysis.
  • Max reboot iterations I_max_R = 10
    Algorithm 1; set experimentally, no sensitivity analysis.
assumptions (4)
  • domain assumption LLM-generated RTL implementations of the same spec have errors that are randomly distributed across scenarios and do not systematically coincide with testbench checker errors.
    Section III-B1: 'their errors tend to be randomly distributed due to the uncertainty of the LLM'. This is the foundation of the RS-matrix validation and is not tested.
  • ad hoc to paper If >=70% of RTLs disagree with the testbench on a scenario, the testbench scenario is functionally wrong.
    Section III-B2, the 70%-wrong criterion, chosen from comparing thresholds on labeled data.
  • ad hoc to paper If >25% of RTLs fully match the testbench, the testbench is correct regardless of scenario-level failures.
    Section III-B2, the extra row rule to reduce false positives.
  • domain assumption The Python checker track is the only source of functional testbench errors, and the Verilog driver can be assumed correct or repairable via the same mechanism.
    The corrector demo and analysis focus on Python checker errors; no validation of the Verilog driver's functional correctness is described.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CorrectBench: Automatic Testbench Generation with Functional Self-Correction using LLMs for HDL Design." pith.science (2026). https://pith.science/paper/M7ZUG4SU

@misc{pith2026241108510,
  author       = {Pith},
  title        = {Pith review of: CorrectBench: Automatic Testbench Generation with Functional Self-Correction using LLMs for HDL Design},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/M7ZUG4SU}},
  note         = {Machine review of arXiv:2411.08510}
}
read the original abstract

Functional simulation is an essential step in digital hardware design. Recently, there has been a growing interest in leveraging Large Language Models (LLMs) for hardware testbench generation tasks. However, the inherent instability associated with LLMs often leads to functional errors in the generated testbenches. Previous methods do not incorporate automatic functional correction mechanisms without human intervention and still suffer from low success rates, especially for sequential tasks. To address this issue, we propose CorrectBench, an automatic testbench generation framework with functional self-validation and self-correction. Utilizing only the RTL specification in natural language, the proposed approach can validate the correctness of the generated testbenches with a success rate of 88.85%. Furthermore, the proposed LLM-based corrector employs bug information obtained during the self-validation process to perform functional self-correction on the generated testbenches. The comparative analysis demonstrates that our method achieves a pass ratio of 70.13% across all evaluated tasks, compared with the previous LLM-based testbench generation framework's 52.18% and a direct LLM-based generation method's 33.33%. Specifically in sequential circuits, our work's performance is 62.18% higher than previous work in sequential tasks and almost 5 times the pass ratio of the direct method. The codes and experimental results are open-sourced at the link: https://github.com/AutoBench/CorrectBench

Figures

Figures reproduced from arXiv: 2411.08510 by the authors.

Figure 1
Figure 1. The outline of CorrectBench workflow. simulation-based verification, preliminary efforts have been made. For instance, [14] investigates the potential of LLMs in generating testbenches for finite state machines (FSMs), while [15] introduces a framework called AutoBench, the first systematic and generic testbench generation framework. Although achieving an average 57% improvement compared with directly generating tes… view at source ↗
Figure 2
Figure 2. The outline of AutoBench workflow [15]. AutoBench is used as the [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. A demo of the test scenario and test stimuli in AutoBench’s Verilog [PITH_FULL_IMAGE:figures/full_fig_p002_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Examples of RS Matrices. The red/green color in the [PITH_FULL_IMAGE:figures/full_fig_p003_4.png]
Figure 5
Figure 5. Figure 5: A Demo of Corrector. The RTL problem is shift18, an arithmetic shifter. Some details are omitted to save space. are different from that described in the testbench for one test scenario, such test scenarios are marked as wrong, and the testbench is marked as wrong. The …
Figure 7
Figure 7. Figure 7: Performance of CorrectBench on Different LLMs. [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SoK: Towards Effective Automated Vulnerability Repair

    cs.CR 2025-01 conditional novelty 6.0 of 10

    This SoK benchmarks ten automated vulnerability repair tools across C/C++ and Java and concludes that learning-based methods are strong on synthetic benchmarks but trail non-learning methods on real-world vulnerabilit...

Reference graph

Works this paper leans on

19 extracted references · 14 canonical work pages · cited by 1 Pith paper

  1. [1]

    Coverage-Directed Test Generation Auto- mated by Machine Learning – A Review,

    C. Ioannides and K. I. Eder, “Coverage-Directed Test Generation Auto- mated by Machine Learning – A Review,” ACM Transactions on Design Automation of Electronic Systems (TODAES) , vol. 17, pp. 1–21, 2012

    work page 2012

  2. [2]

    Improving Coverage of Simulation- Based Verification by Dedicated Stimuli Generation,

    S. Yang, R. Wille, and R. Drechsler, “Improving Coverage of Simulation- Based Verification by Dedicated Stimuli Generation,” in Euromicro Conference on Digital System Design , 2014

    work page 2014

  3. [3]

    A VERT: An Automatic Verilog Testbench Generation Tool for Grammatical Evolution,

    J. McEllin, R. Conway, and C. Ryan, “A VERT: An Automatic Verilog Testbench Generation Tool for Grammatical Evolution,” in 33rd Irish Signals and Systems Conference (ISSC) , 2022

    work page 2022

  4. [4]

    Stimulus Generation for Con- strained Random Simulation,

    N. Kitchen and A. Kuehlmann, “Stimulus Generation for Con- strained Random Simulation,” in IEEE/ACM International Conference on Computer-Aided Design (ICCAD) , 2007

    work page 2007

  5. [5]

    Chip-Chat: Challenges and Opportunities in Conversational Hardware Design,

    J. Blocklove, S. Garg, R. Karri, and H. Pearce, “Chip-Chat: Challenges and Opportunities in Conversational Hardware Design,” in ACM/IEEE International Workshop on Machine Learning for CAD (MLCAD) , 2023

    work page 2023

  6. [6]

    Automated C/C++ Program Repair for High-Level Synthesis via Large Language Models,

    K. Xu, G. L. Zhang, X. Yin, C. Zhuo, U. Schlichtmann, and B. Li, “Automated C/C++ Program Repair for High-Level Synthesis via Large Language Models,” in ACM/IEEE International Symposium on Machine Learning for CAD (MLCAD) , 2024

    work page 2024

  7. [7]

    Rome was Not Built in a Single Step: Hierarchical Prompting for LLM-based Chip Design,

    A. Nakkab, S. Q. Zhang, R. Karri, and S. Garg, “Rome was Not Built in a Single Step: Hierarchical Prompting for LLM-based Chip Design,” in ACM/IEEE International Symposium on Machine Learning for CAD (MLCAD), 2024

    work page 2024

  8. [8]

    ChipGPT: How Far Are We From Natural Language Hardware Design,

    K. Chang, Y . Wang, H. Ren, M. Wang, S. Liang, Y . Han, H. Li, and X. Li, “ChipGPT: How Far Are We From Natural Language Hardware Design,” arXiv preprint:2305.14019, 2023

    arXiv 2023

Show all 19 references
  1. [9]

    VerilogEval: Evaluating large language models for verilog code generation,

    M. Liu, N. Pinckney, B. Khailany, and H. Ren, “VerilogEval: Evaluating large language models for verilog code generation,” in International Conference on Computer Aided Design (ICCAD) , 2023

    2023

  2. [10]

    Machine Learning in Advanced IC Design: A Methodological Survey,

    T. Chen, G. L. Zhang, B. Yu, B. Li, and U. Schlichtmann, “Machine Learning in Advanced IC Design: A Methodological Survey,” IEEE Design and Test, vol. 40, no. 1, pp. 17–33, 2023

    2023

  3. [11]

    Classification-Based Automatic HDL Code Generation Using LLMs,

    W. Sun, B. Li, G. L. Zhang, X. Yin, C. Zhuo, and U. Schlichtmann, “Classification-Based Automatic HDL Code Generation Using LLMs,” Arxiv, 2024

    2024

  4. [12]

    RTLFixer: Automatically Fixing RTL Syntax Errors with Large Language Models,

    Y . Tsai, M. Liu, and H. Ren, “RTLFixer: Automatically Fixing RTL Syntax Errors with Large Language Models,”arXiv preprint: 2311.16543, 2023

    2023 arXiv

  5. [13]

    AutoChip: Automating HDL Generation Using LLM Feedback,

    S. Thakur, J. Blocklove, H. Pearce, B. Tan, S. Garg, and R. Karri, “AutoChip: Automating HDL Generation Using LLM Feedback,” arXiv preprint: 2311.04887, 2023

    2023 arXiv

  6. [14]

    LLM-Aided Testbench Generation and Bug Detection for Finite-State Machines,

    J. Bhandari, J. Knechtel, R. Narayanaswamy, S. Garg, and R. Karri, “LLM-Aided Testbench Generation and Bug Detection for Finite-State Machines,” arXiv preprint arXiv:2406.17132 , 2024

    2024 arXiv

  7. [15]

    Au- toBench: Automatic Testbench Generation and Evaluation Using LLMs for HDL Design,

    R. Qiu, G. L. Zhang, R. Drechsler, U. Schlichtmann, and B. Li, “Au- toBench: Automatic Testbench Generation and Evaluation Using LLMs for HDL Design,” in ACM/IEEE International Symposium on Machine Learning for CAD (MLCAD) , 2024

    2024

  8. [16]

    A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions,

    L. Huang, W. Yu, W. Ma, W. Zhong, Z. Feng, H. Wang, Q. Chen, W. Peng, X. Feng, B. Qin, and T. Liu, “A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions,” arXiv preprint: 2311.05232 , 2023

    2023 arXiv

  9. [17]

    Large Language Models Can be Lazy Learners: Analyze Shortcuts in In-Context Learning,

    R. Tang, D. Kong, L. Huang, and H. Xue, “Large Language Models Can be Lazy Learners: Analyze Shortcuts in In-Context Learning,” in Findings of the Association for Computational Linguistics: ACL 2023 , 2023

    2023

  10. [18]

    The ICARUS verilog compilation system,

    S. Williams, “The ICARUS verilog compilation system,” 2024. [Online]. Available: https://github.com/steveicarus/iverilog

    2024

  11. [19]

    Problem sets - HDLBits,

    H. Wong, “Problem sets - HDLBits,” 2019. [Online]. Available: https://hdlbits.01xz.net/wiki/Problemsets

    2019

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.