{"id":"ba2ca9f5-3f7a-41a0-a3f5-a01246e61ec4","arxiv_id":"2506.03590","paper_version":5,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A VCD-waveform mining framework that classifies failing RTL simulations into likely faulty modules, compressing data by over 120x and reaching 94% top-3 accuracy on tested open-source designs.","lead":"VCDiag is a machine learning toolkit that reads simulator waveform files from failed chip simulations and predicts which part of the design is likely at fault. It compresses huge waveform files into small statistical tables, then trains classifiers that name the top three suspect modules with about 94% accuracy on synthetic test cases.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Synthetic-bug validity is the load-bearing assumption: no experiment links LLM-injected mutations to real RTL failure signatures, so the 77%/94% accuracy claim may not transfer to genuine design bugs.","rationale":"The reader's weakest assumption and my strongest concern are the same: the synthetic-only bug dataset is the load-bearing assumption for the transferability of the central accuracy claim. The reader correctly notes the absence of real-failure experiments and the acknowledged stimulus diversity limit. I add precision about why this is the weakest link: every headline number in Table IV is measured on the LLM-injected population, and the BugMD comparison does not rescue transfer because it compares different bug populations. The limitation statements in Section IV-C and VI-A are flagged and weigh in: they are honest but they explicitly concede that stimulus diversity and realistic bug generation are constraints. The verification step I propose is concrete and would settle the question: hold the trained pipeline fixed and evaluate on independently obtained real failing waveforms. I do not see a stronger concern: the pipeline is reproducible (public artifact, open-source toolchain), the ML methodology is standard with explicit train/test separation, and the compression and parallelization claims are adequately supported by Tables V and Figure 7. Therefore the verdict remains CONDITIONAL (the reader's verdict), not because of internal error but because independent validation on real failures is pending. Since my concern matches the reader's and the proposed check would confirm or refute it, verdict_should_be is UNCHANGED rather than a new verdict.","tokens_in":9749,"tokens_out":1541,"duration_ms":14488,"concrete_test":"Obtain a set of real failing waveforms for FabScalar or an OpenTitan IP from a different source than the authors' generator: e.g., hand-written bug fixes or bugs from a student design exercise, or a second independent LLM pipeline with a different prompt set. Train the VCDiag pipeline only on the authors' synthetic bugs, then evaluate on the real failing waveforms and report top-1/top-3. If accuracy drops by more than roughly 10-15 points versus Table IV, the headline claim is not transferable; if accuracy holds, the synthetic-bug concern is largely resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central accuracy claim (77% top-1, 94% top-3 on FabScalar, Table IV) rests entirely on a dataset of failing waveforms generated by the LLM-based synthetic bug injector (Section IV-C). The class label is the module where a mutation was applied. For the classifier to be useful in real failure triage, the waveform signature of an injected synthetic mutation must resemble the waveform signature of a real RTL bug, and each real failure must be attributable to a single module. The paper does not test either condition: there are no experiments with real failing tests or human-written bug fixes, and Section VI-A admits FabScalar stimulus diversity is limited by fixed program binaries. The comparison to BugMD is indirect (different bug population, different abstraction level, different metrics setup), so it cannot validate transfer. The generator's own limitations (Section IV-C: LLMs struggle with complex designs, auto-revert of ineffective bugs, need for plain-English bug descriptions) mean the injected bugs may be systematically simpler or more localized than real bugs, which would make the classification task easier than real triage. This is not an internal inconsistency: the ML pipeline is internally coherent, and the authors are honest about the synthetic-only evaluation. It is an external-validity gap in the central claim, and it is the weakest link because every headline number is measured on this synthetic population.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes VCDiag, a framework that classifies failing VCD simulation waveforms to predict which RTL module is responsible for a failure, with the goal of accelerating failure triage in design verification. The pipeline consists of AST-based signal selection, parallel simulation and extraction, statistical compression of waveform data into tabular features, optional XGBoost signal ranking, and classical ML classifiers (KNN, Random Forest, XGBoost, LightGBM). The framework is evaluated on OpenTitan AES and other OpenTitan IPs as well as the FabScalar processor, using failures generated by a separate LLM-based synthetic bug injection tool, BugGen. On the largest benchmark, FabScalar, LightGBM achieves 77% top-1 and 94% top-3 accuracy (Table IV), and the reported data compression reduces raw waveform data from 308 GB to 2.5 GB, a 123x reduction. The authors also report end-to-end parallelization speedups and an exploratory result on accuracy versus simulation window length.","tokens_in":10169,"tokens_out":3546,"duration_ms":43789,"significance":"If the reported results transfer to real design failures, VCDiag would be a practical and useful contribution: it targets RTL simulation, uses the standard VCD format, avoids proprietary log parsing, and provides an open-source artifact with explicitly separated train/test splits and reproducible processing scripts. The reported compression ratio and parallelization are genuinely useful engineering results, and the use of multiple open RTL designs (OpenTitan, FabScalar) is a strength relative to much prior work that evaluates on proprietary or very small designs. The main significance is conditional on whether the synthetic-bug benchmark adequately represents real RTL failure signatures; the paper is transparent about this limitation but does not close the gap.","major_comments":[{"comment":"The central accuracy claims in Table IV are measured entirely on failures generated by the authors' LLM-based synthetic bug injector. The paper does not provide any experiment on real failing regression tests, historical bug fixes, or independently authored bug reports, so the 77% top-1 and 94% top-3 numbers are demonstrated only for the distribution of BugGen mutations. Section IV-C itself notes that the generator requires plain-English descriptions and auto-reverts ineffective bugs, which may bias the injected set toward simple, localized mutations; Section VI-A additionally acknowledges that FabScalar stimulus diversity is limited by fixed program binaries. This is not an internal inconsistency, because the held-out test scenarios are legitimately separated from training, but it is a load-bearing external-validity gap for the paper's stated goal of accelerating real failure triage. Please add at least one evaluation on real failing waveforms or human-written bug fixes, or explicitly and pervasively scope all conclusions to synthetic-bug triage.","section":"Section IV-B and Section VI-B"},{"comment":"The abstract and Section VI-B claim that the signal-selection and statistical-compression pipeline preserves features essential for classification, but no ablation is reported that compares classification accuracy with and without these stages. The reader can see the final compressed feature set in Table V, but there is no experiment using, for example, all signals versus selected signals, or raw/rough CSV features versus statistically compressed features, on the same train/test split. Without such an ablation, the claim that a 123x reduction is achieved 'while preserving features essential for classification' is unsupported. I request an ablation study that quantifies the accuracy impact of each pipeline stage; if the accuracy is already high without compression, the compression claim should be reframed as a storage/efficiency result rather than a classification-preservation result.","section":"Section IV-B and Section VI-B"},{"comment":"The comparison with BugMD in Section II and the statement in Section VI-A that 'our approach exceeds BugMD's reported results on FabScalar' is not a same-benchmark comparison. BugMD is evaluated on post-silicon CPU bugs with gate-level injection, a different abstraction level, a different bug population, and different metric definitions. The present paper uses RTL-level bug injection, a different feature representation, and a different simulator setup. As reported, the accuracy gap between VCDiag and BugMD may be explained by any of these differences rather than by the proposed method. Please either adapt BugMD to the same RTL-level synthetic-bug benchmark or clearly characterize the comparison as cross-study and not a direct superiority claim.","section":"Section II and Section VI-A"}],"minor_comments":[{"comment":"All accuracy numbers are point estimates with no confidence intervals, standard deviations, or repeated-seed results; given the variability visible in the FabScalar accuracy versus simulation-window experiment, bootstrap or repeat-run intervals would make the headline numbers more reliable.","section":"Table IV"},{"comment":"In the AES rows, two modules are both labeled 'FSM' with different training/test counts; the later text distinguishes CipherControlFSM and AESCounterFSM, so the table should use these distinct names to avoid ambiguity.","section":"Table II"},{"comment":"Step 3 of the compression pipeline says statistical compression 'shrinks the data size but increases the features'; this trade-off is clear, but the interaction between the optional XGBoost signal count limit (5000 signals) and the number of statistical features per signal (n) is not defined, so the final feature dimensionality is not reproducible from the text alone.","section":"Section IV-B"},{"comment":"The artifact appendix says the tool is built entirely on open-source libraries, but the main experiments use Synopsys VCS and Cadence Xcelium (Table II); please state whether the artifact can reproduce results with an open-source simulator such as Icarus Verilog or Verilator, or list the commercial dependencies explicitly in the reproducibility statement.","section":"Artifact Appendix"}],"recommendation":"major_revision","confidential_remarks":"The paper is honest about its synthetic-only evaluation and the limitations of the LLM-based bug generator, which is a plus. The main risk is that the headline accuracy claims are presented in the abstract and conclusion in a way that implies readiness for real failure triage, while the evidence only supports classification of synthetic mutations. I would also encourage the editor to weigh whether the BugMD comparison, which is cross-study and not same-benchmark, should be softened. These issues are fixable within the manuscript's scope, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe useful thing here is that VCDiag is a working, testbench-agnostic pipeline for turning RTL simulation waveforms into a module-level triage prediction, with public artifacts backing the claims. The ML pieces are off-the-shelf, but the application is genuinely new: nobody has done supervised classification directly on RTL VCD signal activity for failure triage across multiple open designs, and the AST-guided signal selection plus statistical compression is a practical answer to the 100+GB waveform problem. The authors also behave well: train/test separation is explicit, model defaults are used, and limitations like FabScalar's fixed stimulus are admitted rather than buried.\n\nThe soft spot is the one everyone will trip on: every failing waveform in the evaluation comes from bugs injected by the authors' own LLM-based BugGen pipeline. That makes the 77% top-1 / 94% top-3 on FabScalar an internal benchmark on synthetic mutations. There is no experiment with real failing tests or historically known bugs, and no analysis showing the injected mutations produce waveform signatures comparable to real RTL defects. The BugMD comparison in Section II is also indirect, because it uses a different abstraction level and bug population, so it doesn't rescue the transfer claim. This is not an internal inconsistency; the paper is transparent about only using generated bugs. It is a load-bearing external-validity gap: if the synthetic signatures are systematically simpler or more localized than real bugs, the reported accuracy overstates real triage performance.\n\nSecondary issues are minor. There are no confidence intervals across reseeds, and no ablation isolating the contribution of signal selection or statistical compression to the accuracy numbers. The scalability claim in Section VI-D is asserted with plausible rationale, but nothing here demonstrates it on a truly large design.\n\nVerdict: give it a serious referee. The framework is coherent, the artifacts look real and accessible, and the synthetic-bug dataset is a legitimate first step, but the reviewer should push for validation on real RTL bugs, or at minimum a representativeness argument relating injected mutations to naturally occurring bug waveforms. I'd cite it as the current reference for VCD-based triage, and I'd bring it to our group's reading meeting to discuss how much synthetic-bug training can tell you about real failures.","headline":"Solid engineering paper with a real pipeline and honest reporting, but the headline accuracy numbers all rest on synthetic bugs from the authors' own generator, so treat them as internal validation, not field performance.","tokens_in":10546,"tokens_out":2121,"would_cite":true,"duration_ms":23325,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Machine learning on chip-simulation waveforms can rank the failure-causing module among the top three 94% of the time.","keywords":["functional verification","failure triage","VCD waveform","machine learning","RTL debugging","statistical compression","synthetic bug injection","LightGBM"],"falsifier":"Run the trained classifier on a set of real, known-cause RTL failures, for example bugs fixed in an open-source processor's repository, and compare top-3 accuracy against the 94% reported on injected FabScalar bugs; a large drop on real failures would falsify the claim that these waveform signatures generalize beyond synthetic mutations.","tokens_in":9585,"feed_emoji":"🎯","tokens_out":7219,"duration_ms":66660,"temperature":0.7,"pith_summary":"VCDiag is a machine-learning framework that turns failing simulation waveforms into a short ranked list of suspect RTL modules, so verification engineers can start debugging at the likely fault location instead of reading traces by hand. It works by extracting only signals tied to target modules, compressing each waveform with statistical summaries, and training a tree-based classifier on waveforms labeled by synthetic injected bugs. On the largest benchmark, the FabScalar processor, the best model ranks the true failure-causing module in its top three predictions 94% of the time, while shrinking raw waveform data by 123x. The paper argues that this makes learned failure triage practical for large designs and easy to adapt to any Verilog/SystemVerilog testbench.","feed_headline":"Waveform ML classifier ranks buggy module in top 3 at 94% accuracy","feed_subtitle":"Compresses 308 GB of simulation traces to 2.5 GB while keeping enough detail to guide verification engineers.","key_machinery":"The central object is the Value Change Dump (VCD) file, the IEEE-standard waveform format that records every signal's value changes during simulation. VCDiag's load-bearing mechanism is a four-stage reduction pipeline: an AST-based crawler (using Verible) builds a module-to-signal lookup table to prune irrelevant signals; a fixed 2000-tick window around the failure is normalized by trimming and zero-padding; sktime's SummaryTransformer compresses each signal into one row of statistical features (mean, standard deviation, quantiles); and optional XGBoost feature ranking keeps the top 50–70% of signals, capped at 5000. The statistical compression is what makes the classification tractable, turning gigabytes of sparse temporal traces into a compact tabular dataset that classical supervised models can learn from quickly.","core_discovery":"In the paper's own terms, the central discovery is that statistically compressed VCD waveforms retain enough temporal and structural information to classify a failing simulation to its root-cause module: on the 12-module FabScalar processor, LightGBM achieves 77% top-1 and 94% top-3 accuracy, and across the OpenTitan IP blocks the top-3 accuracy reaches 96–99%. The framework formulates the task as multivariate time-series classification, with a waveform $X \\in \\mathbb{R}^{T \\times d}$ mapped by $f: \\mathbb{R}^{T \\times d} \\to \\{1,\\dots,M\\}$ to one of $M$ modules. Rather than feeding raw traces, VCDiag prunes signals via an AST-derived lookup table, truncates to a 2000-tick failure window, and reduces each signal to statistical features (mean, standard deviation, quantiles), which the paper shows is sufficient for accurate module identification. The paper further claims that tree-based gradient boosting models outperform distance- and bagging-based baselines on this representation, and that the compression lowers FabScalar's dataset from 308 GB to 2.5 GB without hurting accuracy.","pith_inferences":["If real RTL bugs produce waveform signatures similar to the injected mutations, VCDiag could be applied directly to industrial regression failures, but the paper gives no evidence for that resemblance, so this remains a conjecture.","The comparable accuracy at 200 ticks suggests the classifier largely exploits timeouts and early-failure signatures on FabScalar; bugs without such timing traces might need different features.","The single-module labeling scheme limits the method to one bug per failing test, so practical multi-bug triage would require the iterative 'onion-peeling' strategy the paper only sketches as future work.","Because signal names must stay consistent and parameter renames invalidate training data, the approach may need retraining whenever a design evolves across regression versions."],"forward_implications":["On FabScalar, ranking the true module in the top three 94% of the time means an engineer can ignore most of the design and focus debugging on three modules.","Because the input is only VCD data, the same pipeline transfers across Verilog/SystemVerilog designs and simulators without changing the ML stack.","The 123x compression and 36-hour parallel generation of 1,600 bug scenarios make a large, ML-ready failure dataset feasible on a single 32-core machine.","Accuracy stays comparable when the trace window is cut from 2000 to 200 ticks, so shorter simulations may be enough for training data.","Adding a new design requires only one automation script and a config file; adapting to additional OpenTitan IPs took hours once the first was integrated."],"supporting_citations":[{"why":"Supplies the BugMD baseline and the top-1/top-3 metrics that VCDiag compares against on FabScalar.","marker":"[7]"},{"why":"Verible's linter generates the ASTs from which the module-to-signal lookup table is built.","marker":"[9]"},{"why":"OpenTitan provides the AES block and additional IP modules used as benchmarks.","marker":"[12]"},{"why":"FabScalar provides the largest benchmark processor behind the 77%/94% accuracy claim.","marker":"[13]"},{"why":"XGBoost serves both as a compared classifier and as the signal-ranking tool for feature selection.","marker":"[14]"},{"why":"sktime's SummaryTransformer performs the statistical compression that turns each signal into summary features.","marker":"[15]"},{"why":"BugGen, the companion LLM pipeline, generates the synthetic bug scenarios that label all training waveforms.","marker":"[18]"},{"why":"LightGBM is the gradient-boosting model that achieves the best reported classification results.","marker":"[20]"}],"fun_headline_variants":["VCDiag: 94% top-3 fault module ID from compressed waveforms","Statistically compressed VCD traces still reveal fault module 94% top-3","VCDiag: 120x smaller traces still give top-3 module accuracy at 94%","Waveform classifier: 94% top-3 module ID after 120x compression"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The accuracy numbers assume that the synthetically injected, single-module bugs produce waveform signatures representative of real chip bugs, an assumption the paper does not test on any real failing testcases.","fun_headline_variants_meta":{"raw":{"variants":["VCDiag: 94% top-3 fault module ID from compressed waveforms","Statistically compressed VCD traces still reveal fault module 94% top-3","VCDiag: 120x smaller traces still give top-3 module accuracy at 94%","Waveform classifier: 94% top-3 module ID after 120x compression"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000567,"raw_usage":{"total_tokens":2675,"prompt_tokens":926,"completion_tokens":1749,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":542,"completion_tokens_details":{"reasoning_tokens":1655}},"tokens_in":542,"tokens_out":1749,"duration_ms":13754,"temperature":1.0,"reasoning_tokens":1655,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:58:51.521525+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the trained classifier on a set of real, known-cause RTL failures, for example bugs fixed in an open-source processor's repository, and compare top-3 accuracy against the 94% reported on injected FabScalar bugs; a large drop on real failures would falsify the claim that these waveform signatures generalize beyond synthetic mutations.","supporting_citations":[{"cited_title":"Bugmd: Automatic mismatch diagnosis for bug triaging,","cited_arxiv_id":null,"evidence_quote":"Supplies the BugMD baseline and the top-1/top-3 metrics that VCDiag compares against on FabScalar."},{"cited_title":"Verible,","cited_arxiv_id":null,"evidence_quote":"Verible's linter generates the ASTs from which the module-to-signal lookup table is built."},{"cited_title":"Opentitan: Open source silicon root of trust (rot),","cited_arxiv_id":null,"evidence_quote":"OpenTitan provides the AES block and additional IP modules used as benchmarks."},{"cited_title":"Fab- scalar: Composing synthesizable rtl designs of arbitrary cores within a canonical superscalar template,","cited_arxiv_id":null,"evidence_quote":"FabScalar provides the largest benchmark processor behind the 77%/94% accuracy claim."},{"cited_title":"sktime documentation,","cited_arxiv_id":null,"evidence_quote":"sktime's SummaryTransformer performs the statistical compression that turns each signal into summary features."},{"cited_title":"BugGen: A Self-Correcting Multi-Agent LLM Pipeline for Realistic RTL Bug Synthesis","cited_arxiv_id":"2506.10501","evidence_quote":"BugGen, the companion LLM pipeline, generates the synthetic bug scenarios that label all training waveforms."}],"review_version":1}