{"id":"620b686d-2902-4b13-bb71-4a31ea403f7e","arxiv_id":"2506.03909","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Solsmith generates valid, diverse Solidity programs and, via differential testing across compiler configurations, uncovered four confirmed defects in the Solidity compiler.","lead":"Solsmith is a random generator that creates valid, diverse Solidity programs to hunt for compiler bugs. In tests it generated 100,000 programs and reports four confirmed defects in the Solidity compiler, found by comparing executions across compiler configurations.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Differential oracle may be unsound: the evaluation-order avoidance in §4.2.2 only handles increment/decrement of variables, not side-effecting function calls in expressions, so cross-configuration mismatches may not be compiler defects.","rationale":"The reader's weakest assumption points to incomplete avoidance of undefined behavior and IR-dependent semantics; my concern is a specific, load-bearing instance of that general problem. The consistency strategy in §4.2.2 is described only for arithmetic expressions with increment/decrement operations, and the paper does not demonstrate that generated expressions are free of order-dependent function-call side effects. Because Solidity explicitly leaves expression evaluation order unspecified, a differential mismatch on such a program would not be a compiler defect. That directly threatens the validity of the four reported defects, which are the strongest evidence for the central claim. I do not allege any misconduct; the repository and reduced examples are positive evidence of real work. But the lack of upstream confirmation artifacts and the omission of viaIR from the stated configuration list mean the current evidence is not sufficient to move beyond a conditional verdict. A targeted static analysis of the generated corpus can settle whether evaluation-order-dependent programs actually occur, and if they do, whether the reported defects survive re-triage. This is a concrete, feasible check rather than a vague appeal for more statistics. My recommendation is therefore to keep the reader's CONDITIONAL verdict, pending that check and upstream confirmation of the four issues.","tokens_in":924,"tokens_out":891,"duration_ms":92122,"concrete_test":"Regenerate or reuse the 100k-program corpus and run a static analysis pass that flags every expression containing more than one function call to a non-pure/non-view function, and every function call whose argument expression contains a function call or a state read/write. If any flagged programs exist, extract a minimal example and run it under the paper's configurations (standard, optimized, and viaIR). If it reproduces a differential outcome that can be explained solely by evaluation order, the consistency assumption in §3.2/§4.2.2 is violated and the four reported defects must be revalidated against the official Solidity issue tracker. If no order-dependent programs exist, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that Solsmith reveals four confirmed solc defects depends on the soundness of its differential oracle: every behavioral difference across the tested configurations must be a genuine compiler bug, not a consequence of Solidity's unspecified semantics. The paper's consistency design (§3.2) promises to remove undefined behavior and IR-dependent semantic differences, but the concrete mechanism for evaluation-order differences (§4.2.2) is incomplete. It tracks variables that have participated in increment/decrement operations and substitutes constants when such variables are reused, but it does not prevent two state-reading or state-writing function calls from appearing as siblings in one expression, nor a side-effecting call from appearing as a function argument. Solidity leaves the evaluation order of expression children unspecified, and the legacy and viaIR code generators are known to differ on this point. Since Solsmith deliberately generates function-call expressions and functions with arbitrary modifiers (not restricted to pure/view), generated programs can legitimately produce different outputs purely because evaluation order differs across configurations; such a mismatch is not evidence of a compiler bug. Reinforcing this concern, the paper provides no upstream issue/PR references for the four 'confirmed' defects, so independent verification is impossible, and §5.1 lists only three configurations without mentioning viaIR even though §5.2.2 and §5.2.4 attribute defects to viaIR. These gaps leave the central claim less secure than the evaluation narrative suggests.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents Solsmith, a random test program generator for the Solidity compiler. Its design is organized around three objectives—consistency, diversity, and compliance—and implements strategies that include avoiding undefined behaviors, mitigating IR-dependent semantic differences, increasing the frequency of optimizer-triggering constructs, and inserting runtime checks for type, division, and array-bound errors. The tool is evaluated by differential testing: 100,000 generated programs are run under three configurations of solc (standard, optimizer with runs=1, and optimizer with runs=2^32−1), and the paper reports four distinct defects in solc 0.8.0, with a minimized example and a root-cause discussion for each. The paper concludes that these results demonstrate the effectiveness and potential of Solsmith.","tokens_in":14161,"tokens_out":6976,"duration_ms":59818,"significance":"If the central claims are substantiated, Solsmith would be a valuable contribution to Solidity compiler testing, an area with relatively few dedicated program generators. The open-source release of the tool and the detailed description of generation strategies are concrete assets. However, the evaluation as presented does not yet establish the claimed effectiveness: the differential oracle is not convincingly sound, the link between Solsmith's generated programs and the four reported defects is not shown, and there is no baseline comparison to random generation or existing Solidity fuzzers. The paper therefore provides a useful design and an interesting set of candidate bugs, but the evidence for the headline contribution is incomplete.","major_comments":[{"comment":"The differential-testing oracle is not sound because the avoidance of expression-evaluation-order differences is incomplete. The strategy in §4.2.2 tracks only variables that have been incremented or decremented and replaces them with constants on reuse; it does not prevent side-effecting function calls from appearing as subexpressions of arithmetic or relational expressions, even though §4.1.3 explicitly includes function-call expressions that return results. Solidity leaves the evaluation order of expression children unspecified, and the legacy and viaIR code generators are known to differ on this point. Consequently, a cross-configuration output difference that involves such a call is not necessarily a compiler defect. The four reported defects must be re-examined under this criterion, or the generator's consistency strategy must be strengthened, before the claim that Solsmith reveals confirmed defects can be accepted.","section":"§4.2.2 and §5.2"},{"comment":"The manuscript does not establish that the four defect-triggering programs are outputs of Solsmith. Figure 9 is a standalone Yul program, not a Solidity contract; Figures 10–12 are Solidity snippets but no reduction process is described, and no statistics are given on how many Solsmith-generated programs triggered each defect before minimization. Without this link, the four 'confirmed defects' cannot be attributed to Solsmith's generation strategy.","section":"§5.2.1–§5.2.4 and Figures 9–12"},{"comment":"The evaluation lacks baseline comparisons and quantitative success metrics. To support the claim that Solsmith is effective, the paper should compare against a random generator that ignores the described strategies, or against an existing Solidity fuzzer such as FUZZOL, using the same differential-testing harness. It should also report the number of defect-triggering programs per 100,000, the false-positive rate, and runtime. Without such a baseline, the four defects cannot be attributed to the generator's design.","section":"§5"},{"comment":"The word 'confirmed' is not verifiable because the paper gives no upstream issue or pull-request identifiers from the Solidity repository. Providing these links is standard for compiler-testing papers and is necessary to substantiate that the reported defects are genuine and not false positives of the comparison setup.","section":"§5.2"}],"minor_comments":[{"comment":"The text refers to 'John et al. [25]', but reference [25] is the YARPGen paper by Livinskii, Babokin, and Regehr; correct the citation.","section":"§4.2.1"},{"comment":"The heading 'Preventing Integer Overflow' is misleading because wrapping arithmetic inside an `unchecked` block is not prevention of overflow; it is a deliberate choice to use wrapping semantics. Rephrase to something like 'Avoiding overflow-induced reverts'.","section":"§3.4"},{"comment":"The three configurations are all variations of cross-optimization testing on the same compiler version, whereas the background section (§2.2) also describes cross-version and cross-compilation strategies, which are not used in the evaluation. Clarify why only cross-optimization is chosen.","section":"§5.1"},{"comment":"Figure 9 is written in Yul, not Solidity; state this in the caption and explain how a Solidity test program produced by Solsmith reduces to this Yul snippet.","section":"§5.2.1"},{"comment":"The term 'cse option' is not defined; state that this is the common-subexpression-elimination step of the Yul optimizer.","section":"§5.2.3"},{"comment":"The phrase 'non-expression-splitted' should be 'non-expression-split'.","section":"§5.2.1"},{"comment":"The text lists 'function argument evaluation order' as undefined behavior in §3.2, but §4.2.2 discusses it as an IR-dependent difference; make the relationship between these two concepts explicit.","section":"§3.2 and §4.2.2"}],"recommendation":"major_revision","confidential_remarks":"The open-source release of Solsmith is a positive feature, and the four candidate defects, if properly linked to Solsmith's output, would be interesting. However, the missing connection between the generator and the minimized examples, together with the unsound-looking oracle for evaluation-order side effects, makes the central claim currently unverifiable. The paper is within the scope of a software-engineering venue, but the evaluation needs substantial strengthening."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Solsmith is a real generator with a sensible design: it adapts Csmith-style random program generation to Solidity, and the three design goals (consistency, diversity, compliance) are spelled out with concrete strategies. The best part is that the paper takes undefined behavior seriously and actively avoids it, which is more than most compiler fuzzers do. The four reported defects, especially Keccak256 and the unchecked overflow one, are reduced to clean minimal examples and look like genuine compiler bugs.\n\nThe soft spots are mostly in the evaluation. First, there is no baseline—no comparison to random generation or to existing Solidity fuzzers like FUZZOL or SpecTest—so we don't know how much Solsmith's strategies actually add. Second, no upstream issue or PR links accompany the 'confirmed' defects, so independent verification is impossible. Third, the experimental setup lists three configurations (default, runs=1, runs=2^32-1) but two of the defects are attributed to viaIR, which is never mentioned in the setup. That is at minimum a presentation problem.\n\nThe bigger concern is about the consistency guarantee. Section 3.2 lists function argument evaluation order as undefined behavior, and Section 4.2.3 says they avoid relying on it. But the mechanism described in 4.2.2 for avoiding evaluation-order differences only tracks increment/decrement of variables. It does not appear to restrict side-effecting function calls appearing as siblings in an expression or as arguments. Since Solidity leaves evaluation order unspecified, a differential mismatch between configurations can be a legitimate semantic difference rather than a compiler bug. The paper's four defects might well be real, but the oracle is less sound than the narrative suggests, and the paper doesn't show how it filters out false positives from this source.\n\nIf you are in the compiler testing or smart contract tooling space, this is a relevant paper. It deserves a serious referee because the tool is real, the bug reports are concrete, and the methodology questions are exactly what reviewers should interrogate. I would not accept it as is; I would ask for a baseline comparison, artifact submission, upstream confirmation links, and a clear statement of how the generator excludes side-effecting function calls from unspecified evaluation order contexts.","headline":"A plausible new Solidity fuzzer with real bug reports, but the evaluation is thin and the consistency oracle may be unsound on evaluation order.","tokens_in":14662,"tokens_out":4201,"would_cite":true,"duration_ms":39616,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Solsmith randomly generates valid Solidity programs that expose confirmed defects in the Solidity compiler, with four distinct bugs found in solc 0.8.0.","keywords":["Solidity compiler testing","random program generation","differential testing","smart contracts","compiler defects","undefined behavior","optimizer testing","IR-based compilation"],"falsifier":"Re-running the four minimal reproducers on solc 0.8.0 with the stated flags and finding that a reported divergence is a documented, intended difference between the old and via-IR code generators, or that the divergence persists after the relevant compiler fix, would overturn the claim that all four are confirmed compiler defects.","tokens_in":13723,"feed_emoji":"🐞","tokens_out":4096,"duration_ms":35485,"temperature":0.7,"pith_summary":"The paper introduces Solsmith, a random generator of Solidity test programs built to find defects in the Solidity compiler. The authors argue that earlier work on Solidity compiler testing focused on testing methodology and left the harder problem of program generation unaddressed. Solsmith is designed around three goals: consistency (each generated program has a single meaning), diversity (wide coverage of language features), and compliance (programs compile cleanly). The evaluation runs 100,000 generated programs under three compiler configurations and reports four distinct, confirmed defects in solc 0.8.0, demonstrating that the generator is effective.","feed_headline":"Random program generator finds four confirmed Solidity compiler bugs","feed_subtitle":"Cross-configuration differential testing of the Solidity compiler turns up optimizer and via-IR defects.","key_machinery":"The central object is the Solsmith generator, whose mechanism is the combination of three design objectives with language-specific strategies. Consistency is achieved by avoiding the four documented Solidity undefined behaviors (inline-assembly length operations on storage arrays, function-parameter evaluation-order dependence, dangling references, and verbatim bytecode) and by suppressing behavioral differences between the old and via-IR code generators, such as repeated placeholder use and expression evaluation order. Diversity comes from a wide feature set and manually adjusted frequencies, including distorting frequencies so optimizations fire more often. Compliance is ensured by explicit type conversions, division-by-zero guards, array-bounds checks, and unchecked blocks for overflow-sensitive code. Differential testing across three solc configurations provides the oracle.","core_discovery":"The central claim is that careful random generation of Solidity programs, guided by language-specific consistency and compliance strategies, can reveal genuine Solidity compiler defects. The paper reports four confirmed defects in solc 0.8.0: a FullInliner defect that reorders function-call arguments during inlining, a function-selector defect under the via-IR pipeline that drops side effects, a keccak256 optimizer defect that treats hashes of different lengths as equal, and an unchecked-block defect where the via-IR pipeline mishandles increment at uint256 maximum and decrement at minimum. For each defect the paper gives a minimal triggering program and a root-cause explanation.","pith_inferences":["The frequency-distortion approach for triggering optimizer paths could be automated with search or learning to maximize bug-finding per generated program.","The reported defects suggest that differential testing across optimizer and runs settings, rather than just compiler versions, is a high-yield oracle for smart-contract compilers.","The same consistency strategies could be adapted to other smart-contract languages whose compilers offer both direct and IR-based code generation paths.","Because the compiler itself can emit keccak256 calls for mappings and indexed events, the keccak256 defect may affect real contracts, not just synthetic test cases."],"forward_implications":["The four confirmed defects give the Solidity team minimal reproducers and root causes for optimizer and via-IR pipeline bugs.","Solidity compilers still harbor configuration-dependent bugs that only appear under certain optimizer step sequences or IR settings.","Compiler testing research for Solidity needs dedicated program generators, not just testing methods, because generic generators do not transfer to Solidity's semantics.","The generator's strategies can be extended to later compiler versions and additional optimization combinations to find further defects."],"supporting_citations":[{"why":"Prior Solidity compiler fuzzing work it directly extends, by replacing mutation-based generation with random generation.","marker":"[17]"},{"why":"The C compiler random generator whose design philosophy for valid program generation Solsmith adapts to Solidity.","marker":"[20]"},{"why":"Supplies the technique of systematically distorting probability distributions to trigger optimizations more frequently.","marker":"[25]"},{"why":"Differential testing as the oracle for detecting compiler defects across configurations.","marker":"[22]"},{"why":"Alternative Solidity compiler testing approach based on specification-based testing, providing context for the generation gap.","marker":"[19]"}],"fun_headline_variants":["Random generator reveals 4 Solidity compiler defects","Solsmith random tests find 4 confirmed Solidity bugs","Four Solidity compiler flaws exposed by random generation","Solsmith: random programs expose 4 solc defects"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that Solsmith's consistency strategies fully eliminate undefined behavior and IR-dependent semantic differences, so any differential behavior across the three compiler configurations is a genuine compiler defect rather than a legitimate semantic difference.","fun_headline_variants_meta":{"raw":{"variants":["Random generator reveals 4 Solidity compiler defects","Solsmith random tests find 4 confirmed Solidity bugs","Four Solidity compiler flaws exposed by random generation","Solsmith: random programs expose 4 solc defects"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00017,"raw_usage":{"total_tokens":1229,"prompt_tokens":864,"completion_tokens":365,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":480,"completion_tokens_details":{"reasoning_tokens":302}},"tokens_in":480,"tokens_out":365,"duration_ms":10473,"temperature":1.0,"reasoning_tokens":302,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:52:22.099691+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-running the four minimal reproducers on solc 0.8.0 with the stated flags and finding that a reported divergence is a documented, intended difference between the old and via-IR code generators, or that the divergence persists after the relevant compiler fix, would overturn the claim that all four are confirmed compiler defects.","supporting_citations":[{"cited_title":"Syntax-aware mutation for testing the solidity compiler,","cited_arxiv_id":null,"evidence_quote":"Prior Solidity compiler fuzzing work it directly extends, by replacing mutation-based generation with random generation."},{"cited_title":"Finding and understanding bugs in c compilers,","cited_arxiv_id":null,"evidence_quote":"The C compiler random generator whose design philosophy for valid program generation Solsmith adapts to Solidity."},{"cited_title":"Random testing for c and c++ compilers with yarpgen,","cited_arxiv_id":null,"evidence_quote":"Supplies the technique of systematically distorting probability distributions to trigger optimizations more frequently."},{"cited_title":"Differential testing for software,","cited_arxiv_id":null,"evidence_quote":"Differential testing as the oracle for detecting compiler defects across configurations."},{"cited_title":"Spectest: Specification-based compiler testing,","cited_arxiv_id":null,"evidence_quote":"Alternative Solidity compiler testing approach based on specification-based testing, providing context for the generation gap."}],"review_version":1}