{"id":"9e5e3dfb-ca3c-4e47-b490-d64e4c611f53","arxiv_id":"2502.07063","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A systematic survey and reproducible benchmark of 25 open-source zero-knowledge proof frameworks, with Docker environments and performance comparisons on SHA-256 and matrix multiplication.","lead":"This paper surveys 25 open-source zero-knowledge proof frameworks, compares 18 of them on matrix multiplication and SHA-256 benchmarks, and releases Docker containers so developers can reproduce the experiments. It is a practical guide for choosing a proof framework and for understanding the tradeoffs between proof size, speed, and setup requirements.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Benchmark comparability is the load-bearing premise: Table IV and Fig. 2 rank frameworks without pinned repository state, per-framework circuit parameters, or raw logs, so the quantitative recommendations are not yet reproducible.","rationale":"The reader's weakest_assumption correctly identifies the comparability of benchmarks as the most fragile load-bearing premise. My independent reading of the manuscript finds the same issue, with concrete support: Section IV-B's explicit caveat, the mixed arithmetization and frontend pipelines, the missing commit hash and raw data, and the malformed Table IV entries. The recommendations in Section IV-C and the Figure 2 decision flowchart are the practical payoff of the survey; if the measurements are not comparable or reproducible, the central claim of offering measured tradeoffs for framework selection is weakened. This is a correctness-and-reproducibility concern, not a novelty or scope issue. I agree with the reader that the paper still has clear qualitative value: it covers 25 frameworks, provides usability/accessibility analysis, and releases Docker environments with documented examples, which is genuine independent support for the survey aspect. Those artifacts do not, however, rescue the quantitative rankings. The condition should therefore remain: accept conditionally, contingent on pinning the repository, publishing raw logs and per-framework configuration, and fixing the table presentation. No change to the reader's verdict is needed; the concern is real but does not rise to rejection, because the qualitative survey and the released environments stand apart from the benchmark table.","tokens_in":32251,"tokens_out":4686,"duration_ms":40914,"concrete_test":"Pin the repository to a specific commit and publish, for every benchmarked configuration: the exact circuit source, arithmetization format, field/curve parameters, security parameters, compiler/frontend versions, and Dockerfile image digests. Then re-run Table IV from that pinned state on a reference machine, 10 runs per cell, and report raw logs and variance. If the relative ranking of frameworks in Table IV (or the branches of Fig. 2) changes by more than one position, or if any single value shifts by more than 20%, the quantitative recommendations are not robust.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim promises measured setup, prover, verifier, proof size, and communication tradeoffs that let developers select a framework. That claim rests on Table IV and Figure 2, which rank 18 configurations across five constructions. The load-bearing premise is that these measurements are comparable and stable across heterogeneous frameworks. The paper itself concedes 'there is no completely fair way to benchmark these' (Section IV-B) and 'we aim to keep settings consistent ... as much as the frameworks allow' (Section IV-C). Three specific gaps make the premise insecure. (1) Circuit equivalence is not established: the same 32x32 matrix multiply and SHA-256 are compiled to R1CS with 32,768 and 59,281 constraints, but frameworks use different arithmetizations (R1CS, PLONK gates, AIR, boolean Bristol circuits), different fields/curves, and different frontend/compiler pipelines. For example, Noir's setup+prover time is reported as a single number, and Diet Mac'n'Cheese's results are dominated by PicoZK's SIEVE IR compilation, as the authors themselves note. Without per-framework constraint counts, field choices, security parameters, and compiler versions, prover/verifier times are not apples-to-apples. (2) Reproducibility is not pinned: the GitHub link is given without a commit hash, no raw logs or variance data are reported, and several Table IV cells are malformed (Noir's row lacks a separate prover time; Gnark PLONK-FRI proof size is missing). (3) The scalability curves in Figure 1 have no data points or error bars. If any of these vary, the rankings and the flowchart recommendations could change. This does not invalidate the qualitative survey or the Docker environment contribution, but it means the central quantitative comparison is not yet supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript surveys 25 open-source zero-knowledge proof frameworks across zk-SNARK, PLONK, MPCitH, VOLE-based, and zk-STARK constructions, evaluates their usability and accessibility, benchmarks 18 configurations on SHA-256 and 32x32 matrix multiplication, and provides Docker containers and documented examples in a public repository. The central claim is that this is the first systematic survey of open-source ZKP frameworks spanning all constructions, with accompanying open-source environments, benchmarks, and documentation.","tokens_in":32497,"tokens_out":6357,"duration_ms":56770,"significance":"If the quantitative comparisons are made interpretable and reproducible, this is a genuinely useful community resource. The paper ships Docker environments for a wide range of frameworks, which directly addresses a real accessibility barrier, and the qualitative usability/accessibility analysis in Section III and Table III is valuable. The framework recommendations and the decision flowchart in Figure 2, however, rest on the benchmark results in Table IV, and those results are not yet controlled or reproducible to the standard the paper's claims require.","major_comments":[{"comment":"The benchmark comparability premise is load-bearing and not established. Section IV-B concedes 'there is no completely fair way to benchmark these,' but Table IV and Figure 2 use the measurements to rank frameworks and drive the recommendations in Section IV-C. The two benchmarks are characterized only by R1CS constraint counts (32,768 and 59,281 constraints), while the evaluated systems use heterogeneous arithmetizations (R1CS, PLONK gates, AIR, boolean Bristol circuits), different fields and curves, and different frontend-to-backend pipelines (e.g., Noir's ACIR and Diet Mac'n'Cheese's PicoZK/SIEVE path). Per-framework constraint counts, field and curve choices, security parameters, compiler versions, and circuit-level parameters are not reported, so the prover/verifier timings and proof sizes cannot distinguish framework capability from implementation and configuration choices.","section":"Section IV-B, Table IV"},{"comment":"The claimed reproducibility is not yet pinned down. Section I gives a GitHub URL without a commit hash or release tag, and Section IV-A reports only means of 10 runs, with no variance, confidence intervals, or raw logs. Table IV also contains malformed or ambiguous cells: the Noir row lacks a separate prover time despite the column structure, and footnote 2 only says setup and prover are measured together; the PLONK-FRI proof-size cells are missing. The authors should provide a pinned repository state, per-run data, environment details, and scripts/logs so that the benchmark results can be independently checked.","section":"Section I, Section IV-A, Table IV"},{"comment":"The comparative recommendations go beyond what the data support. Statements such as 'GNARK-KZG achieves excellent performance', 'we recommend Emp-ZK', and the placement of RISC Zero over Miden VM and Zilch are presented as conclusions from Table IV and Figure 2. Given the comparability gaps noted in Section IV-C ('we aim to keep settings consistent ... as much as the frameworks allow'), these rankings should be made conditional or softened. In particular, the paper itself notes that Diet Mac'n'Cheese's poor results are dominated by PicoZK's SIEVE IR compilation rather than by the underlying proof system; the same kind of caveat applies to every cross-framework ranking in Table IV.","section":"Section IV-C, Figure 2"}],"minor_comments":[{"comment":"The Zokrates rows cite reference [26] (arkworks) instead of the Zokrates reference [64]; the citation should be corrected for both the Groth16 and GM17 rows.","section":"Table IV"},{"comment":"The LEGOSnark row lists 'Brakedown-like [77]' as the proof system, but the text in Section III-A describes LegoSNARK as implementing CP-SNARKs built on libsnark; this inconsistency should be resolved.","section":"Table V"},{"comment":"The framework is referred to as both 'Mozzarella' and 'MozZ2karella'; a single consistent name should be used throughout.","section":"Section III-C, Table III"},{"comment":"The flowchart text is difficult to read in the extracted version, with some labels running together; the authors should ensure the figure is legible in the final PDF and that each decision branch is clearly labeled.","section":"Figure 2"},{"comment":"The hardware description ('128GB RAM, AMD Ryzen 3990X CPU desktop') omits details such as CPU frequency, memory type, Docker resource limits, and background load controls; these should be specified for reproducibility.","section":"Section IV-A"}],"recommendation":"major_revision","confidential_remarks":"The survey's qualitative content and Docker-based tooling are solid contributions, and the benchmark comparability/reproducibility issues are addressable with supplementary data and more cautious claims. The main risk is that Table IV and Figure 2 will be cited as definitive framework rankings; the revision should make their limitations explicit and provide the missing reproducibility artifacts."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague —\n\nThe real value of this paper is the artifact, not the taxonomy: a Docker-based reproducibility environment covering 25 ZKP frameworks, with examples and benchmarks, plus a usability/accessibility rating table. That genuinely lowers the barrier to entry for applied ZK work, and it is the first academic survey I have seen that spans zk-SNARK, PLONK, MPCitH, VOLE, and zk-STARK frameworks side by side. The qualitative framework descriptions are honest, self-aware, and mostly accurate.\n\nThe soft spot is the quantitative ranking. The survey benchmarks 18 frameworks on two workloads, but the measurements are not yet reproducible enough to support the recommendations in Figure 2. There is no commit hash for the repository, no raw logs, no variance or confidence intervals, and several Table IV cells are malformed (Noir's setup+prover lumped together, Gnark PLONK-FRI proof size missing). More importantly, the same high-level circuit is compiled to different arithmetizations with different constraints, fields, curves, and compiler versions, so prover times are not apples-to-apples. The authors themselves concede 'there is no completely fair way to benchmark these,' which is the right attitude, but then they go ahead and rank frameworks anyway. The scalability curves in Figure 1 have no data points or error bars, so they are hard to audit.\n\nNone of this invalidates the qualitative survey or the Docker contribution. The benchmark numbers are probably fine as rough ballpark indicators, and the authors do flag the biggest caveat (Diet Mac'n'Cheese is bottlenecked by PicoZK's SIEVE IR). I also do not think the 'first survey' claim is egregious; prior industry benchmarks exist, but this is the first systematic academic one with this breadth, and that is defensible.\n\nWho this is for: applied researchers and practitioners choosing a ZKP framework, and newcomers who want an overview of the landscape plus working environments to try. It is not a cryptographic theory paper, and it should not be judged as one.\n\nRecommendation: send it to peer review. It deserves referee time because the artifact is useful and the survey fills a gap, but the benchmark section needs a revision: pin the repository state, publish raw logs and per-construction parameters, fix the table, and soften the ranking claims until the measurements are reproducible.","headline":"Useful survey with a genuinely useful Docker artifact; the benchmark rankings need reproducibility fixes before they can carry the paper's recommendations.","tokens_in":33118,"tokens_out":2489,"would_cite":true,"duration_ms":22328,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper surveys 25 open-source zero-knowledge proof frameworks, benchmarks 18 configurations on two circuits, and packages each environment in Docker for reproduction.","keywords":["zero-knowledge proofs","zk-SNARK","zk-STARK","PLONK","MPC-in-the-Head","VOLE-based proofs","framework benchmarking","open-source survey"],"falsifier":"Re-run the two benchmarks at a larger circuit size, for instance 256x256 matrix multiplication, on the paper's released Docker containers; if the relative ordering of prover times or proof sizes changes materially, the decision flowchart's recommendations would not transfer to larger circuits. Recording per-run variance and peak memory would also test whether the reported means are stable.","tokens_in":32018,"feed_emoji":"🔐","tokens_out":7516,"duration_ms":60730,"temperature":0.7,"pith_summary":"The paper sets out to make the zero-knowledge proof ecosystem navigable for application developers by surveying 25 open-source frameworks that together span the major constructions: zk-SNARKs, PLONK-style systems, MPC-in-the-Head, VOLE-based protocols, and zk-STARKs. It claims to be the first survey at this breadth and to include, for each framework, a working Dockerized development environment with documented code so that the same SHA-256 and matrix-multiplication experiments can be rerun. The measured quantities are trusted-setup time, prover time, verifier time, and proof size or communication cost. The payoff the authors aim for is a concrete decision procedure: a developer can look at their computation, bandwidth, and trust assumptions, and pick a framework whose measured tradeoffs match the application.","feed_headline":"25 zero-knowledge proof frameworks benchmarked side by side","feed_subtitle":"Reproducible benchmarks map proof size, setup, prover, and verifier tradeoffs across every major ZKP construction.","key_machinery":"The carrying mechanism is the benchmark harness rather than a single mathematical identity: two standardized circuits, run inside per-framework Docker containers on identical hardware, with four metrics recorded (setup time where applicable, prover time, verifier time, and proof size or communication). The two circuits are chosen because they represent common privacy-preserving workloads, and their constraint counts anchor the comparison across heterogeneous arithmetization schemes. The paper's own caveat that there is no completely fair way to benchmark across such different constructions is part of the methodology; the authors treat the numbers as indicative of algorithmic complexity and supply the environments so readers can re-measure for themselves.","core_discovery":"On the paper's own terms, the central discovery is a reproducible performance and usability map of the current open-source ZKP landscape. Eighteen benchmarked configurations drawn from the 25 surveyed frameworks are measured on two circuits, 32x32 matrix multiplication with about 32,768 R1CS constraints and SHA-256 with about 59,281 constraints, in identical Docker containers on one machine, with each number the mean of 10 runs. The results show a consistent tradeoff structure rather than a single winner: zk-SNARK-style systems give the smallest proofs and cheapest verification but require a trusted setup and a relatively strong prover; zk-STARK-style systems remove the setup and are post-quantum at the price of much larger proofs; VOLE-based systems split work between prover and verifier but are designated-verifier; and MPC-in-the-Head has the fewest accessible general-purpose frameworks. The paper reads these tradeoffs as evidence that framework choice should be application-driven, and it converts the findings into a decision flowchart that routes a developer to a recommendation based on prover strength, interactivity, bandwidth, and ease-of-use preferences.","pith_inferences":["The paper's own analysis of the Python-frontend-to-VOLE-framework path suggests that the intermediate representation and frontend compiler can dominate end-to-end performance as much as the proving system; a fair comparison may need to separate frontend overhead from protocol cost.","Because the reported numbers are means over 10 runs on one machine without variance or memory logs, the ordering of close competitors should be treated as indicative; pinning repository commits and re-running with memory and variance recorded would strengthen the comparison into a durable benchmark.","If post-quantum security hardens into a requirement, the survey's advice would shift decisively toward transparent constructions, so recommendations should be revisited as non-interactive VOLE-based and MPC-in-the-Head frameworks mature.","The maintained Docker repository could grow into a community benchmark where framework authors submit their own containers, making the survey's central promise of reproducibility an ongoing property rather than a point-in-time snapshot."],"forward_implications":["Application developers can reproduce every benchmark on their own hardware using the supplied Docker environments, removing dependency setup as a barrier to evaluating a framework.","For bandwidth-constrained applications that accept a trusted setup, the measurements point to zk-SNARK/PLONK-style frameworks as the practical choice because proof size and verifier time stay small and nearly constant as circuits grow.","For settings that require post-quantum security and no trusted setup, zk-STARK-style frameworks trade transparency for proof sizes that are orders of magnitude larger and prover runtimes that grow more steeply.","VOLE-based frameworks are recommended for applications that already require communication, such as distributed or federated learning, because they shift part of the computational load to the verifier.","The decision flowchart gives a direct route from resource constraints, such as strong-prover availability and interactivity needs, to a short list of suitable frameworks."],"supporting_citations":[{"why":"defines the zk-SNARK construction that anchors the SNARK benchmark category.","marker":"[40]"},{"why":"introduces zk-STARKs, the transparent post-quantum construction benchmarked in the survey.","marker":"[38]"},{"why":"supplies the MPC-in-the-Head transformation underlying the surveyed MPCitH frameworks.","marker":"[87]"},{"why":"defines PLONK, the universal-setup construction evaluated via multiple backends.","marker":"[69]"},{"why":"defines Groth16, the proving system used in most measured zk-SNARK configurations.","marker":"[79]"},{"why":"provides one of the benchmarked Rust-based zk-SNARK ecosystems.","marker":"[26]"},{"why":"provides the Go-based framework benchmarked on both zk-SNARK and PLONK backends.","marker":"[49]"},{"why":"provides the Rust zkVM benchmarked as the representative zk-STARK framework.","marker":"[115]"}],"fun_headline_variants":["25 ZKP frameworks benchmarked head-to-head","ZKP frameworks: 25-way benchmark comparison","Which ZKP framework should you use? 25 tested","ZKP survey: 25 frameworks, one benchmark suite","Benchmark-driven guide to 25 ZKP frameworks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the 18 measured configurations are representative of their frameworks, meaning the same high-level circuits can be implemented equivalently across very different arithmetization schemes and that a handful of runs on one machine captures the performance a developer would see.","fun_headline_variants_meta":{"raw":{"variants":["25 ZKP frameworks benchmarked head-to-head","ZKP frameworks: 25-way benchmark comparison","Which ZKP framework should you use? 25 tested","ZKP survey: 25 frameworks, one benchmark suite","Benchmark-driven guide to 25 ZKP frameworks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000833,"raw_usage":{"total_tokens":3699,"prompt_tokens":1070,"completion_tokens":2629,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":686,"completion_tokens_details":{"reasoning_tokens":2553}},"tokens_in":686,"tokens_out":2629,"duration_ms":17015,"temperature":1.0,"reasoning_tokens":2553,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T13:54:30.346576+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the two benchmarks at a larger circuit size, for instance 256x256 matrix multiplication, on the paper's released Docker containers; if the relative ordering of prover times or proof sizes changes materially, the decision flowchart's recommendations would not transfer to larger circuits. Recording per-run variance and peak memory would also test whether the reported means are stable.","supporting_citations":[{"cited_title":"Ishai, E","cited_arxiv_id":null,"evidence_quote":"supplies the MPC-in-the-Head transformation underlying the surveyed MPCitH frameworks."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"defines Groth16, the proving system used in most measured zk-SNARK configurations."},{"cited_title":"Introduction — risc zero developer docs, 2023","cited_arxiv_id":null,"evidence_quote":"provides the Rust zkVM benchmarked as the representative zk-STARK framework."}],"review_version":1}