{"id":"5c1250c3-7c35-4fb7-b6af-5e5625f32aca","arxiv_id":"2608.04857","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A classification-driven static scanner with a separate rule repository discovers and assesses cryptographic assets at rest, achieving F1 0.75 on a synthetic benchmark and finding 370 assets in ten real services.","lead":"This paper presents a static scanner and classification scheme that discovers cryptographic assets in software source code, configuration files, and key material, producing a machine-readable CBOM inventory. A reader might use it to understand how cryptography is actually deployed in a codebase, which matters for security governance and post-quantum migration planning.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Real-world recall is measured against a self-compiled partial reference list; if manual inspection missed crypto in unsupported languages or unparsed files, the 'practical cryptographic transparency' claim is overstated.","rationale":"I read the paper as a carefully scoped systems contribution: it ships a classification, a rule repository, a scanner, and an honest evaluation. The strongest evidence is the head-to-head comparison with CBOMkit, the component-level benchmark results, and the explicit admission that real-world recall is partial. My concern is the same one the reader identified: the real-world ground truth is a manually compiled list by the authors, and the services are concentrated in Ruby/Go plus configuration files. Because the central claim is about practical transparency for deployed software systems, the evaluation needs some independent evidence that the scanner is not missing substantial crypto that a user would care about. The paper already conditions the claim in Section 8, but the abstract and conclusion present it more broadly. This does not change the reader's CONDITIONAL verdict; it underlines why the condition should be enforced before the result is generalized.","tokens_in":15456,"tokens_out":7391,"duration_ms":90509,"concrete_test":"Select one scanned service, preferably GitLab CE, and commission an independent inventory audit by researchers not involved in the paper, using manual inspection together with a second scanner that also covers JavaScript, Java, and Python. Compare the independent component list and post-quantum candidates against Crypsy's 370-component output. If the independent audit identifies additional crypto assets in unsupported languages or unparsed file types that Crypsy missed, the 'practical transparency' claim must be narrowed to the supported ecosystem subset; if the audit finds no substantial additional material assets, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 7.2 reports 'partial recall of 100%' against a manually compiled reference list and explicitly states this is not complete coverage. The real-world evidence therefore cannot bound the number of missed assets: the 370 discovered components and the 52 post-quantum candidates are lower bounds, not demonstrated inventories. Section 8 concedes that Java, C/C++, and Python ecosystems are not demonstrated, and the ten scanned services are dominated by Ruby, Go, and configuration files. A deployed GitLab instance also contains JavaScript and other file types that Crypsy does not parse, so any crypto in those files would be invisible. The central claim in Section 9 is stated as 'practical cryptographic transparency for deployed software systems' and the abstract states the conclusion without the language-scope caveat that appears later. For that claim to hold, a substantial fraction of deployed crypto must be discoverable. The partial, author-built ground truth cannot establish this; it only establishes that the scanner did not miss assets the authors already knew about.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents a classification-driven static approach for discovering cryptographic assets in software. It introduces three discovery categories (Crypto-Material, Crypto-Artifacts, Crypto-Invocations) and two assessment categories (Weakness, Vulnerability), derives a scanner-independent rule repository called Crypistry, implements a static scanner called Crypsy, and evaluates the artifacts on a synthetic benchmark (Cryben) and on ten deployed services. The reported headline results are an F1 of 0.75 for benchmark discovery (0.86 excluding by-design counting differences, 0.95 at component level), 91% assessment recall on expected identifiers, and, in the real-world scan, 370 discovered cryptographic components with six CVE-linked vulnerabilities and 52 post-quantum migration candidates. The paper is transparent that the real-world ground truth is partial and that language coverage is currently limited to Ruby, Go, and selected configuration formats.","tokens_in":15619,"tokens_out":6248,"duration_ms":64659,"significance":"The work addresses a real and timely gap: organizations lack structured inventories of cryptographic assets for governance and post-quantum migration. The main strengths are the clear separation between classification, rule repository, scanner, and benchmark; the explicit separation of by-design counting differences from true detection gaps in the synthetic evaluation; the head-to-head comparison with CBOMkit-hyperion; and the candid discussion of limitations in Sections 7.2 and 8. If the central claim is read as scoped to the supported ecosystems, the paper provides a useful, reproducible baseline for CBOM-oriented static crypto discovery. The open supplementary material strengthens the contribution by making the artifacts independently inspectable. The main weaknesses are that the real-world evaluation cannot bound missed assets because of its partial, author-built ground truth, and that the synthetic benchmark's ground truth is not fully independent of the taxonomy that generated the rules, so the reported correctness numbers partly measure internal consistency.","major_comments":[{"comment":"The central claim that 'rule-based static discovery, grounded in a structured classification, can provide practical cryptographic transparency for deployed software systems' is stated at a level of generality that the evidence does not support. Section 8 explicitly limits the empirical evidence to Ruby, Go, and selected configuration formats and states that Java, C/C++, and Python are 'not yet demonstrated.' Since those ecosystems hold a substantial fraction of enterprise cryptography, the claim should either be explicitly scoped to the evaluated ecosystems or be accompanied by evidence of transferability. As written, the abstract and conclusion assert practical transparency for deployed systems broadly, while the only deployed evidence comes from ten services dominated by Ruby, Go, and configuration files.","section":"§9 and Abstract"},{"comment":"Section 7.2 reports a 'partial recall of 100%' against a manually compiled reference list and correctly states that this is not exhaustive. However, this design cannot bound the number of missed assets; it only establishes that no asset already known to the authors was missed. The subsequently emphasized counts (370 components, 52 post-quantum candidates, six CVE-linked vulnerabilities) are therefore lower bounds, not demonstrated inventories. Because the manual inspection was conducted by the authors on the same repositories that were scanned, and because unsupported languages and unparsed file types are excluded from the scan, the real-world evaluation does not by itself justify the 'practical cryptographic transparency' conclusion. The manuscript should either report the fraction and language distribution of parsed versus unparsed files and give a best-effort estimate of crypto-bearing content in unparsed files, or explicitly weaken the claim to transparency for the supported languages and file types.","section":"§7.2, real-world recall"},{"comment":"Section 7.1 states that Cryben's ground truth is a CycloneDX CBOM 'constructed independently of Crypsy,' but it does not state that the ground truth was constructed independently of Crypistry or of the classification in Section 4. Given the paper's own account in Section 3 that the classification and the rule repository co-evolved, the benchmark annotations and the discovery rules plausibly share the same conceptual source. The reported F1 therefore partly measures the scanner's consistency with the authors' own taxonomy rather than agreement with an external standard. This is a threat to the 'correctness' interpretation of the benchmark numbers. The authors should either base the benchmark ground truth on an independent source (for example, a second annotation team or a pre-existing inventory constructed without reference to Crypistry), or explicitly frame the benchmark results as a consistency measure and discuss how much of the reported performance is attributable to the shared taxonomy.","section":"§7.1, benchmark construction"},{"comment":"The abstract highlights that the scanner 'correctly annotates 91% of expected weaknesses and vulnerabilities,' but that figure is benchmark-level assessment recall. In the real-world evaluation, Section 7.2 reports that of the 91 components receiving a CWE-mapped weakness annotation, only about 0.3 are actionable after manual review, because weak-hash and weak-PRNG rules fire on non-security digests and identifiers. This discrepancy is material to the assessment contribution. The manuscript should present the real-world actionable-weakness precision prominently and clarify in the abstract or conclusion that the 91% figure is a recall measure against a known-identifier benchmark, not a precision or actionability measure on real-world findings. Without this clarification, the assessment claim reads as stronger than the evidence supports.","section":"§7.2, actionable weakness precision"}],"minor_comments":[{"comment":"As typeset, the Material and Artifacts rows appear to have ten values for eleven columns, so the per-service counts for Redis and PostgreSQL are not fully recoverable from the table. Please realign the columns and recheck the row-wise sums against the P column.","section":"Table 4"},{"comment":"The CVE column labels are abbreviated to year ranges (CVE-2011, CVE-2013, CVE-2015, CVE-2019). Consider listing the full CVE identifiers or providing a legend so that the assessment coverage is directly interpretable.","section":"Figure 5"},{"comment":"The summary rows use the labels 'P Findings', 'P excl. by-design', and 'P component level', but 'P' is elsewhere used for precision. Rename these rows (for example, 'Overall', 'Overall excl. by-design', 'Component level') to avoid confusion with the precision column.","section":"Table 3"},{"comment":"The sentence 'A single rule is not bound to a single file' is unclear on first reading; consider rephrasing to explain that a rule can match multiple files and multiple occurrences, which is what makes the rule repository compact.","section":"§5.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid DSR-style tool paper with a transparent evaluation, but the central claim needs to be scoped more carefully and the synthetic benchmark needs to address the shared-taxonomy concern. I would support publication after major revision. The comparison with CBOMkit-hyperion and the explicit separation of by-design counting differences are valuable and should be retained."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know up front. This is a genuinely useful systems paper: the Material/Artifact/Invocation taxonomy is a clean organizing scheme, and the scanner-independent rule repository is a real contribution. But the headline numbers are softer than the abstract suggests, because the benchmark and the real-world ground truth are both author-built to a significant degree.\n\nWhat is actually new: the paper does not just offer another crypto-misuse detector. It gives a classification of what a cryptographic asset is, separates that knowledge into a versionable rule set (Crypistry), and then builds a scanner (Crypsy) on top. That separation is the right call for maintainability. The CBOM output and the head-to-head comparison with CBOMkit on the Go subset are also useful, and they use the same evaluator and ground truth, which is more than most tool comparisons do.\n\nThe evaluation is careful in several places. The authors distinguish by-design counting differences from true detection gaps, report component-level results where recall is much higher, and their limitations section is candid about missing Java, C/C++, and Python, and about the partial manual ground truth. The real-world scan of 57k files in under six minutes is a nice demonstration.\n\nThe soft spots are real but not disqualifying. The synthetic benchmark ground truth is a CBOM built from the same classification that generated the rules, so the F1 partly measures how consistent the scanner is with the authors' own taxonomy, not against an external standard. And the real-world '100% recall' only means they did not miss assets they already knew about; it cannot bound the number of missed assets, particularly in unsupported languages or files the scanner does not parse. The assessment numbers also need care: only about 30% of weakness annotations are security-actionable, which the paper admits, but the abstract's 'correctly annotates 91%' could easily be misread.\n\nNone of this undermines the central mechanism. For Ruby, Go, and config files, the approach works, and the paper says so with appropriate caveats in the conclusion. What is missing is independent validation on a corpus the authors did not construct, or at least expanded language coverage.\n\nI would send this to peer review. With a tightened abstract and an evaluation against an externally grounded corpus, this could become a standard baseline for crypto inventory tooling.","headline":"Useful taxonomy and scanner-independent rule repository for crypto-asset discovery, with an evaluation that is honest but weaker than the headline F1 suggests.","tokens_in":16125,"tokens_out":3015,"would_cite":true,"duration_ms":31666,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Rule-based static analysis grounded in a three-way asset taxonomy can provide practical cryptographic transparency for deployed software systems.","keywords":["cryptographic transparency","cryptographic asset discovery","static analysis","CBOM","CWE","CVE","post-quantum cryptography","PQC migration"],"falsifier":"Run Crypsy on a set of Java, C/C++, or Python services whose cryptographic assets have been enumerated by an independent, exhaustive ground-truth process; if the scanner's recall falls well below the 0.84–0.96 range reported on the synthetic benchmark, or if it misses large classes of artifacts or material present in those ecosystems, the paper's transparency claim would be refuted for those ecosystems.","tokens_in":15235,"feed_emoji":"🔐","tokens_out":9082,"duration_ms":76470,"temperature":0.7,"pith_summary":"Organizations rarely know what cryptography their software actually contains, which blocks security governance and post-quantum migration planning. This paper claims that a static scanner driven by a structured classification of cryptographic assets can give them that view automatically. The classification separates crypto material (keys, seeds, tokens), crypto artifacts (certificates, keystores), and crypto invocations (API calls, config directives, dependencies), with assessment split into weakness and vulnerability annotations. On a synthetic benchmark the scanner achieves F1 0.75, and on ten real services it processes 57,610 files in under six minutes, finding 370 cryptographic components, six CVE-linked vulnerabilities, and 52 post-quantum migration candidates. If the claim holds, inventories that used to require manual surveys become a repeatable, evidence-linked byproduct of the build.","feed_headline":"Rule-driven scanner maps a codebase's cryptography into a CBOM","feed_subtitle":"New classification of keys, artifacts, and invocations uncovers 370 assets and flags PQC migration candidates.","key_machinery":"The central object is the three-way discovery classification (Crypto-Material, Crypto-Artifacts, Crypto-Invocations) paired with the two-way assessment classification (Weakness, Vulnerability). Its load-bearing role is to make rule design uniform: every discovery rule in Crypistry maps to exactly one asset category, and every assessment rule operates on normalized CryptoComponent objects rather than raw files. This uniformity, plus the separation of the rule repository from the scanner pipeline, is what lets Crypsy recognize new file formats and library APIs by adding rules instead of changing scanner code, and it is what allows the same classification to drive both occurrence-level discovery and CWE/CVE-based assessment.","core_discovery":"The central claim, as stated in Section 9, is that rule-based static discovery grounded in a structured classification can provide practical cryptographic transparency for deployed software systems. The classification defines three discovery categories—value-centric Crypto-Material (private keys, tokens, seeds, nonces), structure-centric Crypto-Artifacts (X.509 certificates, keystores, signed manifests), and context-centric Crypto-Invocations (source-level API calls, TLS/SSH configuration directives, dependency declarations)—plus two assessment categories, weakness (CWE-mapped) and vulnerability (CVE-mapped). From this classification the paper derives Crypistry, a scanner-independent repository of 148 discovery and 66 assessment rules, and Crypsy, a four-phase scanner (rule loading, discovery, assessment, CBOM export) that produces CycloneDX v1.7 CBOM output. The supporting evidence is an occurrence-level F1 of 0.75 on the Cryben synthetic benchmark (0.86 when by-design counting differences are excluded, 0.95 at component granularity), 91% assessment recall, and a real-world scan of ten services that discovered 370 components, including six CVE-linked vulnerabilities and 52 post-quantum migration candidates. The paper is careful to scope the real-world recall as agreement with a manually compiled partial reference list rather than exhaustive coverage.","pith_inferences":["The classification itself is language-agnostic, so the same rule-repository procedure could be applied to Java, C/C++, and Python; the empirical claim, however, would need re-validation in those ecosystems before 'practical transparency' can be generalized beyond Ruby and Go.","Because 43 of the 67 synthetic-benchmark false negatives are by-design counting differences, a component-level or CBOM-level scoring convention may be the fairer metric for inventory tools than occurrence-level F1.","The roughly 0.3 actionable-weakness precision suggests a natural extension: add lightweight taint or dataflow tracking so weakness annotations like weak-hash or weak-PRNG only fire when the result is actually used in a security-sensitive context.","A stronger test than the manual partial reference list would be to compare Crypsy's CBOM against an inventory built by multiple independent analyzers, including binary-level and dynamic tools, to measure true coverage in the presence of unparseable or unsupported files."],"forward_implications":["Organizations could generate a CycloneDX CBOM automatically from source, configuration, and dependency files, replacing ad hoc manual surveys with a repeatable, evidence-linked inventory.","Post-quantum migration planning can begin from a concrete candidate list (the 52 RSA/ECC keys, certificates, and TLS configurations found in the evaluation) instead of a blank sheet.","Because detection and assessment knowledge live in a versioned rule repository, adding a new library, language, or CVE/CWE entry can be done without rewriting scanner pipeline logic.","Weakness and vulnerability annotations arrive pre-mapped to CWE and CVE identifiers, so findings can plug directly into governance and vulnerability-management workflows."],"supporting_citations":[{"why":"The existing CBOM generation tool used as the head-to-head baseline in the synthetic benchmark comparison.","marker":"[4]"},{"why":"Documents how scattered and diverse cryptographic artifacts are in software, motivating a classification-driven discovery scope.","marker":"[8]"},{"why":"A prior CBOM approach that reports invocation-level evidence but does not parse artifacts or extract key material, defining the gap this paper fills.","marker":"[14]"},{"why":"Systematic review showing many detection tools couple detection and assessment knowledge to scanner logic, motivating the scanner-independent rule repository.","marker":"[18]"},{"why":"Literature review showing post-quantum migration depends on reliable knowledge of existing cryptographic assets, the practical need behind the transparency claim.","marker":"[22]"},{"why":"Benchmark showing advanced misuse detectors still miss relevant finding classes, justifying a discovery-first rather than misuse-only approach.","marker":"[31]"}],"fun_headline_variants":["Scanner maps codebase crypto into CBOM, flags PQC candidates","New scanner finds 370 crypto assets, 52 PQC candidates","Rule-based crypto scanner hits 0.75 F1, flags PQC risks","Static crypto discovery yields CBOM with PQC migration leads","Crypto asset scanner produces CBOM, flags PQC"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The real-world evaluation supports the transparency claim only if the manually compiled partial reference list is representative of the cryptography actually deployed in the ten services; the paper itself reads its 100% partial recall as 'no manually identified asset was missed,' not complete coverage, and the claim is explicitly scoped to Ruby, Go, and the covered configuration formats.","fun_headline_variants_meta":{"raw":{"variants":["Scanner maps codebase crypto into CBOM, flags PQC candidates","New scanner finds 370 crypto assets, 52 PQC candidates","Rule-based crypto scanner hits 0.75 F1, flags PQC risks","Static crypto discovery yields CBOM with PQC migration leads","Crypto asset scanner produces CBOM, flags PQC"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000753,"raw_usage":{"total_tokens":3400,"prompt_tokens":1044,"completion_tokens":2356,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":660,"completion_tokens_details":{"reasoning_tokens":2266}},"tokens_in":660,"tokens_out":2356,"duration_ms":16967,"temperature":1.0,"reasoning_tokens":2266,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T14:48:15.041549+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Crypsy on a set of Java, C/C++, or Python services whose cryptographic assets have been enumerated by an independent, exhaustive ground-truth process; if the scanner's recall falls well below the 0.84–0.96 range reported on the synthetic benchmark, or if it misses large classes of artifacts or material present in those ecosystems, the paper's transparency claim would be refuted for those ecosystems.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The existing CBOM generation tool used as the head-to-head baseline in the synthetic benchmark comparison."},{"cited_title":"In: 2024 Annual Computer Security Applica- tions Conference","cited_arxiv_id":null,"evidence_quote":"Documents how scattered and diverse cryptographic artifacts are in software, motivating a classification-driven discovery scope."},{"cited_title":"Master’s thesis, University of Jyväskylä (2024),https://urn","cited_arxiv_id":null,"evidence_quote":"A prior CBOM approach that reports invocation-level evidence but does not parse artifacts or extract key material, defining the gap this paper fills."},{"cited_title":"ACM Comput","cited_arxiv_id":null,"evidence_quote":"Systematic review showing many detection tools couple detection and assessment knowledge to scanner logic, motivating the scanner-independent rule repository."},{"cited_title":"aes.NewCipher(key)","cited_arxiv_id":null,"evidence_quote":"Benchmark showing advanced misuse detectors still miss relevant finding classes, justifying a discovery-first rather than misuse-only approach."}],"review_version":1}