{"id":"e119c96a-a96e-4e1c-abd2-f2ec74a6677b","arxiv_id":"2512.21781","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"CycloneDX tools and projects using them show higher GitHub activity metrics, while SPDX has a larger and older tool ecosystem.","lead":"This paper maps and compares the open-source and proprietary tool ecosystems for the two main software-bill-of-materials formats, SPDX and CycloneDX, using GitHub metrics, issue reports, and project adoption patterns. It finds CycloneDX communities more active while SPDX has a larger tool base, giving SBOM adopters a practical, though imperfect, comparison.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RQ4's top-250 comparison compares top-tier CycloneDX projects against nearly the entire detected SPDX pool, so the reported activity gap may reflect search coverage and selection asymmetry rather than format-driven differences.","rationale":"The reader's weakest_assumption identifies exactly the same threat: the RQ4 comparison rests on unequal pools, unequal search-pattern counts, and unverified CI matching. My stress-test confirms this is the most load-bearing concern because it directly undermines the paper's strongest claim. The recommended concrete test is feasible: the replication package is available, and expanding the SPDX search patterns is a bounded task. Since the reader already assigned CONDITIONAL based on this and related issues, my assessment does not change the verdict; it reinforces the need for re-analysis before the claim can be accepted at face value. I did not find a more serious internal inconsistency; the paper is transparent about its methodology and limitations, and the RQ1-RQ3 findings are largely defensible despite minor statistical issues. The concern is not about fraud or intent; it is about selection bias in the observational design.","tokens_in":35670,"tokens_out":2071,"duration_ms":24031,"concrete_test":"Expand the SPDX CI-pattern set to approximate parity with CycloneDX (including the five dual-format tools and all SPDX build tools listed in Table 10, with equivalent snippet variants), re-run the GitHub search, and redo the RQ4 analysis in two ways: (a) compare the full detected pools rather than the truncated top-250, and (b) after constructing a matched sample of SPDX and CycloneDX projects on age, stars, language, and star-decile before comparing health metrics. If the significant differences persist across all metrics in both the full-pool and matched analyses, the finding is robust; if they shrink or vanish, the claimed format-driven activity advantage is a coverage artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central RQ4 claim (Section 4.4.1) is that projects adopting CycloneDX are more active and mature than SPDX-adopting projects. The evidence rests on a head-to-head comparison of the top 250 most-starred projects from each pool, but the pools are not comparable. Section 3.5 reports 307 SPDX-only projects versus 1,087 CycloneDX-only projects, discovered using 11 SPDX CI patterns versus 50 CycloneDX patterns. Selecting the top 250 by stars from the SPDX pool keeps 81% of all detected SPDX projects, while the same selection keeps only 23% of the CycloneDX pool. Thus the comparison contrasts the best quarter of the CycloneDX population with almost the entire SPDX population. If the SPDX pattern set under-detects smaller or less active SPDX-using projects, the SPDX pool is biased toward large, visible repositories; the observed gap in stars, forks, and commits could then be an artifact of detection coverage rather than a property of the format. The paper itself acknowledges in Section 6.1 that the CI snippets were not executed and SBOM generation was not verified, so snippet presence is a proxy that may be uneven across the two tool ecosystems. Because the load-bearing comparison is structurally confounded by search-pattern imbalance and pool truncation, the RQ4 headline finding is not robust as stated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a comparative empirical study of the SPDX and CycloneDX SBOM tool ecosystems. It manually classifies 170 tools from the official tool centers, compares CHAOSS-based health metrics for 171 CycloneDX and 470 SPDX tool repositories, analyzes 36,990 issue reports, and identifies 1,394 GitHub projects that appear to use SBOM tools in CI pipelines. The headline claim, stated in Section 4.4.1, is that projects adopting CycloneDX tools are more active and mature than those using SPDX tools, while SPDX retains a larger and more mature tool ecosystem. The paper is mostly descriptive and includes a replication package.","tokens_in":35996,"tokens_out":6101,"duration_ms":63265,"significance":"If the findings were robust, this would be a useful reference for SBOM practitioners and tool vendors. The strengths include a manually curated tool inventory, use of CHAOSS metrics, a substantial issue-report analysis, and a publicly available replication kit. The paper does not rely on fitted parameters or circular derivations. However, the central RQ4 claim that CycloneDX-using projects are more active is not currently supported because of a structural selection/detection confound: the top-250 sets are drawn from pools of very different sizes and were identified with very different numbers of search patterns. This concern is load-bearing for the abstract and the conclusions.","major_comments":[{"comment":"The RQ4 comparison is confounded by selection and detection asymmetry. The paper compares the top-250 most-starred projects from a pool of 307 SPDX-using projects with the top-250 from a pool of 1,087 CycloneDX-using projects, where the pools were found using 11 SPDX CI patterns versus 50 CycloneDX patterns. Keeping the top 250 from the SPDX pool retains 81% of all detected SPDX projects, while the same selection retains only 23% of the CycloneDX pool. Even under identical underlying distributions, the CycloneDX top-250 will be drawn from a much higher quantile than the SPDX top-250, so the Mann-Whitney U tests in Table 8 do not test a well-defined population difference. The acknowledgment in Section 6.1 that CI snippets were not executed and SBOM generation was not verified further weakens the identification of actual SBOM usage. To support the headline claim, the authors need to either","section":"§3.5, §4.4.1, Table 8"},{"comment":"The chi-square analysis of use-case support is ambiguous about whether dual-format tools are counted as independent observations in both the SPDX and CycloneDX groups. Table 1 appears to list percentages for all tools supporting each format, while Figure 3 and Table 2 treat dual-format tools as a separate category. If the chi-square test used the Table 1 columns, then dual-format tools are double-counted and the independence assumption is violated, invalidating the p-value and Cramér's V. The manuscript should state the exact contingency table used and, if necessary, re-run the analysis on exclusive categories (SPDX-only, CycloneDX-only, dual-format).","section":"§4.1.1, Table 1, Table 2, Figure 3"}],"minor_comments":[{"comment":"Figure 1 says 'Top-200 projects' but the text consistently says 'Top-250'. Please reconcile.","section":"Figure 1 vs. §3.5/§4.4"},{"comment":"The p-values are formatted ambiguously, e.g., '1.00 −10' and '3.79 −12'. These should be written in standard scientific notation, and values below 0.001 should be reported as '<0.001' rather than '0.000'.","section":"Tables 4 and 8"},{"comment":"'format deviation' should be 'standard deviation' throughout this section.","section":"§4.3.1"},{"comment":"The sentence 'SPDX tools have most (43.12%) of the issue reports related to Feature Development and Enhancement and Code Components' is confusing; 43.12% refers only to Feature Development and Enhancement, while Code Components is 21.57%.","section":"§4.3.3, Figure 12"},{"comment":"The claim sentence lists 'commits' twice in the same enumeration ('stars, watchers, forks, pull requests, releases, contributors, and commits'). Please remove the duplication.","section":"§4.4.1"},{"comment":"The description of the random sample from the CycloneDX tool center (140 of 219) should clarify whether the 170 distinct tools used in the analysis are the same as the sampled 187 after duplicate removal, and how the sampling weights are handled in the reported percentages.","section":"§3.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is a substantial empirical study with a useful replication package, but the central RQ4 comparison is currently not robust. I would ask for a re-analysis that controls for pool-size and detection-pattern asymmetry, or a reframing of the claim as descriptive of the detected subsets. The RQ1 chi-square ambiguity should also be clarified."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The useful thing here is the scale and transparency of the mapping. The manual classification of 170 tools against the NTIA taxonomy, the ecosystem-level health metrics across 641 repositories, and the issue-label analysis across ~34k reports are all real work, and the replication package makes it checkable. RQ1's finding that proprietary tools skew dual-format while OSS tools skew single-format is a clean, credible result, and the chi-square diagnostics there are fine. RQ2 and RQ3 are also defensible: normalized metrics, Mann-Whitney with Bonferroni, effect sizes reported, and the issue-category hand-coding is documented with Kappa agreement and example issues. The paper is honest about snapshot limits and about not executing the CI snippets.\n\nWhere I part company with the authors is RQ4. The claim that CycloneDX-adopting projects are more active and mature rests on comparing the top 250 most-starred projects from pools of 307 SPDX-only and 1,087 CycloneDX-only projects, found via 11 SPDX vs 50 CycloneDX search patterns. That means the SPDX set is almost the entire detected population while the CycloneDX set is the top quarter. If the SPDX pattern set under-detects smaller projects, the comparison is not format-driven; it is coverage-driven. The paper itself concedes the CI snippets were not executed, so snippet presence is a noisy proxy. This is a load-bearing flaw in the headline claim, but it is repairable: matched sampling on stars/language/age, or at minimum a sensitivity analysis that compares the top N for several N and reports how pool truncation changes the gap. The language-preference analysis (Go/Python with CycloneDX, Java/C# with SPDX) is suggestive but inherits the same detection bias.\n\nThe minor issues are minor: the RQ1 chi-square double-counts dual-format tools across both groups, which inflates the test's significance, though the reported Cramér's V and residuals are still informative as descriptive associations. The abstract's phrase about industry adoption goes beyond what GitHub data can support; the threats section says as much.\n\nBottom line: for a reader who wants a descriptive map of SBOM tooling gaps, RQ1–RQ3 are worth the time, and the dataset is a useful starting point for future work. The RQ4 conclusion should not be cited as established until the pools are matched. I would send it to peer review, but with a clear request to redo RQ4's comparison and to soften the abstract. It deserves a serious referee, not a desk reject.","headline":"A genuinely useful, large empirical map of the SPDX/CycloneDX tool ecosystems, with RQ1–RQ3 mostly solid and RQ4's headline comparison confounded by unbalanced search pools.","tokens_in":36468,"tokens_out":649,"would_cite":true,"duration_ms":10617,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GitHub projects that adopt CycloneDX show stronger community activity than those using SPDX.","keywords":["SBOM","SPDX","CycloneDX","software supply chain security","open-source ecosystem health","GitHub mining","CI/CD adoption","tool use-case taxonomy"],"falsifier":"Execute a random sample of the 1,394 identified CI pipelines, confirm that SBOM files are actually generated, and re-run the community-health comparison on verified adopters only. If the CycloneDX advantage shrinks or reverses under verification—or if the same health gap appears when the two sets are matched for project age and language—the RQ4 claim is falsified.","tokens_in":1180,"feed_emoji":"🧾","tokens_out":1359,"duration_ms":60032,"temperature":0.7,"pith_summary":"The paper tries to establish that the two dominant SBOM formats, SPDX and CycloneDX, are best understood as competing tool ecosystems, not just competing specifications. Its central finding is that open-source projects using CycloneDX tools show significantly higher GitHub activity—more stars, forks, watchers, commits, pull requests, releases, and contributors—while SPDX retains a larger, more established tool ecosystem and broader industry adoption. The study also maps which use cases each ecosystem supports, showing that CycloneDX tools skew toward build-time generation and integration, SPDX tools toward diffing and translation, and proprietary tools toward broader coverage than open-source tools. This matters because SBOM adoption is now policy-driven, and the paper gives practitioners evidence that format choice should include ecosystem health, not just the specification.","feed_headline":"CycloneDX projects outpace SPDX in GitHub activity","feed_subtitle":"Across 1,394 SBOM-using repos, CycloneDX adopters show higher stars and commits; SPDX keeps a larger tool ecosystem.","key_machinery":"Three components carry the argument. Use-case coverage is measured by manually classifying 170 tools against the official SBOM tool-use-case taxonomy (produce, consume, transform). Ecosystem health is measured with community-health metrics—stars, forks, watchers, pull requests, releases, contributors, commits—normalized by repository age and compared via Mann-Whitney U tests with Cliff's delta effect sizes. Adoption is measured by mining CI configuration snippets from tool READMEs and searching GitHub to find 1,394 projects that invoke an SBOM tool in their build pipeline; those projects are then compared as top-250 sets per format.","core_discovery":"Analyzing 470 SPDX and 171 CycloneDX open-source tool repositories, plus 36,990 issue reports and the top 250 most-starred GitHub projects using each format, the paper finds a consistent pattern: CycloneDX-using projects have higher normalized medians on every community-health metric, with large effect sizes, and CycloneDX tools resolve issues faster on average (88 vs 130 days). SPDX counters with a larger tool population (470 vs 171), a head start since 2011, faster resolution of licensing issues, and stronger association with translation and diffing use cases. The paper's conclusion is that the ecosystems are complementary rather than one being superior: SPDX is the mature, broad incumbent","pith_inferences":["If the engagement gap is real, it may not be caused by the format itself; CycloneDX tools tend to be promoted more heavily on their official tool center and are more numerous in the search pool, so the top-250 comparison may partly reflect discoverability and marketing rather than intrinsic quality.","A natural test is to run the CI SBOM-generation steps on a sample of the 1,394 projects and verify that SBOM files are actually produced; the paper acknowledges it did not execute projects, so a validation study could confirm or weaken the adoption findings.","The language split (Go/Python toward CycloneDX, Java/C# toward SPDX) suggests that format choice tracks the surrounding dev toolchain; if SPDX 3.0 expands beyond licensing, the activity gap may narrow as enterprise and cloud-native communities converge.","Future studies could use projects that generate both formats' SBOMs as matched controls, isolating format effects from project popularity."],"forward_implications":["SBOM adopters should evaluate the health of a format's tooling, not just the specification, before choosing; a format with fewer tools can still have more active maintainers.","Open-source tool developers have a clear gap to fill: build-time SBOM generation and integration support are well covered, but analysis and visualization features lag proprietary tools.","CycloneDX-using projects' higher engagement suggests that security-focused, modern cloud-native projects (Go, Python) are driving one ecosystem, while enterprise compliance stacks (Java, C#) drive the other.","Issue-resolution data points to concrete priorities: licensing issues are resolved faster by SPDX tools, while bug-fix and feature-development issues are resolved faster by CycloneDX tools.","Both formats have maintained strengths: CycloneDX resolves most issue categories faster, while SPDX resolves licensing issues 51.85% faster."],"fun_headline_variants":["CycloneDX shows stronger GitHub engagement than SPDX","SPDX ecosystem larger, CycloneDX more active","SBOM tools: SPDX mature, CycloneDX engaged","CycloneDX resolves issues faster than SPDX"],"cache_read_input_tokens":37888,"weakest_assumption_plain":"The central claim depends on CI snippet matching correctly identifying real SBOM tool use and on top-250 comparisons from unbalanced pools (307 SPDX vs 1,087 CycloneDX projects) reflecting format effects rather than tool availability or search coverage.","fun_headline_variants_meta":{"raw":{"variants":["CycloneDX shows stronger GitHub engagement than SPDX","SPDX ecosystem larger, CycloneDX more active","SBOM tools: SPDX mature, CycloneDX engaged","CycloneDX resolves issues faster than SPDX"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000432,"raw_usage":{"total_tokens":2013,"prompt_tokens":691,"completion_tokens":1322,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":435,"completion_tokens_details":{"reasoning_tokens":1258}},"tokens_in":435,"tokens_out":1322,"duration_ms":10417,"temperature":1.0,"reasoning_tokens":1258,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T13:59:32.434015+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Execute a random sample of the 1,394 identified CI pipelines, confirm that SBOM files are actually generated, and re-run the community-health comparison on verified adopters only. If the CycloneDX advantage shrinks or reverses under verification—or if the same health gap appears when the two sets are matched for project age and language—the RQ4 claim is falsified.","supporting_citations":[],"review_version":1}