{"id":"a98af7f8-86bf-409f-9a22-5caa1e230698","arxiv_id":"2607.17242","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Across ~97,940 generated AIBOM artifacts, required fields are complete but AI-specific documentation (model cards, ethics, limitations, environment) is largely missing.","lead":"This paper uses a public tool to generate AI Bill of Materials (AIBOM) for about 98,000 Hugging Face models and measures how complete those documents are. It finds that the documents include basic structure but often miss details like limitations, safety risks, and dataset information.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's main inference—that missing AIBOM fields reflect missing repository documentation—is not validated: the OWASP generator's extraction logic is the sole instrument, and Table III's 0.00% fields may be structurally impossible for the tool to emit.","rationale":"The reader's weakest assumption is exactly the one I would stress. The paper studies AIBOMs generated by a single tool, but frames results as repository documentation completeness. The most suspicious signature is Table III: eleven fields at 0.00% and several categories with near-zero variance. A real corpus would be unlikely to produce exactly zero across 97,940 repositories for fields like intended use or ethical considerations unless either the tool never emits them or the model-card template has no such sections. Since the generator is open-source, a code audit can settle this cheaply. I do not recommend rejection: the paper is careful to qualify many statements as 'generated AIBOMs,' and the data release enables reuse. But the strongest headline finding—model-card documentation at ~19.5% and AI-specific fields absent in >90% of artifacts—is not trustworthy until the extraction mapping is validated. If the test shows high sensitivity, the finding would stand; if not, the paper should be reframed as a tool-behavior study. This supports CONDITIONAL, not UNCHANGED, because the condition is necessary for the central contribution.","tokens_in":17413,"tokens_out":4180,"duration_ms":40678,"concrete_test":"Take a stratified random sample of 300 repositories from the 97,940 (oversampling paper/dataset branches), independently annotate the rendered model card and HF metadata for the fields in Table III that are 0.00% or headline (intendedUse, ethicalConsiderations, technicalLimitations, safetyRiskAssessment, datasets, energyConsumption), then run the OWASP generator at a pinned commit on each repository and compute per-field false-negative rate (annotated present but AIBOM absent). Also inspect the generator's source for each field to determine whether any extraction path exists; a field with no code path is structurally untestable. If the false-negative rate exceeds ~20% for any headline field, or any 0.00% field has no extraction path, the field-absence percentages reflect tool behavior, not repository documentation.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim requires that a field absent from a generated AIBOM is absent from the HF repository. The paper never checks this. Section III-C makes the OWASP AIBOM Generator the sole measurement instrument, and Section VI admits 'limitations in the generator's extraction logic may influence which fields appear in AIBOMs.' The strongest result—model-card documentation at 19.51%, with intendedUse, ethicalConsiderations, modelExplainability, informationAboutTraining, informationAboutApplication, metric, SensitivePersonalInformation, energyQuantity, energyUnit, standardCompliance, and autonomyType all at exactly 0.00% in Table III—is consistent with the generator having no code path that maps free-text model-card content to those fields, rather than with 97,940 repositories genuinely lacking such documentation. The generator also inserts CycloneDX scaffolding (bomFormat, specVersion, serialNumber, version) and repository URLs (website, downloadLocation) independently of repository content, so the 100% 'required fields' and 99.99% 'website' results do not measure repository documentation. The only manual validation (Section III-E) checked that the authors' extraction pipeline matched generator score reports, not that the generator's fields matched repository content. Consequently, RQ1/RQ2's headline percentages cannot be interpreted as documentation-completeness measurements until generator extraction fidelity is established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper conducts a large-scale repository-mining study of Hugging Face model repositories to measure the completeness of AI Bills of Materials (AIBOMs) generated by the OWASP GenAI Security Project AIBOM Generator. The authors collect metadata for 2,942,466 HF models, retain 97,940 with more than 100 downloads, generate AIBOMs for each, and compute completeness scores using the generator's built-in scoring scheme. They report a mean completeness score of 54.31/100, with required CycloneDX fields at 100%, metadata at 40%, model-card documentation at 19.51%, and many AI-specific fields (e.g., intendedUse, ethicalConsiderations, modelExplainability, energyQuantity, standardCompliance) absent in all artifacts. They also analyze how coverage varies with paper references, dataset declarations, licenses, tasks, and model families. The paper concludes that generated AIBOMs are structurally valid but weak in AI-specific documentation, and it recommends improved model-card practices and AIBOM tooling.","tokens_in":17802,"tokens_out":4934,"duration_ms":48400,"significance":"If the measurements faithfully reflect the documentation content of HF repositories, this would be the largest empirical characterization of AIBOM completeness to date, with a public dataset and a reproducible pipeline. The paper's decomposition into score, category, field, and repository-characteristic levels is useful, and the emphasis on distinguishing syntactic presence from substantive content is important. The main strength is scale: 97,940 artifacts and 2.94M repository records. The central weakness is that the OWASP generator is the sole measurement instrument, and its extraction fidelity is never validated against actual repository content. This makes the headline percentages, and any conclusions about 'documentation gaps' in HF repositories, currently unsupported. The paper's significance is therefore conditional on a validation study that the current manuscript does not provide.","major_comments":[{"comment":"The central inference—that a field absent from a generated AIBOM is absent from the HF repository—is not established. The OWASP AIBOM Generator is the only field-extraction instrument; the manual check in §III-E only verifies that the authors' parser preserved the generator's score reports, not that the generator's fields match repository content. Table III's 0.00% fields (intendedUse, ethicalConsiderations, modelExplainability, informationAboutTraining, informationAboutApplication, metric, SensitivePersonalInformation, energyQuantity, energyUnit, standardCompliance, autonomyType) are consistent with the generator having no code path that maps model-card text to these fields, rather than with 97,940 repositories genuinely lacking such information. Conversely, the 100% 'required fields' and 'website'/'downloadLocation' are CycloneDX scaffolding and repository URLs, not documentation conte","section":"§III-C, §III-D, §IV-B, Table III, §VI"},{"comment":"The 'meaningfulDescription' diagnostic is an ad hoc, unvalidated heuristic. The paper reports that only 211/97,940 (0.22%) artifacts have a meaningful description, and this result is load-bearing for the claim that field presence overestimates documentation quality (Finding 5). However, the rule itself is never defined: the paper does not state what counts as 'meaningful,' what thresholds were used, or whether the heuristic was validated on a labeled sample. Without this information, the 0.22% number is not reproducible and cannot be interpreted. The authors should either fully specify and validate the heuristic, or remove it and avoid drawing conclusions based on it.","section":"§IV-B, Table III, Finding 5"},{"comment":"The study is not reproducible as reported. The paper does not specify the version or commit of the OWASP AIBOM Generator, the exact invocation parameters, or the generation dates (only that the tool page was accessed on 2026-06-25). The scoring formula in Eq. (1) is also ambiguous: 'FinalScore = Subtotal × P' where P is called a 'penalty factor,' but a penalty factor would normally reduce the score; the text describes penalties as subtracting 10%, 20%, or 5%, so P presumably is (1 - penalty) or similar, but this is not defined. Provide the exact tool version, command-line invocation, generation timestamps, and a precise definition of P and how critical/important field penalties are applied.","section":"§III-D, §III-E, Eq. (1)"},{"comment":"RQ3's hierarchical audit tables use aggregate columns 'Resp./Safe.' and 'Env.' without defining which AIBOM fields compose them. For example, Table IV reports 'responsible/safety coverage' percentages but never states whether this combines safetyRiskAssessment, ethicalConsiderations, intendedUse, or other fields, nor how the union is computed. The 'Env.' column is likewise undefined. Without these definitions, the RQ3 results cannot be audited or compared with the field-level results in Table III. Define the exact field sets and, ideally, report per-field coverage for the aggregates.","section":"§IV-C, Fig. 2, Tables IV–V"}],"minor_comments":[{"comment":"The abstract says 'approximately 97.5K AIBOM artifacts' while the body uses 97,940. Use a consistent number throughout.","section":"Abstract and §IV-B"},{"comment":"The formula labels 'Subtotal' as the sum of category scores and 'FinalScore' as 'Subtotal × P'. If P includes penalties, the notation should be clarified (e.g., P = 1 - penalty_rate) so readers can reproduce score calculations.","section":"Eq. (1)"},{"comment":"The row 'SensitivePersonalInformation' appears in the Model Card section, but the text in §IV-B refers to 'useSensitivePersonalInformation'. Unify the field naming to match the generated AIBOM schema.","section":"Table III"},{"comment":"Figure 1 is referenced in the introduction but not described in the text; its content is vague. Add a caption or a short explanation of what information is being transformed and how the completeness evaluation works.","section":"Fig. 1"},{"comment":"Several references contain extraneous spaces inside URLs (e.g., ref [17] 'OW ASP-AIBOM-Generator' and ref [23]). These should be fixed to point to the correct resources.","section":"References"},{"comment":"The near-zero Pearson correlation with metadata (0.003) is expected because metadata is almost constant (40.00%). The text should note this low-variance caveat rather than interpreting the correlation alone.","section":"§III-F"}],"recommendation":"major_revision","confidential_remarks":"The paper has an impressive scale and a valuable public dataset, but the lack of validation of the OWASP generator's extraction fidelity is a load-bearing issue. The current manuscript cannot support claims about 'repository documentation gaps' because the instrument's behavior is confounded with repository content. This is fixable within the scope of the paper: a validation study comparing generated fields against manual inspection of even a few hundred repositories, plus clear reframing of claims as being about generated AIBOMs, would substantially improve the paper. With that addition, the paper could become a solid empirical contribution to the AI supply-chain literature."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, it is the largest AIBOM completeness measurement I have seen: 97,940 generated artifacts from Hugging Face, with a field-level breakdown that goes well beyond ALOHA's 312-model sample. The scale and the granular Table III are genuinely new and useful for anyone working on AI supply-chain tooling. Second, the central interpretation is not yet supported. The paper uses the OWASP AIBOM Generator as the sole instrument and then reads missing fields as missing repository documentation. That inference is load-bearing and it is never validated.\n\nWhat the paper does well: the data collection is careful, the pipeline is described in enough detail to reproduce, the filtering is transparent, and the descriptive statistics (54.31 mean, 19.51% model-card coverage, the near-zero AI-specific fields) are likely accurate as tool outputs. The manual check on 100 artifacts verified the extraction pipeline against generator reports, and the threats-to-validity section is unusually honest about the generator's extraction logic. The paper also wisely refuses to claim causality in RQ3, framing the associations as associations. Credit where due: this is a solid descriptive study of what the OWASP tool currently produces.\n\nThe soft spots are real and, in one place, severe. The severe one: fields like intendedUse, ethicalConsiderations, modelExplainability, and autonomyType appear at exactly 0.00% across 97,940 artifacts. That uniformity is far more consistent with the generator having no code path for those fields than with 97,940 repositories all independently lacking that documentation. The paper's own threat-to-validity statement admits the generator's extraction logic may influence which fields appear. That admission is buried in Section VI, but it undermines the RQ1/RQ2 headline percentages as documentation-completeness measurements.\n\nThe other soft spots are minor by comparison: the >100-download filter and the 20/20/20/30/10 weights come from outside the paper and shape the results, and the \"meaningful description\" threshold is only briefly described. None of these are fatal on their own.\n\nWho this is for: researchers working on AIBOM tooling, model-card standards, and HF repository mining. The field-level availability table is a useful reference even if the interpretation changes. I would send this to peer review, but with a required revision: either validate the generator against ground-truth repository content on a sample, or reframe the conclusions to be strictly about tool-generated AIBOMs. The paper deserves a serious referee, not a desk reject.\n\nRecommendation: engage with it, but only after the authors do the validation work or narrow their claims.","headline":"A genuinely large empirical baseline for AIBOM completeness, but the headline numbers are tool-output measurements, not repository-documentation measurements, so the paper needs a validation step before the conclusions can be trusted.","tokens_in":18230,"tokens_out":1217,"would_cite":true,"duration_ms":10296,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Across 97,940 generated AIBOMs from Hugging Face model repositories, the mean completeness score is 54.31/100, and model-card documentation covers only 19.51% of fields.","keywords":["AIBOM completeness","Hugging Face","machine learning supply chain","model documentation","model cards","repository mining","CycloneDX","AI governance"],"falsifier":"Take a random sample of, say, 200 of the repositories whose generated AIBOMs report zero for 'intended use' and read the model-card text directly; if a substantial share contain an intended-use statement that the generator failed to map, the claimed documentation gaps are partly an artifact of the extraction tool rather than of the repositories.","tokens_in":17341,"feed_emoji":"📋","tokens_out":6581,"duration_ms":56615,"temperature":0.7,"pith_summary":"The paper tries to establish that AIBOMs — machine-readable bills of materials for AI models — generated at scale from public Hugging Face repositories are structurally complete but substantively weak: every required CycloneDX field appears in all artifacts, yet the average completeness score is 54.31/100, dragged down by metadata at 40% and model-card documentation at 19.51%. It further shows that AI-specific transparency fields — intended use, ethical considerations, safety-risk assessment, environmental impact — are absent in over 90% of artifacts, and that field presence can mask placeholder content: only 0.22% of descriptions are meaningful. The paper argues this matters because AIBOMs are proposed as the traceability and governance layer of the AI supply chain; if they are complete in form but empty in content, downstream audits, reuse decisions, and compliance checks may rest on a false sense of transparency.","feed_headline":"AI model bills of materials average just 54 of 100 in completeness","feed_subtitle":"Required fields are always present, but model-card and responsible-use fields are missing in most artifacts.","key_machinery":"The central mechanism is an open-source AIBOM generator that turns a model repository's public metadata, model-card text, and configuration files into a CycloneDX-formatted JSON artifact, plus the tool's weighted completeness scoring scheme. The score sums five category scores — required fields (20), metadata (20), component-basic information (20), model-card documentation (30), external references (10) — and applies a penalty factor when critical or important fields are missing. This identity converts repository documentation into a single comparable number between 0 and 100, which is what allows the study to measure completeness at scale; the accompanying field-level parser reveals which s","core_discovery":"Generated AIBOMs from nearly 98,000 Hugging Face models achieve a mean completeness score of 54.31/100 with a tight distribution (sd 4.44, range 46.6–68.7). Required CycloneDX fields (bomFormat, specVersion, serialNumber, version) and core component-identification fields are present in 100% of artifacts, and licensing appears in 73.14%. The main deficits are concentrated in metadata (40%, almost constant) and model-card documentation (19.51%), with several responsible-use fields — ethical considerations, intended use, model explainability, metrics — appearing in 0% of artifacts and environmental fields nearly absent. Paper references appear in only 13.50% of artifacts. The paper also shows f","pith_inferences":["Because field-absence counts come solely from the generator's extraction logic, they may overstate documentation gaps: model-card text is free-form, and a field the tool cannot map may still exist in the repository. A manual reading of a sample of model cards would quantify this tool effect.","The near-total absence of environmental fields (0.02% of artifacts) suggests that if model providers adopt standardized energy reporting, this category alone could lift overall scores noticeably — a measurable prediction for a future snapshot of the same ecosystem.","The 0.888 correlation between model-card coverage and overall completeness implies that aligning AIBOM field definitions with the current Hugging Face model-card template would likely raise scores more than any other single change, a hypothesis an AIBOM tool developer could test directly.","The study's 100-download filter means the completeness distribution reflects only reused models; the long tail of low-download repositories is plausibly even less documented, which matters if governance applies to all public models rather than only popular ones."],"forward_implications":["A structurally valid AIBOM does not imply substantive transparency; users must inspect category- and field-level detail before relying on an artifact for audits or compliance.","AIBOM completeness is driven primarily by model-card documentation and external references (correlations 0.888 and 0.662), so improving model-card practices is the highest-leverage fix for completeness scores.","Repositories that provide paper references and dataset declarations show markedly stronger traceability and limitation coverage (e.g., VCS references appear in ~99% of paper+dataset models vs. ~55% without), making these signals useful markers of documentation readiness.","Even the highest-scoring artifacts, concentrated among well-known LLM families and their quantized GGUF variants, reach only 68.7/100, so the most documented public models remain only moderately complete.","Because only 0.22% of artifacts contain a meaningful description, AIBOM tooling should add semantic quality checks rather than equating field presence with documentation quality."],"fun_headline_variants":["AI model docs score 54/100 on completeness","Hugging Face models' AIBOMs hit only 54/100","Model AIBOM completeness averages just 54","Required fields yes, but AIBOM cards missing in HF models","AIBOMs for HF models: 54/100, cards scarce"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The generator's field-extraction logic faithfully captures what each Hugging Face repository actually documents — an assumption the paper itself flags in its threats-to-validity section as a possible influence on which fields appear.","fun_headline_variants_meta":{"raw":{"variants":["AI model docs score 54/100 on completeness","Hugging Face models' AIBOMs hit only 54/100","Model AIBOM completeness averages just 54","Required fields yes, but AIBOM cards missing in HF models","AIBOMs for HF models: 54/100, cards scarce"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000777,"raw_usage":{"total_tokens":3326,"prompt_tokens":849,"completion_tokens":2477,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":593,"completion_tokens_details":{"reasoning_tokens":2388}},"tokens_in":593,"tokens_out":2477,"duration_ms":16290,"temperature":1.0,"reasoning_tokens":2388,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T18:36:11.785312+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of, say, 200 of the repositories whose generated AIBOMs report zero for 'intended use' and read the model-card text directly; if a substantial share contain an intended-use statement that the generator failed to map, the claimed documentation gaps are partly an artifact of the extraction tool rather than of the repositories.","supporting_citations":[],"review_version":1}