{"id":"a1ec76a3-1caf-41e1-adf9-fdd6662659b2","arxiv_id":"2501.02170","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Most safetensors adoptions on Hugging Face came from automated conversions, but most conversion-tool pull requests remain unmerged, and developer discussions show both enthusiasm and practical errors.","lead":"This paper measures how and when developers adopted safetensors, a safer model serialization format, on Hugging Face, using commit histories and pull request records. It finds most adoptions came from an automated conversion tool, while most of the tool's pull requests were left unmerged.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section III-C's filter excludes every repository created after safetensors' release, so the 95.7% automation-share and RQ1/RQ2 trends describe only migration of pre-existing repos and cannot support the unqualified 'most adoptions up to date' claim.","rationale":"I agree with the reader's weakest assumption and think the conditional verdict is right. The central empirical claim is not just about whether old repos migrated; it is about how safetensors got adopted. Excluding every repo created after September 2022 removes exactly the population most likely to adopt safetensors natively, i.e., models born after the format existed. The resulting 95.7% figure conflates \"of the safetensors-adding commits we observed in 1,032 old repos, 95.7% were conversion-tool merges\" with \"most safetensors adoptions on Hugging Face were made by the bot.\" The paper's own RQ3 data show the conversion tool operated across 35,094 repositories, so the authors already have a much broader lens for PR behavior; applying the same lens to adoption commits in post-release repos is feasible. I do not see an internal inconsistency in the commit analysis or the serialization-format identification rules; the issue is external validity of the sample. A secondary concern, also raised by the reader, is that \"ignore\" in the abstract overinterprets an open PR status, since open can mean pending, abandoned, conflicting, or awaiting maintainer action; this is a wording and interpretation issue rather than a computational error. The concrete test above would settle the dataset-filter concern directly. If the post-release cohort shows substantial direct adoption, the verdict should remain conditional pending re-scoping; if it does not, the current conclusions could be accepted with tighter wording.","tokens_in":18452,"tokens_out":5621,"duration_ms":56593,"concrete_test":"Query the Hugging Face Hub API for model repositories with createdAt on or after 2022-09-23, lastModified in 2024, and at least one .safetensors file. Take a random sample (95% confidence, 5% margin of error), clone each repo, and locate the earliest commit that adds a .safetensors file. Classify that commit as either (a) a merge of a SFConvertbot/conversion-tool PR or (b) a direct author commit adding safetensors, using the commit message and the PR metadata already collected in Section III-F. Recompute the percentage of safetensors-adoption commits that are conversion-tool merges for this post-release cohort. If direct commits account for more than a small share (e.g., >20%), the 95.7% automation figure is an artifact of the Section III-C filter and the abstract's \"most adoptions\" claim must be re-scoped or revised.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The load-bearing step is the repository filter in Section III-C: only repos \"created before September 2022\" and \"last modified in 2024\" are analyzed. Because safetensors was released in September 2022, this excludes all model repositories created after the format existed. RQ1's 418 safetensors-adding commits and the 95.7% figure that most adoptions were via Hugging Face's conversion tool are therefore computed only over the pre-existing cohort; any repository whose first safetensors file was added at creation time by a new model is invisible. This is not a minor boundary condition: the Hub had 996,939 model repos at query time, the filter yielded 1,032, and a newer format is plausibly most likely to appear in newer repos. The paper's own RQ3 corpus (43,596 PRs across 35,094 repos) is not restricted by the same filter, so the data to test the comparison exists but is not used. The same filtered set underlies Figure 6 and the RQ2 trend. Consequently the central claim that \"most safetensor adoptions up to date were made via automated conversions\" is an unqualified conclusion about the whole ecosystem drawn from a migration-only sample. The reader's diagnosis is correct: the filter is load-bearing for RQ1, RQ2, and the abstract's headline.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates the adoption of safetensors on Hugging Face, combining a quantitative analysis of commit histories and conversion-tool pull requests with a qualitative open-coding analysis of developer discussions. The quantitative part filters Hugging Face model repositories to those created before September 2022 and last modified in 2024, yielding 1,032 repositories, and finds 418 commits adding safetensors files, 95.7% of which merge PRs created by Hugging Face's conversion tool. The same filtered set underlies the adoption-trend analysis (RQ2). The PR analysis (RQ3) reports that 83.5% of conversion-tool PRs remain open and only 13.9% are merged. The qualitative part (RQ4) codes 418 sampled posts/PRs/discussions and produces a taxonomy of developer perceptions, including loading difficulties, performance concerns, and security awareness. The paper concludes that safetensors is gaining traction but that most adoption is automated while most developers do not merge the conversion tool's PRs.","tokens_in":18660,"tokens_out":4791,"duration_ms":45600,"significance":"If the findings hold, the paper provides a useful empirical baseline for how a security-motivated serialization format spreads through a large ML model hub, and it makes a credible contribution to the software-engineering literature on ecosystem evolution and secure development practices. Strengths include a detailed data-collection pipeline, an explicit rule-based format-identification method inherited from prior validated work, a replication package, and inter-rater reliability (Cohen's kappa = 0.88) for the qualitative coding. The paper is also careful in several places, for example in distinguishing the conversion-tool PR set from the repository-commit set. However, the central quantitative claims are currently stated more strongly than the data support: the sample filter excludes all repositories created after safetensors was released, and the PR-status analysis conflates 'unmerged' with 'ignored'. Both issues are load-bearing for the abstract and for RQ1/RQ3 conclusions, so the contribution is significant but conditional on the authors either broadening the analysis or qualifying the claims.","major_comments":[{"comment":"The repository filter in Section III-C restricts the analysis to repositories created before September 2022 and last modified in 2024. Because safetensors was released in September 2022, this excludes every repository created after the format existed. The 418 safetensors-adding commits and the 95.7% automation share in Section IV-A therefore characterize only migration of pre-existing repositories, not 'most safetensor adoptions up to date' as stated in the RQ1 findings and the abstract. Repositories created after September 2022 that adopt safetensors at creation time are invisible to this analysis, and a newer format is plausibly more likely to appear in newer repositories. Since the RQ3 corpus of 43,596 PRs is not subject to the same filter, the data needed to test the comparison exist but are not used in RQ1. The authors should either re-run the analysis on an unrestricted repository set or explicitly restate the claims as applying only to repositories that predate safetensors; without this change, the headline percentages are an overgeneralization.","section":"Abstract and §IV-C (RQ3)"},{"comment":"The paper equates 'unmerged' PRs with PRs that developers 'ignore.' Figure 7 reports that 83.5% of conversion-tool PRs remain open and 13.9% are merged; an open PR may be pending review, awaiting the repository owner's decision, or intentionally left for later, none of which is the same as being ignored. The abstract's claim that 'a majority of developers ignore the conversion tool's pull requests' is therefore not supported by the reported status data. The authors should report the open/closed/merged breakdown separately and qualify the interpretation of open PRs, or present evidence (for example, PR age or absence of any activity) to support the 'ignore' framing.","section":"Abstract and §IV-C (RQ3)"},{"comment":"The PR counts in Section III are internally inconsistent. Section III-F states that the crawler extracted 43,596 PR URLs and that after retrieving their web pages it obtained 43,248 PRs; Section III-G then states that after merging duplicates and filtering out PRs without metadata and internal bot activity, the final set was 43,596 PRs. Since the filtering steps can only reduce the count, the post-filter total cannot exceed the pre-filter total. This discrepancy affects the reported RQ3 totals and the percentages derived from them; the authors should correct the numbers and verify that the RQ3 statistics are computed on the correct final dataset.","section":"§III-F and §III-G"}],"minor_comments":[{"comment":"Listing 1 contains a typo, 'toch.save', which should read 'torch.save'.","section":"§II-B (Listing 1)"},{"comment":"The word 'deserializaiton' is misspelled and should be 'deserialization'.","section":"§II-A"},{"comment":"The Introduction contains a duplicated word: 'democratize and and further advance'.","section":"§I"},{"comment":"The sentence '394 repositories (98.7%) still kept their old PyTorch model files' does not specify the denominator; the authors should state explicitly how many repositories or commits this percentage is based on.","section":"§IV-A"},{"comment":"The description of the RQ4 sampling is confusing: the paper samples 159 StackOverflow posts, 352 GitHub PRs, and 24 Hugging Face discussions, but then reports 'we obtained a total of 418 posts/discussions/PRs to analyze' without clarifying whether 418 is before or after the true-positive filtering step.","section":"§III-I and §III-J"},{"comment":"The RQ1 findings summary states 'Most of safetensor adoptions (95.7%) up to date' without specifying the data cut-off date or the sample restriction; adding 'among repositories created before September 2022' and the query date would avoid overstatement.","section":"§IV-A"}],"recommendation":"major_revision","confidential_remarks":"The paper's empirical core is solid and the data pipeline is transparent, but the headline claims need to be reined in: the adoption figures are computed on a pre-safetensors repository cohort only, and the 'ignore' language in the abstract overinterprets open PR statuses. The PR-count inconsistency in Section III also needs to be resolved before the RQ3 percentages can be trusted. I would be willing to look at a revised version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid empirical study with one load-bearing sampling limitation the authors don't acknowledge, and one overinterpretation in the abstract. The data pipeline is detailed, the magic-byte identification is sensible, and the qualitative taxonomy is a genuine contribution. But the headline claim that 95.7% of safetensors adoptions were automated conversions only holds for repositories created before safetensors existed. Section III-C filters to repos created before September 2022 and last modified in 2024, which excludes every repo created after the format was released. Any direct adoption by newer repos is invisible, so the figure is really about migrations of pre-existing repos, not 'adoptions up to date.' That doesn't invalidate the study, but it changes the claim, and the findings summary doesn't carry the caveat.\n\nWhat's new and good: it's the first large-scale measurement of safetensors adoption on Hugging Face, and it quantifies what many suspected—that the conversion bot drives most of the movement. The RQ3 merge-rate analysis is useful, especially the contrast between HF-staff PRs (93% merged) and everyone else (15.7%). The open-coding taxonomy with Cohen's kappa 0.88 is a reasonable qualitative contribution.\n\nSoft spots, in order. The filter is the big one. The authors frame it as wanting to observe how existing repositories changed after safetensors was released, which is a legitimate question, but the results are presented as ecosystem-wide. The abstract's 'developers ignore the conversion tool's pull requests' is also stronger than the data: 83.5% unmerged doesn't mean ignored, since open PRs could be pending or consciously declined. A time-to-close analysis or an explicit check for comments would support the 'ignore' wording. Notably, the RQ3 corpus of 43,596 PRs is not filtered the same way, so the data to test the comparison exists but isn't used. Minor: 1,032 repos is a small slice of the nearly million models on the Hub, so the trend lines in Figure 6 should be read as describing that cohort, not the whole ecosystem.\n\nOverall: the work is honest, the methods are appropriate, and the limitation is a scope issue rather than a fatal flaw. I'd bring this to a reading group to talk about sampling bias in repository studies.\n\nRecommendation: send to peer review with major revisions. The authors should re-run or stratify the sample, or at least qualify the automation-share claim, and soften the 'ignore' language. It deserves referee time.","headline":"Solid empirical study with a load-bearing sampling filter that makes the headline automation-share claim overbroad; worth a major revision, not a desk reject.","tokens_in":19222,"tokens_out":3532,"would_cite":false,"duration_ms":30975,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Most safetensors adoptions on Hugging Face come from an automated conversion bot, not deliberate developer choices.","keywords":["safetensors","model serialization","Hugging Face","supply chain security","empirical software engineering","pull request merge rate","developer perceptions","automated conversion tool"],"falsifier":"A concrete check is to run the same analysis on all Hugging Face model repositories, including those created after September 2022, and compute the percentage of safetensors-addition commits that come from the conversion tool; if that share is far below 95.7%, the paper's central claim about the tool's dominance does not generalize to newer models.","tokens_in":1543,"feed_emoji":"🤖","tokens_out":1818,"duration_ms":40730,"temperature":0.7,"pith_summary":"This paper asks how quickly and why developers on Hugging Face adopted safetensors, a serialization format designed to prevent malicious code execution when loading machine learning models. By analyzing commit histories of over a thousand model repositories and coding hundreds of developer discussions, it finds that adoption is real and growing, but most of the observed conversions were performed automatically by Hugging Face's own tool. The paper's central surprise is that the vast majority of the conversion tool's pull requests remain unmerged, and that many developers report problems with converted models. If correct, the finding reframes safetensors' rise as a story about automation and developer trust, not just enthusiasm for a safer format.","feed_headline":"A bot drives 95.7% of safetensors adoption","feed_subtitle":"But 83.5% of its conversion pull requests on Hugging Face go unmerged, a study finds.","key_machinery":"The central mechanism is Hugging Face's safetensors format and its automated conversion tool, which opens pull requests that replace PyTorch's torch.save files with safetensors files. The study's engine is a three-part analysis: commit-history extraction from 1,032 Hugging Face repositories created before September 2022, a rule-based file-header classifier that identifies serialization formats by magic bytes and zip structure, and open coding of 418 sampled discussions from Stack Overflow, GitHub, and Hugging Face, with a Cohen's kappa of 0.88 for inter-rater agreement.","core_discovery":"Safetensors is gaining traction on Hugging Face, but the adoption observed among existing repositories is dominated by an automated pipeline: 95.7% of commits adding safetensors files were merges of pull requests opened by Hugging Face's conversion tool. Despite that, 83.5% of the tool's pull requests across the platform remain unmerged, and only 13.9% are merged overall, with internal Hugging Face PRs merging at 93% versus outside developers' much lower rate. Developer discussions show both a genuine appetite for safe serialization and recurring pain points, including loading errors and reports of models behaving differently after conversion, which undermines trust in the format and the tool. The paper concludes that automated conversion is the main engine of safetensors adoption so far, but that verification and reliability concerns remain the key bottleneck.","pith_inferences":["If the tool's unmerged PRs reflect a lack of trusted verification, a testable extension is whether adding automatic fidelity checks (e.g., comparing model outputs before and after conversion) to the PR would raise merge rates.","The paper's dataset excludes repositories created after safetensors' release, so the 95.7% automation share likely overstates the tool's role among newer, safetensors-native models; re-running the analysis on the full hub snapshot would test that.","Keeping both model files side by side may create a split-brain problem where users accidentally load the outdated or unsafe file; a follow-up could measure which file downstream users actually load.","The reported 'model not working as expected' cases suggest a direct test: convert a set of models with the tool and compare hidden states, not just final outputs, to see if serialization changes weights or tied-weight handling."],"forward_implications":["Safetensors adoption in pre-existing Hugging Face repositories is largely an artifact of automated conversion, meaning the tool, not developer initiative, drove the measured shift.","Because 83.5% of conversion pull requests go unmerged, the real adoption bottleneck is developer acceptance and verification of converted models, not awareness of the format.","The high merge rate for Hugging Face staff PRs (93%) versus outside PRs suggests that organizational familiarity and responsibility strongly affect whether automated conversions are accepted.","Keeping old PyTorch files in 98.7% of converted repositories means the security benefit of conversion is diluted, since downstream users can still load the vulnerable originals.","Developers' reported model-fidelity problems after conversion, alongside their eagerness to learn the format, point to a need for better verification tooling and documentation."],"supporting_citations":[{"why":"Prior study of serialization formats and supply chain risks on Hugging Face; supplies the format-detection method and the baseline showing unsafe serialization is rampant.","marker":"[16]"},{"why":"Defines the safetensors format itself, including the header layout the paper uses to detect safetensors files.","marker":"[13]"},{"why":"Documents Hugging Face's conversion tool, the object whose pull requests and merge rates are the paper's core dataset.","marker":"[15]"},{"why":"Provides the pre-April 2023 conversions dataset that gives the paper its early list of conversion pull requests.","marker":"[30]"},{"why":"Supplies the SFConvertBot community-activity feed that yields the post-March 2023 pull request URLs.","marker":"[31]"},{"why":"Identifies the dedicated SFConvertBot account whose PRs are the study's central objects.","marker":"[32]"},{"why":"Reports the HiddenLayer hijacking technique that the paper links to the March 2024 conversion spike.","marker":"[40]"}],"fun_headline_variants":["Hugging Face bot drives 95.7% of safetensors adoption","83.5% of safetensors bot PRs never get merged","Automated conversion powers safetensors growth","Developers dodge most safetensors bot PRs","Safetensors adoption is bot-driven, but PRs fail"],"cache_read_input_tokens":21376,"weakest_assumption_plain":"The dataset only includes repositories created before September 2022 and last modified in 2024, so the adoption timeline and the 95.7% automation share describe only migration of pre-existing repositories; if developers of newer models adopted safetensors directly without the conversion tool, the tool's role is overstated and the measured shift is incomplete.","fun_headline_variants_meta":{"raw":{"variants":["Hugging Face bot drives 95.7% of safetensors adoption","83.5% of safetensors bot PRs never get merged","Automated conversion powers safetensors growth","Developers dodge most safetensors bot PRs","Safetensors adoption is bot-driven, but PRs fail"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000761,"raw_usage":{"total_tokens":3356,"prompt_tokens":903,"completion_tokens":2453,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":519,"completion_tokens_details":{"reasoning_tokens":2365}},"tokens_in":519,"tokens_out":2453,"duration_ms":16780,"temperature":1.0,"reasoning_tokens":2365,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:13:24.549925+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete check is to run the same analysis on all Hugging Face model repositories, including those created after September 2022, and compute the percentage of safetensors-addition commits that come from the conversion tool; if that share is far below 95.7%, the paper's central claim about the tool's dominance does not generalize to newer models.","supporting_citations":[{"cited_title":"Safetensors — huggingface.co,","cited_arxiv_id":null,"evidence_quote":"Defines the safetensors format itself, including the header layout the paper uses to detect safetensors files."},{"cited_title":"Convert to Safetensors - a Hugging Face Space by safetensors","cited_arxiv_id":null,"evidence_quote":"Documents Hugging Face's conversion tool, the object whose pull requests and merge rates are the paper's core dataset."},{"cited_title":"safetensors/conversions · Datasets at Hugging Face","cited_arxiv_id":null,"evidence_quote":"Provides the pre-April 2023 conversions dataset that gives the paper its early list of conversion pull requests."},{"cited_title":"SFconvertbot (Safetensors convertbot)","cited_arxiv_id":null,"evidence_quote":"Supplies the SFConvertBot community-activity feed that yields the post-March 2023 pull request URLs."},{"cited_title":"SFconvertbot (Safetensors convertbot)","cited_arxiv_id":null,"evidence_quote":"Identifies the dedicated SFConvertBot account whose PRs are the study's central objects."},{"cited_title":"Hijacking safetensors conversion on hugging face — hid- denlayer,","cited_arxiv_id":null,"evidence_quote":"Reports the HiddenLayer hijacking technique that the paper links to the March 2024 conversion spike."}],"review_version":1}