{"id":"3f2f3775-2f1f-40b9-ad8e-d7d0cf48958a","arxiv_id":"2504.15038","paper_version":2,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"Using over 13,000 hybrid journals, the study shows that open metadata from Crossref and OpenAlex produce country-level open access estimates consistent with Scopus and Web of Science, with transformative agreements enabling most hybrid open access by 2023.","lead":"This paper compares free, open metadata with the paid databases Scopus and Web of Science to measure how much open access publishing is driven by transformative agreements. It finds the sources largely agree, and that by 2023 these agreements funded the majority of open access articles in hybrid journals.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Web of Science upper-bound estimates (68–69%) are not adjusted for Elsevier open-archive articles that violate the paper's own CC-license definition; the reported agreement range may be inflated by a known source misclassification.","rationale":"The reader identified the ROR-to-proprietary matching error as the weakest assumption, and I agree that issue is real but mitigated by native ROR hoaddata, the lower Scopus error rate, and the cited de Jonge validation. However, the more direct threat to the paper's headline claim is that the WoS benchmark does not implement the paper's own open access definition. Section 3.2 explicitly shows that WoS counts Elsevier open-archive articles as hybrid OA, even though the study defines hybrid OA as CC-licensed and WoS's own documentation specifies CC variants. This is not a subtle definitional choice: it is a known measurement inconsistency in the benchmark used to produce the highest estimates in the abstract's range. The paper's scatterplots and correlations may still hold under this misclassification, but the specific 68–69% figures are not comparable to hoaddata and Scopus without a robustness check. A simple exclusion test would settle whether the upper bound is genuine. I would not reject the paper; the core contribution and the hoaddata/Scopus alignment remain valuable. I would make acceptance conditional on running and reporting that check, or on softening the claim to a CC-licensed range based only on hoaddata and Scopus.","tokens_in":19943,"tokens_out":14204,"duration_ms":144027,"concrete_test":"Recompute the 2023 Web of Science first- and corresponding-author agreement share after excluding articles that are not CC-licensed, particularly Elsevier items tagged as open archive or user license. Use WoS open access type/license fields, or restrict to Unpaywall publisher-hosted OA locations with a CC license. If the share drops by 3 or more percentage points, or falls below the hoaddata/Scopus range, the reported 68–69% upper bound is an artifact of the OA misclassification, and the paper should report a CC-licensed range or exclude WoS from the headline range.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim's upper bound rests on Web of Science, but Section 3.2 and Figure 5 document that WoS labels Elsevier's delayed 'open archive' articles (published under the publisher's user license, not a CC license) as hybrid open access, contrary to WoS's own documentation and to the study's stated CC-license definition. If those articles are also attributed to transformative agreements through institutional mappings, the WoS first-author share of 68% and corresponding-author share of 69% are not measuring the same quantity as hoaddata (61%) or Scopus (64–65%). The paper reports the Elsevier discrepancy qualitatively but does not quantify its effect on the agreement-attributed share; it then uses WoS as the upper confirmation of the headline range. Because this range is the paper's flagship result, the WoS estimate should be corrected or excluded before claiming that the majority share reaches 68–69%.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper compares hoaddata, an open metadata pipeline built from Crossref, OpenAlex, and the cOAlition S Journal Checker Tool, with Scopus and Web of Science to estimate the impact of transformative agreements on hybrid open access publishing between 2019 and 2023. For more than 13,000 hybrid journals, the author computes open access uptake and the share of open access articles attributable to transformative agreements, using first- and corresponding-author attributions where available. The central claims are that the three data sources agree strongly at country level (Spearman rho > 0.9 for most comparisons) and that by 2023 transformative agreements enabled the majority of hybrid open access, with agreement-attributed shares of 61% (hoaddata), 64-65% (Scopus), and 68-69% (Web of Science). The paper also documents coverage differences, including journals missing from one or more sources and discrepancies in open access labelling, particularly for Elsevier open-archive articles in Web of Science and Wiley redirect issues in Unpaywall.","tokens_in":20036,"tokens_out":4043,"duration_ms":37901,"significance":"If the results hold, this is a valuable contribution to bibliometric methodology and open access monitoring. The study uses a transparent, reproducible pipeline with public code, data, and a test suite, and it validates an open dataset against independent proprietary databases rather than against itself, which makes the comparison informative. The detailed coverage diagnostics and the explicit acknowledgement of known labelling discrepancies are notable strengths. The finding that open metadata can produce country-level estimates consistent with Scopus and Web of Science is practically important for consortia and funders that cannot afford proprietary data. However, the reliability of the headline range depends on resolving the Web of Science open-archive issue and on the robustness of the institutional identifier mapping, so the contribution needs revision before it can be fully endorsed.","major_comments":[{"comment":"The Web of Science upper-bound estimates of 68% (first author) and 69% (corresponding author) include Elsevier 'open archive' articles that are labelled as hybrid open access in Web of Science even though they are not published under a CC licence, as the paper itself documents in the discussion of Figure 5 and the surrounding text. The manuscript reports this discrepancy qualitatively but does not quantify its effect on the agreement-attributed share. Because the 68-69% figures are used as the upper endpoint of the flagship claim that transformative agreements enabled the majority of hybrid open access by 2023, the paper should either correct the Web of Science estimates by excluding Elsevier open-archive articles, or report the Web of Science values with and without those articles and adjust the conclusion accordingly. As it stands, the headline range mixes a quantity measured under the paper's CC-licence definition (hoaddata, Scopus) with a quantity that includes a known source misclassification (Web of Science).","section":"Section 3.2, Figure 5"},{"comment":"The quality check for the ROR-to-proprietary identifier mapping is based on a random sample of 50 pairs, with 22% mismatches for Web of Science and 6% for Scopus. The paper acknowledges that mismatches mainly involve less-represented institutions, but the mapping is used to attribute every agreement-enabled article, so a 22% mismatch rate in the Web of Science mapping could materially affect the country-level and global estimates. The high country-level correlations (rho > 0.9) could be partly an artifact of aggregation if errors average out across institutions within countries. To support the claim that results are consistent across data sources, the paper should provide a confidence interval for the mismatch rate, stratify the mismatches by institution size or country, and report a sensitivity analysis, for example by recomputing the main indicators after excluding low-confidence institutional matches.","section":"Section 2.2, harmonising author affiliations"}],"minor_comments":[{"comment":"The word 'affilation' should be 'affiliation' in the description of hoaddata, and the Introduction contains 'open accesss' instead of 'open access'.","section":"Section 2.1"},{"comment":"The sentence 'Since 2021 (hoadata, Scopus) resp. 2022 (Web of Science), transformative agreements have enabled the majority...' should be rephrased as formal prose rather than abbreviated notation, and 'hoadata' should be 'hoaddata' for consistency.","section":"Section 3.2"},{"comment":"The sentence beginning 'Because neither Web of Science nor Scopus supported ROR-ID at the time of data retrieval, the institution identifier used by the JCT, a two-step matching process...' is a sentence fragment and should be rewritten.","section":"Section 2.2"},{"comment":"In the Figure 2 caption, 'The x-axis represents the set intersections in a matrix layout' is imprecise; the x-axis of an UpSet plot represents the intersection membership matrix, not the intersections themselves.","section":"Section 3.1"},{"comment":"The table note should state more explicitly that hoaddata counts are based on DOI-bearing articles, while Scopus and Web of Science counts use database identifiers; this distinction is described in the text but the table could be misread without it.","section":"Section 2.3, Table 1"}],"recommendation":"major_revision","confidential_remarks":"The Web of Science open-archive issue is the main obstacle to acceptance; it affects the paper's headline range and needs a quantitative remedy. The other major concern, the small institutional-mapping validation sample, is also fixable with additional robustness analyses. The paper is otherwise methodologically transparent and well suited to a bibliometrics journal. I would not recommend rejection, as the core finding that transformative agreements account for a majority of hybrid open access also holds under the hoaddata (61%) and Scopus (64-65%) estimates, even if the Web of Science upper bound is corrected downward."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth your time. This is the first systematic comparison of hoaddata (Crossref+OpenAlex+JCT) against Scopus and Web of Science on the same set of >13,000 hybrid journals under transformative agreements, 2019–2023. It gives journal and article coverage overlap, open access detection differences, and agreement-attributed share by publisher and country. The comparative design is genuinely new in this specific context, and the author does the field a service by documenting concrete discrepancies: Elsevier's open archive mislabelled in WoS, Wiley redirects breaking Unpaywall, and publishers that don't share CC license metadata with Crossref. Code, data, an R package, and a dashboard are all shipped, so the work is reproducible and checkable. That's real evidence and deserves credit.\n\nThe main result holds. By 2023, transformative agreements enabled the majority of hybrid OA in all sources: 61% (hoaddata), 64% (Scopus), 68–69% (WoS). Even if you drop WoS entirely, hoaddata and Scopus both show a majority. So the headline is robust.\n\nSoft spots, in proportion. The stress-test note is correct: WoS labels Elsevier's delayed open archive as hybrid OA even though it's not CC-licensed, and the paper doesn't quantify how much that inflates the WoS agreement-attributed share. The paper mentions this and shows it in Figure 5, but then still uses WoS as the upper confirmation of the range. That is a genuine reporting gap, not a fatal one. A revision should either correct the WoS estimate or bound the effect. Second, the ROR-to-proprietary affiliation mapping has a 22% mismatch in WoS based on a 50-pair sample. The author acknowledges it and notes mismatches concentrate in less-represented institutions, but the validation sample is small and the institutional-level attribution is noisier than the country-level results suggest. The high correlations (rho>0.9) could be partly aggregation artifacts. Third, OpenAlex's lack of corresponding-author data limits hoaddata to first authors, which the author handles transparently; that's a limitation, not a flaw. Missing confidence intervals for correlations is minor.\n\nThe citation pattern is honest. The author cites his own hoaddata and prior QSS paper, but he validates against independent commercial databases, which breaks the circularity. The paper is clearly written and takes its limitations seriously.\n\nMy take: send it to peer review. This deserves a serious referee. Ask the author to quantify the WoS–Elsevier effect and to strengthen the mapping validation discussion. The core result is solid and the transparency is exemplary.","headline":"A transparent large-scale three-way comparison that validates open metadata for monitoring transformative agreements; the WoS Elsevier misclassification is a real but non-fatal caveat.","tokens_in":20587,"tokens_out":2621,"would_cite":true,"duration_ms":24227,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Open metadata matches Scopus and Web of Science on transformative agreements, a 13,000-journal comparison finds.","keywords":["hybrid open access","transformative agreements","open metadata","bibliometric data sources","hoaddata","Crossref","OpenAlex","curative bibliometrics"],"falsifier":"Recompute the country-level Spearman correlations after removing all institutions involved in mismatched affiliation pairs or with fewer than about ten articles; if the correlations drop materially below 0.9, the equivalence claim fails. A stronger test would compare open-metadata estimates with actual invoicing data from several national consortia: if the per-country error is no smaller than the cross-source differences, the open metadata are not in fact reproducing the proprietary databases.","tokens_in":19679,"feed_emoji":"🔓","tokens_out":9710,"duration_ms":74768,"temperature":0.7,"pith_summary":"The paper asks whether openly available metadata can replace or complement the proprietary databases Scopus and Web of Science when measuring how much open access in hybrid journals is enabled by transformative agreements. Hybrid journals are subscription journals that let individual articles be made open access, and transformative agreements are licensing deals meant to flip them to full open access. It applies the same journal- and institution-matching method to all three sources — hoaddata, an open dataset built from Crossref, OpenAlex, and the cOAlition S Journal Checker Tool, alongside Scopus and Web of Science — covering more than 13,000 hybrid journals from 2019 to 2023. The central finding is that the sources agree: by 2023, transformative agreements enabled the majority of hybrid open access, and country-level ranks correlate above 0.9 across data sources and author roles. The paper concludes that open metadata alone already supports reliable large-scale monitoring of the open access transition, while using multiple data sources together is safer than relying on any single one.","feed_headline":"Open metadata matches paywalled databases on open-access deals","feed_subtitle":"Free metadata matches Scopus and Web of Science when estimating agreement-funded open access in 13,000 hybrid journals.","key_machinery":"The load-bearing mechanism is a DOI-based matching pipeline that connects the cOAlition S Journal Checker Tool's ROR (Research Organization Registry) institution and journal-agreement records to each database's own affiliation identifiers: OpenAlex ROR IDs for hoaddata, Scopus Affiliation IDs, and Web of Science enhanced organization names. A two-step algorithm pairs millions of articles by DOI, selects the most frequent ROR-to-proprietary-identifier match, then attributes an article to a transformative agreement only when the agreement was active in that year according to the ESAC registry. This pipeline is what lets the same estimation logic run inside all three data sources. The paper's quality check on fifty random pairs found a 6% mismatch rate for Scopus and a 22% mismatch rate for Web of Science, concentrated among institutions with few publications.","core_discovery":"The discovery is that agreement-enabled hybrid open access can be estimated consistently from open metadata and from proprietary databases, and that by 2023 such agreements had become the dominant route to open access in hybrid journals. Depending on the source, 61% (hoaddata), 64-65% (Scopus), or 68-69% (Web of Science) of hybrid open access articles in 2023 were attributed to transformative agreements; in absolute terms, hoaddata linked 501,649 articles to agreements over 2019-2023. Agreement between sources is strongest at the country level, where Spearman rank correlations exceed 0.9 for article volume, open access share, agreement-enabled output, and agreement share, and remain above 0.85 when small countries are included. First-author and corresponding-author attributions give nearly the same national picture, which matters because open metadata often lack corresponding-author fields. The discrepancies that do appear, such as Web of Science counting Elsevier's delayed open access as hybrid or Unpaywall missing Wiley articles, are used to argue that multi-source verification is the safer path.","pith_inferences":["If the matching errors are concentrated in small institutions, institutional-level league tables built from Web of Science without ROR support are likely to be noisier than country-level tables; the paper's quality check points in that direction but does not demonstrate it.","The same pipeline could be pointed at funder-level or discipline-level subsets, where a testable prediction is that agreement shares will vary more across disciplines than across data sources.","As more publishers begin depositing CC-license metadata in Crossref, the open-metadata share of detected hybrid open access should rise, so the 61% figure is probably a lower bound for current hoaddata coverage rather than an upper bound.","The high first-versus-corresponding correlation may not survive in fields with large international author teams, so first-author proxies should be checked within disciplines before use at finer granularity."],"forward_implications":["Country-level monitoring of transformative agreements can be done with open metadata alone, without buying Scopus or Web of Science access.","First-author affiliation is an adequate proxy for corresponding-author affiliation in national-level agreement uptake statistics.","Any single database, open or proprietary, mislabels some hybrid open access; Web of Science's treatment of Elsevier delayed open access and Unpaywall's Wiley parsing failures show that triangulation changes conclusions.","Journals missing from selective databases, especially newly launched hybrid titles, are visible in Crossref-based open metadata, so relying exclusively on Scopus or Web of Science undercounts the agreement landscape.","Agreements accounted for 61-69% of hybrid open access by 2023, meaning the remaining open access is paid for outside agreements, and most hybrid articles are still paywalled."],"supporting_citations":[{"why":"Supplies the hoaddata method - combining the Journal Checker Tool, Crossref, and OpenAlex - that this study adapts to Scopus and Web of Science.","marker":"Jahn (2025)"},{"why":"Validates the Journal Checker Tool matching approach against Dutch NWO invoicing data, supporting the accuracy of agreement attribution.","marker":"de Jonge et al. (2025)"},{"why":"Defines the Unpaywall-based hybrid open access evidence used by Scopus and Web of Science and the paratext exclusion approach adapted by hoaddata.","marker":"Piwowar et al. (2018)"},{"why":"Establishes what transformative agreements are and why agreement coverage is usually limited to original research articles and reviews.","marker":"Borrego et al. (2021)"},{"why":"Describes Crossref as the open metadata source hoaddata uses for publication volume and CC-license open access status.","marker":"Hendricks et al. (2020)"},{"why":"Describes OpenAlex, the affiliation data source supplying ROR IDs and author roles for hoaddata.","marker":"Priem et al. (2022)"},{"why":"Motivates the ESAC workflow assumption that CC licenses should be deposited in Crossref during DOI registration, which underlies hoaddata's open access measure.","marker":"Geschuhn & Stone (2017)"},{"why":"Documents the uniform annual snapshots of Scopus and Web of Science that supply the proprietary-side indicators.","marker":"Schmidt et al. (2024)"}],"fun_headline_variants":["Open metadata matches paywalled indexes for agreement impact","Transformative agreements now dominate hybrid open access","Free data rivals Scopus and WoS on hybrid OA deals","Open metadata validates agreement-funded hybrid open access","Study: Open metadata tracks transformative agreements accurately"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The estimates stand on the assumption that the DOI-based mapping from ROR institutions to Scopus and Web of Science affiliation identifiers is accurate enough that errors mostly cancel out at the country level, despite a 22% mismatch rate found for Web of Science in a fifty-pair quality check.","fun_headline_variants_meta":{"raw":{"variants":["Open metadata matches paywalled indexes for agreement impact","Transformative agreements now dominate hybrid open access","Free data rivals Scopus and WoS on hybrid OA deals","Open metadata validates agreement-funded hybrid open access","Study: Open metadata tracks transformative agreements accurately"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000215,"raw_usage":{"total_tokens":1435,"prompt_tokens":960,"completion_tokens":475,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":576,"completion_tokens_details":{"reasoning_tokens":404}},"tokens_in":576,"tokens_out":475,"duration_ms":4950,"temperature":1.0,"reasoning_tokens":404,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:33:43.804510+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the country-level Spearman correlations after removing all institutions involved in mismatched affiliation pairs or with fewer than about ten articles; if the correlations drop materially below 0.9, the equivalence claim fails. A stronger test would compare open-metadata estimates with actual invoicing data from several national consortia: if the per-country error is no smaller than the cross-source differences, the open metadata are not in fact reproducing the proprietary databases.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the hoaddata method - combining the Journal Checker Tool, Crossref, and OpenAlex - that this study adapts to Scopus and Web of Science."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Validates the Journal Checker Tool matching approach against Dutch NWO invoicing data, supporting the accuracy of agreement attribution."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes what transformative agreements are and why agreement coverage is usually limited to original research articles and reviews."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents the uniform annual snapshots of Scopus and Web of Science that supply the proprietary-side indicators."}],"review_version":1}