{"id":"d05a5463-8d19-428d-88fe-ea93ab8adb2b","arxiv_id":"2506.08300","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"About 983,000 public domain books from Harvard Library's Google Books digitization, containing roughly 242B tokens, were processed and released with metadata, language and topic labels, dedup hints, and post-processed OCR.","lead":"Researchers at Harvard released Institutional Books 1.0, a dataset of about 983,000 public domain books containing roughly 242 billion tokens of OCR text, along with catalog metadata, deduplication hints, language labels, and processed text versions. The release is meant to give AI researchers and libraries a large, documented source of historic text with a clear provenance chain for training and study.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'public domain' claim is materially overbroad: roughly 20% of released volumes are HathiTrust 'pdus' (US-only public domain), and the initial release is under a noncommercial click-through license.","rationale":"The reader's weakest assumption focused on HathiTrust as a single external authority; that is a legitimate epistemic risk but not the most immediate threat to the central claim. The sharper issue is internal: the paper's own rights table distinguishes 'pd' from 'pdus', and Section 5.3 disclaims non-US rights. Even if HathiTrust's determinations are perfectly correct, the released set is not uniformly in the public domain worldwide. HathiTrust's 'pdus' status is specifically used when a work is public domain in the US but copyright protection may exist in the country of origin. Because the collection is heavily European (German 17.3% and French 14.0% of detected tokens, for example), a substantial share of the pdus volumes will be under copyright in their home countries until 70 years after the author's death. Calling the corpus 'public domain books' without qualification is therefore misleading to the international research community, which is a primary audience for this dataset. The paper's transparency in Section 5.3 and Appendix J is commendable, and the underlying release remains extremely valuable; a CONDITIONAL verdict is appropriate, requiring the authors to (a) replace or qualify 'public domain' with 'public domain in the United States' when describing pdus volumes, (b) state the pdus volume/token share in the abstract and dataset card, and (c) clearly mark the dataset license (noncommercial, click-through) so users understand the artifact is not CC0. This is consistent with the reader's CONDITIONAL verdict, so the verdict itself does not change.","tokens_in":41909,"tokens_out":7267,"duration_ms":87291,"concrete_test":"Using the released metadata fields hathitrust_data_ext.rights_code and token_count_o200k_base_gen, compute the count and token-weighted fraction of the 983,004 released volumes whose rights_code is 'pdus'. Separately, download the Hugging Face dataset card and record its license field and terms-of-use URL. If the pdus share is at least 5% of tokens (Appendix J suggests about 20% of volumes) and the dataset card does not explicitly state 'US public domain only' and 'noncommercial dataset license at initial release', the abstract and Contribution 2 must be revised to qualify the public-domain claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract and Contribution 2 describe the released 983,004 volumes as 'in the public domain' and with 'no known copyright.' The paper's own Appendix J shows that 196,171 of the rights-matched volumes (18.23% of the collection, about 19.95% of the 983,510 rights-positive set) carry HathiTrust status 'pdus' — public domain only in the United States. Section 5.3 concedes that 'some volumes in this dataset may be in the public domain in the United States but still subject to copyright or other rights protections in other jurisdictions.' For a corpus explicitly aimed at global LLM training and research, this is not a minor edge case: under life-plus-70 regimes, a pre-1929 US-public-domain work by an author who died after 1955 remains in copyright in the EU and many other countries. In addition, Section 1 states that the OCR text is initially released under a noncommercial license with a click-through, so the compiled dataset artifact is not public domain/CC0 even though many underlying books may be. The unqualified 'public domain' claim is therefore not supported for roughly a fifth of the corpus and for the dataset as distributed. This concern does not require doubting HathiTrust's accuracy; it follows from the status codes the paper itself reports.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This technical report (arXiv:2506.08300) describes the construction of Institutional Books 1.0, a corpus of OCR-extracted text from Harvard Library's participation in the Google Books project. The authors document retrieval from GRIN, rights determination via HathiTrust's API, and a processing pipeline that includes language detection, topic classification, near-duplicate detection, OCR artifact analysis, text metrics, and a line-type classifier used to post-process OCR text for the five most frequent languages. The released dataset is claimed to contain 983,004 public-domain volumes, approximately 242B o200k_base tokens, with both original and post-processed OCR text plus extensive metadata. The paper also releases the processing pipeline and the fine-tuned classifier.","tokens_in":42185,"tokens_out":4194,"duration_ms":53238,"significance":"If the claims are properly qualified, this is a significant resource: a multilingual, multi-century book corpus at the 100B-token scale with unusually detailed provenance documentation, per-volume metadata, duplicate flags, OCR quality scores, and both original and post-processed text. The authors ship the pipeline and classifier, report collection-level statistics in extensive tables and appendices, and are explicit about several limitations. The main contributions, however, are currently presented with an unqualified 'public domain' label that is contradicted by the paper's own rights breakdown, and the usability improvements are supported by metrics that are partly constructed by the post-processing heuristics themselves.","major_comments":[{"comment":"The unqualified 'public domain' claim is not supported for the released dataset. Appendix J reports 196,171 volumes with HathiTrust status 'pdus' (18.23% of the collection, roughly 19.95% of the 983,510 rights-positive set), and §5.3 concedes that these volumes may be in the public domain in the United States but still subject to copyright or other rights protections in other jurisdictions. In addition, §1 states that the OCR text is initially released under a noncommercial license with a click-through, so the compiled dataset artifact is not public domain or CC0 even though many underlying volumes may be. Because 'a public dataset containing ... volumes ... for which there is no known copyright' is Contribution 2, the abstract and contribution list must be revised to say 'public domain in the United States' and to state the dataset's actual license and access terms.","section":"Abstract, §2, §5.3, Appendix J"},{"comment":"The evidence that post-processing improves 'usability' is partly circular. The two reported improvements are the o200k_base tokenizability score and average detected sentence length, and the post-processing pipeline was explicitly designed to merge line fragments, remove hyphenations, and strip page numbers and running heads. These operations directly change token-per-word ratios and sentence segmentation results, so observing movement in those metrics is partly by construction. To support the claim that the post-processed text is easier to filter, read, and use in ML/NLP contexts, the paper needs an independent evaluation—for example, human readability annotation, downstream task performance, perplexity, or an external OCR-quality measure. As it stands, Figures 14–17 show that the heuristics move their own targets, not that the output is more usable.","section":"§4.9.2, §4.8, Figures 14–17"},{"comment":"The line-type detector's reported 71% accuracy is measured against 10% of the same Phi-4-generated annotations used for training, with no human-validated held-out set described. Because the training labels were produced by Phi-4 and the benchmark is drawn from the same machine-generated distribution, the benchmark largely measures agreement with the labeling model rather than accuracy on real OCR lines. This matters because the line-type detections are a load-bearing component of the post-processing claim; independent, manually verified evaluation of the detector's predictions is needed before the post-processed output can be presented as an accuracy improvement.","section":"§4.9.1, Appendix I"}],"minor_comments":[{"comment":"The percentage '91,41%' appears to be a typographical error and should read '91.41%'.","section":"§5.2"},{"comment":"The field description for language_gen says 'ISO 693-3' in two places; this should be 'ISO 639-3'.","section":"Appendix A"},{"comment":"Table App. A4 is numbered twice: once for text_analysis_gen and once for identifiers_src; the second table should be renumbered, with subsequent tables adjusted accordingly.","section":"Appendix A"},{"comment":"The spelling of the OCRoscope tool is inconsistent: both 'OCROscope' and 'OCRoscope' appear in the text; the paper should use a single spelling.","section":"§4.7"},{"comment":"The token counts for openai/gpt-4 and microsoft/phi-4 are identical (275,637,216,999); this is plausible if phi-4 uses the same cl100k_base tokenizer, but the paper should state that explicitly to avoid the appearance of a copying error.","section":"§4.2.2, Table 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a valuable dataset release, and the central technical documentation appears careful and internally consistent. The main obstacle is the framing: the 'public domain' characterization is materially overbroad given the paper's own Appendix J and the noncommercial click-through release. This is correctable by revising the claims and highlighting the license and pdus status. I would ask the authors to make those changes prominently in the abstract, contribution list, and dataset card, and to add an independent evaluation for the post-processing claim before recommending acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a real release worth engaging with. The dataset — roughly 983,000 volumes, 242B tokens by o200k_base, original and post-processed OCR, language and topic metadata, collection-level dedup hints — is new at this scale, and the provenance documentation is unusually careful. The retrieval pipeline, post-processing code, and trained classifier are all shipped, which means the processing steps can be checked and reused.\n\nThe soft spot is the rights framing. The paper's own Appendix J shows 196,171 of the rights-matched volumes (18.23%) are HathiTrust 'pdus' — public domain only in the United States. For a corpus explicitly aimed at global LLM training and research, that is not a minor edge case: under life-plus-70 regimes, a pre-1929 US-public-domain work by an author who died after 1955 is still in copyright in the EU and many other countries. The abstract and Contribution 2 call these volumes 'public domain' with 'no known copyright' without that qualifier. Section 5.3 concedes the point, but the headline claim is still overbroad. Also, the initial release is under a noncommercial click-through license, so the compiled dataset itself is not public domain/CC0 even though many underlying books may be.\n\nThe accuracy claims are shakier but not fatal. The post-processing evaluation uses tokenizability and sentence length, both of which the pipeline was explicitly designed to move, so seeing them improve is partly by construction. The line-type classifier was trained on Phi-4-generated labels and benchmarked on a sample of those same labels; 71% against machine labels is not an external accuracy estimate. The topic classifier's benchmark comes from the same mapping that built its training set. The authors do label these outputs as experimental and ship the original OCR alongside the post-processed version, so users can ignore the processing. Still, the 'refined for accuracy' part of the pitch should be read with those caveats.\n\nNo fatal flaw in the dataset itself. The central artifact — a large, documented, multilingual book corpus with a clear provenance chain — stands. The paper deserves serious review; a referee should push for the rights language to be qualified and for at least one external evaluation of the post-processing. I'd take it.","headline":"A genuinely useful 242B-token book corpus with strong provenance documentation; the rights claims are overstated — roughly a fifth of volumes are US-public-domain only, and the distributed dataset is noncommercial.","tokens_in":42723,"tokens_out":2900,"would_cite":true,"duration_ms":33400,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Nearly a million public-domain books from Harvard Library's Google Books digitization — 983,004 volumes, about 242B tokens — are released with documented provenance, per-volume metadata, and cleaned OCR text.","keywords":["public domain corpus","large language model training data","OCR post-processing","HathiTrust rights database","Google Books digitization","multilingual text corpus","near-duplicate detection","dataset provenance"],"falsifier":"Take a random sample of roughly 1,000 released barcodes, re-query the HathiTrust API, and compare the returned rights codes with the dataset's records; then independently verify the public-domain basis of a subsample of volumes dated after 1928, for example by checking for US government authorship or non-US publication. If a material share of post-1928 volumes have no valid public-domain basis, or if the barcode-to-hvd matching misaligns for a large fraction of the sample, the claim that this is a public-domain corpus would be refuted.","tokens_in":41690,"feed_emoji":"📚","tokens_out":14378,"duration_ms":142355,"temperature":0.7,"pith_summary":"This paper claims that Harvard Library's Google Books digitization can be converted into a documented, reusable public-domain training corpus, and reports doing so at record scale: 983,004 volumes totaling 242B tokens (o200k_base), drawn from a 1,075,899-volume collection spanning over 250 languages. The authors argue this matters because publicly available, high-quality training data is scarce and because most existing corpora are English-dominated web text with weak provenance chains. The release pairs each volume's raw OCR text with a post-processed version for the five most common languages and attaches bibliographic, source, and generated metadata designed to let users filter the collection for their own uses. The central promise, if correct, is that a documented multi-hundred-billion-token public-domain book corpus now exists as a building block for language-model research.","feed_headline":"Nearly a million public-domain books released as a 242B-token dataset","feed_subtitle":"Cleaned OCR, per-volume metadata, and documented provenance make it a large public text resource for model research.","key_machinery":"The load-bearing mechanism is the barcode-keyed volume record: each Harvard barcode, prefixed with 'hvd', is matched to the HathiTrust API, and the returned rights status decides whether the volume enters the release. Around that gate the paper assembles a processing stack: a custom retrieval pipeline for the Google Return Interface (GRIN); a multilingual BERT classifier fine-tuned to assign each volume one of 20 Library of Congress top-level topics; a Simhash-based near-duplicate detector using 7-character shingles; and an OCR line-type classifier — a static embedding model distilled from LaBSE — that labels each OCR line as paragraph chunk, heading, page number, running head, separator, or noise, guiding a post-processing pass that reassembles sentences and removes page furniture.","core_discovery":"The paper's central claim is that the public-domain portion of Harvard Library's Google Books digitization — 983,004 volumes, 91.37% of the 1,075,899-volume collection, about 242B o200k_base tokens — has been retrieved, analyzed, documented, and released. Inclusion in the release is gated by a single external judgment: each volume's barcode is prefixed with 'hvd' and matched against the HathiTrust rights database, and only volumes returning pd, pdus, or cc-zero statuses are included. The released text comprises the original OCR output plus a post-processed version for English, German, French, Italian, and Spanish, together with per-volume topic labels from a fine-tuned classifier, text-level language distributions, near-duplicate flags, OCR-quality scores, and text statistics. The authors present the dataset not as a finished training set but as a substrate for community filtering and future iteration.","pith_inferences":["Beyond the paper: if other Google Books partner libraries adopt the same barcode-prefix matching and pipeline, the approach generalizes into a family of institutional public-domain corpora that could multiply the supply of documented book text for training.","Beyond the paper: a direct test of the post-processing claim would be to train matched small models on original versus post-processed text and compare perplexity or downstream-task scores; the paper reports proxy metrics but does not run that experiment.","Beyond the paper: because roughly 32% of volumes lack usable dates and the rights determinations are US-centric, users outside the United States should expect to perform their own rights review despite the authors' good-faith assessment."],"forward_implications":["Researchers gain a documented public-domain corpus of roughly 242B tokens, with 230 languages each contributing at least a million detected tokens.","The corpus is concentrated in the nineteenth and early twentieth centuries (about 60% of datable volumes fall between 1820 and 1920), so it fits historical research and long-context work better than contemporary-web-style pretraining.","The 41,366 volumes flagged as likely near-duplicates are listed in the dataset rather than removed, letting each user decide whether to exclude them from training runs.","Because post-processing removed only 0.97% of characters while raising tokenizability scores and lowering detected sentence length, the cleaned text offers a more machine-usable alternative to the raw OCR without replacing it.","At preliminary launch the text is released under a noncommercial click-through license with raw scan images withheld, so the practical reach of the corpus depends on how those access terms evolve."],"supporting_citations":[{"why":"The sole authority for the public-domain determinations that gate which volumes enter the release.","marker":"1 (HathiTrust rights database footnote)"},{"why":"Supplies the Common Corpus precedent of library collections as ethical LLM training data and the OCRoscope tool used to compute secondary OCR quality scores.","marker":"Langlais et al., 2025"},{"why":"FineWeb is cited as the model for careful deduplication and description to improve training results, which this paper's analysis design follows.","marker":"Penedo et al., 2024"},{"why":"Provides the evidence that deduplicating training data makes language models better, grounding the collection-level deduplication experiment.","marker":"Lee et al., 2022"},{"why":"Supplies the Simhash algorithm that the paper uses to identify near-duplicate OCR-extracted texts.","marker":"Charikar, 2002"},{"why":"Evidence that Simhash can be used effectively on the OCR-extracted text of books, justifying its application to this collection.","marker":"Vladimir et al., 2015"},{"why":"Supplies bert-base-multilingual-uncased, fine-tuned into the 20-class topic classifier applied to the whole collection.","marker":"Devlin et al., 2019"},{"why":"Supplies LaBSE, distilled via Model2Vec into the static embedding model that classifies OCR line types for post-processing.","marker":"Feng et al., 2022"},{"why":"Phi-4 generates the labeled line-type training data used to train the OCR post-processing classifier.","marker":"Abdin et al., 2024"}],"fun_headline_variants":["Harvard opens 242B tokens of public-domain books","983K public-domain books, 242B tokens: Harvard dataset","242B tokens from Harvard's digitized book archive","Nearly a million public books become 242B tokens","Harvard Library releases 242B-token public dataset"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire release depends on one external judgment: the HathiTrust rights database, matched to each volume through a simple barcode prefix, correctly labels everything it marks as public domain, so if those determinations or that match are wrong for a significant number of volumes, the dataset's central promise fails regardless of how well the processing pipeline works.","fun_headline_variants_meta":{"raw":{"variants":["Harvard opens 242B tokens of public-domain books","983K public-domain books, 242B tokens: Harvard dataset","242B tokens from Harvard's digitized book archive","Nearly a million public books become 242B tokens","Harvard Library releases 242B-token public dataset"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000566,"raw_usage":{"total_tokens":2729,"prompt_tokens":1039,"completion_tokens":1690,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":655,"completion_tokens_details":{"reasoning_tokens":1611}},"tokens_in":655,"tokens_out":1690,"duration_ms":14938,"temperature":1.0,"reasoning_tokens":1611,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:13:49.061868+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of roughly 1,000 released barcodes, re-query the HathiTrust API, and compare the returned rights codes with the dataset's records; then independently verify the public-domain basis of a subsample of volumes dated after 1928, for example by checking for US government authorship or non-US publication. If a material share of post-1928 volumes have no valid public-domain basis, or if the barcode-to-hvd matching misaligns for a large fraction of the sample, the claim that this is a public-domain corpus would be refuted.","supporting_citations":[],"review_version":1}