{"id":"7acdadea-e209-4c37-8915-2c287266d409","arxiv_id":"2507.14752","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"The authors describe and release a 27.3 million URL sample from the Wayback Machine, built to support studies of how long web pages survive.","lead":"This paper documents how the authors built a sample of 27.3 million archived web addresses from the Internet Archive, spanning 1996 to 2021. It is a methods and dataset paper for researchers who need a large, time-balanced collection of historical web pages.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central claim is under-specified: the 27.3M-URL dataset's composition and reproducibility depend on an undocumented 107M→92M step and hand-tuned post hoc parameters, so the stated pipeline is not fully reconstructible.","rationale":"The reader's verdict is CONDITIONAL and my stress-test largely confirms it, but I identify a slightly different load-bearing concern. The reader focused on the absence of design weights and hand-tuned parameters; I agree that is the core statistical gap. However, I want to emphasize an even more basic issue: the exact composition of the 107M-URL set and the exact pipeline that maps the 92M set to the final 27.3M set is not fully specified in a way that would allow an independent reproduction of the central claim. The paper provides code links and detailed heuristics, but the specific version of the heuristics (e.g., the exact regex set for extensions) and the exact order of operations (e.g., removal of popular domains before extension filtering, per-year long-tail reduction threshold of 900,000 domains) are not given as a single algorithmic description. This is not a claim of fraud or a 'fatal' flaw; it is a reproducibility concern that is load-bearing because the paper's contribution is a dataset, not a theorem. A dataset whose construction cannot be independently re-run is unverifiable, and the paper itself admits several hand-tuned parameters (K, C, 900K threshold) and an unquantified false-negative rate for the extension filter. The concrete test I propose would settle whether the pipeline is reproducible: if an independent run yields the same intermediate rates and yearly distributions, the central claim survives; if not, the dataset's value is substantially reduced. The paper is honest about many limitations (Section 5), and the code links are a real positive, but the lack of a formal sampling design and inclusion probabilities remains the main weakness. The reader's weakest_assumption and my concern overlap substantially; I would rate agreement as 'partial' because the reader emphasizes bias/weights while I emphasize exact reproducibility of the multi-stage pipeline. Both point to the same underlying issue: the dataset's construction is not fully specified or statistically supportable as a basis for inference. The verdict CONDITIONAL is appropriate: the paper should be accepted only on condition that the authors release the exact code version, the explicit list of the 107M/92M URL sets (at least as SURT hashes), and the design weights or inclusion probabilities for the final 27.3M URLs.","tokens_in":24867,"tokens_out":2861,"duration_ms":27484,"concrete_test":"Independently reconstruct the 107M-URL set from the 2021 ZipNum file using the precisely-specified extension heuristics (and the same SURT library version), then run the CDX API first-entry queries on a deterministic 1% subsample and compare the text/html rate and first-archive-year distribution to Tables 4 and 5 and Figure 8. If the reproduced rate and per-year counts deviate by more than a few percent, the reported 92M intermediate set is not reproducible and the central 27.3M claim is weakened; if they match, the concern is largely resolved.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central claim is the existence and composition of a 27.3M-URL, 3.8B-memento longitudinal sample. This claim requires the multi-stage pipeline to be reproducible and the stated per-year counts to be internally consistent. The most load-bearing gap is that the 107M-URL set (the actual 'likely HTML' URL set used for CDX queries) is never defined as an explicit enumeration. Section 4.5.1 says extension heuristics reduced 282.7M URLs to 222M likely-HTML URLs; Section 4.5.1 then says excluding popular domains 'reduced the dataset to 107 million likely HTML URLs'; Section 4.5.2 queries 40M and then 'the remaining 67 million' to total 107M, and reports 92M text/html. But Section 4.5.2 also states that the 40M evaluation found 88.12% precision, while the full 107M set was found to be 86.12% text/html. These rates are inconsistent with the reported raw counts: 88.12% of 40.3M gives 35.5M true positives (close to the 35.6M in Table 4), but 86.12% of 107M gives 92.15M, consistent with the 92M/92.6M. The discrepancy itself is minor, but the absence of the exact regex/heuristic version and the absence of an auditable ordered list of the 107M URLs means the pipeline cannot be independently reproduced from the paper alone. Tighter than this, the later downstream steps (root-URL upsampling in §4.7, popular-domain handling in §4.8, long-tail reduction in §4.9, and log-downsampling with hand-tuned K and C in Table 7) are each applied to the 92M set but the paper does not give a single algorithmic expression of how the final 27.3M URLs were selected from the 92M set, including the exact ordering of operations (e.g., whether popular-domain URLs were removed before or after the long-tail reduction, and how the 'reintroduced' popular-domain URLs were merged into the yearly samples).","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes the construction of the Not Your Parents' Web (NYPW) dataset, a longitudinal sample of URLs drawn from the Internet Archive's Wayback Machine. The authors start from the ZipNum index (every 6000th CDX entry), filter to 'likely HTML' URLs by filename-extension heuristics, verify MIME types via CDX API queries, group URLs by year of first capture, upsample early years by extracting root URLs from deep links, and then apply long-tail reduction and logarithmic downsampling to balance domain representation. The final dataset is reported to contain TimeMaps for 27.3 million URLs and 3.8 billion mementos spanning 1996-2021. The paper also documents several practical issues encountered, including session-ID aliases, wildcard URLs, warc/revisit records, and a SURT canonicalization bug.","tokens_in":25287,"tokens_out":8398,"duration_ms":94971,"significance":"If the dataset is reproducible and its biases are adequately characterized, it would be a valuable public resource for studying web page longevity, complementing prior studies that start from external URL corpora. The authors provide GitHub links for much of the sampling code, a clear funnel diagram of the pipeline, and an honest discussion of several known limitations (e.g., no true negatives/false negatives for the HTML filter, and acknowledged archival selectivity). The Memento terminology and the detailed account of TimeMap collection issues (revisit-record rehydration, wildcard URLs, SURT bugs) are useful lessons for the web-archiving community. However, because the paper's central claim is the existence and composition of the dataset, the internal numerical inconsistencies, the absence of false-negative analysis, and the lack of design weights are significant gaps that must be addressed before the dataset can be used for reliable downstream inference.","major_comments":[{"comment":"The counts at the critical 107M-to-92M stage are internally inconsistent, which undermines the reproducibility of the central claim. Section 4.5.1 says 115 million URLs from popular domains were excluded, but Section 4.8 says 111 million; the text states that 92 million of the 107 million likely-HTML URLs were text/html, while Table 5 sums to 92.6 million; and the reported precision of 88.12% for the 40M sample does not match Table 4's totals (35.6M/40.3M ≈ 88.3%). The exact version of the extension regexes/heuristics and an auditable enumeration of the 107M URL set are not provided. Please correct the counts, specify the exact regex and code version, and describe how the 107M set was formed and stored so that the pipeline is independently reconstructible.","section":"§4.5.1, §4.5.2, §4.8"},{"comment":"The extension-based HTML filter has no false-negative (recall) analysis. The authors explicitly state that the evaluation only measured precision on URLs already predicted as HTML, so the number of HTML pages excluded by the heuristic is unknown. Because the dataset is intended to represent 'web pages,' a missing recall estimate means the filtering stage introduces an unquantified selection bias. Please add a validation experiment on a random sample of URLs that do not pass the extension heuristics, query their MIME types via the CDX API, and report the estimated false-negative rate, ideally stratified by year and TLD.","section":"§4.5.2"},{"comment":"The multi-stage sample design (ZipNum probability proportional to memento count, popular-domain exclusion/reintroduction, root-URL upsampling, long-tail reduction, and per-year logarithmic downsampling with hand-tuned K and C) is not accompanied by inclusion probabilities or design weights. Without these, the final dataset cannot be treated as a probability sample for estimating quantities like 'how long does a web page last.' Moreover, the popular-domain reintroduction is described only for twitter.com, yet Table 8 shows many other popular domains in the final sample. Please provide per-URL inclusion weights (or the code and parameters sufficient to compute them), clarify the exact procedure used for all excluded popular domains, and state the target population and estimands the dataset is intended to support.","section":"§4.7-§4.9"},{"comment":"The downsampling parameters K and C, as well as the 900,000-domain long-tail reduction threshold, were chosen post hoc by trial-and-error to hit target sample sizes. The resulting per-year sample sizes and the reported 'heavy-tailed' domain/memento distributions are therefore partly artifacts of the tuning procedure rather than empirical properties of the archive. The Discussion should more explicitly separate design-induced properties from observational findings, and a sensitivity analysis around K, C, and the long-tail threshold would help readers understand how much the final dataset's shape depends on these choices.","section":"§4.9, Table 7, §5"}],"minor_comments":[{"comment":"In the paragraph following Equation (1), the text says 'The term log(N)+C sets a baseline...' but the equation is K·log(N)+C; please fix this to avoid confusion about whether C is inside the logarithm.","section":"§4.9"},{"comment":"The caption contains the typo 'response turnicated' and should read 'response truncated.'","section":"Figure 1 caption"},{"comment":"The sentence 'We decided to retain the root URLs that were first archived between 1996 and 2002' is unclear relative to Figure 9, which shows root-URL counts for all years; please clarify whether newly extracted root URLs outside 1996-2002 were discarded or included in subsequent downsampling steps.","section":"§4.7"},{"comment":"The relationship between the 292 million ZipNum entries, 285 million unique SURTs, and the 'sampled 285 million URLs' phrasing in the abstract should be stated more precisely, since duplicate removal occurs before URL conversion.","section":"§4.2, §4.5.1"},{"comment":"The paper mentions GitHub code and the NYPW project but does not provide a persistent identifier or explicit access instructions for the final 27.3M-URL dataset and the 1.4TB of TimeMaps; please add a data-availability statement with a DOI or archive.org identifier.","section":"Dataset access"},{"comment":"The text says '92 million out of 107 million URLs' while Table 5 sums to 92.6 million; use a single consistent figure (and corresponding percentage) throughout the manuscript.","section":"§4.5.2, Table 5"}],"recommendation":"major_revision","confidential_remarks":"This is a dataset/methods paper, and its value depends on the dataset being reproducible and usable for inference. The internal inconsistencies and the missing false-negative analysis and design weights are fixable but require real additional work and validation. I would encourage the editor to request a revised version that either provides design weights or explicitly reframes the dataset as a non-probability sample with clearly stated limitations for downstream inference. There is no indication of misconduct; the issues are about completeness and statistical rigor."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a serious look: this is a detailed, candid dataset paper that gives us the first complete pipeline for the NYPW sample, and the soft spots are mostly about reproducibility rather than the sample's existence.\n\nThe new stuff is real: the revisit-record rehydration at scale, the wildcard-URL handling, the SURT canonicalization bug, and the step-by-step accounting from ZipNum to 27.3M TimeMaps. The authors ship code links and are unusually honest about biases, acknowledging the sample is not representative and that archived web is not the live web. The precision table for extension heuristics is useful, and the log-downsampling equation is simple enough to evaluate.\n\nSoft spots: the pipeline is not fully reconstructible from the paper alone. The exact version of the extension regex is not pinned, and the ordering of operations after the 92M set (popular-domain removal vs long-tail reduction vs log-downsampling) is not given as a single algorithmic expression. The 107M set is never explicitly enumerated, so an independent audit would have to reconstruct it. There are also minor count inconsistencies (115M vs 111M for popular domains; 92M vs 92.6M text/html), but those don't threaten the scale claim. The bigger issue for downstream users is the absence of design weights or inclusion probabilities; the authors tell you what they did but not how to undo it for inference. The hand-tuned K and C are described, so that's transparent, but the lack of a false-negative analysis for the HTML filter means the sample's coverage of actual HTML pages is unknown.\n\nNone of this kills the paper. The central claim—that the dataset exists at the stated scale—is plausible from the internal counts, and the code links make verification possible. For a dataset description, that's the right bar. What I'd want in revision is a reproducibility appendix: exact versioned code, an explicit list of the 107M URLs or a hash, the ordered pipeline, and at least a statement about why design weights weren't provided.\n\nThe paper is for web archiving and web science people building or using large web samples. It deserves a serious referee—not desk reject—but the referee should push on reproducibility. I'd cite it if I were working with NYPW or similar data.","headline":"A candid and detailed dataset paper whose central claim holds up; the main work in review should be on reproducibility, not on whether the sample exists.","tokens_in":25878,"tokens_out":2106,"would_cite":true,"duration_ms":24494,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 26-year sample of 27.3 million archived URLs is built to measure how long web pages last.","keywords":["web archives","URL sampling","Wayback Machine","Memento protocol","longitudinal dataset","web page lifespan","CDX index","TimeMaps"],"falsifier":"Compute the per-URL inclusion probability under the full pipeline, multiplying the ZipNum every-6000th-line draw, the extension heuristic, the CDX MIME-type verification, the root-URL upsampling, and the log-domain downsampling; if those probabilities vary by more than an order of magnitude within a year and no weights are published with the dataset, then pooled lifespan estimates from the 27.3 million URLs cannot be taken as unbiased for the archived web.","tokens_in":1780,"feed_emoji":"🌐","tokens_out":2340,"duration_ms":112372,"temperature":0.7,"pith_summary":"This paper documents a large longitudinal sample of archived web pages: 27.3 million URLs with 3.8 billion archived captures, first archived between 1996 and 2021, drawn from the Wayback Machine. The authors' aim is to make a dataset that can revisit the question 'How long does a web page last?' at a scale far beyond earlier studies that tracked hundreds or millions of pages for short periods. They sample directly from the archive's own index rather than starting from an external URL list, and they walk through the sequence of filters and corrections that balance temporal and domain coverage. The paper is explicit that the result is a sample of the archived web, carrying the archive's own biases, not a representative sample of the live web.","feed_headline":"27.3M URLs and 3.8B captures revisit web lifespan","feed_subtitle":"The Wayback Machine sample spans 1996–2021 with about a million pages per year.","key_machinery":"The central machinery is the ZipNum index: a compressed form of the Wayback Machine's CDX index containing every 6000th entry, which makes it possible to sample URLs without scanning hundreds of billions of records. Sampling from it makes a URL's inclusion probability proportional to its memento count, with URLs having more than 6,000 captures guaranteed inclusion; the paper then layers a sequence of corrections on top of that frame, including extension-based HTML filtering, CDX API MIME-type verification, root-URL upsampling for early years, and logarithmic downsampling per domain using $\\min(N, \\text{round}(K \\cdot \\log(N) + C))$. The unit of the final dataset is the Memento-protocol TimeMap, the full list of archived captures for one original URL, and collecting these TimeMaps for all 27.3 million URLs yields the 3.8 billion mementos.","core_discovery":"The central claim is that it is feasible to build a web-lifespan dataset of 27.3 million URLs and 3.8 billion mementos by taking every 6000th line of the Wayback Machine's CDX index, filtering for likely HTML by filename extension, confirming text/html MIME types through the CDX API, upsampling sparse early years with root URLs extracted from deep links, and log-downsampling domains so popular sites do not dominate. The final sample has roughly one million HTML URLs per year across 26 years and 7 million unique hosts, each with its full TimeMap. The authors present the resulting dataset as the foundation for estimating the size, nature, and prevalence of the publicly archivable web, and for re-examining page lifespan.","pith_inferences":["The paper does not provide inclusion probabilities or sampling weights for the final 27.3 million URLs, so any pooled estimate of how long a web page lasts will need a separate design-based analysis or sensitivity study before it can be treated as a population quantity; this is my inference from the described procedure.","Because the extension-based HTML filter was validated only for precision and never for recall, the sample's coverage of extensionless or unusual HTML URLs is unknown; checking recall would require querying the CDX API for a random set of URLs that failed the filter.","A testable extension of the same pipeline would apply the sampling stages to non-HTML MIME types such as images, CSS, and JavaScript, letting the lifespan question extend to embedded resources that are part of the user experience.","Comparing survival curves from this archive-driven sample with curves from small manually revisited random samples would quantify how much archive selection changes lifespan estimates; this comparison is not performed in the paper."],"forward_implications":["Lifespan and survival-curve estimates for HTML pages first archived in each year from 1996 to 2021 can be computed from a dataset of roughly one million URLs per year, something earlier studies could not support.","Because the dataset stores full TimeMaps, analyses can look beyond first and last capture to archiving intensity, gaps between captures, and reuse of the same content over time.","The logarithmic downsampling keeps the heavy-tailed distribution of URLs per domain while preventing a few platforms from dominating, so domain-level diversity is retained in each yearly bin.","The documented failure modes, including wildcard asterisk URLs, revisit records with missing status codes, session-ID aliases, and SURT canonicalization errors, give future archive-sampling projects a concrete checklist.","Researchers studying link rot and reference rot can use this sample as a comparison point for URL sets drawn from scholarly articles or news stories."],"supporting_citations":[{"why":"Defines the ZipNum index file, the sampling frame used to draw every 6000th entry from the CDX index.","marker":"[64]"},{"why":"Explains how sampling from a CDX index biases toward URLs with many mementos, the central bias the paper's later steps mitigate.","marker":"[47]"},{"why":"Provides the CDX API used to query first capture dates, MIME types, and full TimeMaps.","marker":"[15]"},{"why":"Specifies the space-separated CDX line format that the paper parses for timestamps, MIME types, and status codes.","marker":"[13]"},{"why":"Defines the Memento protocol terms URI-R, URI-M, URI-T, and Memento-Datetime, including the TimeMap structure that makes up the final dataset.","marker":"[26]"},{"why":"Supplies the reservoir sampling algorithm used to select a random subset of URLs per domain after downsampling.","marker":"[66]"},{"why":"Provides the concrete reservoir sampling utility the paper uses for randomized URL selection.","marker":"[5]"},{"why":"Defines SURT canonicalization, which the paper uses to sort and group URLs and whose preprocessing triggered a deduplication bug.","marker":"[14]"}],"fun_headline_variants":["3.8B archived pages, 27M URLs: web lifespan revisited","Web page lifespan reexamined via 26-year Wayback sample","How long do web pages last? 27M Wayback URLs provide answers","Sampling every 6000th Wayback URL yields 3.8B pages for lifespan study"],"cache_read_input_tokens":27776,"weakest_assumption_plain":"The dataset can support lifespan estimates only if the biases introduced by the multi-stage sampling procedure, namely memento-count-weighted ZipNum selection, extension-based HTML filtering, early-year root-URL upsampling, and hand-tuned log downsampling, are small enough or documented enough to be corrected for.","fun_headline_variants_meta":{"raw":{"variants":["3.8B archived pages, 27M URLs: web lifespan revisited","Web page lifespan reexamined via 26-year Wayback sample","How long do web pages last? 27M Wayback URLs provide answers","Sampling every 6000th Wayback URL yields 3.8B pages for lifespan study"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000316,"raw_usage":{"total_tokens":1848,"prompt_tokens":1061,"completion_tokens":787,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":677,"completion_tokens_details":{"reasoning_tokens":704}},"tokens_in":677,"tokens_out":787,"duration_ms":9055,"temperature":1.0,"reasoning_tokens":704,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:48:20.338009+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the per-URL inclusion probability under the full pipeline, multiplying the ZipNum every-6000th-line draw, the extension heuristic, the CDX MIME-type verification, the root-URL upsampling, and the log-domain downsampling; if those probabilities vary by more than an order of magnitude within a year and no weights are published with the dataset, then pooled lifespan estimates from the 27.3 million URLs cannot be taken as unbiased for the archived web.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the ZipNum index file, the sampling frame used to draw every 6000th entry from the CDX index."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the CDX API used to query first capture dates, MIME types, and full TimeMaps."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Specifies the space-separated CDX line format that the paper parses for timestamps, MIME types, and status codes."},{"cited_title":"Nelson, Robert Sanderson, Lyudmila Balakireva, Scott Ainsworth, and Harihar Shankar","cited_arxiv_id":null,"evidence_quote":"Defines the Memento protocol terms URI-R, URI-M, URI-T, and Memento-Datetime, including the TimeMap structure that makes up the final dataset."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the reservoir sampling algorithm used to select a random subset of URLs per domain after downsampling."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the concrete reservoir sampling utility the paper uses for randomized URL selection."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines SURT canonicalization, which the paper uses to sort and group URLs and whose preprocessing triggered a deduplication bug."}],"review_version":1}