{"id":"2794ddba-8024-488c-9214-4c4b49b3db2d","arxiv_id":"2412.00479","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Scraping URLs from browsing histories misrepresents the news content users were exposed to, with at least one third of article content unrecoverable and biases concentrated in specific news categories.","lead":"This study compared the web pages that 534 German internet users actually saw with versions of the same pages obtained later by automated scraping, and found large, category-dependent gaps. The gaps are driven mainly by the different collection environment, not by waiting 30 to 90 days, which matters because many research datasets are built by scraping URL histories.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The in-situ extension is the reference for every discrepancy; if its own capture is incomplete or state-dependent, the 33.8% estimate is inflated, and the 30-case validation is too thin to rule this out.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the in-situ extension is treated as ground truth for what the participant saw, but its validity is not independently established. This is the most load-bearing issue because every quantity in the paper—the 33.8% lower bound, the environment-vs-delay decomposition, and the category-specific bias estimates—is defined as a distance from the extension's capture. If the extension misses dynamically loaded content, records a post-navigation state, or reflects session-specific elements that the participant did not actually view, then the measured gap is not attributable to ex-situ scraping alone. The paper provides useful supporting evidence: two waves, a large number of visits, low HTTP error rates, and a conservative focus on news articles whose content is relatively stable. The 30-case manual check for dynamic loading is a reasonable start, but it is too small and too narrowly targeted to validate the full reference distribution, and the keyword search found a non-negligible set of potentially affected pages. Other concerns, such as the absence of clustered standard errors or the outcome-informed refresh threshold, affect precision or framing but do not threaten the core existence of a gap; even with clustering, a mean distance of 33.8% would likely remain statistically significant. The ground-truth concern, by contrast, questions whether the gap is actually due to ex-situ scraping or to artifacts of the reference measurement. Thus the reader's conditional verdict is appropriate: the paper should be accepted only with the explicit condition that the in-situ reference be validated against an independent capture, or that the claims be softened to describe a difference between two capture methods rather than a loss of content attributable to ex-situ scraping.","tokens_in":21039,"tokens_out":7625,"duration_ms":85642,"concrete_test":"In a random sample of about 500 news-article visits, instrument the extension to capture the DOM at three points: immediately on navigation, at the first substantial scroll/read event, and on page leave. For the same URLs, run a headless browser with the participant's cookies and session state to produce an independent render. Compute the in-situ start-vs-leave Levenshtein distance and the leave-vs-headless distance. If the average start-vs-leave distance or the average leave-vs-headless distance is nontrivial (e.g., more than 5 percentage points), re-estimate the 33.8% lower bound after excluding visits where the reference capture is unstable; if the estimate changes materially, the headline should be reframed as a comparison of two capture methods rather than a pure measure of ex-situ scraping loss.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is defined against the in-situ extension capture: every Levenshtein distance compares ex-situ scraped HTML to the HTML the extension happened to record. If that record is not a faithful representation of what the participant actually saw, the measured 'disparity' is partly an artifact of the reference, not a scraping loss. The extension transmits content only when the participant leaves the page, so pages that update during a visit, lazy-load content after scrolling, or change after cookie-consent interaction are captured in their final state, not necessarily in the state the participant read. The paper's own validation of dynamic loading is limited to a manual sample of 30 cases selected at distances between .2 and .8, plus a keyword search for 'javascript'/'js'; this does not cover the full distribution, especially high-distance cases where an incomplete in-situ capture would look identical to a scraping failure (empty text). A keyword search also found 487 URLs containing such messages, mostly from subscription-based outlets, which is not negligible. Because the headline 33.8% is a mean distance, even a small number of false-failure cases could move it materially, and all category-level bias estimates inherit the same problem since the ML classifier is trained and applied using the in-situ text as ground truth. The concern is not that the authors are careless; it is that the reference instrument itself is assumed valid rather than independently verified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper uses browser-extension data from 534 German desktop users (34,108 news article visits) to quantify the discrepancy between content captured in the user's browser (in-situ) and content scraped from the logged URLs on the authors' server infrastructure (ex-situ), with scraping delays of 0, 30, 60, and 90 days. The outcome is a normalized Levenshtein distance between the in-situ and ex-situ documents, computed for cleaned text, raw text, and full HTML. The central empirical claims are that the cleaned-text mean distance is about 33.8% in wave 1 and up to 46.6% in the post-hoc subsample; that the ex-situ environment, rather than temporal delay, is the main source of the discrepancy, with a 90-day delay adding only about 6.5 percentage points; and that the errors are not uniformly distributed across news categories, regardless of whether categories are derived from domains, URL paths, or a machine-learning content classifier. The paper closes with recommendations for interpreting and mitigating the biases in ex-situ scraping.","tokens_in":21204,"tokens_out":8367,"duration_ms":85210,"significance":"If the estimates hold, the paper provides a valuable empirical benchmark for a large body of web-tracking research that scrapes URLs to measure news exposure. It goes beyond prior work such as Dahlke et al. (2023) by comparing ex-situ scraping directly against in-situ captures in the user's environment, by separating environment effects from time-delay effects, and by showing that the errors are non-random with respect to news categories. The strengths include the large naturalistic dataset, the two-wave design, the controlled delay manipulation, the use of multiple text representations, the transparent HTTP-error accounting, and the explicit lower-bound framing of the headline estimate. The main risks are the assumed validity of the in-situ browser extension as ground truth, the outcome-informed choice of the page-refresh threshold, and the absence of clustered inference for nested observations; these are addressable in revision and do not, in my view, invalidate the core comparison.","major_comments":[{"comment":"Every discrepancy in the paper is a Levenshtein distance between a scraped ex-situ document and the HTML recorded by the browser extension, so the extension must be a faithful record of what the participant actually saw. The extension sends the HTML only after the participant leaves the page, and pages that update, lazy-load, or change after cookie or paywall interactions are captured in their final state, not necessarily in the state that was read. The validation in the Results section (a manual sample of 30 cases in the 0.2-0.8 distance range plus a 'javascript'/'js' keyword search that itself flags 487 URLs, mostly from subscription-based outlets) is too thin to rule out non-negligible under-capture, especially among the distance-1 cases where an incomplete in-situ capture would look identical to a scraping failure. Because 33.8% is a mean, a small number of false-failure cases can move it materially, and the category-level estimates inherit the same dependence. The authors should provide an independent validation of the extension capture (for example, a screen-level or rendered-page comparison on a stratified sample that includes high-distance cases) and report a robustness analysis excluding or recoding the flagged cases.","section":"Data collection; Results, Figure 1D"},{"comment":"The choice of 20 seconds as the page-refresh threshold is explicitly made by minimizing the measured Levenshtein distances, which is an outcome-informed modeling decision. This is defensible for a lower-bound estimate, but it needs to be transparent: the paper should report the threshold search space, the objective function, and a sensitivity analysis of the main estimates and category-level results to thresholds around 20 seconds. As written, readers cannot separate the empirical disparity from the design choice, and the headline 33.8% is partly a function of this decision.","section":"Data filtering; Appendix S16"},{"comment":"The statistical analyses model 34,108 article visits as independent observations with linear regression and ANOVA, but the data are nested in participants (412) and news domains (hundreds), and per-participant and per-domain residuals are almost certainly correlated, as illustrated by the spiegel.de blocking episode. Without participant- and domain-level random effects or clustered standard errors, the reported 95% and 99% confidence intervals, Tukey contrasts, and chi-square tests likely overstate precision. The paper should re-estimate the marginal means and pairwise differences with multilevel models or cluster-robust variance estimation.","section":"Statistics"},{"comment":"The paper uses 'lower-bound estimate' in two potentially incompatible ways. The abstract states that at least 33.8% of news exposure content cannot be determined by static scraping, while the Discussion says that 46.6% 'is a better lower-bounded estimate for a more intensive and prolonged scraping process' and describes the environment effect as 'up to 46.6%'. If the discrepancy is intensity-dependent, then 33.8% is a lower bound for low-intensity scraping and 46.6% for higher-intensity scraping, but this conditional reading should be stated explicitly. The authors should define what 'lower bound' means, report the intensity-dependent range, and adjust the abstract so that it is not read as a universal bound across scraping intensities.","section":"Abstract; Discussion"},{"comment":"The only strategy that aligns the ex-situ and in-situ category distributions (removing the Non-thematic, Domestic Commerce, and Technology categories) is implemented with the biased categories identified using the in-situ distribution. Since a researcher relying on ex-situ data does not have access to that distribution, RQ4 does not demonstrate a deployable ex-situ correction; it diagnoses the source of the bias. The Discussion acknowledges this caveat, but the RQ4 framing and the recommendations section present category removal as a mitigation strategy. Please reframe RQ4 as diagnostic, or develop a procedure that identifies the biased categories from ex-situ features alone (for example, automated detection of paywall, login, and cookie-consent markers).","section":"Results, 'Mitigating biases'; Discussion"}],"minor_comments":[{"comment":"The word 'Levenshtein' is misspelled as 'Levensthein' in the Statistics section and in the Figure 1 caption; please correct throughout.","section":"Statistics; Figure 1 caption"},{"comment":"The caption labels the three bottom-row panels as '(D) removing the two most problematic domains, (E) excluding the Non-thematic content category and (E) removing the biased categories'; the third label should be (F).","section":"Figure 4 caption"},{"comment":"The text refers to 'spiegel.com' in the description of Figure 4D, while the rest of the paper refers to 'spiegel.de'; please standardize the domain name.","section":"Results, 'Mitigating biases'"},{"comment":"The paper refers to 'near real-time' ex-situ scraping, but the ex-situ request is triggered only after the participant leaves the page, and the in-situ capture also occurs at leave time; a brief definition of what 'near real-time' means here would improve precision.","section":"Data filtering; Methods"},{"comment":"The statement that 'anonymized data sets can be shared to reproduce the presented analysis and plots upon request' is weaker than the reproducibility standards common in this literature; consider a fuller replication statement or deposit of analysis code and derived data where ethically possible.","section":"Data protection"}],"recommendation":"major_revision","confidential_remarks":"This is a strong empirical contribution to a methodologically important problem, and the authors are clearly aware of several limitations. The main issue for me is not the central measurement design but the strength of the inference drawn from it: the in-situ reference instrument needs independent validation, the nesting of observations should be modeled, and the lower-bound language should be made conditional on scraping intensity. These are fixable within the manuscript's scope. The paper is a good fit for the journal's audience. I saw no citation or novelty concerns."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Paper does something new: benchmarks ex-situ scraping against in-situ captured HTML with per-visit timestamps and 0/30/60/90 day delays, on a large German tracking panel. The headline estimate—at least 33.8% of news content is unrecoverable by static scraping—is a direct measurement, not a derivation. The finding that the ex-situ environment matters more than delay, with 90 days adding only ~6.5 percentage points, is the kind of result that should change how people interpret scraped URL-logged data.\n\nThe design is generally careful. URLs were randomly assigned to two scrapers, requests were spread over time to mimic user traffic, HTTP errors were tracked, and the identification of news articles was validated with a weighted F1 of .957. The category-level analyses are useful: the paywall-driven misclassification of Domestic Commerce pieces as Non-thematic, and the role of login/paywall pages in 73% of the high-distance errors, give concrete mechanisms, not just a mean distance.\n\nThe soft spots are real but mostly fixable. The in-situ extension is the reference for every discrepancy, and its captures happen when the participant leaves the page; the validation for dynamic loading is a manual sample of 30 cases plus a keyword search. That is thin for a result with this level of precision. Relatedly, the 20-second refresh threshold was selected by minimizing measured Levenshtein distance, which is outcome-informed; and the regression models don't cluster by participant or domain. None of these sink the central claim, but they should be handled in revision. The debiasing exercise in RQ4 is circular in the way the reader says—the biased categories are identified using the in-situ distribution and then removed—but the authors say so themselves, and it doesn't contribute to the headline estimate.\n\nThe data and code are not public, only available on request; for a methodological paper, releasing anonymized artifacts would materially raise its value.\n\nWho is it for? Anyone using URL-logged browsing histories to measure news exposure, and researchers building error-correction frameworks. It deserves a serious referee. I'd want the robustness analyses around the threshold and the ground-truth validation strengthened, but the core measurement is solid enough to engage with.","headline":"Direct in-situ versus ex-situ benchmark on a large tracking panel with per-visit timestamps; the 33.8% lower-bound discrepancy is real and important, though the ground-truth assumption and a few methodological choices need tighter robustness work.","tokens_in":21810,"tokens_out":2039,"would_cite":true,"duration_ms":20197,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that static web scraping of URL-logged browsing histories fails to recover a substantial, non-random share of online news exposure—at least 33.8%—with the ex-situ collection environment, not the delay, as the main culprit.","keywords":["web scraping","web tracking","online news exposure","measurement error","in-situ data collection","ex-situ data collection","Levenshtein distance","paywalls"],"falsifier":"Re-run the comparison with a screen-recording or browser-device-protocol capture of the visible rendered page on the same 34,108 visits and compute Levenshtein distances against the extension-captured HTML; if those reference-versus-extension distances average near zero, the extension is valid, but if they approach the 33.8% magnitude, the ground-truth baseline and therefore the headline number are called into question.","tokens_in":20757,"feed_emoji":"🕸️","tokens_out":5403,"duration_ms":49633,"temperature":0.7,"pith_summary":"Many studies of online news exposure work from URL logs of browsing histories and infer what people read by scraping those URLs later from a server. This paper tests that practice against a direct record of what participants actually saw: HTML captured by a browser extension at the moment of the visit. It finds that even scraping in near real time from a remote server cannot recover at least 33.8% of the content of participants' news-article visits, that delaying the scrape by 90 days adds only another 6.5 percentage points of loss, and that the losses concentrate in certain news categories and outlets. If these numbers hold, published web-tracking studies that scraped URLs carry substantial, non-random measurement error in content-based measures of news exposure.","feed_headline":"Web scraping hides at least 33.8% of news exposure","feed_subtitle":"URL-logged news studies miss a third of article content; delays matter less than the scraping environment.","key_machinery":"The central measuring device is the normalized Levenshtein distance between the in-situ and ex-situ versions of each page, computed on three content representations: cleaned text (boilerplate removed), raw text, and full HTML. The in-situ baseline comes from a browser extension that uploads the page's HTML when the participant leaves a tab; the ex-situ arm assigns each URL at random to one of two standard static scraping libraries and fetches it at four delays. Around this comparison, the paper layers three independent news categorizations—outlet type from a domain list, topic from URL-path keywords, and content topic from a press-release-trained classifier—to test whether scraping losses are uniform across news types. The key contrast is that all three categorization schemes show systematic, non-uniform losses, which is what turns a generic data-quality concern into a validity threat for content-based exposure measures.","core_discovery":"Using 34,108 news-article visits from 412 German participants, the paper compares HTML captured in the user's browser (in-situ) with HTML obtained by scraping the same URLs from dedicated server infrastructure (ex-situ), at delays of 0, 30, 60, and 90 days. The discrepancy, measured as normalized Levenshtein distance between cleaned texts, has an estimated marginal mean of about 40.1% across waves; the paper's lower-bound estimate for the share of news exposure content that static scraping cannot determine is 33.8%. The ex-situ collection environment is the dominant source of loss; time delay contributed about 5.5 percentage points at 30 days and only about 6.5 at 90 days, mostly front-loaded. The errors are not random: they vary systematically across outlet types, URL categories, and machine-learned content categories. A large share of the worst mismatches trace to pages requiring user interaction, such as paywalls and login prompts, which returned boilerplate or inaccessible text to the scraper.","pith_inferences":["The authors stop short of quantifying the bias for non-news pages; an extension of the same comparison to homepages, search-engine results, and social-media feeds would probably reveal larger ex-situ losses, since those pages are more personalized and dynamic.","One testable extension is to train an interaction-detection classifier on page features such as HTML structure and login or payment keywords, and use its output to weight or exclude scraped pages; the paper's own 73% attribution to paywall and login pages suggests such a classifier could recover much of the 33.8%.","The finding also implies that studies comparing desktop and mobile news diets may conflate device differences with scraping loss, since the ex-situ environment is constant while the in-situ user environment differs.","If the result generalizes beyond Germany, it would justify re-analysis of past web-tracking datasets by re-scraping with dynamic interaction and comparing category distributions; significant shifts would indicate prior substantive conclusions need robustness checks."],"forward_implications":["Existing URL-scraping studies of news exposure likely under-represent or misattribute content from paywalled and login-gated outlets, and the distortion is correlated with news category.","Correcting for scrape timing alone cannot fix the problem; the dominant error is introduced by the collection environment before any delay.","Content-based classifications built on scraped text inherit the non-random gaps, so measures such as political topic shares or selective-exposure estimates can be biased.","Capturing HTML in the participant's environment, or supplementing scraper text with interaction-detection heuristics, is the direct remedy; domain removal and classifier thresholding did not fix the bias.","The 33.8% figure is a lower bound for static scraping of news articles; other page types with more dynamic or personalized content are expected to fare worse."],"supporting_citations":[{"why":"Introduces the content- and long-tail sensitive browser-extension tracking approach used here as the in-situ ground truth.","marker":"Adam et al., 2024"},{"why":"Previous study of time delays in scraping URL-logged news, which this paper extends by adding an in-situ baseline and shorter delays.","marker":"Dahlke et al., 2023"},{"why":"Supplies the 679-domain list and outlet-type classification used to identify German news domains and categorize them by type.","marker":"Stier et al., 2020"},{"why":"Trained press-release classifier whose content categories are used to compare in-situ and ex-situ topic distributions.","marker":"Erfort et al., 2023"},{"why":"Describes the trafilatura scraping and text-extraction library, one of the two ex-situ scrapers and the source of cleaned text.","marker":"Barbaresi, 2021"},{"why":"Simulation showing how measurement error in content analysis and media-use reports attenuates media-effect estimates, motivating the validity concern.","marker":"Scharkow & Bachl, 2017"},{"why":"Argues that web-scraped data can be biased by the scraping environment, which the paper operationalizes as ex-situ versus in-situ.","marker":"Foerderer, 2023"}],"fun_headline_variants":["Web scraping misses 33.8% of news content","News scraping bias stems from environment, not delay","URL-scraped news data skewed by collection method","Scraping news: the how matters more than the when"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The in-situ browser extension is treated as the true record of what participants saw; every ex-situ discrepancy is measured against it, so if the extension misses dynamically loaded content, captures a page after the user navigated away, or is itself altered by other browser extensions, part of the measured disparity is a measurement artifact rather than scraping error.","fun_headline_variants_meta":{"raw":{"variants":["Web scraping misses 33.8% of news content","News scraping bias stems from environment, not delay","URL-scraped news data skewed by collection method","Scraping news: the how matters more than the when"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000522,"raw_usage":{"total_tokens":2540,"prompt_tokens":978,"completion_tokens":1562,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":594,"completion_tokens_details":{"reasoning_tokens":1497}},"tokens_in":594,"tokens_out":1562,"duration_ms":13152,"temperature":1.0,"reasoning_tokens":1497,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T05:21:22.123879+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the comparison with a screen-recording or browser-device-protocol capture of the visible rendered page on the same 34,108 visits and compute Levenshtein distances against the extension-captured HTML; if those reference-versus-extension distances average near zero, the extension is valid, but if they approach the 33.8% magnitude, the ground-truth baseline and therefore the headline number are called into question.","supporting_citations":[{"cited_title":"Improving the quality of individual-level online information tracking: challenges of existing approaches and introduction of a new content- and long-tail sensitive academic solution","cited_arxiv_id":"2403.02931","evidence_quote":"Introduces the content- and long-tail sensitive browser-extension tracking approach used here as the in-situ ground truth."}],"review_version":1}