Beyond time delays: How web scraping distorts measures of online news consumption

Felix Schmidt; Frank Mangold; Judith Gilsbach; Roberto Ulloa; Sebastian Stier

REVIEW 5 major objections 5 minor 3 references

Beyond time delays: How web scraping distorts measures of online news consumption

T0 review · 5 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper argues that static web scraping of URL-logged browsing histories fails to recover a substantial, non-random share of online news exposure—at least 33.8%—with the ex-situ collection environment, not the delay, as the main culprit.

desk verdict Direct in-situ versus ex-situ benchmark on a large tracking panel with per-visit timestamps; the 33.8% lower-bound discrepancy is real and important, though the ground-truth assumption and a few methodological choices need tighter robustness work. read the letter →

arxiv 2412.00479 v1 pith:Z2TYEIIZ submitted 2024-11-30 cs.CY

Roberto Ulloa , Frank Mangold , Felix Schmidt , Judith Gilsbach , Sebastian Stier This is my paper · ORCID

classification cs.CY

keywords webscrapingtrackingonlinenewsexposuremeasurementerrorin-situdatacollectionex-situLevenshteindistancepaywalls

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Many studies of online news exposure work from URL logs of browsing histories and infer what people read by scraping those URLs later from a server. This paper tests that practice against a direct record of what participants actually saw: HTML captured by a browser extension at the moment of the visit. It finds that even scraping in near real time from a remote server cannot recover at least 33.8% of the content of participants' news-article visits, that delaying the scrape by 90 days adds only another 6.5 percentage points of loss, and that the losses concentrate in certain news categories and outlets. If these numbers hold, published web-tracking studies that scraped URLs carry substantial, non-random measurement error in content-based measures of news exposure.

What carries the argument

The central measuring device is the normalized Levenshtein distance between the in-situ and ex-situ versions of each page, computed on three content representations: cleaned text (boilerplate removed), raw text, and full HTML. The in-situ baseline comes from a browser extension that uploads the page's HTML when the participant leaves a tab; the ex-situ arm assigns each URL at random to one of two standard static scraping libraries and fetches it at four delays. Around this comparison, the paper layers three independent news categorizations—outlet type from a domain list, topic from URL-path keywords, and content topic from a press-release-trained classifier—to test whether scraping losses are uniform across news types. The key contrast is that all three categorization schemes show systematic, non-uniform losses, which is what turns a generic data-quality concern into a validity threat for content-based exposure measures.

What would settle it

Re-run the comparison with a screen-recording or browser-device-protocol capture of the visible rendered page on the same 34,108 visits and compute Levenshtein distances against the extension-captured HTML; if those reference-versus-extension distances average near zero, the extension is valid, but if they approach the 33.8% magnitude, the ground-truth baseline and therefore the headline number are called into question.

Watch

Extended reading notes

Core claim

Using 34,108 news-article visits from 412 German participants, the paper compares HTML captured in the user's browser (in-situ) with HTML obtained by scraping the same URLs from dedicated server infrastructure (ex-situ), at delays of 0, 30, 60, and 90 days. The discrepancy, measured as normalized Levenshtein distance between cleaned texts, has an estimated marginal mean of about 40.1% across waves; the paper's lower-bound estimate for the share of news exposure content that static scraping cannot determine is 33.8%. The ex-situ collection environment is the dominant source of loss; time delay contributed about 5.5 percentage points at 30 days and only about 6.5 at 90 days, mostly front-loaded. The errors are not random: they vary systematically across outlet types, URL categories, and machine-learned content categories. A large share of the worst mismatches trace to pages requiring user interaction, such as paywalls and login prompts, which returned boilerplate or inaccessible text to the scraper.

Load-bearing premise

The in-situ browser extension is treated as the true record of what participants saw; every ex-situ discrepancy is measured against it, so if the extension misses dynamically loaded content, captures a page after the user navigated away, or is itself altered by other browser extensions, part of the measured disparity is a measurement artifact rather than scraping error.

Editorial extensions

If this is right

Existing URL-scraping studies of news exposure likely under-represent or misattribute content from paywalled and login-gated outlets, and the distortion is correlated with news category.
Correcting for scrape timing alone cannot fix the problem; the dominant error is introduced by the collection environment before any delay.
Content-based classifications built on scraped text inherit the non-random gaps, so measures such as political topic shares or selective-exposure estimates can be biased.
Capturing HTML in the participant's environment, or supplementing scraper text with interaction-detection heuristics, is the direct remedy; domain removal and classifier thresholding did not fix the bias.
The 33.8% figure is a lower bound for static scraping of news articles; other page types with more dynamic or personalized content are expected to fare worse.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The authors stop short of quantifying the bias for non-news pages; an extension of the same comparison to homepages, search-engine results, and social-media feeds would probably reveal larger ex-situ losses, since those pages are more personalized and dynamic.
One testable extension is to train an interaction-detection classifier on page features such as HTML structure and login or payment keywords, and use its output to weight or exclude scraped pages; the paper's own 73% attribution to paywall and login pages suggests such a classifier could recover much of the 33.8%.
The finding also implies that studies comparing desktop and mobile news diets may conflate device differences with scraping loss, since the ex-situ environment is constant while the in-situ user environment differs.
If the result generalizes beyond Germany, it would justify re-analysis of past web-tracking datasets by re-scraping with dynamic interaction and comparing category distributions; significant shifts would indicate prior substantive conclusions need robustness checks.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Paper does something new: benchmarks ex-situ scraping against in-situ captured HTML with per-visit timestamps and 0/30/60/90 day delays, on a large German tracking panel. The headline estimate—at least 33.8% of news content is unrecoverable by static scraping—is a direct measurement, not a derivation. The finding that the ex-situ environment matters more than delay, with 90 days adding only ~6.5 percentage points, is the kind of result that should change how people interpret scraped URL-logged data.

The design is generally careful. URLs were randomly assigned to two scrapers, requests were spread over time to mimic user traffic, HTTP errors were tracked, and the identification of news articles was validated with a weighted F1 of .957. The category-level analyses are useful: the paywall-driven misclassification of Domestic Commerce pieces as Non-thematic, and the role of login/paywall pages in 73% of the high-distance errors, give concrete mechanisms, not just a mean distance.

The soft spots are real but mostly fixable. The in-situ extension is the reference for every discrepancy, and its captures happen when the participant leaves the page; the validation for dynamic loading is a manual sample of 30 cases plus a keyword search. That is thin for a result with this level of precision. Relatedly, the 20-second refresh threshold was selected by minimizing measured Levenshtein distance, which is outcome-informed; and the regression models don't cluster by participant or domain. None of these sink the central claim, but they should be handled in revision. The debiasing exercise in RQ4 is circular in the way the reader says—the biased categories are identified using the in-situ distribution and then removed—but the authors say so themselves, and it doesn't contribute to the headline estimate.

The data and code are not public, only available on request; for a methodological paper, releasing anonymized artifacts would materially raise its value.

Who is it for? Anyone using URL-logged browsing histories to measure news exposure, and researchers building error-correction frameworks. It deserves a serious referee. I'd want the robustness analyses around the threshold and the ground-truth validation strengthened, but the core measurement is solid enough to engage with.

Referee Report

5 major / 5 minor

Summary. The paper uses browser-extension data from 534 German desktop users (34,108 news article visits) to quantify the discrepancy between content captured in the user's browser (in-situ) and content scraped from the logged URLs on the authors' server infrastructure (ex-situ), with scraping delays of 0, 30, 60, and 90 days. The outcome is a normalized Levenshtein distance between the in-situ and ex-situ documents, computed for cleaned text, raw text, and full HTML. The central empirical claims are that the cleaned-text mean distance is about 33.8% in wave 1 and up to 46.6% in the post-hoc subsample; that the ex-situ environment, rather than temporal delay, is the main source of the discrepancy, with a 90-day delay adding only about 6.5 percentage points; and that the errors are not uniformly distributed across news categories, regardless of whether categories are derived from domains, URL paths, or a machine-learning content classifier. The paper closes with recommendations for interpreting and mitigating the biases in ex-situ scraping.

Significance. If the estimates hold, the paper provides a valuable empirical benchmark for a large body of web-tracking research that scrapes URLs to measure news exposure. It goes beyond prior work such as Dahlke et al. (2023) by comparing ex-situ scraping directly against in-situ captures in the user's environment, by separating environment effects from time-delay effects, and by showing that the errors are non-random with respect to news categories. The strengths include the large naturalistic dataset, the two-wave design, the controlled delay manipulation, the use of multiple text representations, the transparent HTTP-error accounting, and the explicit lower-bound framing of the headline estimate. The main risks are the assumed validity of the in-situ browser extension as ground truth, the outcome-informed choice of the page-refresh threshold, and the absence of clustered inference for nested observations; these are addressable in revision and do not, in my view, invalidate the core comparison.

major comments (5)

[Data collection; Results, Figure 1D] Every discrepancy in the paper is a Levenshtein distance between a scraped ex-situ document and the HTML recorded by the browser extension, so the extension must be a faithful record of what the participant actually saw. The extension sends the HTML only after the participant leaves the page, and pages that update, lazy-load, or change after cookie or paywall interactions are captured in their final state, not necessarily in the state that was read. The validation in the Results section (a manual sample of 30 cases in the 0.2-0.8 distance range plus a 'javascript'/'js' keyword search that itself flags 487 URLs, mostly from subscription-based outlets) is too thin to rule out non-negligible under-capture, especially among the distance-1 cases where an incomplete in-situ capture would look identical to a scraping failure. Because 33.8% is a mean, a small number of false-failure cases can move it materially, and the category-level estimates inherit the same dependence. The authors should provide an independent validation of the extension capture (for example, a screen-level or rendered-page comparison on a stratified sample that includes high-distance cases) and report a robustness analysis excluding or recoding the flagged cases.
[Data filtering; Appendix S16] The choice of 20 seconds as the page-refresh threshold is explicitly made by minimizing the measured Levenshtein distances, which is an outcome-informed modeling decision. This is defensible for a lower-bound estimate, but it needs to be transparent: the paper should report the threshold search space, the objective function, and a sensitivity analysis of the main estimates and category-level results to thresholds around 20 seconds. As written, readers cannot separate the empirical disparity from the design choice, and the headline 33.8% is partly a function of this decision.
[Statistics] The statistical analyses model 34,108 article visits as independent observations with linear regression and ANOVA, but the data are nested in participants (412) and news domains (hundreds), and per-participant and per-domain residuals are almost certainly correlated, as illustrated by the spiegel.de blocking episode. Without participant- and domain-level random effects or clustered standard errors, the reported 95% and 99% confidence intervals, Tukey contrasts, and chi-square tests likely overstate precision. The paper should re-estimate the marginal means and pairwise differences with multilevel models or cluster-robust variance estimation.
[Abstract; Discussion] The paper uses 'lower-bound estimate' in two potentially incompatible ways. The abstract states that at least 33.8% of news exposure content cannot be determined by static scraping, while the Discussion says that 46.6% 'is a better lower-bounded estimate for a more intensive and prolonged scraping process' and describes the environment effect as 'up to 46.6%'. If the discrepancy is intensity-dependent, then 33.8% is a lower bound for low-intensity scraping and 46.6% for higher-intensity scraping, but this conditional reading should be stated explicitly. The authors should define what 'lower bound' means, report the intensity-dependent range, and adjust the abstract so that it is not read as a universal bound across scraping intensities.
[Results, 'Mitigating biases'; Discussion] The only strategy that aligns the ex-situ and in-situ category distributions (removing the Non-thematic, Domestic Commerce, and Technology categories) is implemented with the biased categories identified using the in-situ distribution. Since a researcher relying on ex-situ data does not have access to that distribution, RQ4 does not demonstrate a deployable ex-situ correction; it diagnoses the source of the bias. The Discussion acknowledges this caveat, but the RQ4 framing and the recommendations section present category removal as a mitigation strategy. Please reframe RQ4 as diagnostic, or develop a procedure that identifies the biased categories from ex-situ features alone (for example, automated detection of paywall, login, and cookie-consent markers).

minor comments (5)

[Statistics; Figure 1 caption] The word 'Levenshtein' is misspelled as 'Levensthein' in the Statistics section and in the Figure 1 caption; please correct throughout.
[Figure 4 caption] The caption labels the three bottom-row panels as '(D) removing the two most problematic domains, (E) excluding the Non-thematic content category and (E) removing the biased categories'; the third label should be (F).
[Results, 'Mitigating biases'] The text refers to 'spiegel.com' in the description of Figure 4D, while the rest of the paper refers to 'spiegel.de'; please standardize the domain name.
[Data filtering; Methods] The paper refers to 'near real-time' ex-situ scraping, but the ex-situ request is triggered only after the participant leaves the page, and the in-situ capture also occurs at leave time; a brief definition of what 'near real-time' means here would improve precision.
[Data protection] The statement that 'anonymized data sets can be shared to reproduce the presented analysis and plots upon request' is weaker than the reproducibility standards common in this literature; consider a fuller replication statement or deposit of analysis code and derived data where ethically possible.

Circularity Check

1 steps flagged · score 2.0 of 10

Central disparity estimate is a direct measurement; the only self-referential step is the RQ4 debiasing demonstration, which the authors explicitly acknowledge depends on access to the in-situ distribution.

other [Results, 'Mitigating biases' (RQ4); Discussion]
"Fourth, only removingDomestic Commerce, Technology, and Non-thematic categories aligns the two distributions; χ²(19, N = 28,764) = 16.87, p = .599. ... The only successful strategy was to remove the biased categories, with the caveat that we were able to identify them because we could access the precise distribution via the in-situ collection."

The categories removed in the 'successful' debiasing strategy were selected because their in-situ vs ex-situ counts deviated most (Domestic Commerce overrepresented in in-situ; Non-thematic and Technology overrepresented in ex-situ). The success criterion is closeness of the ex-situ distribution to the in-situ distribution, so choosing which categories to drop using the in-situ distribution and then dropping them makes the distributions align almost by construction. The reported chi-square is therefore not independent evidence for the strategy; it merely shows that with oracle-like knowledge of the target distribution, one can remove the categories that caused the mismatch.

full rationale

The paper's central quantitative claim is a direct measurement, not a derivation from prior equations: normalized Levenshtein distances are computed between HTML captured by the browser extension (in-situ) and HTML later scraped from the logged URLs (ex-situ). The 33.8% lower-bound and the ~6.5 percentage-point delay increment are descriptive marginal means from the collected data, and no fitted parameter is renamed as a prediction. Reliance on the in-situ extension as a reference is a measurement-validity assumption, but it is not circular: if the extension captured content imperfectly, the comparison would be inaccurate, not tautological. Self-citations such as Adam et al. (2024) and Stier et al. (2020) supply the measurement instrument and a domain taxonomy; they are not unverified premises that force the discrepancy result. The one genuinely self-referential element is the RQ4 mitigation exercise, where biased categories are identified using the target distribution and then removed to match that same target distribution; the authors acknowledge this caveat. Because this concerns an auxiliary strategy evaluation rather than the central exposure-discrepancy estimate, the overall circularity score is low.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

This is an empirical measurement paper, not a theoretical model. The load-bearing assumptions are about the validity of the in-situ baseline and the adequacy of the scraping and classification pipeline. No new entities or mediators are introduced.

free parameters (2)

page-refresh threshold = 20 seconds
Chosen by identifying a time threshold that minimizes the measured Levenshtein distances (Data filtering, Appendix S16). This is an outcome-informed tuning parameter that affects the sample and therefore the headline discrepancy estimate.
in-situ/ex-situ time difference exclusion = 12 hours
Visits where the difference between user visit and scraping time exceeds 12 hours are excluded. This hand-chosen inclusion rule affects which observations enter the analysis.

assumptions (5)

domain assumption The in-situ browser extension captures the content as displayed to the user.
The entire comparison treats extension-captured HTML as ground truth for exposure; this premise enters in the Data collection section and is load-bearing for all discrepancy estimates.
domain assumption News article content is stable enough that large Levenshtein distances reflect scraping failure rather than legitimate editing.
Invoked in the Levenshtein distance section to justify interpreting distances as measurement error; if articles are frequently rewritten, distances would overstate scraping bias.
domain assumption The PARTYPRESS classifier, trained on party press releases, can be applied to news articles for comparative category distributions.
Used for content-based categorization in the Newscategorization section; the authors note that absolute classifications may be affected but assume errors affect both collections equally.
domain assumption Static scraping without JavaScript execution is adequate for news articles.
The scraping setup omits Selenium-like tools; the authors present supporting evidence, but this adequacy assumption underlies all ex-situ collections.
domain assumption The 679-domain German news list and URL-based article detection identify the relevant population of news articles.
The sample frame comes from Stier et al. (2020) plus URL heuristics; all downstream estimates are conditional on this frame and on the manual evaluation of the article-detection rules.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Beyond time delays: How web scraping distorts measures of online news consumption." pith.science (2026). https://pith.science/paper/Z2TYEIIZ

@misc{pith2026241200479,
  author       = {Pith},
  title        = {Pith review of: Beyond time delays: How web scraping distorts measures of online news consumption},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Z2TYEIIZ}},
  note         = {Machine review of arXiv:2412.00479}
}

read the original abstract

As the exploration of digital behavioral data revolutionizes communication research, understanding the nuances of data collection methodologies becomes increasingly pertinent. This study focuses on one prominent data collection approach, web scraping, and more specifically, its application in the growing field of research relying on web browsing data. We investigate discrepancies between content obtained directly during user interaction with a website (in-situ) and content scraped using the URLs of participants' logged visits (ex-situ) with various time delays (0, 30, 60, and 90 days). We find substantial disparities between the methodologies, uncovering that errors are not uniformly distributed across news categories regardless of classification method (domain, URL, or content analysis). These biases compromise the precision of measurements used in existing literature. The ex-situ collection environment is the primary source of the discrepancies (~33.8%), while the time delays in the scraping process play a smaller role (adding ~6.5 percentage points in 90 days). Our research emphasizes the need for data collection methods that capture web content directly in the user's environment. However, acknowledging its complexities, we further explore strategies to mitigate biases in web-scraped browsing histories, offering recommendations for researchers who rely on this method and laying the groundwork for developing error-correction frameworks.

Figures

Figures reproduced from arXiv: 2412.00479 by the authors.

**Figure 2.** Post-hoc ex-situ scraping experiment. Plot (A) shows the average distances per library for each of the (post-hoc) ex-situ collections: 0 days (i.e., near real-time) and 30/60/90 days following user interaction compared to the in-situ collection. The vertical lines on top of the averages represent 99% confidence intervals. Plot (B) shows a time series using 7D rolled average distances for the same period. The first s… view at source ↗

**Figure 3.** Normalized Levensthein distances for different news classifications. [PITH_FULL_IMAGE:figures/full_fig_p016_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Strategies to counteract systematic biases in the distributions. [PITH_FULL_IMAGE:figures/full_fig_p018_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

3 extracted references · 2 canonical work pages

[1]

Improving the quality of individual-level online information tracking: challenges of existing approaches and introduction of a new content- and long-tail sensitive academic solution

Adam, S., Makhortykh, M., Maier, M., Aigenseer, V., Urman, A., Lopez, T. G., Christner, C., de León, E., & Ulloa, R. (2024).Improving the quality of individual-level online information tracking: Challenges of existing approaches and introduction of a new content- and long- tail sensitive academic solution(arXiv:2403.02931). arXiv. https://doi.org/10.48550...

work page Pith review arXiv doi:10.48550/arxiv.2403.02931 2024
[2]

https://doi.org/10.5117/CCR2019.1.004.HAIM Jürgens, P., & Stark, B. (2022). Mapping Exposure Diversity: The Divergent Effects of Algorithmic Curation on News Consumption.Journal of Communication, 72(3), 322–344. https://doi.org/10.1093/joc/jqac009 Kalogeropoulos, A., Fletcher, R., & Nielsen, R. K. (2019). News brand attribution in distributed environments...

arXiv 2022
[2019]

Wojcieszak, M., Clemm von Hohenberg, B., Casas, A., Menchen-Trevino, E., de Leeuw, S., Gonçalves, A., & Boon, M. (2022). Null effects of news exposure: A test of the (un)desirable effects of a ‘news vacation’ and ‘news binging.’Humanities and Social Sciences Communications, 9(1), 1–10. https://doi.org/10.1057/s41599-022-01423-x Yang, T., Majó-Vázquez, S.,...

work page doi:10.1057/s41599-022-01423-x 2022

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

Improving the quality of individual-level online information tracking: challenges of existing approaches and introduction of a new content- and long-tail sensitive academic solution

Adam, S., Makhortykh, M., Maier, M., Aigenseer, V., Urman, A., Lopez, T. G., Christner, C., de León, E., & Ulloa, R. (2024).Improving the quality of individual-level online information tracking: Challenges of existing approaches and introduction of a new content- and long- tail sensitive academic solution(arXiv:2403.02931). arXiv. https://doi.org/10.48550...

work page Pith review arXiv doi:10.48550/arxiv.2403.02931 2024

[2] [2]

https://doi.org/10.5117/CCR2019.1.004.HAIM Jürgens, P., & Stark, B. (2022). Mapping Exposure Diversity: The Divergent Effects of Algorithmic Curation on News Consumption.Journal of Communication, 72(3), 322–344. https://doi.org/10.1093/joc/jqac009 Kalogeropoulos, A., Fletcher, R., & Nielsen, R. K. (2019). News brand attribution in distributed environments...

arXiv 2022

[3] [2019]

Wojcieszak, M., Clemm von Hohenberg, B., Casas, A., Menchen-Trevino, E., de Leeuw, S., Gonçalves, A., & Boon, M. (2022). Null effects of news exposure: A test of the (un)desirable effects of a ‘news vacation’ and ‘news binging.’Humanities and Social Sciences Communications, 9(1), 1–10. https://doi.org/10.1057/s41599-022-01423-x Yang, T., Majó-Vázquez, S.,...

work page doi:10.1057/s41599-022-01423-x 2022