REVIEW 4 major objections 5 minor 28 references
Vaccine Hesitancy on YouTube: a Competition between Health and Politics
T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read This paper claims that YouTube's vaccination discourse is split between health-oriented pro-vaccine content from institutions and politically framed anti-vaccine commentary from individual creators, and that only 2.7% of vaccine-hesitant…
desk verdict The thematic findings are solid and useful, but the moderation headline is misstated and Table II's odds ratios are inverted; the paper needs major revision before it can be trusted. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The method that carries the argument is a daily, keyword-based YouTube audit combined with large-language-model stance labeling. For each of the 3,496 videos, up to twenty 100-word excerpts around vaccine keywords were labeled by gpt-4o-mini as in favor, against, or neutral; the average of those excerpt scores was converted into five categories from strongly in favor to strongly against. The paper then compares these stance groups across video topics, video tags, channel topics, manually annotated channel categories, and channel owner type, using odds ratios to identify which features are more associated with strongly pro-vaccine than strongly anti-vaccine videos. The moderation claim rests on re-crawling metadata at the end of the study and categorizing the 96 missing videos by the error message YouTube returned.
What would settle it
Take a new sample of roughly 500 excerpts from these videos, covering sarcastic, quoted, and purely newsworthy passages, and have human annotators label them; if agreement with gpt-4o-mini drops well below the reported 84% on that larger set, or if re-weighted video scores move the hesitant share materially away from 20.8%, the moderation and framing conclusions would need to be re-estimated.
Extended reading notes
Core claim
On its own terms, the paper establishes two claims. First, stance is systematically tied to framing: videos labeled strongly in favor of vaccination are more likely to carry tags naming preventable diseases (HPV, measles, whooping cough, HIV) and health agencies, while videos labeled strongly against are more likely to mention politicians, political slogans, and formats such as podcasts, reports, and news analysis; the same split holds at channel level, with health and science channels and organizations leaning pro-vaccine and Society or Politics channels and individual creators leaning against, including channels self-identifying as doctors. Second, moderation is rare: only 2.7% of the collected videos were unavailable at the end of the collection, and although every video removed for violating YouTube's Community Guidelines was labeled against or strongly against vaccination, the absolute removal rate leaves the 20.8% hesitant share largely untouched online.
Load-bearing premise
The whole stance distribution depends on the AI labeler's judgment of short text excerpts, validated on only 50 human-labeled excerpts, so a systematic misreading of sarcasm, quoted opponents, or neutral political discussion would change the reported 20.8% hesitant share and every stance-stratified comparison.
Editorial extensions
If this is right
- If the central claim is right, YouTube's vaccination discourse is not primarily a medical debate: the clearest anti-vaccine videos are framed as politics and opinion, while disease-specific health content is predominantly pro-vaccine.
- If right, individual creators, not organizations, are the main source of strongly anti-vaccine videos in this sample, and some channels presenting themselves as doctors contribute to the hesitant side.
- If right, moderation enforcement is currently removing only a small fraction of vaccine-hesitant videos, and the removals that do occur are concentrated on the strongly against categories.
- If right, metadata signals such as tags naming podcasts, reports, and news analysis could help platforms and public-health communicators flag likely vaccine-hesitant content before full transcript analysis.
- If right, public-health communication policy should treat political commentary and opinion formats, not just health misinformation channels, as the main competitive arena on YouTube.
Reading between the lines
- The paper reports subscriber counts for the largest channels but does not aggregate reach; weighting the 20.8% hesitant share by subscribers or views would likely show hesitant content captures more watch time than its video count suggests, because the top hesitant channels have multi-million subscriber bases comparable to major news outlets.
- A natural extension is to test whether the political framing is event-driven: because the collection runs daily, stance labels and tag frequencies could be aligned with the US political calendar, such as nominations and confirmation hearings, to see whether hesitant video production spikes around political events rather than health news.
- The tag odds ratios suggest a cheap detection signal: videos tagged with podcast, reports, or news analysis are disproportionately hesitant, so public-health monitoring could use these metadata signals as a pre-filter before full transcript labeling.
- If the LLM labeler treats neutral political coverage of vaccine debates as against because it quotes anti-vaccine arguments, the 20.8% figure could overstate hesitancy; re-labeling with a prompt that explicitly handles quoted speech would quantify this.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper reports a three-month longitudinal collection of YouTube videos mentioning vaccination (3,496 videos, Nov 20 2024–Feb 19 2025), labeled for vaccine stance using GPT-4o-mini on transcript excerpts, and analyzes associations between stance and video topics, tags, channel categories, and channel owner types. The main empirical claims are a thematic split—pro-vaccine videos focus on specific diseases and health topics, while vaccine-hesitant videos are more likely to mention politicians, podcasts, and news commentary—and a moderation claim that only 2.7% of videos were taken down despite 20.8% of videos being vaccine-hesitant, which the authors interpret as evidence of lax platform moderation of hesitant content.
Significance. If the findings hold, the paper provides a valuable, current snapshot of vaccine discourse on a major platform, with a transparent and reproducible collection design and a scalable LLM-based labeling approach. The thematic contrast between health-centered pro-vaccine content and politically framed vaccine-hesitant content is a useful contribution to public-health communication research. However, the moderation headline, which is the most policy-relevant claim, rests on a denominator error as currently stated, and the tag-level odds-ratio presentation is internally inconsistent. These issues must be resolved before the paper's central conclusions can be accepted.
major comments (4)
- [Abstract; §I; §IV.C] The moderation claim is not supported by the reported statistics. The 96 of 3,496 videos that failed re-crawl (2.7%) is the overall disappearance rate across all stance categories, not the conditional takedown rate for the 20.8% of vaccine-hesitant videos. The abstract's juxtaposition of 2.7% with 20.8% hesitant is therefore a denominator error, and the Introduction's wording that 'only 2.7% of the videos against vaccination have been taken down' is inconsistent with the methods. In fact, the data reported in Section IV.C show platform removals concentrated in A/SA videos, with all Community-Guideline removals in those categories. The authors should recompute the stance-conditional takedown rates and, ideally, compare them with a platform-wide or content-matched baseline before drawing any conclusion about the absence of moderation.
- [Table II; Fig. 2] The odds-ratio definitions in Table II and Figure 2 contradict each other. Figure 2 states that OR > 1 means the feature is more associated with SIF and OR < 1 means association with SA. Table II, however, lists tags on the left (labeled 'more likely to appear in videos labeled Strongly In Favor') with odds ratios between 0.02 and 0.09, while the right column, labeled 'Strongly Against,' contains odds ratios from 1.86 to 5.05. Under the stated definition these assignments are reversed. The authors must correct either the definition or the table and verify that the narrative about disease- and health-related tags in SIF videos and political/media tags in SA videos is based on correctly oriented odds ratios.
- [§III.B; §V, Limitations] The central prevalence estimate (20.8% hesitant) and all stance-stratified comparisons inherit uncertainty from the LLM labeling, but the paper quantifies this only as 84% agreement on 50 human-labeled excerpts. This small validation set does not establish how label error propagates to the video-level proportions, the odds ratios, or the estimated takedown rates by stance. The authors should provide confidence intervals or a sensitivity analysis, at minimum by bootstrapping the excerpt-to-video aggregation and by reporting agreement separately for the against/in-favor distinction, which is the one that matters most for the headline claims.
- [§IV.C; Discussion] Even after correcting the denominator, the interpretation of 'lack of moderation activity for hesitant content' requires a meaningful baseline. Without a comparison to takedown rates for other health misinformation, non-health content, or a platform-wide benchmark, a low absolute removal rate cannot be attributed to lax moderation specifically for hesitant content. The paper should either supply such a baseline or soften the conclusion to state only that hesitant videos were observed to have a low absolute takedown rate in this sample.
minor comments (5)
- [§IV.B] The phrase 'videos with each of the annotated label categories' would be clearer as 'videos in each of the annotated label categories,' and the sentence beginning 'We begin by examining the videos with each' should be revised for grammatical flow.
- [Fig. 4] The y-axis label '% of videos removed' would benefit from specifying that the percentages are computed within each stance category, since the accompanying text discusses rates per category.
- [§III.A] The filtering chain is presented as percentages of the original search results ('about 32%', '17%', '13%', '9%'); the authors should clarify the base population at each step and whether the final 3,496 videos correspond to the 9% figure or to a later stage.
- [Table III] The table lists 'Find the Best' as a top SA channel with a description about equipment and services; given the topic, a brief explanation of why this channel appears would help the reader interpret the results.
- [General] The 'Declaration of AI use' statement is potentially confusing, since the paper extensively uses an LLM as a measurement tool; a note distinguishing 'used as an analysis tool' from 'used in writing the manuscript' would preempt misunderstanding.
Circularity Check
No circularity: the paper is an observational audit whose measurements do not reduce to their inputs; the moderation-rate concern is a logical/denominator error, not a circular derivation.
full rationale
The paper is purely observational: it measures stance prevalences from LLM excerpt annotations, channel categories from manual annotation, and video disappearance from a re-crawl. There are no fitted parameters, no equations, and no derived quantity that is equivalent by construction to an input. The stance labels are produced by gpt-4o-mini and then used to compute proportions and odds ratios; this is ordinary measurement, and the paper validates the labels against 50 human-annotated excerpts (84% agreement). The 2.7% takedown figure is the overall 96/3496 re-crawl failure rate, not the conditional removal rate for vaccine-hesitant videos; the abstract's juxtaposition of 2.7% with 20.8% hesitant is a statistical/interpretation error, not a circular step. Indeed, Section IV.C reports that all Community-Guideline removals were for A/SA videos, which cuts against the 'lack of moderation' headline but again is a correctness issue rather than circularity. The only self-citations, [25] and [26], are used to justify the keyword-based search logic; they are methodological precedents and are not load-bearing for any central claim. No uniqueness theorem, imported ansatz, or renaming of known results appears. The paper is self-contained against external benchmarks and its findings are falsifiable, so the circularity score is 0.
Assumptions & free parameters
free parameters (2)
- stance category thresholds =
SIF [0.5,1], IF (0,0.5), N [0], A (-0.5,0), SA [-1,-0.5]
- excerpt sampling cap =
20 excerpts, randomly sampled when exceeded
assumptions (3)
- domain assumption YouTube Search API returns a representative sample of the previous day's vaccine-related videos for the specified keywords, region, and language.
- domain assumption Transcripts are faithful representations of the video content, and the absence of a transcript is ignorable.
- domain assumption GPT-4o-mini excerpt labels are a valid proxy for human judgment of stance, based on 84% agreement on 50 excerpts.
Cite this review
Pith. "Pith review of Vaccine Hesitancy on YouTube: a Competition between Health and Politics." pith.science (2026). https://pith.science/paper/KH6LMXOL
@misc{pith2026250707517,
author = {Pith},
title = {Pith review of: Vaccine Hesitancy on YouTube: a Competition between Health and Politics},
year = {2026},
howpublished = {\url{https://pith.science/paper/KH6LMXOL}},
note = {Machine review of arXiv:2507.07517}
}
read the original abstract
YouTube has rapidly emerged as a predominant platform for content consumption, effectively displacing conventional media such as television and news outlets. A part of the enormous video stream uploaded to this platform includes health-related content, both from official public health organizations, and from any individual or group that can make an account. The quality of information available on YouTube is a critical point of public health safety, especially when concerning major interventions, such as vaccination. This study differentiates itself from previous efforts of auditing YouTube videos on this topic by conducting a systematic daily collection of posted videos mentioning vaccination for the duration of 3 months. We show that the competition for the public's attention is between public health messaging by institutions and individual educators on one side, and commentators on society and politics on the other, the latest contributing the most to the videos expressing stances against vaccination. Videos opposing vaccination are more likely to mention politicians and publication media such as podcasts, reports, and news analysis, on the other hand, videos in favor are more likely to mention specific diseases or health-related topics. Finally, we find that, at the time of analysis, only 2.7% of the videos have been taken down (by the platform or the channel), despite 20.8% of the collected videos having a vaccination hesitant stance, pointing to a lack of moderation activity for hesitant content. The availability of high-quality information is essential to improve awareness and compliance with public health interventions. Our findings help characterize the public discourse around vaccination on one of the largest media platforms, disentangling the role of the different creators and their stances, and as such, they provide important insights for public health communication policy.
Figures
Reference graph
Works this paper leans on
-
[1]
Explore how americans watch tv across platforms,
Nielsen, “Explore how americans watch tv across platforms,” 2025. [Online]. Available: https://www.nielsen.com/data-center/the-gauge/ 1
work page 2025
-
[2]
Youtube as a source of information on the h1n1 influenza pandemic,
A. Pandey, N. Patni, M. Singh, A. Sood, and G. Singh, “Youtube as a source of information on the h1n1 influenza pandemic,” American journal of preventive medicine , vol. 38, no. 3, pp. e1–e3, 2010. 1
work page 2010
-
[3]
Youtube as a source of misinformation on covid-19 vaccination: a systematic analysis,
H. O.-Y . Li, E. Pastukhova, O. Brandts-Longtin, M. G. Tan, and M. G. Kirchhof, “Youtube as a source of misinformation on covid-19 vaccination: a systematic analysis,” BMJ global health , vol. 7, no. 3, p. e008334, 2022. 1, 2
work page 2022
-
[4]
From Facebook to YouTube: The potential exposure to COVID-19 anti-vaccine videos on social media,
A. Gruzd, D. Abul-Fottouh, M. Y . Song, and A. Saiphoo, “From Facebook to YouTube: The potential exposure to COVID-19 anti-vaccine videos on social media,” Social media+ society , vol. 9, no. 1, p. 20563051221150403, 2023, publisher: SAGE Publications Sage UK: London, England. 1, 2, 6
work page 2023
-
[5]
Pandemic profiteers. the business of anti-vaxx,
Center for Countering Digital Hate, “Pandemic profiteers. the business of anti-vaxx,” 6 2021. [Online]. Available: https://counterhate.com/ research/pandemic-profiteers/ 1
work page 2021
-
[6]
Social media use and vaccine hesi- tancy in the european union,
M. Mascherini and S. Nivakoski, “Social media use and vaccine hesi- tancy in the european union,” Vaccine, vol. 40, no. 14, pp. 2215–2225,
-
[7]
Social media behavior is associated with vaccine hesitancy,
S. Rathje, J. K. He, J. Roozenbeek, J. J. Van Bavel, and S. Van Der Lin- den, “Social media behavior is associated with vaccine hesitancy,” PNAS nexus, vol. 1, no. 4, p. pgac207, 2022. 2
work page 2022
-
[8]
Covid-19 misinformation cost at least 2,800 lives and $300m, new report says,
D. Major, “Covid-19 misinformation cost at least 2,800 lives and $300m, new report says,” 2023. [Online]. Available: https://www.cbc. ca/news/politics/cost-of-covid-19-misinformation-study-1.6726356 2
work page 2023
Show all 28 references
-
[9]
Potentials and limitations of educational videos on YouTube for science communication,
S. Kohler and T. C. Dietrich, “Potentials and limitations of educational videos on YouTube for science communication,” Frontiers in communi- cation, vol. 6, p. 581302, 2021, publisher: Frontiers Media SA. 2
2021
-
[10]
YouTube videos as health decision aids for the public: an integrative review,
K. Haslam, H. Doucette, S. Hachey, T. MacCallum, D. Zwicker, M. Smith-Brilliant, and R. Gilbert, “YouTube videos as health decision aids for the public: an integrative review,” Canadian Journal of Dental Hygiene, vol. 53, no. 1, p. 53, 2019, publisher: Canadian Dental Hygienis...
2019
-
[11]
Youtube as a source of information on immunization: a content analysis,
J. Keelan, V . Pavri-Garcia, G. Tomlinson, and K. Wilson, “Youtube as a source of information on immunization: a content analysis,” jama, vol. 298, no. 21, pp. 2482–2484, 2007. 2
2007
-
[12]
Misinformation on vaccination: A quan- titative analysis of youtube videos,
G. Donzelli, G. Palomba, I. Federigi, F. Aquino, L. Cioni, M. Verani, A. Carducci, and P. Lopalco, “Misinformation on vaccination: A quan- titative analysis of youtube videos,” Human vaccines & immunothera- peutics, vol. 14, no. 7, pp. 1654–1659, 2018. 2
2018
-
[13]
Spread of vaccine hesitancy in france: What about youtube™?
M. Lahouati, A. De Coucy, J. Sarlangue, and C. Cazanave, “Spread of vaccine hesitancy in france: What about youtube™?” Vaccine, vol. 38, no. 36, pp. 5779–5782, 2020. 2
2020
-
[14]
Exploring YouTube’s recommendation system in the context of COVID-19 vac- cines: Computational and comparative analysis of video trajectories,
Y . M. M. Ng, K. Hoffmann Pham, and M. Luengo-Oroz, “Exploring YouTube’s recommendation system in the context of COVID-19 vac- cines: Computational and comparative analysis of video trajectories,” Journal of medical Internet research, vol. 25, p. e49061, 2023, publisher: JMIR ...
2023
-
[15]
Science communication on YouTube: Factors that affect channel and video popularity,
D. J. Welbourne and W. J. Grant, “Science communication on YouTube: Factors that affect channel and video popularity,” Public understanding of science , vol. 25, no. 6, pp. 706–718, 2016, publisher: SAGE Publi- cations Sage UK: London, England. 2
2016
-
[16]
The institutionalization of youtube: From user-generated con- tent to professionally generated content,
J. Kim, “The institutionalization of youtube: From user-generated con- tent to professionally generated content,” Media, culture & society , vol. 34, no. 1, pp. 53–67, 2012. 2
2012
-
[17]
Expressions of pro- and anti-vaccine sentiment on youtube,
N. Yiannakoulias, C. E. Slavik, and M. Chase, “Expressions of pro- and anti-vaccine sentiment on youtube,” Vaccine, vol. 37, no. 15, pp. 2057–2064, 2019. 2
2019
-
[18]
Ty- pologies of the popular science web video,
J. Munoz Morcillo, K. Czurda, and C. Y . Robertson-von Trotha, “Ty- pologies of the popular science web video,” J. Sci. Commun. , vol. 15, no. 04, p. A02, May 2016. 2
2016
-
[19]
Science and environmental communication on YouTube: Strategically distorted communications in online videos on climate change and climate engineering,
J. Allgaier, “Science and environmental communication on YouTube: Strategically distorted communications in online videos on climate change and climate engineering,” Frontiers in communication , vol. 4, p. 36, 2019, publisher: Frontiers Media SA. 2
2019
-
[20]
Algorithmic extremism: Exam- ining YouTube’s rabbit hole of radicalization,
M. Ledwich and A. Zaitsev, “Algorithmic extremism: Exam- ining YouTube’s rabbit hole of radicalization,” arXiv preprint arXiv:1912.11211, 2019. 2
1912 arXiv
-
[21]
When vaccines go viral: an analysis of hpv vaccine coverage on youtube,
R. Briones, X. Nan, K. Madden, and L. Waks, “When vaccines go viral: an analysis of hpv vaccine coverage on youtube,”Health communication, vol. 27, no. 5, pp. 478–485, 2012. 2
2012
-
[22]
A survey of text classification with transformers: How wide? how large? how long? how accurate? how expensive? how safe?
J. Fields, K. Chovanec, and P. Madiraju, “A survey of text classification with transformers: How wide? how large? how long? how accurate? how expensive? how safe?” IEEE Access, vol. 12, pp. 6518–6531, 2024. 2
2024
-
[23]
Use of large language models as a scalable approach to understanding public health discourse,
L. Espinosa and M. Salath ´e, “Use of large language models as a scalable approach to understanding public health discourse,” PLOS Digital Health , vol. 3, no. 10, pp. 1–15, 10 2024. [Online]. Available: https://doi.org/10.1371/journal.pdig.0000631 2
2024 doi
-
[24]
Accuracy of a large language model in distinguishing anti- and pro-vaccination messages on social media: The case of human papillomavirus vaccination,
S. Kim, K. Kim, and C. Wonjeong Jo, “Accuracy of a large language model in distinguishing anti- and pro-vaccination messages on social media: The case of human papillomavirus vaccination,” Preventive Medicine Reports, vol. 42, p. 102723, 2024. [Online]. Available: https: //www...
2024
-
[25]
Echoes through Time: Evolution of the Italian COVID-19 Vaccination Debate,
G. Crupi, Y . Mejova, M. Tizzani, D. Paolotti, and A. Panisson, “Echoes through Time: Evolution of the Italian COVID-19 Vaccination Debate,” Proceedings of the International AAAI Conference on Web and Social Media , vol. 16, pp. 102–113, May 2022. [Online]. Available: https://...
2022
-
[26]
Political context of the european vaccine debate on twitter,
G. Paoletti, L. Dall’Amico, K. Kalimeri, J. Lenti, Y . Mejova, D. Paolotti, M. Starnini, and M. Tizzani, “Political context of the european vaccine debate on twitter,” Sci. Rep., vol. 14, no. 1, p. 4397, Feb. 2024. 2
2024
-
[27]
Science communication on YouTube: Factors that affect channel and video popularity,
D. J. Welbourne and W. J. Grant, “Science communication on YouTube: Factors that affect channel and video popularity,” Public Understanding of Science , vol. 25, no. 6, pp. 706–718, Aug. 2016, publisher: SAGE Publications Ltd. [Online]. Available: https: //doi.org/10.1177/0963...
2016 doi
-
[28]
Reply providing just the label
“neither” if the text is not in favor or against vaccines. Reply providing just the label. Here is the text: [text]
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.