Pith. sign in

REVIEW 3 major objections 6 minor 18 references

Bridging the Gap: Leveraging Retrieval-Augmented Generation to Better Understand Public Concerns about Vaccines

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper reports that VaxPulse Query Corner, a retrieval-augmented generation tool, answers complex questions about public vaccine concerns from social media, with up to 0.96 faithfulness and 0.94 relevance on 35,103 Shingrix posts.

desk verdict Applied RAG with a real dataset but a circular evaluation; the abstract's best numbers are not the ones that matter. read the letter →

arxiv 2507.12840 v1 pith:QPZ5Z5LU submitted 2025-07-17 cs.IR cs.LGcs.SI

Muhammad Javed , Sedigh Khademi Habibabadi , Christopher Palmer , Hazel Clothier , Jim Buttery , Gerardo Luis Dimaguila This is my paper · ORCID
classification cs.IRcs.LGcs.SI
keywords VaccinehesitancyRetrieval-augmentedgenerationSocialmediaanalysisPublichealthsurveillanceShingrixQuestionansweringLLMhallucinationVaxPulse
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

VaxPulse Query Corner is a retrieval-augmented generation (RAG) tool, meaning it grounds answers in documents retrieved from an external store instead of relying on the language model's memory alone. The paper's central claim is that this tool can answer complex queries about public vaccine concerns—such as whether people believe vaccines cause long-term side effects—by retrieving and synthesising relevant social media comments. Analysing 35,103 Shingrix posts from X, Reddit, YouTube, and Facebook, it reports faithfulness of 0.96 and relevance of 0.94 for its summarisation answers, and lower but still strong scores for question answering, discussion topics, and public concerns. The stated payoff is that public health administrators could understand community concerns near-real-time and adjust communication, closing a gap left by topic modelling and by hallucination-prone large language models. The evaluation rests on 276 synthetic test questions generated by an automated framework, so the central assumption to check is whether those questions represent what real stakeholders would ask.

What carries the argument

The engine of the system is a two-iteration retrieve–compress–re-rank loop over a vector database, where retrieval-augmented generation means a language model's answers are grounded in documents retrieved from an external store rather than generated from memory alone. The first iteration retrieves a large set of top-k comments by semantic similarity, re-orders them to avoid the 'lost in the middle' problem, and re-ranks them with a lightweight model. The second iteration compresses the retrieved comments, keeps only the half above an 80% similarity threshold, re-ranks again, and feeds the condensed context to a large language model to produce the final answer. This two-stage design is what the paper credits for the measured gains in context precision and recall.

What would settle it

Have public health administrators write 50 real questions about Shingrix concerns, have two human annotators write reference answers from the same 35,103 posts, run VaxPulse Query Corner on those questions, and check whether faithfulness and relevance scores stay within roughly 0.1 of the reported 0.88–0.96 range; a large drop would show the synthetic test questions overstate real-world usefulness.

Watch

Extended reading notes

Core claim

The paper's central claim is that a two-iteration RAG pipeline can turn a large, noisy collection of social media comments into grounded, query-specific answers about vaccine concerns. Starting from 60,935 collected comments and 35,103 Shingrix-specific comments, the system embeds each comment as a 1536-dimensional vector, retrieves the top-ranked comments semantically, applies a two-stage re-ranking step, then in a second iteration compresses and re-ranks the best 50% of those comments at an 80% similarity threshold, and finally has a large language model formulate the answer. The reported effect of the second iteration is retrieval improvement: context precision rises from 0.56 to 0.66 and context recall from 0.85 to 0.91. Answer-generation scores vary by query type: 0.90/0.89 for question answering, 0.90/0.91 for discussion topics, 0.96/0.94 for summarisation, and 0.88/0.82 for public concerns, where faithfulness is the first number and relevance the second. The authors present these results as evidence that RAG can bridge the gap between raw public sentiment and the specific information needs of public health administrators.

Load-bearing premise

The evaluation assumes that automatically generated synthetic test questions, with reference answers produced by a language model from the same Shingrix posts the tool searches, stand in for the questions public health officials actually ask and for what an expert would judge as correct.

Editorial extensions

If this is right

  • Public health teams could query a live social media stream for specific vaccine concerns—such as reactions to a second dose—and receive a grounded summary in near-real time.
  • The reported gain in context recall and precision shows that the compression step measurably concentrates the evidence, making a case for multi-stage retrieval in other health monitoring applications.
  • Because the tool retrieves from the same comment base each week, it could track how a concern changes over time rather than just listing topics once.
  • The approach is not limited to Shingrix: the vaccine-segregation step lets administrators apply the same pipeline to any vaccine or adverse event of special interest in the VaxPulse framework.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the synthetic test questions were replaced by a set of real questions written by public health administrators, the faithfulness and relevance gap between query types would be a more trustworthy basis for deciding where the tool needs human oversight.
  • The headline 0.96/0.94 figures come from the summarisation query type; for the directly decision-relevant public-concerns query type the scores are lower (0.88/0.82), so comparisons between tools should use the query-type breakdown rather than the abstract headline.
  • A natural testable extension is to vary the compression threshold and top-k size to map how faithfulness trades off against answer completeness for different vaccines, something the paper does not report.
  • Since reference answers are generated from the same dataset the system retrieves from, the scores likely reflect how well the system reproduces the dataset's own content; measuring against an external expert-curated knowledge base would test whether the tool can answer questions whose answers are not already in the retrieved comments.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces VaxPulse Query Corner (VaxPulse QC), a retrieval-augmented generation (RAG) tool for answering complex queries about public vaccine concerns from social media. The system is built on 35,103 Shingrix-related comments collected from X, Reddit, YouTube, and Facebook. The pipeline consists of dataset creation, a first iteration of top-k retrieval with ranking and re-ranking, a second iteration with contextual compression and a similarity-threshold-based selection, and a gpt-4o-based answer formulation step supporting four query categories: Answer the Question, Topics of Discussion, Summarise, and Public Concerns. The evaluation uses 276 synthetic test cases generated by RAGAs from the same Shingrix corpus, with RAGAs-based context precision/recall and LLM-judged faithfulness and answer relevance scores. The abstract reports overall answer faithfulness of 0.96 and relevance of 0.94, while the detailed table reports category-specific scores ranging from 0.88 to 0.96 for faithfulness and 0.82 to 0.94 for relevance.

Significance. If the results were supported by a valid evaluation, the paper would offer a practical RAG application for infodemiology and vaccine-hesitancy monitoring, with a clearly documented two-iteration retrieval and compression pipeline. The authors are honest about the labor-intensive nature of manual evaluation and choose an automated framework, and they provide a useful comparison to prior RAG implementations in medicine. However, the central claim is currently not established because the evaluation is self-referential: synthetic queries and LLM-generated ground truths come from the same corpus the system retrieves from, and the same LLM family both produces candidate answers and judges them. The abstract also reports the highest category scores as if they were overall performance. These are load-bearing issues that require additional evidence, not merely presentational fixes.

major comments (3)
  1. [Section 4.1 and Table 2 (Answer Generation)] The evaluation protocol cannot support the central claim. The 276 test cases are generated by RAGAs from the same Shingrix corpus that the retriever searches; the reference answers are generated by an LLM from those same posts; the candidate answers are generated by gpt-4o from retrieved chunks of the same corpus; and the RAGAs faithfulness and relevance scorers are themselves LLM-based. This creates a closed loop in which systematic errors or biases of the LLM family appear on both sides, so high scores may reflect self-consistency rather than actual usefulness for public health administrators. I recommend adding human expert evaluation on real administrator queries or an independently curated gold set, and reporting agreement with human judgments. Without this, the headline faithfulness and relevance numbers are uninformative.
  2. [Abstract and Section 4.2] The abstract's claim of '0.96 answer faithfulness and 0.94 relevance' is based only on the Summarise category, not on overall performance. Table 2 reports Answer the Question at 0.90/0.89 and Public Concerns at 0.88/0.82. Reporting only the highest category scores overstates the system's capability for the task-relevant query types. The abstract and conclusion should report either all category scores or a properly weighted aggregate, and the discussion should not imply that the system uniformly reaches 0.96/0.94.
  3. [Section 4] The paper contains no baseline comparison. The contribution is framed as addressing limitations of topic modelling and mitigating LLM hallucinations, but there is no comparison with plain gpt-4o without retrieval, with topic modelling, or with any prior method. As a result, the reported scores cannot be attributed to the RAG pipeline or to the second-iteration compression step. Adding such baselines and, where feasible, statistical significance tests over the 276 test cases is necessary to support the claimed advantage.
minor comments (6)
  1. [Section 4] Both tables in Section 4 are labelled 'Table 2'; the answer-generation table should be renumbered as Table 3 and referenced consistently in the text.
  2. [Section 3.3] The 80% similarity threshold and 50% selection ratio are introduced without justification or sensitivity analysis; please explain how these values were chosen and whether results are robust to them.
  3. [Section 4.1] Please specify the exact RAGAs version and the LLM judge model used for faithfulness and relevance scoring, since reproducibility depends on this configuration.
  4. [Section 4.2] The entry 'Highest Scores 1 1' in the context-retrieval table is unclear; please state the percentage of test cases achieving perfect scores for context precision and context recall separately.
  5. [Section 3] There are several presentation issues: 'Tweeter' should be 'Twitter', 'Therapeutic Goods Administrator' should be 'Therapeutic Goods Administration', and the section headings in Section 3.1 contain spacing artifacts. Please proofread the manuscript.
  6. [Section 3.2.2] The description of the early exit strategy is terse; please clarify the criterion for skipping re-ranking when fewer than 10 comments are retrieved.

Circularity Check

1 steps flagged · score 6.0 of 10

Evaluation closed loop: RAGAs-generated test cases and LLM-generated ground truths from the same Shingrix corpus make the reported faithfulness/relevance scores self-consistent rather than independently validated.

  1. fitted input called prediction [Section 4.1 Test Cases Generation; Section 4.2 Results (Table 2)]
    "RAGAs can also generate synthetic test cases to evaluate each component of an RAG pipeline. It uses a unique approach to generate questions with varying levels of difficulty, as well as different characteristics such as reasoning, conditioning, and multi-context, all derived from the provided dataset. ... It not only generates the questions and their reference answers (also known as ground truths) but also extracts the context for the validation step. ... RAGAs generated a total of 276 test cases from the Shingrix dataset."

    The test questions, the reference answers, and the validation context are all generated by RAGAs from the same 35,103-post Shingrix corpus that VaxPulse QC retrieves from, and the candidate answers are produced by gpt-4o over retrieved chunks of that same corpus. The faithfulness and relevance scores are therefore a comparison between two LLM-generated views of the same source material, not between the system output and an independent human-validated ground truth. The reported 0.96/0.94 evidence for 'addresses complex queries' is a measure of internal self-consistency within one corpus and generative family, so the central claim is not supported by an external standard.

full rationale

The paper's central evidence is the Section 4 evaluation. That evaluation is closed-loop: RAGAs synthesized 276 queries and 'ground truth' answers from the Shingrix dataset, and the same dataset is the retrieval source for the gpt-4o answer generator, with LLM-based RAGAs metrics judging the result. This makes the headline faithfulness and relevance scores partly self-referential and unable to establish real-world usefulness for public-health queries. The abstract also reports the highest category (Summarise: 0.96/0.94) while the task-relevant categories are lower (Answer the Question: 0.90/0.89; Public Concerns: 0.88/0.82), and no external baseline or human expert comparison is provided. The paper's self-citations ([13], [14]) concern data creation and are not load-bearing to the RAG claim; there is no imported uniqueness theorem or ansatz. Because the system itself is a legitimate engineering contribution but its central numerical claims reduce to a self-generated evaluation, the circularity score is 6 rather than 0.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The central claim rests on several hand-chosen parameters and unvalidated domain assumptions. The 80% similarity threshold and 50% selection ratio are set without sensitivity analysis. The models used for comment segregation and retrieval quality are assumed reliable without reported accuracy. The evaluation assumes that LLM-generated synthetic ground truths are a valid measure of answer quality.

free parameters (4)
  • second iteration similarity threshold = 0.80
    Hand-chosen in Section 3.3 to filter compressed comments; no sensitivity analysis.
  • second iteration selection ratio = 0.50
    Selects 50% of top-ranked comments after compression; hand-chosen.
  • early exit threshold = 10
    Skip re-ranking when fewer than 10 comments retrieved; arbitrary.
  • RAGAs test case distribution = 50% simple, 25% reasoning, 25% multi-context
    Set in Section 4.1 using default settings; influences the distribution of evaluated query difficulty.
assumptions (3)
  • domain assumption Fine-tuned BERT and GPT-4o can reliably segregate vaccine-specific and Shingrix-specific comments from social media.
    Section 3.1.3 relies on these models without reported accuracy or validation.
  • domain assumption Semantic similarity retrieval with text-embedding-ada-002 and re-ranking captures the comments relevant to public health queries.
    Core retrieval premise in Sections 3.2 and 3.3; no evaluation against human relevance judgments.
  • domain assumption RAGAs synthetic test cases and LLM-based judges are a valid proxy for human assessment of answer quality.
    Section 4.1 adopts RAGAs without validating against human expert judgments.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Bridging the Gap: Leveraging Retrieval-Augmented Generation to Better Understand Public Concerns about Vaccines." pith.science (2026). https://pith.science/paper/QPZ5Z5LU

@misc{pith2026250712840,
  author       = {Pith},
  title        = {Pith review of: Bridging the Gap: Leveraging Retrieval-Augmented Generation to Better Understand Public Concerns about Vaccines},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QPZ5Z5LU}},
  note         = {Machine review of arXiv:2507.12840}
}
read the original abstract

Vaccine hesitancy threatens public health, leading to delayed or rejected vaccines. Social media is a vital source for understanding public concerns, and traditional methods like topic modelling often struggle to capture nuanced opinions. Though trained for query answering, large Language Models (LLMs) often miss current events and community concerns. Additionally, hallucinations in LLMs can compromise public health communication. To address these limitations, we developed a tool (VaxPulse Query Corner) using the Retrieval Augmented Generation technique. It addresses complex queries about public vaccine concerns on various online platforms, aiding public health administrators and stakeholders in understanding public concerns and implementing targeted interventions to boost vaccine confidence. Analysing 35,103 Shingrix social media posts, it achieved answer faithfulness (0.96) and relevance (0.94).

Figures

Figures reproduced from arXiv: 2507.12840 by the authors.

Figure 1
Figure 1. Method 3.1Dataset Creation We created the dataset using the following steps. Our method paper [13] provides further details on data creation, and we also utilised initial steps in [14] for social bot detection. 3.1.1Online media data scraping We collect vaccination-related posts from X (former Tweeter), Reddit, YouTube, and Facebook using authorised APIs Tweepy, PRAW, and the Google API Client Library. 3.1.2Preproce… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

18 extracted references · 15 canonical work pages

  1. [1]

    J., Crawford, N

    Clothier, H. J., Crawford, N. W., Kempe, A., and Bu ttery, J. P. 2011. Surveillance of adverse events f ollowing immunisation: the model of SAEFVIC, Victoria. Commun. Dis. Intell. Q. Rep., 35 , 4 (Dec.), 294–8

    work page 2011

  2. [2]

    Walsh, L., Hill, S., and Waterhouse, T. 2019. Guide to producing and sourcing quality health informati on: Guidelines to support high-quality health information for consumers and carers in Vict oria. Centre for Health Communication and Participa tion, La Trobe University, Victoria

    work page 2019

  3. [3]

    Thangaraju, P., and Venkatesan, S. 2019. WHO Ten th reats to global health in 2019: antimicrobial resis tance. Cukurova Med. J., 44, 1150–1

    work page 2019

  4. [4]

    S., Wei , J., Chung, H

    Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei , J., Chung, H. W., et al. 2023. Large language mod els encode clinical knowledge. Nature, 620, 7972 (Aug.), 172–80

    work page 2023

  5. [5]

    Y., Ning, K

    Yao, J. Y., Ning, K. P., Liu, Z. H., Ning, M. N., a nd Yuan, L. 2023. LLM lies: Hallucinations are not bugs, but features as adversarial examples. arXiv [Preprint]. (Oct. 3, 2023) [cited 2 025 Mar 31]. Available from: https://arxiv.org/abs/ 2310.01469

    arXiv 2023

  6. [6]

    Lewis, P., Perez, E., Piktus, A., Petroni, F., Karp ukhin, V., Goyal, N., et al. 2020. Retrieval-augmen ted generation for knowledge-intensive NLP tasks. In Adv. Neural Inf. Process. Syst., 33, 9459–74

    work page 2020

  7. [7]

    Kim, J., and Min, M. 2024. From RAG to QA-RAG: Inte grating Generative AI for Pharmaceutical Regulatory Compliance Process. arXiv preprint arXiv:2402.01717. (Feb. 2024)

    arXiv 2024

  8. [8]

    A., and Cheungpasitporn, W

    Miao, J., Thongprayoon, C., Suppadungsuk, S., Garci a Valencia, O. A., and Cheungpasitporn, W. 2024. In tegrating Retrieval-Augmented Generation with Large Language Models in Nephrology : Advancing Practical Applications. Medicina, 60, 3 , 445

    work page 2024

Show all 18 references
  1. [9]

    C., Pletcher, M

    Ge, J., Sun, S., Owens, J., Galvez, V., Gologorskay a, O., Lai, J. C., Pletcher, M. J., and Lai, K. 202 3. Development of a Liver Disease-Specific Large Language Model Chat Interface using Retrieval Augmented Generation. medRxiv. (2023)

    2023

  2. [10]

    Y., Adavi, U., Sharma, A., G ara, S., Das, N., et al

    Al Ghadban, Y., Lu, H. Y., Adavi, U., Sharma, A., G ara, S., Das, N., et al. 2023. Transforming healthc are education: Harnessing large language models for frontline health worker capacit y building using retrieval-augmented generation. me dRxiv [Preprint]. (Dec. 1, 2023) [cit...

    2023 doi

  3. [11]

    Saba, W., Wendelken, S., and Shanahan, J. 2024. Que stion-Answering Based Summarization of Electronic H ealth Records using Retrieval Augmented Generation. arXiv preprint arXiv:2401.014 69. (Jan. 2024)

    2024

  4. [12]

    R., K im, J

    Zakka, C., Shad, R., Chaurasia, A., Dalal, A. R., K im, J. L., Moor, M., Fong, R., Phillips, C., Alexan der, K., Ashley, E., and Boyd, J. 2024. Almanac—retrieval-augmented language models for cli nical medicine. NEJM AI, 1, 2, AIoa2300068

    2024

  5. [13]

    and Dimaguila, G.L., 2025

    Javed, M., Khademi, S., Hickman, J., Buttery, J., C lothier, H. and Dimaguila, G.L., 2025. VaxPulse: Mo nitoring of Online Public Concerns to Enhance Post-licensure Vaccine Surveillance. arXiv [preprint] https://doi.org/10.48550/arXiv.2507.0465 6. (Jun. 2025). 7

    2025 doi

  6. [14]

    and Buttery, J., 2023

    Javed, M., Dimaguila, G.L., Habibabadi, S.K., Palme r, C. and Buttery, J., 2023. Learning from machines ? social bots influence on covid-19 vaccination-related discussions: 2021 in review. In Proceedings of the 2023 Australasian Computer Scien ce Week (pp. 190-197)

    2023

  7. [15]

    Chase, H. 2022. LangChain [Computer software]. [cit ed 2025 Mar 31]. Available from: https://github.com /langchain-ai/langchain

    2022

  8. [16]

    Damodaran, P. 2023. FlashRank, Lightest and Fastest 2nd Stage Reranker for search pipelines (Version 1 .0.0) [Computer software]. (Dec. 22,

    2023

  9. [17]

    Es, S., James, J., Espinosa-Anke, L., and Schockaer t, S. 2023. Ragas: Automated evaluation of retrieva l augmented generation. arXiv [Preprint]. (Sept. 26, 2023) [cited 2025 Mar 31]. A vailable from: https://arxiv.org/abs/2309.15217

    2023 arXiv

  10. [2023]

    Available from: https:// doi.org/10.5281/zenodo.10426927

    [cited 2025 Mar 31]. Available from: https:// doi.org/10.5281/zenodo.10426927

    2025 doi

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.