Pith. sign in

REVIEW 5 major objections 6 minor 82 references

Can LLMs Assist Annotators in Identifying Morality Frames? -- Case Study on Vaccination Debate on Social Media

T0 review · 5 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Few-shot LLM prompts with explanations identify morality frames in vaccine tweets at 90.79% accuracy.

desk verdict The 90.79% headline measures annotator endorsement of GPT-4o's suggestions, not objective accuracy, because the human judges saw the model's label and explanation before saying yes. read the letter →

arxiv 2502.01991 v2 pith:YB3OGNVL submitted 2025-02-04 cs.CL cs.AIcs.CYcs.HCcs.SI

Tunazzina Islam , Dan Goldwasser This is my paper · ORCID
classification cs.CLcs.AIcs.CYcs.HCcs.SI
keywords moralityframesmoralfoundationsfew-shotpromptingin-contextlearningLLM-assistedannotationvaccinedebatesocialmediacognitiveload
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper aims to show that large language models can assist human annotators in identifying morality frames—the moral foundation expressed in a text plus the actor and target roles with positive or negative polarity—in COVID-19 vaccine debates on social media. Using GPT-4o with seven in-context examples and explanations, the authors report an overall accuracy of 90.79% against the majority vote of nine annotators, with 92.67% accuracy and 93.51% F1 on moral foundations alone. When annotators did not see the LLM's explanations, endorsement dropped to 64.81%. Survey responses indicate the AI labels and explanations made the task easier, reduced cognitive load, and roughly halved annotation time. The paper concludes that integrating LLM-generated labels and explanations into annotation improves accuracy and reduces burden.

What carries the argument

The load-bearing mechanism is few-shot prompting with explanations: the prompt begins with task instructions and definitions of the six moral foundations and actor-target roles, then provides seven examples covering all six foundations plus a non-moral case, each with its label followed by an 'Explanation:' line. The model is forced to choose from the given moral foundation categories, and GPT-4o generates both the moral foundation label and the actor-target polarity roles with explanations. A web-based think-aloud annotation tool presents the LLM output, records yes or no judgments and corrections, and the study aggregates results by majority vote, reporting an inter-annotator agreement coefficient of 0.979.

What would settle it

Have a fresh set of annotators label the same 150 tweets without ever seeing the LLM outputs; if their independent labels agree with GPT-4o far below 90.79%, or if a control condition in which the suggested labels are deliberately wrong still draws majority 'yes' responses, then the reported accuracy is endorsement rather than correctness.

Watch

Extended reading notes

Core claim

The central claim is that few-shot prompting with explanations enables an LLM to produce morality frame annotations that human annotators endorse. On 150 randomly selected tweets from the COVID-19 vaccine debate dataset, GPT-4o prompted with seven examples and explanations achieved 90.79% overall accuracy by the annotators' majority vote, with 92.67% accuracy and 93.51% macro F1 for moral foundation prediction alone. When annotators judged the LLM labels without seeing the explanations, endorsement fell to 64.81%, which the paper interprets as evidence that explanations are what make the collaboration work. All nine annotators reported that the explanations were helpful and reduced cognitive load, and the paper argues this supports a human-AI collaborative model for psycholinguistic annotation.

Load-bearing premise

The whole result rests on the premise that annotators who have just seen the AI's suggested label and explanation are still able to judge that label independently; if they simply defer to the AI suggestion, the 90.79% figure measures agreement rather than correctness.

Editorial extensions

If this is right

  • LLM-generated explanations, not just labels, are what push endorsement from 64.81% to 90.79%, so explanation quality is central to the assistive benefit.
  • Integrating LLM outputs into annotation can cut per-batch time by at least 50% and reduce self-reported exhaustion by over 60%, making larger psycholinguistic datasets more feasible to build.
  • Even when the LLM's label was wrong, annotators reported that the explanation helped them arrive at the correct annotation, supporting a collaborative rather than fully automated workflow.
  • Human oversight remains necessary because LLMs sometimes classify factual statements as moral and can exhibit bias in explanations, as the Pentagon vaccine mandate example shows.
  • The framework is presented as domain-agnostic and could be applied to other polarized debates, such as political discourse or climate discussion.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the anchoring concern is real, the with-versus-without-explanation gap may partly reflect trust in the AI rather than improved understanding; a control that deliberately injects wrong labels would separate deference from learning.
  • The 90.79% figure is an endorsement rate from nine volunteer annotators with academic degrees who reside in the United States, so generalization to other annotator populations or cultural contexts is untested.
  • The tool's correction records could be mined as a human-only benchmark for morality frames on the subset of tweets where a majority rejected the AI label, providing a check on LLM bias that the paper does not perform.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes using GPT-4o with few-shot prompting and in-context explanations to generate morality frame annotations (moral foundation, actor/target roles with polarity, and explanations) for COVID-19 vaccine tweets, and it evaluates these outputs with nine human annotators through a purpose-built web interface. The reported results include an overall morality-frame accuracy of 90.79% (Table 2), an inter-annotator agreement of Krippendorff's alpha = 0.979, and survey responses suggesting that LLM-generated labels and explanations reduce task difficulty, cognitive load, and annotation time. The authors argue that this demonstrates a promising human-AI collaborative annotation framework for complex psycholinguistic tasks.

Significance. If the central claims were valid, the paper would offer a practical contribution to human-AI collaboration for subjective annotation tasks, with a concrete tool and an evaluation on a socially relevant corpus. Strengths include the use of an existing COVID-19 vaccine tweet dataset, a relatively detailed description of the annotation interface and participant recruitment, an ablation comparison, and a candid Limitations section. However, the main empirical claim is not established by the current experimental design: the accuracy metric is computed against the majority vote of annotators who were shown the LLM's labels and explanations before judging them, so the reported accuracy conflates correctness with annotator endorsement. This is a load-bearing issue for the paper's central conclusion and cannot be resolved by a purely local revision.

major comments (5)
  1. [Sec. 4.2 and Sec. 4.3] The evaluation protocol makes the headline accuracy measure endogenous. Annotators are shown the LLM's predicted moral foundation, actor/target roles, and explanation, and are asked to click 'yes' if they agree; the LLM prediction is counted as a 'win' when a majority of the nine annotators click 'yes.' This measures acceptance of AI-generated suggestions, not correctness against an independent gold standard. Because moral frame identification is a subjective psycholinguistic task, and the paper itself reports a three-way disagreement in Sec. 5.1, displaying the model's answer before eliciting the human judgment can anchor annotators. The claim in Sec. 5.1 that 'LLMs can identify morality frames ... with an overall accuracy of 90.79%' is therefore not supported by Table 2. The paper should either use an independent blind annotation as ground truth or compare against a pre-existing gold-standard set, such as the labels in the source dataset of Pacheco et al. [57].
  2. [Sec. 4.2] The reported Krippendorff's alpha of 0.979 requires clarification and is difficult to reconcile with the described annotation task. If alpha is computed over annotators' yes/no responses to the displayed LLM label, it measures agreement with a shared AI suggestion rather than independent coding reliability. If alpha is computed over moral foundation labels, a value of 0.979 is implausibly high given that Sec. 5.1 describes annotators disagreeing among 'sanctity/degradation,' 'none,' and 'care/harm' for at least one tweet. The manuscript should specify the unit of coding, the number of coders, and the disagreement metric used; as reported, the number cannot serve as evidence of reliable annotation.
  3. [Sec. 4.5] The ablation comparison confounds the prompting condition with what the annotators could see. The 64.81% figure in the 'few-shot w/o expl' row is described as resulting from annotators providing judgments 'without looking at explanations,' but it is not stated whether these annotators also saw the LLM labels, whether the same annotators scored the same texts in both conditions, or in what order the conditions were administered. Without a counterbalanced or within-subject design, the 26-percentage-point gap between rows in Table 2 cannot be attributed to the presence of explanations rather than to task order, fatigue, learning, or differences between participant groups. The ablation also does not report inter-annotator agreement for the no-explanation condition.
  4. [Sec. 5.2 and Table 3] The survey-based claims about time reduction and exhaustion are not supported by the data presented. Table 3 reports only a single 'Avg. Time/Batch (min)' column, with no paired measurement for the condition without LLM labels and explanations, no statistical test, and no definition of how the claimed 'at least 50%' time reduction or 'over 60%' decrease in exhaustion were computed. These statements should be presented either as qualitative participant opinions clearly separated from quantitative findings, or substantiated with paired measurements and appropriate tests.
  5. [Sec. 4.3 and Table 2] The accuracy and F1 definitions are underspecified. It is unclear how 'overall accuracy' combines moral foundation correctness with actor-target polarity correctness, whether partial credit is allowed, how the majority vote over corrected annotator labels is aggregated when annotators disagree, and over which classes the macro F1 is computed. The paper should provide a precise scoring rule and a confusion matrix or per-component breakdown, especially because Sec. 5.1 states that some annotators rejected LLM predictions due to role/polarity errors despite correct moral foundations.
minor comments (6)
  1. [Table 1] There is a typo in the Loyalty/Betrayal row: 'for the froup' should read 'for the group.'
  2. [Sec. 4.2.1] It is unclear whether the pilot test with a separate batch of 10 tweets was fully excluded from the analysis and whether any pilot feedback changed the annotation interface or instructions before the main 150-tweet evaluation.
  3. [Sec. 4.6] The correlation heatmaps in Fig. 6 are reported without significance levels or multiple-comparison corrections; with 150 tweets, many of the displayed correlations may be unstable, and the text should acknowledge this.
  4. [Sec. 3.2 and Sec. 5] The abstract and introduction describe a 'think-aloud' tool, but no think-aloud verbal protocol data are reported in the results; the paper should either describe how think-aloud responses were collected and analyzed or remove the term.
  5. [Sec. 5.1] The claim that 'LLMs can identify moral cases better than non-moral ('none') cases' is not quantified; per-class accuracy or a confusion matrix for the 'none' class should be reported to support this statement.
  6. [Sec. 7] The Limitations section candidly notes novelty bias and voluntary participation, but the manuscript does not describe any procedural controls (e.g., debriefing questions, attention checks, or a control condition) that would mitigate these threats.

Circularity Check

1 steps flagged · score 6.0 of 10

The reported 90.79% accuracy is measured against a majority vote taken after annotators viewed the LLM's labels and explanations, so the benchmark is not independent of the prediction.

  1. self definitional [Sec 4.2 'Human Evaluation' and Sec 4.3 'Results']
    "Our specialized web tool (Fig. 5) allows annotators to assess moral foundations (with explanation) and actor-target polarity roles (with explanation) generated by LLMs. If they believe that the LLMs-generated answer is correct, they can select ‘yes’ and move on to the next (Fig. 5b). ... If majority vote from annotators’ choose ‘yes’ on LLMs-generated labels for a text, we consider that a ‘win’ situation. ... We calculate the overall accuracy of the LLMs-generated morality frame prediction task as well as the accuracy and macro average F1 score for the moral foundation prediction task of LLMs."

    The paper's only correctness criterion is the majority 'yes' of annotators who were shown the LLM's label and explanation before judging. Thus 'accuracy' is, by construction, the fraction of texts on which annotators endorse the displayed LLM output, not agreement with an independent gold standard. Because the candidate answer is present before the judgment, anchoring or deference can inflate the 90.79% figure; the metric cannot distinguish correct identification from acquiescence. The 64.81% 'without explanations' condition is the same self-referential measure—annotators still respond to the LLM labels they were given—so the ablation does not provide an independent baseline for accuracy.

full rationale

The central numerical claim—that few-shot prompting with explanations lets LLMs identify morality frames with 90.79% accuracy—is operationalized as the rate at which annotators say 'yes' to the LLM-generated label and explanation they were just shown. Because no external gold-standard labels are used, the 'ground truth' is generated after, and potentially influenced by, the prediction being evaluated. This makes the headline accuracy a measure of annotator endorsement of displayed AI outputs rather than an independent measure of correctness, a self-referential evaluation for the accuracy claim. The 64.81% ablation row is subject to the same issue, since annotators were still judging the LLM labels, not an independent annotation. Other elements—the use of the authors' prior morality-frame formalism and dataset, and self-citations for stances/themes—are not load-bearing circularity for the main accuracy claim, because they supply the task definition and materials rather than the evidence for the claimed prediction quality. The survey claims about reduced difficulty and cognitive load are self-reports, which are not circular but are less probative. Overall, the accuracy result partially reduces to agreement with the displayed prediction, warranting a score of 6.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim sits on three imported pillars: MFT as the label taxonomy, the morality frame role formalism from the authors' earlier work, and the use of the same annotators' post-hoc majority vote as ground truth. The latter is the most fragile, because the annotators judge the LLM outputs after seeing them.

free parameters (2)
  • Majority threshold for 'win' (LLM prediction counted correct) = >=5 of 9 annotators answering 'yes'
    Sec 4.3 defines a text as correct if a majority of annotators endorse the LLM label; a lower bar than requiring consensus or agreement with an external gold standard.
  • Few-shot prompt composition = 7 examples covering six moral foundations plus 'none'
    Sec 4.1: the choice of which examples to include and their order is hand-picked and could affect LLM performance; no sensitivity analysis is reported.
assumptions (4)
  • domain assumption Moral Foundation Theory's six foundations are a valid and complete taxonomy for the moral content in the vaccine tweets.
    The label set is fixed by MFT (Table 1). If the taxonomy does not fit the data, the annotation task and agreement numbers lose meaning.
  • domain assumption The morality frame role formalism (actor, target, polarity) from Roy et al. 2021 and Pacheco et al. 2022 is adopted without independent validation.
    Sec 1 and Sec 3.1 define the output structure using this formalism, which comes from the authors' prior work.
  • ad hoc to paper Annotators' majority vote after viewing LLM outputs is an unbiased measure of correctness.
    Sec 4.3 uses this vote as ground truth; no external labels are used, and the risk of anchoring on the AI answer is not controlled.
  • domain assumption The 150-tweet sample is representative of the vaccination debate on Twitter in the April-October 2021 period.
    Sec 4.2 samples from the Pacheco et al. dataset; no power analysis or representativeness check is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Can LLMs Assist Annotators in Identifying Morality Frames? -- Case Study on Vaccination Debate on Social Media." pith.science (2026). https://pith.science/paper/YB3OGNVL

@misc{pith2026250201991,
  author       = {Pith},
  title        = {Pith review of: Can LLMs Assist Annotators in Identifying Morality Frames? -- Case Study on Vaccination Debate on Social Media},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YB3OGNVL}},
  note         = {Machine review of arXiv:2502.01991}
}
read the original abstract

Nowadays, social media is pivotal in shaping public discourse, especially on polarizing issues like vaccination, where diverse moral perspectives influence individual opinions. In NLP, data scarcity and complexity of psycholinguistic tasks, such as identifying morality frames, make relying solely on human annotators costly, time-consuming, and prone to inconsistency due to cognitive load. To address these issues, we leverage large language models (LLMs), which are adept at adapting new tasks through few-shot learning, utilizing a handful of in-context examples coupled with explanations that connect examples to task principles. Our research explores LLMs' potential to assist human annotators in identifying morality frames within vaccination debates on social media. We employ a two-step process: generating concepts and explanations with LLMs, followed by human evaluation using a "think-aloud" tool. Our study shows that integrating LLMs into the annotation process enhances accuracy, reduces task difficulty, lowers cognitive load, suggesting a promising avenue for human-AI collaboration in complex psycholinguistic tasks.

Figures

Figures reproduced from arXiv: 2502.01991 by the authors.

Figure 1
Figure 1. Morality frame identification with explanation. In [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Prompt template for morality frame identification [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Screenshot of our graphical interface. After click [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Example interface. Annotators can view explana [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Task interface for assessing LLMs generated moral [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Correlation Heatmaps. MF Top 2 Reasons for MF Most Freq. Entity Roles VaccineAgainstReligion 100% anti-vax (vaccine, actor, negative) (Christian, target, negative) sanctity/degradation VaccineNotAgainstReligion 100% pro-vax (vaccine, actor, positive) (I, target, positi…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

82 extracted references · 70 canonical work pages

  1. [57]

    María Leonor Pacheco, Tunazzina Islam, Monal Mahajan, Andrey Shor, Ming Yin, Lyle Ungar, and Dan Goldwasser. 2022. A Holistic Framework for Analyz- ing the COVID-19 Vaccine Debate. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 5821–5839

    work page 2022

  2. [1]

    Woo-kyoung Ahn, William F Brewer, and Raymond J Mooney. 1992. Schema acquisition from a single example. Journal of Experimental Psychology: Learning, Memory, and Cognition 18, 2 (1992), 391

    1992

  3. [2]

    Lola Al-Uqdah, F Abron Franklin, Chu-Chuan Chiu, and Brianna N Boyd. 2022. Associations between social media engagement and vaccine hesitancy. Journal of Community Health 47, 4 (2022), 577–587

    2022

  4. [3]

    Avnika B Amin, Robert A Bednarczyk, Cara E Ray, Kala J Melchiori, Jesse Graham, Jeffrey R Huntsinger, and Saad B Omer. 2017. Association of moral values with vaccine hesitancy. Nature Human Behaviour 1, 12 (2017), 873–880

    work page 2017

  5. [4]

    Pablo Barberá, Anita R Gohdes, Evgeniia Iakhnis, and Thomas Zeitzoff. 2024. Distract and divert: How world leaders use social media during contentious politics. The International Journal of Press/Politics 29, 1 (2024), 47–73

    work page 2024

  6. [5]

    Eric PS Baumer et al . 2020. Topicalizer: reframing core concepts in machine learning visualization by co-designing for interpretivist scholarship. Human– Computer Interaction 35, 5-6 (2020), 452–480

    work page 2020

  7. [6]

    Mariano Gastón Beiró, Jacopo D’Ignazi, Victoria Perez Bustos, María Florencia Prado, and Kyriaki Kalimeri. 2023. Moral narratives around the vaccination debate on facebook. In Proceedings of the ACM Web Conference 2023 . 4134–4141

    work page 2023

  8. [7]

    Sandeep Bhanot. 2012. Use of social media by companies to reach their customers. SIES Journal of Management 8, 1 (2012)

    work page 2012

Show all 82 references
  1. [8]

    Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. Lan- guage (Technology) is Power: A Critical Survey of “Bias” in NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics . 5454– 5476

    2020

  2. [9]

    Erika Bonnevie, Allison Gallegos-Jeffrey, Jaclyn Goldbarg, Brian Byrd, and Joseph Smyser. 2021. Quantifying the rise of vaccine opposition on Twitter during the COVID-19 pandemic. Journal of communication in healthcare 14, 1 (2021), 12–19

    2021

  3. [10]

    Petter Bae Brandtzaeg, Marita Skjuve, and Asbjørn Følstad. 2022. My AI friend: How users of a social chatbot understand their human–AI friendship. Human Communication Research 48, 3 (2022), 404–429

    2022

  4. [11]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901

    2020

  5. [12]

    Joel Brynielsson et al . 2018. Informing crisis alerts using social media: Best practices and proof of concept. Journal of contingencies and crisis management (2018)

    2018

  6. [13]

    Eugene Y Chan. 2021. Moral foundations underlying behavioral compliance during the COVID-19 pandemic. Personality and individual differences 171 (2021), 110463

    2021

  7. [14]

    Nan-Chen Chen, Margaret Drouhard, Rafal Kocielnik, Jina Suh, and Cecilia R Aragon. 2018. Using machine learning to support qualitative coding in social sci- ence: Shifting the focus to ambiguity. ACM Transactions on Interactive Intelligent Systems (TiiS) 8, 2 (2018), 1–20

    2018

  8. [15]

    Wen-ying Sylvia Chou, Yvonne M Hunt, Ellen B Beckjord, Richard P Moser, and Bradford W Hesse. 2009. Social media use in the United States: implications for health communication. Journal of medical Internet research 11, 4 (2009), e1249

    2009

  9. [16]

    Aakanksha Chowdhery, Sharan Narang, et al . 2023. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research (2023)

    2023

  10. [17]

    Rodrigo Diaz and Florian Cova. 2021. Reactance, morality, and disgust: The relationship between affective dispositions and compliance with official health recommendations during the COVID-19 pandemic. Cognition and Emotion (2021), 1–17

    2021

  11. [18]

    Bosheng Ding, Chengwei Qin, Linlin Liu, Yew Ken Chia, Boyang Li, Shafiq Joty, and Lidong Bing. 2023. Is GPT-3 a Good Data Annotator?. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 11173–11195

    2023

  12. [19]

    Emilio Ferrara, Herbert Chang, Emily Chen, Goran Muric, and Jaimin Patel. 2020. Characterizing social media manipulation in the 2020 US presidential election. First Monday (2020)

    2020

  13. [20]

    Emilio Ferrara, Stefano Cresci, and Luca Luceri. 2020. Misinformation, manipula- tion, and abuse on social media in the era of COVID-19.Journal of Computational Social Science 3 (2020), 271–277

    2020

  14. [21]

    Simret Araya Gebreegziabher, Zheng Zhang, Xiaohang Tang, Yihao Meng, Elena L Glassman, and Toby Jia-Jun Li. 2023. Patat: Human-ai collaborative qualitative coding with explainable interactive rule synthesis. In Proceedings of the 2023 CHI Conference on Human Factors in Computi...

    2023

  15. [22]

    Rebekah Getman, Mohammad Helmi, Hal Roberts, Alfa Yansane, David Cutler, and Brittany Seymour. 2018. Vaccine hesitancy and online information: the influence of digital networks. Health Education & Behavior 45, 4 (2018), 599–606

    2018

  16. [23]

    Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023. ChatGPT outperforms crowd workers for text-annotation tasks. Proceedings of the National Academy of Sciences 120, 30 (2023), e2305016120

    2023

  17. [24]

    Jesse Graham, Jonathan Haidt, and Brian A Nosek. 2009. Liberals and conserva- tives rely on different sets of moral foundations. JPSP (2009)

    2009

  18. [25]

    Jonathan Haidt and Jesse Graham. 2007. When morality opposes justice: Con- servatives have moral intuitions that liberals may not recognize. Social Justice Research 20, 1 (2007), 98–116

    2007

  19. [26]

    Jonathan Haidt and Craig Joseph. 2004. Intuitive ethics: How innately prepared intuitions generate culturally variable virtues. Daedalus 133, 4 (2004), 55–66

    2004

  20. [27]

    Raquel G Hernandez, Loni Hagen, Kimberly Walker, Heather O’Leary, and Cecile Lengacher. 2021. The COVID-19 vaccine social media infodemic: healthcare providers’ missed dose in addressing misinformation and vaccine hesitancy. Human Vaccines & Immunotherapeutics 17, 9 (2021), 2962–2964

    2021

  21. [28]

    Gabriela Hoefer et al . 2022. Bridging the Social Distance: Offline to Online Social Support during the COVID-19 Pandemic. Proceedings of the ACM on Human-Computer Interaction CSCW2 (2022)

    2022

  22. [29]

    Fan Huang, Haewoon Kwak, and Jisun An. 2023. Is chatgpt better than human annotators? potential and limitations of chatgpt in explaining implicit hate speech. In Companion proceedings of the ACM web conference 2023 . 294–297

    2023

  23. [30]

    Tunazzina Islam and Dan Goldwasser. 2022. Understanding COVID-19 Vaccine Campaign on Facebook using Minimal Supervision. In 2022 IEEE International Conference on Big Data (Big Data) . IEEE, 585–595. WebSci’25, May 20–24,2025, New Brunswick, NJ, USA Tunazzina Islam and Dan Goldwasser

    2022

  24. [31]

    Tunazzina Islam and Dan Goldwasser. 2024. Discovering Latent Themes in Social Media Messaging: A Machine-in-the-Loop Approach Integrating LLMs. arXiv preprint arXiv:2403.10707 (2024)

    2024 arXiv

  25. [32]

    Tunazzina Islam and Dan Goldwasser. 2024. Post-hoc Study of Climate Mi- crotargeting on Social Media Ads with LLMs: Thematic Insights and Fairness Evaluation. arXiv preprint arXiv:2410.05401 (2024)

    2024

  26. [33]

    Tunazzina Islam and Dan Goldwasser. 2024. Uncovering Latent Arguments in Social Media Messaging by Employing LLMs-in-the-Loop Strategy.arXiv preprint arXiv:2404.10259 (2024)

    2024 arXiv

  27. [34]

    Tunazzina Islam, Shamik Roy, and Dan Goldwasser. 2023. Weakly Supervised Learning for Analyzing Political Campaigns on Facebook. In Proceedings of the International AAAI Conference on Web and Social Media , Vol. 17. 411–422

    2023

  28. [35]

    Tunazzina Islam, Ruqi Zhang, and Dan Goldwasser. 2023. Analysis of Climate Campaigns on Social Media Using Bayesian Model Averaging. In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society . 15–25

    2023

  29. [36]

    Jialun Aaron Jiang, Kandrea Wade, Casey Fiesler, and Jed R Brubaker. 2021. Sup- porting serendipity: Opportunities and challenges for Human-AI Collaboration in qualitative analysis. Proceedings of the ACM on Human-Computer Interaction 5, CSCW1 (2021), 1–23

    2021

  30. [37]

    Kristen Johnson and Dan Goldwasser. 2018. Classification of moral foundations in microblog political discourse. In Proceedings of the 56th annual meeting of the association for computational linguistics (volume 1: long papers) . 720–730

    2018

  31. [38]

    S Mo Jones-Jang and Myojung Chung. 2024. Can we blame social media for polarization? Counter-evidence against filter bubble claims during the COVID-19 pandemic. New Media & Society 26, 6 (2024), 3370–3389

    2024

  32. [39]

    Minsol Kim, Aliea L Nallbani, and Abby Rayne Stovall. 2024. Exploring LLM- based Chatbot for Language Learning and Cultivation of Growth Mindset. In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems

    2024

  33. [40]

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems 35 (2022), 22199–22213

    2022

  34. [41]

    Mahi Kolla, Siddharth Salunkhe, Eshwar Chandrasekharan, and Koustuv Saha

  35. [42]

    Klaus Krippendorff. 2004. Measuring the reliability of qualitative text analysis data. Quality and quantity 38 (2004), 787–800

    2004

  36. [43]

    Andrew Lampinen, Ishita Dasgupta, et al . 2022. Can language models learn from explanations in context?. In Findings of the Association for Computational Linguistics: EMNLP 2022

    2022

  37. [44]

    Teven Le Scao, Angela Fan, et al. 2022. Bloom: A 176b-parameter open-access multilingual language model. (2022)

    2022

  38. [45]

    Susan Lin, Jeremy Warner, et al. 2024. Rambler: Supporting Writing With Speech via LLM-Assisted Gist Manipulation. In CHI

    2024

  39. [46]

    Tania Lombrozo. 2006. The structure and function of explanations. Trends in cognitive sciences 10, 10 (2006), 464–470

    2006

  40. [47]

    Tania Lombrozo and Susan Carey. 2006. Functional explanation and the function of explanation. Cognition 99, 2 (2006), 167–204

    2006

  41. [48]

    Xinhao Mei et al. 2024. Wavcaps: A chatgpt-assisted weakly-labelled audio cap- tioning dataset for audio-language multimodal research. IEEE/ACM Transactions on Audio, Speech, and Language Processing (2024)

    2024

  42. [49]

    Joy Ming, Sharon Heung, Shiri Azenkot, and Aditya Vashistha. 2021. Accept or address? Researchers’ perspectives on response bias in accessibility research. In Proceedings of the 23rd International ACM SIGACCESS Conference on Computers and Accessibility. 1–13

    2021

  43. [50]

    Swaroop Mishra, Daniel Khashabi, Chitta Baral, Yejin Choi, and Hannaneh Ha- jishirzi. 2022. Reframing Instructional Prompts to GPTk’s Language. In Findings of the Association for Computational Linguistics: ACL 2022 . 589–612

    2022

  44. [51]

    Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2022. Cross-Task Generalization via Natural Language Crowdsourcing Instructions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . 3470–3487

    2022

  45. [52]

    Patricia Moravec, Randall Minas, and Alan R Dennis. 2018. Fake news on social media: People believe what they want to believe when it makes no sense at all. Kelley School of Business research paper 18-87 (2018)

    2018

  46. [53]

    Marcia Mundt, Karen Ross, and Charla M Burnett. 2018. Scaling social movements through social media: The case of Black Lives Matter. Social media+ society 4, 4 (2018), 2056305118807911

    2018

  47. [54]

    OpenAI. 2024. Hello gpt-4o. https://openai.com/index/hello-gpt-4o/, 2024. (2024)

    2024

  48. [55]

    Daniela Orr, Ayelet Baram-Tsabari, and Keren Landsman. 2016. Social media as a platform for health-related public debates and discussions: the Polio vaccine on Facebook. Israel journal of health policy research 5 (2016), 1–11

    2016

  49. [56]

    Maria Leonor Pacheco and Dan Goldwasser. 2021. Modeling Content and Context with Deep Relational Learning. Transactions of the Association for Computational Linguistics (2021)

    2021

  50. [58]

    Maria Leonor Pacheco, Tunazzina Islam, Lyle Ungar, Ming Yin, and Dan Gold- wasser. 2022. Interactively uncovering latent arguments in social media platforms: A case study on the covid-19 vaccine debate. InProceedings of the Fourth Workshop on Data Science with Human-in-the-Loo...

    2022

  51. [59]

    Maria Leonor Pacheco, Tunazzina Islam, Lyle Ungar, Ming Yin, and Dan Gold- wasser. 2023. Interactive Concept Learning for Uncovering Latent Themes in Large Text Collections. InFindings of the Association for Computational Linguistics: ACL 2023

    2023

  52. [60]

    Stefano Pagliaro et al. 2021. Trust predicts COVID-19 prescribed and discretionary behavioral intentions in 23 countries. PloS one (2021)

    2021

  53. [61]

    Neha Puri, Eric A Coomes, Hourmazd Haghbayan, and Keith Gunaratne. 2020. Social media and vaccine hesitancy: new updates for the era of COVID-19 and globalized infectious diseases. Human vaccines & immunotherapeutics (2020)

    2020

  54. [62]

    Yasmim Mendes Rocha et al. 2021. The impact of fake news on social media and its influence on health during the COVID-19 pandemic: A systematic review. Journal of Public Health (2021), 1–10

    2021

  55. [63]

    Jon Roozenbeek, Claudia R Schneider, et al. 2020. Susceptibility to misinformation about COVID-19 around the world. R. Soc. Open Sci. (2020)

    2020

  56. [64]

    Isabel Rossen, Mark J Hurlstone, Patrick D Dunlop, and Carmen Lawrence. 2019. Accepters, fence sitters, or rejecters: Moral profiles of vaccination attitudes.Social Science & Medicine 224 (2019), 23–27

    2019

  57. [65]

    Shamik Roy, Nishanth Sridhar Nakshatri, and Dan Goldwasser. 2022. Towards Few-Shot Identification of Morality Frames using In-Context Learning. In Pro- ceedings of the Fifth Workshop on Natural Language Processing and Computational Social Science (NLP+ CSS) . 183–196

    2022

  58. [66]

    Shamik Roy, María Leonor Pacheco, and Dan Goldwasser. 2021. Identifying Morality Frames in Political Tweets using Relational Learning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing . 9939–9958

    2021

  59. [67]

    Arnaldo Santoro, Alessandro Galeazzi, Teresa Scantamburlo, Andrea Baronchelli, Walter Quattrociocchi, and Fabiana Zollo. 2023. Analyzing the changing land- scape of the Covid-19 vaccine debate on Twitter. Social Network Analysis and Mining 13, 1 (2023), 115

    2023

  60. [68]

    Ana Lucía Schmidt, Fabiana Zollo, Antonio Scala, Cornelia Betsch, and Walter Quattrociocchi. 2018. Polarization of the vaccination debate on Facebook.Vaccine 36, 25 (2018), 3606–3612

    2018

  61. [69]

    Eva Sharma, Koustuv Saha, Sindhu Kiranmai Ernala, Sucheta Ghoshal, and Mun- mun De Choudhury. 2017. Analyzing ideological discourse on social media: A case study of the abortion debate. In Proceedings of the 2017 international conference of the computational social science soc...

    2017

  62. [70]

    Laura M Smith, Linhong Zhu, Kristina Lerman, and Zornitsa Kozareva. 2013. The role of social media in the discussion of controversial topics. In2013 International Conference on Social Computing . IEEE, 236–243

    2013

  63. [71]

    pandemic

    Fabio Tagliabue, Luca Galassi, and Pierpaolo Mariani. 2020. The “pandemic” of disinformation in COVID-19. SN comprehensive clinical medicine (2020)

    2020

  64. [72]

    Kathie M d’I Treen, Hywel TP Williams, and Saffron J O’Neill. 2020. Online misinformation about climate change. Wiley Interdisciplinary Reviews: Climate Change 11, 5 (2020), e665

    2020

  65. [73]

    Shuohang Wang, Yang Liu, Yichong Xu, Chenguang Zhu, and Michael Zeng. 2021. Want To Reduce Labeling Cost? GPT-3 Can Help. InFindings of the Association for Computational Linguistics: EMNLP 2021 . 4195–4205

    2021

  66. [74]

    Yuping Wang, Chen Ling, and Gianluca Stringhini. 2023. Understanding the use of images to spread COVID-19 misinformation on Twitter. Proceedings of the ACM on Human-Computer Interaction 7, CSCW1 (2023), 1–32

    2023

  67. [75]

    Jason Wei et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems 35 (2022), 24824–24837

    2022

  68. [76]

    Maxwell A Weinzierl and Sanda M Harabagiu. 2022. From hesitancy framings to vaccine hesitancy profiles: A journey of stance, ontological commitments and moral foundations. In Proceedings of the International AAAI Conference on Web and Social Media, Vol. 16. 1087–1097

    2022

  69. [77]

    Liang Wu, Fred Morstatter, Kathleen M Carley, and Huan Liu. 2019. Misinfor- mation in social media: definition, manipulation, and detection. ACM SIGKDD explorations newsletter 21, 2 (2019), 80–90

    2019

  70. [78]

    Xiaotong Xu, Jiayu Yin, et al. 2024. Jamplate: Exploring LLM-Enhanced Templates for Idea Reflection. In IUI

    2024

  71. [79]

    Zhiying Yue, Renwen Zhang, and Jun Xiao. 2022. Passive social media use and psychological well-being during the COVID-19 pandemic: The role of social comparison and emotion regulation. Computers in Human Behavior (2022)

    2022

  72. [80]

    Bazarova, and Madhu Reddy

    Renwen Zhang, Natalya N. Bazarova, and Madhu Reddy. 2021. Distress disclosure across social media platforms during the COVID-19 pandemic: Untangling the effects of platforms, affordances, and audiences. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems

    2021

  73. [81]

    Wei Zhao et al. 2023. Older adults using technology for meaningful activities during COVID-19: An analysis through the lens of self-determination theory. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems

    2023

  74. [2024]

    In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems

    LLM-Mod: Can Large Language Models Assist Content Moderation?. In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.