Pith. sign in

REVIEW 4 major objections 3 minor 37 references

Hybrid Annotation for Propaganda Detection: Integrating LLM Pre-Annotations with Human Intelligence

T0 review · 4 major / 3 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper shows that LLM-generated pre-annotations with human verification raise fine-grained propaganda-label agreement and cut annotation time.

desk verdict The three-way propaganda taxonomy and LLM span pre-annotation are useful, but the uncontrolled Study 2 design does not establish that LLM assistance caused the agreement and time gains. read the letter →

arxiv 2507.18343 v1 pith:HCMZXGLQ submitted 2025-07-24 cs.CL

Ariana Sahitaj , Premtim Sahitaj , Veronika Solopova , Jiaao Li , Sebastian Möller , Vera Schmitt This is my paper · ORCID
classification cs.CL
keywords propagandadetectionLLM-assistedannotationhuman-in-the-loopinter-annotatoragreementspanextractionknowledgedistillationhierarchicaltaxonomysocialmediadisinformation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a hybrid annotation workflow—a large language model first extracts propagandistic spans, writes short explanations, and assigns fine-grained technique labels, then human annotators verify by choosing coarse and fine labels for the whole tweet—produces more consistent and faster propaganda annotations than fully manual labeling. In the unassisted study, three annotators fully agreed on only 7.61% of fine-grained labels; in the assisted study, full three-way agreement rose to 47.89%, Krippendorff's alpha went from 0.1233 to 0.5941, and average annotation time fell from 151.70 to 41.14 seconds per tweet. The paper also groups the 14 propaganda techniques into three coarse categories intended to ease annotator decisions, and it distills the LLM's structured outputs into smaller open models that can detect spans without human-labeled training data. If the claimed improvement holds, it points to a scalable way to build larger, more consistent propaganda-detection datasets for social media.

What carries the argument

The load-bearing mechanism is the LLM pre-annotation pipeline: a large open model (LLaMA3-70B-Instruct) extracts propagandistic spans, explains why each is propagandistic, assigns a local fine-grained label to each span, and then selects a global label for the tweet, with outputs forced into a structured JSON format. The global label is hidden from human annotators, who verify by selecting coarse and fine labels for the whole tweet with the spans and local labels visible. That design converts the hardest part of the task—finding and naming the manipulative segment from scratch—into a verification task, which is what carries the reported gains in agreement and speed.

What would settle it

A controlled study using the same annotators on the same tweet sample under three conditions—manual labeling, LLM pre-annotation with correct spans, and LLM pre-annotation with randomly generated spans—would settle the claim. If the random-span condition shows the same agreement and time gains, the effect comes from task simplification or anchoring, not from the LLM's content; if only the correct-span condition improves, the central claim is supported.

Watch

Extended reading notes

Core claim

The paper's central claim is that LLM-generated pre-annotations, when combined with human verification, substantially improve both consistency and efficiency in fine-grained propaganda annotation. Presented with the LLM's extracted spans and local labels but not its global prediction, human annotators reached full three-way agreement on 47.89% of tweets, up from 7.61% with unassisted labeling, while Krippendorff's alpha improved from 0.1233 to 0.5941. The paper also reports that the human majority-vote global label agreed strongly with the LLM's hidden global label (Cohen's kappa 0.8438), and that annotation time dropped by roughly 73%. On tweets where all annotators agreed on the coarse category, fine-grained 2/3 agreement reached 99.55%, which the paper reads as evidence that the three-category taxonomy and the span-level pre-annotation jointly make fine-grained decisions tractable.

Load-bearing premise

The load-bearing assumption is that Study 2's higher agreement and lower annotation time are caused by the LLM pre-annotations rather than by the simultaneous change of annotators, the switch to stratified sampling, the shift from free labeling to verification, or the annotators' growing familiarity with the task.

Editorial extensions

If this is right

  • Fine-grained propaganda annotation can be scaled beyond fully manual effort: verification takes about 41 seconds per tweet versus 152 seconds for unassisted labeling, so larger corpora become feasible at similar cost.
  • Global-label-only annotation is insufficient, because most propagandistic tweets contain multiple spans; keeping span extraction and local labels as the primary output preserves information that a single label discards.
  • Small open models distilled on LLM outputs can find propaganda spans (fuzzy-span F1 of 0.60–0.67) but remain weak at assigning local techniques (0.21–0.29 F1), so human verification is most valuable at the label-assignment step.
  • The coarse taxonomy acts as a useful intermediate layer: fine-grained 2/3 agreement reaches 99.55% on tweets where all annotators agree on the coarse category, suggesting hierarchical labeling reduces confusion among fine-grained techniques.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported gains are not cleanly attributable to the LLM pre-annotations, because the second study also replaced the most experienced annotator, switched from random to stratified sampling, and changed the task from free labeling to verification; a controlled comparison holding annotators and sampling fixed would be needed to separate causes.
  • If the agreement gain comes mainly from anchoring on the displayed spans rather than their correctness, then even wrong pre-annotations could raise agreement, which would inflate confidence in LLM-assisted pipelines; a study with deliberately incorrect spans would reveal this.
  • The alignment between majority local labels and global labels (83.55% when a majority exists) could be used as a model-confidence signal to route only uncertain tweets to human verification, making the pipeline cheaper without losing consistency.
  • Because the student models inherit the teacher's training-data biases, the distilled pipeline should be audited on text from opposing political directions; the paper flags this but does not test it.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The paper proposes a hybrid annotation framework for propaganda detection on tweets: a hierarchical taxonomy of 14 propaganda techniques into 3 intent-based categories; an initial manual annotation study (Study 1) on a sample of 355 tweets from the HQP dataset; an LLM pipeline (LLaMA-3.3-70B) that extracts spans, explains them, and assigns local and global labels; a second human verification study (Study 2) where annotators see the LLM's spans and local labels and select the global label; and knowledge distillation to fine-tune smaller models on LLM-generated annotations. The central claim, stated in Section 3.4.2, is that Study 2 achieves higher inter-annotator agreement and lower annotation time than Study 1, and that the student models learn to reproduce the teacher's structured annotations.

Significance. If the causal claim were established, the hybrid workflow would be a valuable contribution to scalable propaganda annotation. The paper ships publicly available code, uses a fully open-source teacher model, reports detailed ablations on prompt randomness and text normalization, and provides a large LLM-annotated dataset. These are concrete strengths. However, the central claim is not currently supported because the comparison between Study 1 and Study 2 is confounded by simultaneous changes in annotators, sampling, and task design, and the secondary human-LLM agreement metric is mechanically inflated by imputing random LLM labels as human labels. The student-model evaluation is a teacher-student consistency check rather than a quality evaluation. With additional controlled experiments or appropriately weakened claims, the work could still be a useful proof-of-concept, but the current evidence does not justify the stated conclusions.

major comments (4)
  1. [Section 3.4 vs. Section 3.2, Table 5] The improvement in IAA and annotation time between Study 1 and Study 2 is not attributable to the LLM pre-annotation because several factors changed simultaneously: the annotator pool (the most experienced annotator was removed and replaced), the sampling method (random vs. stratified), and the task itself (free coarse/fine labeling of whole tweets vs. selecting global labels while being shown LLM spans and local labels). The paper further acknowledges a possible learning effect in Section 4 and in the Limitations. No significance tests are reported for the differences in Krippendorff's alpha or annotation time. Consequently, the central claim in Section 3.4.2 that the integration 'resulted in higher IAA and reduced annotation time' is confounded and should be reframed as an observational comparison, or the studies need to be redesigned to control these variables.
  2. [Section 3.4.2 (Cohen's Kappa)] The reported Cohen's Kappa of 0.8438 between human majority-vote labels and LLM global labels is mechanically inflated. When no 2/3 human majority is reached, a random LLM prediction is imputed as the 'human' label, so those cases contribute LLM self-agreement rather than human-LLM agreement. The Kappa should be computed only on cases with a genuine human majority, or with a proper treatment of disagreement, and should be reported with confidence intervals.
  3. [Section 3.3.1 and Section 3.4 (hidden global label)] The independence of the human global-label choice in Study 2 is not assured. Section 3.3.1 shows that for tweets with at least three spans, the first local label matches the global label in 76.65% of cases and the majority local label matches in 83.55%. Since annotators in Study 2 see these local labels, the 'hidden' global label is largely inferable from the visible spans and labels. The higher agreement in Table 5 may therefore partly reflect annotators converging on the LLM's pre-annotations (anchoring) rather than independent verification. The authors should test this by comparing Study 2 agreement on cases where local labels are predictive of the global label versus cases where they are not.
  4. [Section 3.5, Table 7] The evaluation of the student models in Table 7 is a teacher-student consistency measure: the students are trained on the teacher's (LLaMA-3.3-70B) annotations and evaluated on a held-out split of the same teacher's labels. This does not measure the quality of propaganda detection. There is no human-annotation baseline, no comparison to models trained on human labels, and no measure of agreement between student outputs and human judgment. The claims about scalable propaganda detection therefore need to be supported by human evaluation or at least by comparison on a dataset with gold human labels.
minor comments (3)
  1. [Section 3.5] The text says 'we fine-tune SMLs'; this should be 'SLMs' (small language models).
  2. [Section 2 and References] There is a typo 'SemEvak-2020' in Section 2, which should be 'SemEval-2020'. Also, the reference for the SemEval-2020 paper lists the first author as 'G Martino' instead of 'Giovanni Da San Martino', and the two Hasanain et al. references use inconsistent spellings of the second author (Ahmad vs. Ahmed).
  3. [Appendix A.1 and Table 8] The label list in Appendix A.1 omits an entry 'n' and continues with 'o', and Table 8 lists 'whataboutism', 'straw_man', and 'red_herring' as separate labels, whereas the taxonomy defines them as a combined category. Please clarify how the 14 labels relate to the 17 entries in Table 8.

Circularity Check

2 steps flagged · score 6.0 of 10

Partial circularity: LLM labels define the 'human' labels in the kappa computation, and student-model F1 scores measure agreement with the teacher that produced both training and test data.

  1. self definitional [Section 3.4.2, Cohen's Kappa computation between human majority-vote labels and LLM-generated global labels]
    "Additionally, Cohen's Kappa was calculated to measure agreement between human majority-vote labels and LLM-generated global labels. If no 2/3 majority was reached, a random LLM prediction was used as the human label. The resulting Cohen's Kappa score of 0.8438 indicates strong agreement between human annotations and LLM-generated global labels."

    Whenever the three annotators failed to produce a 2/3 majority, the 'human' label is defined as a random LLM prediction. The comparison 'human vs. LLM' then contains cases where both sides are LLM outputs, so the reported agreement is partly LLM self-agreement. The metric is constructed from the very model it is supposed to validate, making the 0.8438 kappa a self-referential quantity rather than an independent measure.

  2. fitted input called prediction [Section 3.5 and 3.5.1 (Knowledge Distillation setup and Results, Table 7)]
    "In this knowledge-distillation-inspired setup, the 70B model as described in Section 3.3 serves as the teacher, providing structured propaganda annotations for every data point. ... We utilize a stratified 80/20 split and learn on the train split for three epochs. ... We report six evaluation metrics on the unseen test set as reported in Table 7. ... All four student models achieve reasonable performance on each metric."

    The teacher generates the labels for every data point; the 80/20 split is taken from that teacher-annotated corpus, so the 'unseen test set' has no human gold labels. The reported F1/G metrics therefore measure how closely the students reproduce the teacher's outputs, not how accurately they detect propaganda. Presenting these distillation-fidelity numbers as 'student model evaluation results' with 'reasonable performance' treats agreement with the training-source model as if it were external prediction quality.

full rationale

The human annotation studies themselves are not definitionally circular: Study 1 reports direct IAA on human labels, and Study 2 reports human IAA under a changed protocol, so those numbers are real measurements. The strongest circularity is in the Cohen's Kappa computation, where missing human majorities are filled with random LLM predictions, making human--LLM agreement partially self-agreement by construction. A second partial circularity appears in the knowledge-distillation evaluation: student models are trained on teacher-generated labels and evaluated on a split of the same teacher-generated labels, so the reported F1 scores quantify teacher--student consistency rather than independent detection quality. The paper's central causal claim that LLM pre-annotation 'resulted in' higher IAA and lower annotation time is also threatened by confounds (changed task, changed annotators, stratified sampling, and the acknowledged learning effect), but that is an internal-validity problem rather than a definitional circularity. No load-bearing self-citation chain was found; the taxonomy and LLM pipeline are independently presented.

Assumptions & free parameters 4 free parameters · 7 assumptions · 0 invented entities

The paper introduces a hierarchical taxonomy and a pipeline, but no new physical or formal entities. The axiom ledger mainly captures the assumptions that the HQP binary labels are sound, that the LLM outputs are trustworthy, and that the teacher's labels can serve as ground truth for student evaluation. The free parameters include thresholds and rules that directly affect the reported agreement and F1 scores.

free parameters (4)
  • Fuzzy span similarity threshold = 0.8
    Defines Spanf and Localf metrics; changing it would change reported student F1 scores.
  • Majority-vote imputation rule = random LLM prediction
    When no 2/3 human majority, the paper substitutes a random LLM prediction as the human label in the Cohen's kappa computation, mechanically inflating human-LLM agreement.
  • Fine-tuning epochs = 3
    Student models trained for three epochs on the teacher-generated data; choice affects distillation performance.
  • Stability repetitions = 5
    Stability test repeats LLM annotation five times; the 5/5 and >=4/5 thresholds are arbitrary.
assumptions (7)
  • domain assumption The 14 propaganda techniques of Martino et al. (2020) are an appropriate and complete label set for short-form propaganda.
    Invoked throughout; the paper adopts the 14 labels without validating them on HQP tweets.
  • domain assumption The binary HQP propaganda labels are reliable, so the analysis can be restricted to the 4,534 propagandistic tweets without confounding binary misclassification.
    Section 3.2 states 'Assuming that the binary classification of propaganda versus non-propaganda is reliable'.
  • ad hoc to paper Grouping the 14 techniques into three intent-based categories reduces cognitive load and improves annotator consistency.
    Section 3.1 introduces the taxonomy; no independent evidence is provided beyond the internal Study 1/Study 2 comparison.
  • domain assumption The LLM's span extractions and explanations are accurate enough to serve as helpful pre-annotations.
    Section 3.4 relies on LLM outputs being useful; the manual analysis in Section 3.3.1 found that 30 of 94 no-span cases were actually propagandistic.
  • standard math The Stuart-Maxwell test is appropriate for paired categorical comparisons of normalized versus non-normalized tweets.
    Section 3.3.2 applies the test without a citation or a check of its assumptions for this data.
  • domain assumption The LLM-generated labels can serve as high-quality supervision for training student models without human verification on the training set.
    Section 3.5 trains exclusively on LLM outputs; the paper acknowledges teacher bias in Limitations.
  • ad hoc to paper The teacher's outputs on the held-out test split provide a valid evaluation of student model quality.
    Section 3.5.1 uses teacher-generated labels as ground truth, meaning student F1 scores measure imitation of the teacher, not independent detection quality.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Hybrid Annotation for Propaganda Detection: Integrating LLM Pre-Annotations with Human Intelligence." pith.science (2026). https://pith.science/paper/HCMZXGLQ

@misc{pith2026250718343,
  author       = {Pith},
  title        = {Pith review of: Hybrid Annotation for Propaganda Detection: Integrating LLM Pre-Annotations with Human Intelligence},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HCMZXGLQ}},
  note         = {Machine review of arXiv:2507.18343}
}
read the original abstract

Propaganda detection on social media remains challenging due to task complexity and limited high-quality labeled data. This paper introduces a novel framework that combines human expertise with Large Language Model (LLM) assistance to improve both annotation consistency and scalability. We propose a hierarchical taxonomy that organizes 14 fine-grained propaganda techniques into three broader categories, conduct a human annotation study on the HQP dataset that reveals low inter-annotator agreement for fine-grained labels, and implement an LLM-assisted pre-annotation pipeline that extracts propagandistic spans, generates concise explanations, and assigns local labels as well as a global label. A secondary human verification study shows significant improvements in both agreement and time-efficiency. Building on this, we fine-tune smaller language models (SLMs) to perform structured annotation. Instead of fine-tuning on human annotations, we train on high-quality LLM-generated data, allowing a large model to produce these annotations and a smaller model to learn to generate them via knowledge distillation. Our work contributes towards the development of scalable and robust propaganda detection systems, supporting the idea of transparent and accountable media ecosystems in line with SDG 16. The code is publicly available at our GitHub repository.

Figures

Figures reproduced from arXiv: 2507.18343 by the authors.

Figure 1
Figure 1. Our proposed LLM output for a reduced input [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Methodological Overview GPT-3 and GPT-4 variants for multi-label classifi￾cation of 14 propaganda techniques at article-level using the SemEval-2020 Task 11 dataset (Martino et al., 2020), employing a range of prompt engi￾neering and fine-tuning strategies. Their results show that GPT-4 can approach state-of-the-art per￾formance. Our work builds on these efforts by grouping the 14 fine-grained techniques (Martino et… view at source ↗
Figure 3
Figure 3. Example of LLM-assisted annotation, show [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: The example highlights how fact-checking [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]
Figure 6
Figure 6. Figure 6: Initially annotated as propaganda and ’loaded [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 5
Figure 5. Figure 5: An example initially annotated as propa [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 7
Figure 7. Figure 7: Prompt (Part 1): Initial instructions for the [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8: Prompt (Part 2): JSON output format defini [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

37 extracted references · 25 canonical work pages

  1. [1]

    Malak Abdullah, Ola Altiti, and Rasha Obiedat. 2022. Detecting propaganda techniques in english news articles using pre-trained transformers. In 2022 13th International Conference on Information and Communication Systems (ICICS), pages 301--308. IEEE

    work page 2022

  2. [2]

    Sirwan Khalid Ahmed. 2024. How to choose a sampling technique and determine sample size for research: a simplified guide for researchers. Oral Oncology Reports, 12:100662

    work page 2024

  3. [3]

    AI@Meta. 2024. https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md Llama 3 model card

    2024

  4. [4]

    Meysam Alizadeh, Ma \"e l Kubli, Zeynab Samei, Shirin Dehghani, Mohammadmasiha Zahedivafa, Juan D Bermeo, Maria Korobeynikova, and Fabrizio Gilardi. 2025. Open-source llms for text annotation: a practical guide for model setting and fine-tuning. Journal of Computational Social Science, 8(1):1--25

    work page 2025

  5. [5]

    Alberto Barr \'o n-Cedeno, Israa Jaradat, Giovanni Da San Martino, and Preslav Nakov. 2019. Proppy: Organizing the news based on their propagandistic content. Information Processing & Management, 56(5):1849--1864

    work page 2019

  6. [6]

    Judit Bayer, Bernd Holznagel, Katarzyna Lubianiec, Adela Pintea, Josephine B Schmitt, Judit Szak \'a cs, and Erik Uszkiewicz. 2021. Disinformation and propaganda: impact on the functioning of the rule of law and democratic processes in the eu and its member states. European Union

    work page 2021

  7. [7]

    Giovanni Da San Martino, Alberto Barron-Cedeno, and Preslav Nakov. 2019. Findings of the nlp4if-2019 shared task on fine-grained propaganda detection. In Proceedings of the second workshop on natural language processing for internet freedom: censorship, disinformation, and propaganda, pages 162--170

    work page 2019

  8. [8]

    Bosheng Ding, Chengwei Qin, Linlin Liu, Yew Ken Chia, Shafiq Joty, Boyang Li, and Lidong Bing. 2022. Is gpt-3 a good data annotator? arXiv preprint arXiv:2212.10450

    arXiv 2022

Show all 37 references
  1. [9]

    Aaron Erlich and Calvin Garner. 2023. Is pro-kremlin disinformation effective? evidence from ukraine. The International Journal of Press/Politics, 28(1):5--28

    2023

  2. [10]

    Gabriel C Gherasim. 2022. Reductio ad hitlerum: Reflections on the russian propaganda of de-nazification in ukraine. Romanian Journal of Political Sciences, 22(1):75--86

    2022

  3. [11]

    Fabrizio Gilardi, Meysam Alizadeh, and Ma \"e l Kubli. 2023. Chatgpt outperforms crowd workers for text-annotation tasks. Proceedings of the National Academy of Sciences, 120(30):e2305016120

    2023

  4. [12]

    Maram Hasanain, Fatema Ahmad, and Firoj Alam. 2024. https://aclanthology.org/2024.lrec-main.244/ Can GPT -4 identify propaganda? annotation and detection of propaganda spans in news articles . In Proceedings of the 2024 Joint International Conference on Computational Linguisti...

    2024

  5. [13]

    Maram Hasanain, Fatema Ahmed, and Firoj Alam. 2023. Large language models for propaganda span annotation. arXiv preprint arXiv:2311.09812

    2023 arXiv

  6. [14]

    Julia Jose and Rachel Greenstadt. 2025. Are large language models good at detecting propaganda? arXiv preprint arXiv:2505.13706

    2025 arXiv

  7. [15]

    Garth S Jowett and Victoria O'donnell. 2018. Propaganda & persuasion. Sage publications

    2018

  8. [16]

    Minhyeok Lee. 2023. A mathematical investigation of hallucination and creativity in gpt models. Mathematics, 11(10):2320

    2023

  9. [17]

    Irina Lock and Ramona Ludolph. 2020. Organizational propaganda on the internet: A systematic review. Public Relations Inquiry, 9(1):103--127

    2020

  10. [18]

    Abdurahman Maarouf, Dominik B \"a r, Dominique Geissler, and Stefan Feuerriegel. 2023. Hqp: a human-annotated dataset for detecting online propaganda. arXiv preprint arXiv:2304.14931

    2023 arXiv

  11. [19]

    G Martino, Alberto Barr \'o n-Cedeno, Henning Wachsmuth, Rostislav Petrov, and Preslav Nakov. 2020. Semeval-2020 task 11: Detection of propaganda techniques in news articles. arXiv preprint arXiv:2009.02696

    2020 arXiv

  12. [20]

    Giovanni Da San Martino, Seunghak Yu, Alberto Barr \'o n-Cede \ n o, Rostislav Petrov, and Preslav Nakov. 2019. Fine-grained analysis of propaganda in news articles. arXiv preprint arXiv:1910.02517

    2019 arXiv

  13. [21]

    Eric Mwangi. 2023. Technology and fake news: shaping social, political, and economic perspectives. Political, and Economic Perspectives (May 29, 2023)

    2023

  14. [22]

    E Perez. 2022. Strategic disinformation: Russia, ukraine and crisis communication in digital era

    2022

  15. [23]

    Jakub Piskorski, Nicolas Stefanovitch, Giovanni Da San Martino, and Preslav Nakov. 2023. https://doi.org/10.18653/v1/2023.semeval-1.317 S em E val-2023 task 3: Detecting the category, the framing, and the persuasion techniques in online news in a multi-lingual setup . In Proce...

    2023 doi

  16. [24]

    Darius Plikynas, Ieva Rizgelien \.e , and Gra z ina Korvel. 2025. Systematic review of fake news, propaganda, and disinformation: Examining authors, content, and social impact through machine learning. IEEE Access

    2025

  17. [25]

    Hannah Rashkin, Eunsol Choi, Jin Yea Jang, Svitlana Volkova, and Yejin Choi. 2017. Truth of varying shades: Analyzing language in fake news and political fact-checking. In Proceedings of the 2017 conference on empirical methods in natural language processing, pages 2931--2937

    2017

  18. [26]

    Ariana Sahitaj, Premtim Sahitaj, Salar Mohtaj, Sebastian M \"o ller, and Vera Schmitt. 2024. Towards a computational framework for distinguishing critical and conspiratorial texts by elaborating on the context and argumentation with llms. Working Notes of CLEF

    2024

  19. [27]

    Premtim Sahitaj, Iffat Maab, Junichi Yamagishi, Jawan Kolanowski, Sebastian M \"o ller, and Vera Schmitt. 2025. https://doi.org/10.48550/arXiv.2502.08909 Towards Automated Fact-Checking of Real-World Claims : Exploring Task Formulation and Assessment with LLMs . Preprint, arXi...

    2025 doi arXiv

  20. [28]

    Kilian Sprenkamp, Daniel Gordon Jones, and Liudmila Zavolokina. 2023. Large language models for propaganda detection. arXiv preprint arXiv:2310.06422

    2023 arXiv

  21. [29]

    Ivan Srba, Olesya Razuvayevskaya, Jo \ a o A Leite, Robert Moro, Ipek Baris Schlicht, Sara Tonelli, Francisco Moreno Garc \' a, Santiago Barrio Lottmann, Denis Teyssou, Valentin Porcellini, et al. 2024. A survey on automatic credibility assessment of textual credibility signal...

    2024

  22. [30]

    Muhammad Tanvir and Azeem Malik. 2024. The information battlefield: How cyber propaganda affects thoughts and shape the public opinion. Wah Academia Journal of Social Sciences, 3(2):258--279

    2024

  23. [31]

    Sebasti \'a n Vallejo Vera and Hunter Driggers. 2024. Bias in llms as annotators: The effect of party cues on labelling decision by large language models. arXiv preprint arXiv:2408.15895

    2024 arXiv

  24. [32]

    Prashanth Vijayaraghavan and Soroush Vosoughi. 2022. Tweetspin: Fine-grained propaganda detection in social media using multi-view representations. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langu...

    2022

  25. [33]

    Claire Wardle and Hossein Derakhshan. 2017. Information disorder: Toward an interdisciplinary framework for research and policymaking, volume 27. Council of Europe Strasbourg

    2017

  26. [34]

    Brandon T Willard and R \'e mi Louf. 2023. Efficient guided generation for large language models. arXiv preprint arXiv:2307.09702

    2023 arXiv

  27. [35]

    Mariia Zhdanova and Dariya Orlova. 2017. Computational propaganda in ukraine: Caught between external threats and internal challenges

    2017

  28. [36]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  29. [37]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.