REVIEW 3 major objections 4 minor 1 cited by
Are Large Language Models Good at Detecting Propaganda?
T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read Despite advanced prompting, none of the GPT-3.5, GPT-4, and Claude 3 Opus models outperforms a RoBERTa-CRF ensemble baseline on phrase-level propaganda detection in news articles.
desk verdict A transparent and largely convincing negative result on LLMs for fine-grained propaganda detection, but the gold-label completeness issue and a few reporting choices keep it from being the final word. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the evaluation setup: the PTC dataset's phrase-level gold annotations for six propaganda techniques, combined with a macro-F1 metric that averages per-technique F1. The comparison is anchored by two baselines—a RoBERTa-CRF model with an ensemble classifier from the SemEval 2020 task, used as the upper bound, and the Multi-Granularity Network released with the dataset, used as the lower bound. The prompting strategies (zero-shot, one-shot, chain-of-thought, generated knowledge, self-consistency) are the intervention being tested. The metric and baselines together determine the claim: any LLM score is interpreted relative to those anchors.
What would settle it
Take a random sample of LLM predictions scored as false positives, have fresh annotators judge whether each span matches the published technique definitions, and compare the adjudicated labels with the original gold set. If a substantial share of the false positives are genuine missed annotations, the reported F1 gap would shrink or disappear.
Extended reading notes
Core claim
The central claim is that large language models, even with advanced prompting, are not yet competitive with a fine-tuned transformer baseline for detecting propaganda techniques in news articles. On the Propaganda Techniques Corpus, the strongest LLM result, GPT-4 with generated-knowledge prompting, reaches a macro-F1 of 0.16, far below the RoBERTa-CRF ensemble's 0.67. The paper reports this gap for every one of the six techniques: none of the LLMs exceeds the baseline on name-calling, loaded language, doubt, appeal to fear, flag-waving, or exaggeration/minimization. Against a lower-bound baseline, the MGN model, GPT-4 wins on macro-F1 under one-shot, generated-knowledge, and self-consistency prompting, and all LLMs beat it on name-calling.
Load-bearing premise
The PTC human annotations are treated as the complete set of correct propaganda spans, so every LLM detection that the annotators did not record is counted as an error.
Editorial extensions
If this is right
- Fine-tuned transformer models remain the practical choice for automated propaganda annotation; LLM prompting alone is not a drop-in replacement.
- For name-calling, and for appeal to fear and flag-waving with certain prompts, LLMs beat the weaker MGN baseline, so the technology has partial traction on emotionally loaded techniques.
- Prompting strategy matters: GPT-4's best macro-F1 comes only under one-shot, generated-knowledge, and self-consistency prompting, not zero-shot.
- The large precision-recall imbalance in LLM outputs means any deployment would need a separate verification step to be usable.
Reading between the lines
- The low precision values might mean LLMs flag correct propaganda instances the PTC annotators missed; if so, the headline gap is partly an artifact of treating the gold set as complete. Re-annotating the disagreements would settle this.
- A hybrid pipeline—LLMs proposing span candidates and the RoBERTa-CRF model confirming labels—could combine the LLM's recall on some techniques with the baseline's precision.
- Because the study covers only six of eighteen techniques, the results may not transfer to more subtle or context-dependent techniques such as repetition or whataboutism.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper investigates whether large language models can detect propaganda techniques in news articles. Using the Propaganda Techniques Corpus (PTC) and focusing on six techniques, the authors evaluate GPT-3.5, GPT-4, and Claude 3 Opus under five prompting settings (zero-shot, one-shot, chain-of-thought, generated knowledge, and self-consistency), plus a fine-tuned GPT-3.5 model. They compare these against two transformer baselines: a RoBERTa-CRF ensemble (Jurkiewicz et al. 2020) and a Multi-Granularity Network (MGN, Da San Martino et al. 2019). The central finding is that none of the LLMs reach the macro-F1 of the RoBERTa-CRF baseline (0.67), with GPT-4 achieving the highest LLM macro-F1 at 0.16. The paper also reports that GPT-3.5 and GPT-4 outperform the MGN baseline on a few individual techniques, and GPT-4 outperforms the MGN model in macro-F1 under some prompting strategies.
Significance. If the central result is accepted, the paper provides a useful, if sobering, data point: general-purpose LLMs are not yet competitive with fine-tuned supervised models on fine-grained propaganda technique detection, despite strong performance on many other NLP tasks. The study also contributes a comparison across prompting strategies and across three LLMs, and it is careful to compare against a strong SemEval-winning baseline and a dataset-author baseline. The reporting of precision and recall in the appendix is a strength. However, the validity of the headline comparison depends on methodological choices that need to be addressed, in particular the selection of the best prompt per setting and the completeness of the gold annotation set.
major comments (3)
- [Discussions] The paper reports, for each LLM and prompt setting, the F1 score of the prompt variation that gave the highest macro-F1 (Discussions: 'The F1 scores reported in Table 2 correspond to prompts that gave us the highest macro-F1 across these variations'). This is a post-hoc selection on the test set, which inflates the reported scores and makes the comparison with the baselines unfair in an unpredictable direction. The central claim that LLMs underperform the RoBERTa-CRF baseline is large enough to survive this bias, but the narrower claims about which prompting strategy is best, and the detailed GPT-4-vs-MGN comparisons, are not supported without reporting the full distribution of results across prompt variations or an averaged score with standard deviation.
- [Results / Discussions] The comparison assumes the PTC human annotations are the complete and correct set of propaganda instances. The paper itself concedes in the Discussions that low precision 'raises the question of whether the LLMs are identifying instances of propaganda techniques that were missed by human annotators in the original dataset.' This is load-bearing: the RoBERTa-CRF baseline is trained on the same gold labels and can achieve high F1 by reproducing annotator decisions, including omissions, while an LLM that correctly identifies an instance absent from the gold set is penalized as a false positive. The reported low inter-annotator agreement (0.24 and 0.28 in the initial annotation stage) makes incomplete annotations plausible. The paper should include an error analysis or an adjudicated/relabeled sample to distinguish genuine false positives from potentially correct detections missing from the gold set; without this, the size of the gap (0.16 vs 0.67) does not by itself rule out a systematic underestimation of LLM precision.
- [Results / Appendix Tables 4-6] No error bars, confidence intervals, or significance tests are reported, and it is not stated whether the LLM API calls were run at temperature 0 or with multiple seeds, nor how many runs were averaged. Several claimed individual-technique improvements over the MGN baseline are numerically small (e.g., GPT-4 macro-F1 0.16 vs MGN 0.14; GPT-3.5 generated-knowledge appeal-to-fear F1 0.16 vs MGN 0.09). Without run-to-run variance, these differences cannot be distinguished from noise. At minimum, the temperature/decoding settings and number of runs should be given, and for the close comparisons the authors should report variance or a small statistical test.
minor comments (4)
- [Abstract] The abstract states 'we find that all three LLMs outperform a MultiGranularity Network (MGN) baseline in detecting instances of one out of six propaganda techniques (name-calling)' while the Results say all versions of the LLMs outperformed the MGN for name-calling. The abstract phrasing is slightly awkward; consider saying 'for name-calling' instead of 'in detecting instances of one out of six techniques'.
- [Methods / Models] The paper does not specify how the LLM JSON outputs were parsed, how malformed outputs were handled, or whether the model was allowed to abstain. These details affect precision and recall and should be reported for reproducibility.
- [Table 2] Table 2 is dense and does not include precision and recall, which are central to the annotation-completeness concern. Consider moving precision and recall into the main table, or at least highlighting the low-precision cells in the discussion.
- [Discussions] There are minor grammatical issues, e.g., 'GPT-3.5 and GPT-4 performs better' should be 'perform better', and 'GPT-3.5 and GPT-4 outperforms' should be 'outperform'. These do not affect the technical content but should be corrected.
Circularity Check
Empirical benchmark against external baselines with no circular derivation.
full rationale
This paper is an empirical evaluation rather than a derivation. The central comparison is between LLM outputs and two external baselines: the ApplicaAI RoBERTa-CRF model (Jurkiewicz et al. 2020) and the Multi-Granularity Network (Da San Martino et al. 2019). Neither baseline is authored by the present authors, and neither is fitted within this paper. All F1 scores in Table 2 are measured against the PTC gold annotations rather than constructed from the models' own outputs. The only model trained in the paper is the fine-tuned GPT-3.5, and its results are also reported as measured performance. The Discussion's caveat that low precision may reflect propaganda instances missed by human annotators is a validity limitation about gold-label completeness, not a circular step: the paper does not define the evaluation target in terms of the LLMs' outputs, and the conclusion is conditional on the PTC annotation set. Reporting the best prompt variation per setting is a mild form of test-set selection, but that is a statistical reporting concern, not an equivalence between inputs and predictions. No self-citation chain, uniqueness theorem, or redefinition is load-bearing in the paper, so there is no circularity to flag.
Assumptions & free parameters
free parameters (1)
- Best-prompt selection (reported F1) =
Highest macro-F1 across 1-6 prompt variations per model
assumptions (2)
- domain assumption PTC dataset annotations are accurate and complete.
- domain assumption Baseline F1 values from Jurkiewicz et al. (2020) and Da San Martino et al. (2019) are correctly transcribed.
Cite this review
Pith. "Pith review of Are Large Language Models Good at Detecting Propaganda?." pith.science (2026). https://pith.science/paper/5OIYAGBS
@misc{pith2026250513706,
author = {Pith},
title = {Pith review of: Are Large Language Models Good at Detecting Propaganda?},
year = {2026},
howpublished = {\url{https://pith.science/paper/5OIYAGBS}},
note = {Machine review of arXiv:2505.13706}
}
read the original abstract
Propagandists use rhetorical devices that rely on logical fallacies and emotional appeals to advance their agendas. Recognizing these techniques is key to making informed decisions. Recent advances in Natural Language Processing (NLP) have enabled the development of systems capable of detecting manipulative content. In this study, we look at several Large Language Models and their performance in detecting propaganda techniques in news articles. We compare the performance of these LLMs with transformer-based models. We find that, while GPT-4 demonstrates superior F1 scores (F1=0.16) compared to GPT-3.5 and Claude 3 Opus, it does not outperform a RoBERTa-CRF baseline (F1=0.67). Additionally, we find that all three LLMs outperform a MultiGranularity Network (MGN) baseline in detecting instances of one out of six propaganda techniques (name-calling), with GPT-3.5 and GPT-4 also outperforming the MGN baseline in detecting instances of appeal to fear and flag-waving.
Forward citations
Cited by 1 Pith paper
-
Hybrid Annotation for Propaganda Detection: Integrating LLM Pre-Annotations with Human Intelligence
An LLM pre-annotation pipeline with span extraction and hierarchical labels improves human inter-annotator agreement and speed on Russian propaganda tweets, with smaller models distilled from the LLM outputs.
Reference graph
Works this paper leans on
-
[1]
is a key and the technique (< technique > ) is its value (eg, “phrase”:“< technique > ”,...). Technique: “< technique > ”: “< technique def inition > ”. Text: Generated Knowledge Prompting Detect the propaganda technique < technique > in the input article. Follow the example format given below: Input: Article: < example article > are there instances of < ...
work page 2020
-
[4]
arXiv preprint arXiv:2302.09210
How good are gpt models at machine translation? a comprehensive evaluation. arXiv preprint arXiv:2302.09210. Hollis;, E. V
-
[5]
arXiv preprint arXiv:2005.07934
ApplicaAI at SemEval-2020 task 11: On RoBERTa- CRF, span CLS and whether self-training helps them. arXiv preprint arXiv:2005.07934. Kojima, T.; Gu, S. S.; Reid, M.; Matsuo, Y .; and Iwasawa, Y
arXiv 2020
-
[8]
Martino, G.; Barr ´on-Cedeno, A.; Wachsmuth, H.; Petrov, R.; and Nakov, P
Large lan- guage model is not a good few-shot information extrac- tor, but a good reranker for hard samples! arXiv preprint arXiv:2303.08559. Martino, G.; Barr ´on-Cedeno, A.; Wachsmuth, H.; Petrov, R.; and Nakov, P
-
[9]
arXiv preprint arXiv:2009.02696
SemEval-2020 task 11: Detection of propaganda techniques in news articles. arXiv preprint arXiv:2009.02696. Nashid, N.; Sintaha, M.; and Mesbah, A
arXiv 2020
-
[10]
In 2023 IEEE/ACM 45th International Conference on Soft- ware Engineering (ICSE), 2450–2462
Retrieval- based prompt selection for code-related few-shot learning. In 2023 IEEE/ACM 45th International Conference on Soft- ware Engineering (ICSE), 2450–2462. IEEE. Wan, Z.; Cheng, F.; Mao, Z.; Liu, Q.; Song, H.; Li, J.; and Kurohashi, S
work page 2023
-
[11]
arXiv preprint arXiv:2305.02105
Gpt-re: In-context learning for rela- tion extraction using large language models. arXiv preprint arXiv:2305.02105. Wang, L.; Lyu, C.; Ji, T.; Zhang, Z.; Yu, D.; Shi, S.; and Tu, Z
-
[12]
arXiv preprint arXiv:2304.02210
Document-level machine translation with large language models. arXiv preprint arXiv:2304.02210. Wang, X.; Wei, J.; Schuurmans, D.; Le, Q.; Chi, E.; Narang, S.; Chowdhery, A.; and Zhou, D
Show all 17 references
-
[14]
arXiv preprint arXiv:2302.10205
Zero- shot information extraction via chatting with chatgpt. arXiv preprint arXiv:2302.10205. Zhong, Q.; Ding, L.; Liu, J.; Du, B.; and Tao, D
-
[15]
arXiv preprint arXiv:2302.10198
Can chatgpt understand too? a comparative study on chatgpt and fine-tuned bert. arXiv preprint arXiv:2302.10198. Ziems, N.; Yu, W.; Zhang, Z.; and Jiang, M
-
[16]
phrase”:“ < technique >
Large language models are built-in autoregressive search engines. arXiv preprint arXiv:2305.09612. Appendix Prompt Set- ting Prompt Zero-shot I want you to perform a data annotation task. In your output, I want you to return a json dictio- nary with key as phrase and value as ...
-
[1939]
Li, L.; Fan, L.; Atreja, S.; and Hemphill, L
The fine art of propaganda. Li, L.; Fan, L.; Atreja, S.; and Hemphill, L. 2023a. “HOT” ChatGPT: The promise of ChatGPT in detecting and dis- criminating hateful, offensive, and toxic comments on social media. ACM Transactions on the Web. Li, X.; Zhu, X.; Ma, Z.; Liu, X.; and S...
-
[2019]
Fine-grained analysis of propaganda in news article. In Proceedings of the 2019 conference on empirical methods in natural language pro- cessing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), 5636–5646. Asso- ciation for Computational...
2019
-
[2020]
The International Journal of Press/Politics, 25(3): 357–389
Cross-platform state propaganda: Rus- sian trolls on twitter and YouTube during the 2016 US Presi- dential Election. The International Journal of Press/Politics, 25(3): 357–389. Graham, M. M. W
2016
-
[2021]
arXiv preprint arXiv:2110.08387
Generated knowledge prompting for commonsense reasoning. arXiv preprint arXiv:2110.08387. Ma, Y .; Cao, Y .; Hong, Y .; and Sun, A
-
[2022]
arXiv preprint arXiv:2203.11171
Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171. Wang, Y .; Zhao, Y .; and Petzold, L
-
[2023]
arXiv preprint arXiv:2302.04023
A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. arXiv preprint arXiv:2302.04023. Booth;, G. C
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.