REVIEW 4 major objections 7 minor 43 references
Comparing and Modeling Argumentation in German Political Communication across Arenas
T0 review · 4 major / 7 minor · reviewed 2026-08-04 · deepseek-v4-flash
Pith's one-line read In German COVID-19 politics, press conferences show a higher share of evidence-based, expert-style justifications than health committee meetings, according to a newly annotated corpus across three political arenas.
desk verdict A valuable new corpus, but the headline cross-arena finding rests on annotation categories whose reliability is never reported. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The key machinery is the annotation scheme, built on a claim-plus-justification definition of argument (Toulmin-inspired). It categorizes justifications into three types — evidence-based/disciplinary, normative/pragmatic, and ideological/analogical — and only the first counts as expert argumentation. A further binary distinguishes medical from non-medical evidence. This scheme yields the cross-arena comparison at the heart of the paper and also drives the modeling pilot, where argument-boundary recognition is evaluated with gamma agreement and F1 scores.
What would settle it
Reproduce the annotation with a broader definition of expert argumentation — e.g., counting explicit references to scientific experts, studies, or expert institutions as expert justifications even when not framed as measurable evidence — and recompute per-arena shares; if committee meetings then outrank press conferences, the paper's central reversal rests on its narrow coding.
Extended reading notes
Core claim
The paper's main discovery is a corpus-level fact: in German federal political discourse during the pandemic, the share of justifications coded as evidence-based (disciplinary) is highest in government press conferences (16.7%), followed by plenary speeches (6.6%), with health committee protocols last (5.7%). Because the annotation scheme treats only evidence-based disciplinary justifications as expert arguments, this inverts the usual frontstage/backstage expectation that expert argumentation concentrates in committee deliberation. The paper attributes the reversal to the press conference's institutional logic: officials produce reasonable, evidence-citing answers to journalists in a formal
Load-bearing premise
That expert argumentation can be captured solely as evidence-based, disciplinary justification, and that the 87 pre-screened documents (including only 12 press conferences) accurately represent each arena's true rates of evidence-based justification.
Editorial extensions
If this is right
- If the finding holds, expertise-based argumentation in political communication is tied to audience and communicative function rather than simply to institutional proximity to expert knowledge.
- Press conferences become a more important data source for argument-mining and deliberative-quality research because they combine low overall argument density with a high proportion of evidence-based justification.
- Plenary speeches' dominance of normative justification is consistent with representative-deliberation theory, but implies that measuring 'argument quality' purely as evidence-use would rank arenas counter-intuitively.
- The modeling pilot implies that zero-shot LLM argument detection needs density calibration or pre-screening; across models and arenas, accuracy of predicted argument density correlates inversely with F1 error.
Reading between the lines
- One could test the institutional-logic explanation by comparing COVID press conferences with non-crisis government press conferences: if the evidence-based share drops when no acute crisis is being communicated, the effect belongs to crisis-communication genre rather than to the arena as such.
- The operationalization of expert argumentation only as evidence-based/disciplinary justification likely undercounts explicit references to experts, studies, or expert institutions; coding those as expert arguments might close the gap between committees and press conferences.
- Because the 87 annotated documents were pre-screened to ensure enough positive examples, the per-arena density numbers (e.g., 3.4 arguments per 100 sentences in press conferences) are upper-bound estimates; a random sample could either narrow or erase the observed gap.
- The finding that some models reached gamma scores comparable to human annotators on press conferences (0.44 vs. 0.73 for gold) suggests that treating boundary disagreement as noise, rather than as a phenomenon worth modeling, may be the wrong lesson to draw.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents a manually annotated corpus of German political communication during the COVID-19 pandemic, covering three arenas: Bundestag plenary speeches, health committee protocols, and government press conferences. Arguments and justifications are annotated with span boundaries, topics, and justification types (evidence-based, normative/pragmatic, ideological/analogical). The central empirical claim is that, contrary to the expectation that expert argumentation concentrates in committees, evidence-based justifications are more frequent in press conferences (16.7%) than in plenary speeches (6.6%) or health committee protocols (5.7%). A second contribution is a zero-shot LLM pilot study for argument-span extraction, reporting that boundary identification is difficult and that models are biased toward predicting non-empty argument lists.
Significance. If the central comparative finding is reliable, the paper makes a substantive contribution to political communication research by suggesting that expertise-based justification is tied to communicative function and audience rather than institutional proximity to experts. The corpus itself is a useful resource, and the annotation design is documented in unusual detail, including full guidelines in an appendix and public dataset release. The modeling pilot is honest about its limitations and provides a clean negative result: zero-shot LLMs struggle with span-level argument recognition and are biased by argument-density assumptions. These strengths make the paper a reasonable candidate for a resource/empirical contribution, provided the validity of the outcome variable is established.
major comments (4)
- [§3.4, §3.5, Table 2] The central comparative claim in §3.5 rests entirely on the JUST-type variable, yet no category-level annotation reliability is reported. Section 3.4 explicitly states: 'We do not carry out more detailed analysis of category correspondences, since these are hard to align.' Only span-level Gamma is reported. Because the 16.7% vs. 6.6% vs. 5.7% ranking is a claim about justification types, the absence of per-category agreement (e.g., κ/γ for the three-level type label among JUST spans) means the central finding could reflect annotator disagreement rather than arena differences. The authors should report category-level agreement or, failing that, substantially weaken the comparative claim.
- [§3.5, Table 2] The cross-arena percentages are reported without uncertainty quantification. The BPK estimate is based on 12 documents and 216 JUST spans, and the GA estimate on 15 documents and 1245 JUST spans. A difference between 16.7% and 5.7% could be within sampling variability. The paper should provide confidence intervals, a permutation test, or a hierarchical model that treats documents as random effects. Without this, the abstract's 'contrary to expectations' finding is not statistically grounded.
- [§3.4] The pre-screening procedure described in §3.4 may bias the arena-level density estimates that underlie the comparative analysis. Documents were selected to ensure a sufficient number of positive examples, using an automatic claim-detection classifier to quantify claim presence. This makes the annotated sample non-random and likely enriches for argumentative content, especially in the smaller BPK and GA samples. The paper should either justify why pre-screening does not affect the relative type distribution, or reframe the result as conditional on a claim-rich sample rather than as a property of the arenas.
- [Appendix A, §3.5 footnote 3] The coding scheme's boundary between 'evidence-based' and 'normative/pragmatic' is demonstrably fuzzy at exactly the point that drives the central finding. Appendix A lists 'Die Infektionszahlen sind gefallen. Die Maßnahmen waren also effektiv.' as an example of normative/pragmatic justification, although it describes measurable, empirical evidence. Conversely, evidence-based justification is defined broadly enough that 'der Verweis auf eine Studie reicht aus' (a mere reference to a study suffices). In §3.5, footnote 3, a quantitative statement about a 98% staff vaccination rate is presented as normative. These examples suggest that coders may have applied the boundary inconsistently across arenas, particularly between factual numeric statements from press-conference speakers and similar statements from committee speakers. The authors should clarify the decision rules and ideally conduct
minor comments (7)
- [§3.5, Table 2] The text says 'On average, this corresponds to 18.35 ARG spans per 100 sentences and 20.65 JUST spans per 100 sentences,' but Table 2 reports total rates of 9.87 and 11.20 per 100 sentences. The numbers 18.35 and 20.65 appear to be half the sum of the three per-arena rates, which is not a meaningful overall rate. Please correct the aggregation.
- [§3.5] The sentence 'Speeches show the highest average, with 23.02 ARG and JUST spans per text' is inconsistent with Table 2. The value 23.02 is not a per-text count; it is roughly the mean of the ARG and JUST per-100-sentence rates for BT. Similar issues affect the GA (12.43) and BPK (3.54) values. Please rephrase to match the table.
- [Table 2] The total for contributions is 2465, but the row sums to 60 + 1046 + 1450 = 2556. Please correct the total.
- [Table 1 caption] Typo: 'Agumentative' should be 'Argumentative'.
- [§2] Typo: 'evalutation' should be 'evaluation'. Also, the sentence fragment 'with the health minister regularly appearing alongside leading scientific experts. framing the severity of the crisis' should be joined or rephrased.
- [§3.1] Typo: 'Or period of observation' should be 'Our period of observation'.
- [Figure 1] The caption says 'Return rates of unfaithful quotations' but the term 'unfaithful' is not defined in the caption or the text. Please define it explicitly, e.g., as quotations that do not match the input verbatim.
Circularity Check
Mild definitional step: 'expert argumentation' is operationalized as evidence-based justification, so the flagship cross-arena ranking restates the codebook label distribution; no fitted prediction or self-citation chain is circular.
-
self definitional
[Section 3.3 (Annotation Schema) and Section 3.5 (Corpus Findings)]
"To capture expert argumentation, we assumed that only evidence-based (disciplinary) arguments qualify as expert arguments and added a binary indicator for whether the justification falls within the subject area. ... The data reveal a discipline-specific concentration of evidence-based arguments, which is in line with our assumption that expert argumentation primarily takes place in evidence-based arguments."
The abstract's central finding ('justification by domain-specific expertise is more frequent in press conferences than in committee meetings') is computed in Section 3.5 from the share of evidence-based JUST spans (BPK 16.7%, BT 6.6%, GA 5.7%). That share is the same variable introduced in Section 3.3, where expert argumentation is defined as evidence-based justification. Consequently, saying the data are 'in line with our assumption' that expert argumentation is evidence-based is a restatement of the codebook, not an independent confirmation, and the cross-arena 'expertise' ranking is by construction the distribution of one manually labeled category. This is a definitional/construct-validity caveat rather than a fitted-input prediction: the percentages are still contingent empirical count
full rationale
The paper contains no equations, fitted parameters, or 'predictions' constructed from their own inputs, so the strongest circularity failure modes do not apply. The flagship corpus result is a descriptive annotation count; the LLM pilot is evaluated against the gold standard, and prompt selection uses only the unfaithful-quotation rate, not the target labels. The only definitional issue is that the paper equates 'expert argumentation' with evidence-based justification in the codebook (Section 3.3), and then reports the cross-arena ranking of that label as a finding about expertise (Section 3.5); this is an explicit operationalization, and the ranking is therefore about the codebook category. The acknowledged absence of category-level inter-coder agreement (Section 3.4: 'We do not carry out more detailed analysis of category correspondences...') and the small BPK sample (n=12) are validity/reliability risks, not circularity. Self-citations (Blokker et al. 2020) are used only for claim pre-screening and are not load-bearing for the main result.
Assumptions & free parameters
assumptions (4)
- domain assumption Expert argumentation can be operationalized as justifications that are classified as evidence-based/disciplinary and fall within the subject area.
- domain assumption Pre-screening documents with an automatic claim classifier to ensure enough positive examples does not materially bias the cross-arena comparison.
- domain assumption The adjudicated gold standard, built from two student annotators plus a third expert, is a valid consensus for measuring argument boundaries and types.
- domain assumption A single policy domain (COVID-19) and a single committee (health) suffice to draw conclusions about political arenas more generally.
Cite this review
Pith. "Pith review of Comparing and Modeling Argumentation in German Political Communication across Arenas." pith.science (2026). https://pith.science/paper/4WSSQPYN
@misc{pith2026260800288,
author = {Pith},
title = {Pith review of: Comparing and Modeling Argumentation in German Political Communication across Arenas},
year = {2026},
howpublished = {\url{https://pith.science/paper/4WSSQPYN}},
note = {Machine review of arXiv:2608.00288}
}
read the original abstract
Deliberation, involving the formulation and exchange of arguments, forms an integral part of political decision making in democracies. Argumentation patterns however differ substantially across different political arenas, such as plenary speeches and committee meetings. However, despite a lot of interest in argumentation, there is comparatively little computational work on analyzing differences in patterns of political argumentation between arenas. Our work addresses this research gap. First, we present a 17k-sentence corpus with annotation for argumentative passages (argument and their justifications, both their boundaries and their categories) across three German political arenas (plenary speeches, committee meetings, and press conferences), keeping the topic (COVID-19) constant. Our analysis of the corpus finds that contrary to expectations, justification by domain-specific expertise is more frequent in press conferences than in committee meetings. Second, we present a pilot study on automatically identifying such argumentative passages. The results show that boundaries are hard to pin down, and models predictions additionally suffer from confirmation bias.
Figures
Reference graph
Works this paper leans on
-
[1]
Detecting Argumentative Fallacies in the Wild: Problems and Limitations of Large Language Models
Ruiz-Dolz, Ramon and Lawrence, John. Detecting Argumentative Fallacies in the Wild: Problems and Limitations of Large Language Models. Proceedings of the 10th Workshop on Argument Mining. 2023. doi:10.18653/v1/2023.argmining-1.1
-
[2]
On Selecting Training Corpora for Cross-Domain Claim Detection
Schaefer, Robin and Knaebel, Ren \'e and Stede, Manfred. On Selecting Training Corpora for Cross-Domain Claim Detection. Proceedings of the 9th Workshop on Argument Mining. 2022
2022
-
[3]
Accounting for Sycophancy in Language Model Uncertainty Estimation
Sicilia, Anthony and Inan, Mert and Alikhani, Malihe. Accounting for Sycophancy in Language Model Uncertainty Estimation. Findings of the Association for Computational Linguistics: NAACL 2025. 2025. doi:10.18653/v1/2025.findings-naacl.438
-
[4]
Computational Linguistics , volume =
Mathet, Yann and Widlöcher, Antoine and Métivier, Jean-Philippe , title =. Computational Linguistics , volume =. 2015 , month =. doi:10.1162/COLI_a_00227 , url =
-
[5]
From Debates to Diplomacy: Argument Mining Across Political Registers
Poiaganova, Maria and Stede, Manfred. From Debates to Diplomacy: Argument Mining Across Political Registers. Proceedings of the 12th Argument mining Workshop. 2025. doi:10.18653/v1/2025.argmining-1.20
-
[6]
Pfister, Jan and Wunderle, Julia and Hotho, Andreas. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025. doi:10.18653/v1/2025.acl-long.111
-
[7]
2025 , eprint=
Gemma 3 Technical Report , author=. 2025 , eprint=
2025
-
[8]
2025 , eprint=
Qwen3 Technical Report , author=. 2025 , eprint=
2025
Show all 43 references
-
[9]
Aaron Grattafiori and Abhimanyu Dubey and Abhinav Jauhri and Abhinav Pandey and Abhishek Kadian and Ahmad Al-Dahle and Aiesha Letman and Akhil Mathur and Alan Schelten and Alex Vaughan and Amy Yang and Angela Fan and Anirudh Goyal and Anthony Hartshorn and Aobo Yang and Archi ...
-
[10]
Guerreiro and Ricardo Rei and Duarte M
Pedro Henrique Martins and Patrick Fernandes and João Alves and Nuno M. Guerreiro and Ricardo Rei and Duarte M. Alves and José Pombal and Amin Farajian and Manuel Faysse and Mateusz Klimaszewski and Pierre Colombo and Barry Haddow and José G. C. de Souza and Alexandra Birch an...
-
[11]
2503.01743 , archivePrefix=
Abdelrahman Abouelenin and Atabak Ashfaq and Adam Atkinson and Hany Awadalla and Nguyen Bach and Jianmin Bao and Alon Benhaim and Martin Cai and Vishrav Chaudhary and Congcong Chen and Dong Chen and Dongdong Chen and Junkun Chen and Weizhu Chen and Yen-Chun Chen and Yi-ling Ch...
-
[12]
What is the Essence of a Claim? C ross-Domain Claim Identification
Daxenberger, Johannes and Eger, Steffen and Habernal, Ivan and Stab, Christian and Gurevych, Iryna. What is the Essence of a Claim? C ross-Domain Claim Identification. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. 2017. doi:10.18653/v1...
2017 doi
-
[13]
Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence (IJCAI 2018) , pages =
Elena Cabrio and Serena Villata , title =. Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence (IJCAI 2018) , pages =. 2018 , doi =
2018
-
[14]
Aho and Jeffrey D
Alfred V. Aho and Jeffrey D. Ullman , title =. 1972
1972
-
[15]
Publications Manual , year = "1983", publisher =
1983
-
[16]
Opening up
Liesenfeld, Andreas and Lopez, Alianda and Dingemanse, Mark , year = 2023, booktitle =. Opening up
2023
-
[17]
Chandra and Dexter C
Ashok K. Chandra and Dexter C. Kozen and Larry J. Stockmeyer , year = "1981", title =. doi:10.1145/322234.322243
1981
-
[18]
Scalable training of
Andrew, Galen and Gao, Jianfeng , booktitle=. Scalable training of
-
[19]
Dan Gusfield , title =. 1997
1997
-
[20]
Tetreault , title =
Mohammad Sadegh Rasooli and Joel R. Tetreault , title =. Computing Research Repository , volume =. 2015 , url =
2015
-
[21]
A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =
Ando, Rie Kubota and Zhang, Tong , Issn =. A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =. Journal of Machine Learning Research , Month = dec, Numpages =
-
[22]
and Mansbridge, Jane and Warren, Mark E
Bächtiger, André and Dryzek, John S. and Mansbridge, Jane and Warren, Mark E. , year=. Deliberative Democracy. An Introduction , ISBN=. The Oxford Handbook of Deliberative Democracy , publisher=
-
[23]
Argument Quality in Public Deliberations , volume=
Steffensmeier, Timothy and Schenck-Hamlin, William , year=. Argument Quality in Public Deliberations , volume=. Argumentation and Advocacy , publisher=. doi:10.1080/00028533.2008.11821693 , number=
2008
-
[24]
Comparative European Politics , author=
Measuring Political Deliberation: A Discourse Quality Index , volume=. Comparative European Politics , author=. 2003 , pages=. doi:10.1057/palgrave.cep.6110002 , number=
2003 doi
-
[25]
Strategies of legitimization in political discourse: From words to actions , volume=
Reyes, Antonio , year=. Strategies of legitimization in political discourse: From words to actions , volume=. Discourse & Society , publisher=. doi:10.1177/0957926511419927 , number=
-
[26]
Parliamentary Affairs , author=
Do Members of Parliament Express More Opposition in the Plenary than in the Committee?. Parliamentary Affairs , author=. 2024 , month=. doi:10.1093/pa/gsac016 , number=
2024 doi
-
[27]
Computational Argumentation Quality Assessment in Natural Language , url=
Wachsmuth, Henning and Naderi, Nona and Hou, Yufang and Bilu, Yonatan and Prabhakaran, Vinodkumar and Thijm, Tim Alberdingk and Hirst, Graeme and Stein, Benno , editor=. Computational Argumentation Quality Assessment in Natural Language , url=. Proceedings of the 15th Conferen...
2017
-
[28]
How to Do Politics with Words: Investigating Speech Acts in Parliamentary Debates , url=
Reinig, Ines and Rehbein, Ines and Ponzetto, Simone Paolo , editor=. How to Do Politics with Words: Investigating Speech Acts in Parliamentary Debates , url=. Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation...
2024
-
[29]
Argumentation , author=
Argumentative Patterns in the Political Domain: The Case of. Argumentation , author=. 2016 , month=. doi:10.1007/s10503-015-9372-4 , number=
2016 doi
-
[30]
Overview of P erspective A rg2024 The First Shared Task on Perspective Argument Retrieval
Falk, Neele and Waldis, Andreas and Gurevych, Iryna. Overview of P erspective A rg2024 The First Shared Task on Perspective Argument Retrieval. Proceedings of the 11th Workshop on Argument Mining (ArgMining 2024). 2024. doi:10.18653/v1/2024.argmining-1.14
2024 doi
-
[31]
2023 , publisher=
Blaette, Andreas and Leonhardt, Christoph , title=. 2023 , publisher=. doi:10.5281/zenodo.10416536 , url=
2023 doi
-
[32]
Yes, we can! Mining Arguments in 50 Years of US Presidential Campaign Debates
Haddadan, Shohreh and Cabrio, Elena and Villata, Serena. Yes, we can! Mining Arguments in 50 Years of US Presidential Campaign Debates. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2019. doi:10.18653/v1/P19-1463
2019 doi
-
[33]
O nto N otes: The 90 \
Hovy, Eduard and Marcus, Mitchell and Palmer, Martha and Ramshaw, Lance and Weischedel, Ralph. O nto N otes: The 90 \. Proceedings of the Human Language Technology Conference of the NAACL , Companion Volume: Short Papers. 2006
2006
-
[34]
Proceedings of the AAAI Conference on Artificial Intelligence , author=
Argument Mining from Speech: Detecting Claims in Political Debates , volume=. Proceedings of the AAAI Conference on Artificial Intelligence , author=. 2016 , month=. doi:10.1609/aaai.v30i1.10384 , abstractNote=
2016 doi
-
[35]
Visser and B
J. Visser and B. Konat and R. Duthie and M. Koszowy and K. Budzynska and C. Reed , title =. Language Resources and Evaluation , volume =. 2020 , doi =
2020
-
[36]
undated , note=
Word list for the. undated , note=
-
[37]
undated , note=
Bundespressekonferenz transcripts , url=. undated , note=
-
[38]
The INCEpTION Platform: Machine-Assisted and Knowledge-Oriented Interactive Annotation , booktitle=
Klie, Jan-Christoph and Bugert, Michael and Boullosa, Beto and Eckart de Castilho, Richard and Gurevych, Iryna , year=. The INCEpTION Platform: Machine-Assisted and Knowledge-Oriented Interactive Annotation , booktitle=
-
[39]
and Schiappa, Edward , year=
Nordin, John P. and Schiappa, Edward , year=. Argumentation: Keeping Faith with Reason , ISBN=. doi:10.4324/9781003415268 , publisher=
-
[40]
Toulmin, Stephen E. , year=. The Uses of Argument , publisher=
-
[41]
Ideology: A Multidisciplinary Approach , url=
van Dijk, Teun , year=. Ideology: A Multidisciplinary Approach , url=. doi:10.4135/9781446217856 , publisher=
-
[42]
Media and Communication , author =
Media Framing of Government Crisis Communication During. Media and Communication , author =. doi:10.17645/mac.7774 , abstract =
-
[43]
Swimming with the Tide? Positional Claim Detection across Political Text Types
Blokker, Nico and Dayanik, Erenay and Lapesa, Gabriella and Pad \'o , Sebastian. Swimming with the Tide? Positional Claim Detection across Political Text Types. Proceedings of the Fourth Workshop on Natural Language Processing and Computational Social Science. 2020. doi:10.186...
2020 doi
Reviewed August 4, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.