Annotation Tool and Dataset for Fact-Checking Podcasts

Adam James Becker; Vinay Setty

REVIEW 3 major objections 6 minor 15 references

Annotation Tool and Dataset for Fact-Checking Podcasts

T0 review · 3 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims to build the first podcast annotation tool that plays audio while showing the transcript, and that the released multilingual labels let a fine-tuned XLM-RoBERTa-Large compete with GPT-4 on claim and stance detection.

desk verdict A genuinely useful podcast annotation tool and a small but well-structured dataset, with a model comparison that is too underpowered to support its headline claim. read the letter →

arxiv 2502.01402 v1 pith:24KC7SLB submitted 2025-02-03 cs.CL

Vinay Setty , Adam James Becker This is my paper · ORCID

classification cs.CL

keywords podcastsfact-checkingdataannotationclaimdetectionstanceclassificationmultilingualdatasetscrowdsourcingspokencontent

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that podcast fact-checking can be supported end to end by a lightweight open-source pipeline: transcribe the audio, assign speaker labels, resolve mentions, and let crowdworkers mark claims while listening to the recording. It claims this is the first tool designed for podcast annotation that plays audio and shows the transcript at the same time, and it releases transcripts for 531 episodes in English, Norwegian, and German plus fine-grained claim annotations for selected episodes from seven podcasts. The paper then argues the annotations are useful by fine-tuning XLM-RoBERTa-Large and showing it competes with GPT-4 in few-shot settings on claim detection and stance classification. A sympathetic reader would care because spoken fact-checking needs both the annotation interface and the labeled data, which have been missing since the large podcast corpus used by earlier work became inaccessible.

What carries the argument

The load-bearing mechanism is the annotation interface itself: a web application that plays the podcast audio while displaying a Whisper transcription with word-level timestamps, speaker labels from a diarization model, sentence splits, and coreference-resolved utterances, so annotators can label claims in real time. Around it sits a crowdsourcing pipeline that assigns each utterance to three crowdworkers and keeps only unanimous labels, and an evaluation pipeline that fine-tunes XLM-RoBERTa-Large on the resulting binary claim-detection and stance-classification tasks. The interface is what makes the claimed first-of-its-kind contribution possible.

What would settle it

Re-annotate a held-out sample of podcast utterances with more than three independent workers, measure inter-annotator agreement, and count how many labels the unanimous rule discards; then rerun the fine-tuned XLM-RoBERTa-Large and the GPT-4 prompts on a larger test set built from those labels. If the stance-detection advantage (0.79 versus 0.56 F1) shrinks or flips, or if a large share of labels is discarded, the dataset is too weak to support the comparative conclusions.

Watch

Extended reading notes

Core claim

The central discovery, on the paper's own terms, is that simultaneous audio-plus-transcription annotation changes what can be collected from podcasts: annotators can correct transcription errors, mark check-worthy claims and claim spans, choose claim types and fact-checking motivations, and assign verdicts while the audio plays, producing fine-grained labels that generic text tools cannot capture. With those labels, the paper reports that a fine-tuned XLM-RoBERTa-Large reaches a weighted F1 of 0.85 for claim detection and 0.67 for stance detection, while GPT-4 reaches 0.86 and 0.55, respectively; GPT-4 is better at identifying true check-worthy claims (0.57 versus 0.45), while XLM-RoBERTa is better at identifying supporting evidence (0.79 versus 0.56). The paper reads this as evidence that smaller multilingual models are competitive for podcast fact-checking and that the released data can support both tasks.

Load-bearing premise

The entire label set rests on the assumption that unanimous agreement among just three crowdworkers, with no reported inter-annotator agreement and no count of discarded non-unanimous labels, produces gold-standard annotations strong enough to support the model comparisons.

Editorial extensions

If this is right

If the tool is adopted, podcast fact-checking no longer requires a separate transcription step: annotators can correct automatic speech recognition errors and mark claims in one pass, making the process faster and cheaper than listening and annotating separately.
The released transcripts and annotations give the community a multilingual resource for claim detection and stance classification in English, Norwegian, and German, filling the gap left by the closed large podcast corpus.
Fine-grained labels for claim types and motivations allow downstream systems to prioritize numerical claims and public-interest claims, which together dominate the check-worthy annotations.
The competitive performance of XLM-RoBERTa-Large suggests that specialized smaller models can serve as a lightweight alternative to large API-based language models for podcast fact-checking.
The claim-type taxonomy and the annotation schema can be reused for other long-form audio domains, such as lectures, interviews, and meeting recordings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The unanimous-only labeling rule probably makes the gold standard conservative: ambiguous utterances are discarded, so the reported F1 values are likely measured on the easier instances and may overstate how the models would perform on borderline real-world claims.
Because numerical claims make up 62.6 percent of the check-worthy labels, a natural next step the paper does not pursue is a dedicated numerical-claim verifier that checks statistics against external sources, and the dataset would support training such a system.
A testable extension is to re-annotate a sample with more workers and measure inter-annotator agreement; if agreement is low, the three-worker unanimous rule would need to be replaced by majority voting or adjudication.
The same simultaneous audio-and-transcription design could be applied to live audio fact-checking by feeding a real-time transcript into the annotation interface, connecting this offline dataset work to streaming fact-checking systems the paper cites.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

I\'ll be blunt: this is a resource paper, and judged as that, it is basically sound. The genuinely new thing is a purpose-built podcast annotation tool that plays audio and lets the annotator correct transcripts, label check-worthy spans, and assign fine-grained claim types and fact-checking motivations in one interface. That combination does not exist in the open-source ecosystem, and the annotation scheme is more detailed than ClaimBuster or debate-focused check-worthiness datasets. The authors also transcribe 531 episodes across three languages and release the tool code. For the fact-checking community, that is a real contribution, especially since the Spotify Podcast Dataset is no longer public.

The paper does not oversell the scale. Section 3 is explicit that these are 7 episodes and that larger annotation is planned. But the experimental section overreaches. Table 2 reports F1 gaps (0.45 vs 0.57 for true claims, 0.79 vs 0.56 for supports) on a test set of 24 check-worthy, 18 refute, and 32 support instances. There are no confidence intervals or significance tests. Those differences are within the noise for counts that small. The claim that fine-tuned XLM-RoBERTa is \"competitive with GPT-4\" is therefore not supported by the data. It is a plausible preliminary observation, but the paper should either add intervals or soften the wording.

The more load-bearing soft spot is the gold-label reliability. Section 2.2 says each task was assigned to three annotators and unanimous labels were retained, but it reports no agreement statistics and no retention rate. If unanimity discarded many judgments, the surviving labels are biased toward easy cases, which would weaken both the dataset and the model comparison. The authors point to a master\'s thesis for details, but a resource paper should include that information on the page.

Also, the abstract says \"we release the annotated podcast transcripts and sample annotations,\" but the only URL is the tool repository. No dataset link appears. That is a concrete missing artifact for a paper whose contribution is a dataset.

None of this is fatal. The tool and the annotation scheme stand on their own, and the experiments are explicitly preliminary. The missing agreement numbers and dataset URL are fixable in revision. I\'d send this to review. The right outcome is a conditional acceptance that requires releasing the data, reporting inter-annotator agreement, and either adding confidence intervals or clearly labeling the comparisons as illustrative.

Referee Report

3 major / 6 minor

Summary. The paper presents an open-source web tool for transcribing and annotating podcasts, with simultaneous audio playback, and describes a multilingual (English, Norwegian, German) dataset built from 531 episodes of 38 podcasts, of which 7 episodes were annotated for check-worthy claims, claim types, fact-checking motivations, and stance/verification. The authors fine-tune XLM-RoBERTa-Large on the annotated utterances for binary claim detection and stance classification, and compare against GPT-4 in few-shot settings. The central claims are that the tool is the first designed for simultaneous audio playback and transcription annotation for podcasts, and that the released annotations allow small multilingual models to be competitive with GPT-4.

Significance. If the claims hold, the contribution is a useful new resource: an open tool for spoken-document annotation, a released multilingual podcast transcript corpus, and fine-grained claim/verification annotations that are currently scarce. The paper's strengths are that the tool is released as open source, the transcript collection is large (531 episodes, three languages), and the proposed annotation taxonomy is concrete and task-oriented. However, the experimental support for the comparative model claim is thin: the test sets are very small, no significance testing is reported, and the quality of the gold labels is not documented beyond a unanimous-agreement rule. The resource may still be valuable for the community, but the current evidence as presented is not yet sufficient to establish the stated competitive-performance conclusion.

major comments (3)

[§2.2] The gold-label quality is not established. The paper states that each task was assigned to three annotators and 'unanimous labels retained to account for subjectivity,' but it does not report inter-annotator agreement, the number or proportion of labels discarded, or any adjudication procedure for non-unanimous cases. This is load-bearing because if a large fraction of non-unanimous judgments was discarded, the retained 1,960 utterances and the 176-instance test split would be biased toward easy instances, and the subsequent fine-tuning results in Table 2 would overstate model utility. Please report per-task agreement metrics (e.g., Fleiss' kappa or Krippendorff's alpha), retention rates, and an explicit adjudication policy, or otherwise justify the filtering rule.
[§4, Table 2 and Table 3] The comparison between fine-tuned XLM-RoBERTa and GPT-4 is statistically unsupported. Table 3 shows that the test set contains only 24 check-worthy utterance labels, 18 stance 'True/Supports' instances, and 32 'False/Refutes' instances, yet Table 2 reports F1 differences such as 0.45 vs. 0.57 for true claims and 0.79 vs. 0.56 for supports. With double-digit denominators, these differences are compatible with sampling noise. No confidence intervals, bootstrap estimates, or significance tests are provided. Please add uncertainty quantification (e.g., bootstrap CIs) and, where appropriate, a paired test such as McNemar's test, or explicitly reframe the results as exploratory with no comparative claim.
[§1 (dataset release)] The claimed release of the annotated dataset is not verifiable from the manuscript. Section 1 gives only the tool repository URL (https://github.com/factiverse/factcheck-podcasts) and says 'we release transcripts for 531 episodes... alongside an annotated dataset,' but no dataset URL, archival link, license, or download procedure is provided. The abstract's phrase 'sample annotations' also leaves unclear whether the full annotations are released. Please specify the data repository, exact contents, licensing, and version, and ensure the URLs are resolvable.

minor comments (6)

[§2.2] The manuscript repeatedly defers annotation details to the self-cited master's thesis [1]; the thesis cannot substitute for reporting agreement and filtering statistics in the paper itself, and the self-citation should be supported by the primary data in the paper.
[§4, Table 3] The column headers 'True' and 'False' in Table 3 are ambiguous because §4 defines stance labels as 'Supports' and 'Refutes.' Please unify the terminology and state clearly whether True=Supports and False=Refutes.
[§4, Table 4] The transcription evaluation lacks details: it is not stated on what corpus, language, or audio conditions the Whisper error rates were computed, whether the transcripts were compared with human reference transcripts, or how the 'prompted' condition differs from the standard one. Please provide this context or remove the table if it is not comparable.
[§4.1] The fine-tuning setup is underspecified: no training hyperparameters, number of runs, seeds, early-stopping criteria, or label imbalance handling are reported, which makes it difficult to reproduce the XLM-RoBERTa results in Table 2.
[§1] The sentence 'the pipeline supports over 90 languages (excluding co-reference resolution)' is presented without supporting evidence; the paper only evaluates Whisper on what appears to be a small sample, and no multilingual evaluation of the full pipeline is reported. Please either provide such evidence or soften the claim.
[§3] The dataset analysis is based on only 7 annotated episodes, a limitation the paper acknowledges only in passing ('We plan larger-scale annotations in the future'); this scale should also be reflected in the abstract and conclusion, where the resource is described as an 'annotated dataset specifically for end-to-end fact-checking.'

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation: held-out supervised evaluation and a fixed prior-work GPT-4 prompt; self-citations are supplementary, not load-bearing.

full rationale

The paper's empirical chain is self-contained. The check-worthiness and stance labels are crowd-sourced in Section 2.2 and then used to fine-tune XLM-RoBERTa, with evaluation on held-out splits (Table 3: train/dev/test), which is standard supervised evaluation rather than a prediction that reduces to its fitted input. The GPT-4 comparison in Table 2 uses a prompt fixed in the author's prior SIGIR paper [11]; the prompt is an external published artifact and the F1 scores are freshly computed on this paper's own test set, so the comparison is not forced by construction. The Whisper WER results in Table 4 benchmark the ASR model against the pipeline's transcripts and show monotone improvement with model size, again an externally checkable measurement. The only self-citations are [1] (co-author's master's thesis, referenced for 'more details on the annotation process'), [10], [11], and [13] (same first author's related fact-checking editor, GPT-4 prompt, and live fact-checking system). None of these carries the derivation: the annotation procedure is stated in the paper (three annotators, unanimous labels retained), and the GPT-4 prompt serves as a fixed experimental setting rather than as evidence that the author's models are better. Concerns about missing inter-annotator agreement, the small test-set counts (24 check-worthy, 18 refutes, 32 supports), and absent confidence intervals are validity and statistical-power issues, not circularity, and are outside this pass's remit.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper does not rely on mathematical axioms, but its evaluation depends on assumptions about transcription fidelity, annotation schema validity, crowdworker agreement, and test-set size. These assumptions are load-bearing for the claimed utility of the dataset and the comparative results.

assumptions (4)

domain assumption Whisper transcription output is accurate enough for sentence-level annotation and downstream claim detection.
Used in Section 2.1 to populate transcripts; if transcription errors are high, the labels may be misaligned with the audio.
ad hoc to paper The annotation taxonomy (claim types, motivations, check-worthiness) is a complete and appropriate schema for podcast fact-checking.
Proposed in Section 2.2 with examples but no validation from external experts.
domain assumption Labels agreed on by three crowdworkers form a reliable gold standard.
Section 2.2 retains only unanimous labels; no agreement metrics are reported.
ad hoc to paper The small test split (24 check-worthy, 18 refutes, 32 supports) is sufficient for reliable performance comparison.
Underlies the F1 claims in Table 2; with tens of examples, differences may be noise.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Annotation Tool and Dataset for Fact-Checking Podcasts." pith.science (2026). https://pith.science/paper/24KC7SLB

@misc{pith2026250201402,
  author       = {Pith},
  title        = {Pith review of: Annotation Tool and Dataset for Fact-Checking Podcasts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/24KC7SLB}},
  note         = {Machine review of arXiv:2502.01402}
}

read the original abstract

Podcasts are a popular medium on the web, featuring diverse and multilingual content that often includes unverified claims. Fact-checking podcasts is a challenging task, requiring transcription, annotation, and claim verification, all while preserving the contextual details of spoken content. Our tool offers a novel approach to tackle these challenges by enabling real-time annotation of podcasts during playback. This unique capability allows users to listen to the podcast and annotate key elements, such as check-worthy claims, claim spans, and contextual errors, simultaneously. By integrating advanced transcription models like OpenAI's Whisper and leveraging crowdsourced annotations, we create high-quality datasets to fine-tune multilingual transformer models such as XLM-RoBERTa for tasks like claim detection and stance classification. Furthermore, we release the annotated podcast transcripts and sample annotations with preliminary experiments.

Figures

Figures reproduced from arXiv: 2502.01402 by the authors.

**Figure 1.** Podcast annotation interface for claim detection. Right hand size shows the options for fine-grained claim annotation. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

**Figure 2.** Fact-checking annotation interface. To facilitate downstream NLP tasks, the transcriptions were segmented into sentences using SpaCy, which split the transcribed text based on punctuation marks. Co-reference resolution was performed using the F-coref package [6]. This step ensured that pronouns and proper nouns were linked to the correct entities, creating self-contained utterances that were ready for fact-checki… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

15 extracted references · 10 canonical work pages

[1]

Adam James Becker. 2023. Automated Fact-Checking of Podcasts . Master’s thesis. University of Stavanger. https://hdl.handle.net/11250/3088365

work page 2023
[2]

Ann Clifton, Sravana Reddy, Yongze Yu, Aasish Pappu, Rezvaneh Rezapour, Hamed Bonab, Maria Eskevich, Gareth Jones, Jussi Karlgren, Ben Carterette, et al

work page
[3]

Naeemul Hassan, Gensheng Zhang, Fatma Arslan, Josue Caraballo, Damian Jimenez, Siddhant Gawsane, Shohedul Hasan, Minumol Joseph, Aaditya Kulkarni, Anil Kumar Nayak, et al . 2017. Claimbuster: The first-ever end-to-end fact- checking system. Proceedings of the VLDB Endowment 10, 12 (2017), 1945–1948

2017
[4]

Petar Ivanov, Ivan Koychev, Momchil Hardalov, and Preslav Nakov. 2024. Detect- ing Check-Worthy Claims in Political Debates, Speeches, and Interviews Using Audio Data. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 12011–12015

work page 2024
[5]

Potsawee Manakul and Mark JF Gales. 2022. Podcast Summary Assessment: A resource for evaluating summary assessment methods. arXiv preprint arXiv:2208.13265 (2022)

work page Pith review arXiv 2022
[6]

Shon Otmazgin, Arie Cattan, and Yoav Goldberg. 2022. F-coref: Fast, accurate and easy to use coreference resolution. arXiv preprint arXiv:2209.04280 (2022)

work page Pith review arXiv 2022
[7]

Alexis Plaquet and Hervé Bredin. 2023. Powerset multi-class cross entropy loss for neural speaker diarization. In Proc. INTERSPEECH 2023

2023
[8]

Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In International conference on machine learning . PMLR, 28492–28518

2023

Show all 15 references

[9]

Michael Schlichtkrull, Zhijiang Guo, and Andreas Vlachos. 2023. AVeriTeC: A dataset for real-world claim verification with evidence from the web. arXiv preprint arXiv:2305.13117 (2023)

2023 arXiv
[10]

Vinay Setty. 2024. Factcheck editor: Multilingual text editor with end-to-end fact-checking. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval . 2744–2748

2024
[11]

Vinay Setty. 2024. Surprising Efficacy of Fine-Tuned Transformers for Fact- Checking over Larger Language Models. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’24). 2842–2846

2024
[12]

James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal
[13]

Venktesh V and Vinay Setty. 2024. LiveFC: A System for Live Fact-Checking of Audio Streams. (2024). arXiv:2408.07448 [cs.CL]

2024 arXiv
[2018]

In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)

FEVER: a Large-scale Dataset for Fact Extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). 809–819

2018
[2020]

In Proceedings of the 28th International Conference on Computational Linguistics

100,000 podcasts: A spoken English document corpus. In Proceedings of the 28th International Conference on Computational Linguistics . 5903–5917

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.

[1] [1]

Adam James Becker. 2023. Automated Fact-Checking of Podcasts . Master’s thesis. University of Stavanger. https://hdl.handle.net/11250/3088365

work page 2023

[2] [2]

Ann Clifton, Sravana Reddy, Yongze Yu, Aasish Pappu, Rezvaneh Rezapour, Hamed Bonab, Maria Eskevich, Gareth Jones, Jussi Karlgren, Ben Carterette, et al

work page

[3] [3]

Naeemul Hassan, Gensheng Zhang, Fatma Arslan, Josue Caraballo, Damian Jimenez, Siddhant Gawsane, Shohedul Hasan, Minumol Joseph, Aaditya Kulkarni, Anil Kumar Nayak, et al . 2017. Claimbuster: The first-ever end-to-end fact- checking system. Proceedings of the VLDB Endowment 10, 12 (2017), 1945–1948

2017

[4] [4]

Petar Ivanov, Ivan Koychev, Momchil Hardalov, and Preslav Nakov. 2024. Detect- ing Check-Worthy Claims in Political Debates, Speeches, and Interviews Using Audio Data. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 12011–12015

work page 2024

[5] [5]

Potsawee Manakul and Mark JF Gales. 2022. Podcast Summary Assessment: A resource for evaluating summary assessment methods. arXiv preprint arXiv:2208.13265 (2022)

work page Pith review arXiv 2022

[6] [6]

Shon Otmazgin, Arie Cattan, and Yoav Goldberg. 2022. F-coref: Fast, accurate and easy to use coreference resolution. arXiv preprint arXiv:2209.04280 (2022)

work page Pith review arXiv 2022

[7] [7]

Alexis Plaquet and Hervé Bredin. 2023. Powerset multi-class cross entropy loss for neural speaker diarization. In Proc. INTERSPEECH 2023

2023

[8] [8]

Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In International conference on machine learning . PMLR, 28492–28518

2023