{"id":"bbe45b1b-647c-48ce-8168-be2c4857cf59","arxiv_id":"2507.18343","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"An LLM pre-annotation pipeline with span extraction and hierarchical labels improves human inter-annotator agreement and speed on Russian propaganda tweets, with smaller models distilled from the LLM outputs.","lead":"This paper tests whether letting an LLM pre-annotate propaganda tweets before humans label them improves agreement and speed. It reports higher annotator agreement and lower annotation time, and distills the LLM's judgments into smaller models, but the comparison is confounded by changes in annotators and task design.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed IAA/time gains are not attributable to LLM pre-annotation: Study 2 altered the task by showing LLM spans and local labels, and the hidden global label is largely inferable from those labels (Section 3.3.1), so the improvement may be an anchoring artifact rather than a verification…","rationale":"The reader's conditional verdict is appropriate. I agree with the reader's weakest assumption: the Study 1 versus Study 2 comparison is not controlled, and the causal claim in Section 3.4.2 is not supported as stated. The most load-bearing single issue is not just the annotator swap or the sampling change; it is that Study 2 changed the annotation task itself by displaying the LLM's span-level labels, and the paper's own statistics in Section 3.3.1 show that the hidden global label is strongly predictable from those displayed local labels. This makes the IAA gain an expected consequence of anchoring, not evidence that human-plus-LLM verification is better than manual annotation. The time gain is also confounded by learning effects, which the paper acknowledges, and by the simpler response format. I nonetheless recommend no change to the verdict because the paper has independent value: the taxonomy, the open-source LLM pipeline with stability ablations in Section 3.3.2, and the distillation results are reportable contributions, and the central quantitative claim can be repaired with a matched control condition or by re-framing the claim as descriptive. The reported Cohen's Kappa is also inflated by the random-LLM-prediction rule for non-majority cases, which should be corrected in revision. These are exactly the kinds of issues a conditional accept should require the authors to fix.","tokens_in":14431,"tokens_out":7263,"duration_ms":75889,"concrete_test":"Run a matched control experiment on the same stratified sample and the same annotator pool used in Study 2, with three conditions: (A) the current interface showing the LLM's spans and local labels; (B) the same interface showing only the tweet, with no LLM output; and (C) showing spans but withholding their local labels. Measure IAA and per-tweet annotation time in all arms. If condition B reproduces Study 1's low IAA and about 152 s per tweet while condition A reproduces Study 2's high IAA and about 41 s per tweet, the improvement is caused by the visible LLM cues, not by the human-verification workflow. A cheaper preparatory check: compute, for each Study 2 tweet, the global label inferred from the displayed local labels (majority or first label) and compare it with the human-majority label; near-perfect agreement would confirm anchoring before running the experiment.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing problem is that Study 2 does not compare the same annotation task with and without LLM assistance. In Study 1 (Section 3.2), annotators assigned coarse- and fine-grained labels to whole tweets with no additional cues. In Study 2 (Section 3.4), annotators are shown the LLM's extracted spans and their local labels, are told not to verify the spans, and only select global labels. The global label is hidden, but the paper's own analysis in Section 3.3.1 shows that for tweets with at least three spans, the first local label matches the global label in 76.65% of cases and the majority local label matches in 83.55%. Thus the 'independent' human global labels are strongly predictable from the visible local labels; the high IAA in Table 5 may simply reflect annotators converging on the LLM's pre-annotations. This is compounded by simultaneous changes in annotators (the most experienced annotator was removed), sampling (random vs stratified), and the acknowledged learning effect (Section 4). Consequently, the central claim that the integration 'resulted in' higher IAA and reduced time is not established by this design. A secondary metric, the Cohen's Kappa of 0.8438 between humans and LLM, is also inflated because cases without a 2/3 human majority are assigned a random LLM prediction as the 'human' label, so it partially measures LLM self-agreement.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a hybrid annotation framework for propaganda detection on tweets: a hierarchical taxonomy of 14 propaganda techniques into 3 intent-based categories; an initial manual annotation study (Study 1) on a sample of 355 tweets from the HQP dataset; an LLM pipeline (LLaMA-3.3-70B) that extracts spans, explains them, and assigns local and global labels; a second human verification study (Study 2) where annotators see the LLM's spans and local labels and select the global label; and knowledge distillation to fine-tune smaller models on LLM-generated annotations. The central claim, stated in Section 3.4.2, is that Study 2 achieves higher inter-annotator agreement and lower annotation time than Study 1, and that the student models learn to reproduce the teacher's structured annotations.","tokens_in":14718,"tokens_out":5008,"duration_ms":46677,"significance":"If the causal claim were established, the hybrid workflow would be a valuable contribution to scalable propaganda annotation. The paper ships publicly available code, uses a fully open-source teacher model, reports detailed ablations on prompt randomness and text normalization, and provides a large LLM-annotated dataset. These are concrete strengths. However, the central claim is not currently supported because the comparison between Study 1 and Study 2 is confounded by simultaneous changes in annotators, sampling, and task design, and the secondary human-LLM agreement metric is mechanically inflated by imputing random LLM labels as human labels. The student-model evaluation is a teacher-student consistency check rather than a quality evaluation. With additional controlled experiments or appropriately weakened claims, the work could still be a useful proof-of-concept, but the current evidence does not justify the stated conclusions.","major_comments":[{"comment":"The improvement in IAA and annotation time between Study 1 and Study 2 is not attributable to the LLM pre-annotation because several factors changed simultaneously: the annotator pool (the most experienced annotator was removed and replaced), the sampling method (random vs. stratified), and the task itself (free coarse/fine labeling of whole tweets vs. selecting global labels while being shown LLM spans and local labels). The paper further acknowledges a possible learning effect in Section 4 and in the Limitations. No significance tests are reported for the differences in Krippendorff's alpha or annotation time. Consequently, the central claim in Section 3.4.2 that the integration 'resulted in higher IAA and reduced annotation time' is confounded and should be reframed as an observational comparison, or the studies need to be redesigned to control these variables.","section":"Section 3.4 vs. Section 3.2, Table 5"},{"comment":"The reported Cohen's Kappa of 0.8438 between human majority-vote labels and LLM global labels is mechanically inflated. When no 2/3 human majority is reached, a random LLM prediction is imputed as the 'human' label, so those cases contribute LLM self-agreement rather than human-LLM agreement. The Kappa should be computed only on cases with a genuine human majority, or with a proper treatment of disagreement, and should be reported with confidence intervals.","section":"Section 3.4.2 (Cohen's Kappa)"},{"comment":"The independence of the human global-label choice in Study 2 is not assured. Section 3.3.1 shows that for tweets with at least three spans, the first local label matches the global label in 76.65% of cases and the majority local label matches in 83.55%. Since annotators in Study 2 see these local labels, the 'hidden' global label is largely inferable from the visible spans and labels. The higher agreement in Table 5 may therefore partly reflect annotators converging on the LLM's pre-annotations (anchoring) rather than independent verification. The authors should test this by comparing Study 2 agreement on cases where local labels are predictive of the global label versus cases where they are not.","section":"Section 3.3.1 and Section 3.4 (hidden global label)"},{"comment":"The evaluation of the student models in Table 7 is a teacher-student consistency measure: the students are trained on the teacher's (LLaMA-3.3-70B) annotations and evaluated on a held-out split of the same teacher's labels. This does not measure the quality of propaganda detection. There is no human-annotation baseline, no comparison to models trained on human labels, and no measure of agreement between student outputs and human judgment. The claims about scalable propaganda detection therefore need to be supported by human evaluation or at least by comparison on a dataset with gold human labels.","section":"Section 3.5, Table 7"}],"minor_comments":[{"comment":"The text says 'we fine-tune SMLs'; this should be 'SLMs' (small language models).","section":"Section 3.5"},{"comment":"There is a typo 'SemEvak-2020' in Section 2, which should be 'SemEval-2020'. Also, the reference for the SemEval-2020 paper lists the first author as 'G Martino' instead of 'Giovanni Da San Martino', and the two Hasanain et al. references use inconsistent spellings of the second author (Ahmad vs. Ahmed).","section":"Section 2 and References"},{"comment":"The label list in Appendix A.1 omits an entry 'n' and continues with 'o', and Table 8 lists 'whataboutism', 'straw_man', and 'red_herring' as separate labels, whereas the taxonomy defines them as a combined category. Please clarify how the 14 labels relate to the 17 entries in Table 8.","section":"Appendix A.1 and Table 8"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a computational linguistics venue, but the evaluation design is too confounded for the current claims. I would advise the editor that the central claim requires either a controlled comparison (same annotators, same sampling, task-to-task with and without pre-annotation) or a substantial weakening of the causal language. The student-model evaluation also needs a human baseline."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the organizing move—grouping Martino's 14 propaganda techniques into three coarse categories (emotional appeal, simplification/distortion, trust/authority games)—is a genuinely practical contribution for annotation work, and the LLM span-extraction-with-explanations pipeline is a sensible assembly of existing ideas. Second, the headline result, that LLM pre-annotation raised inter-annotator agreement from alpha 0.12 to 0.59 and cut annotation time from 152 to 41 seconds, is not actually demonstrated. Study 1 and Study 2 differ in annotator pool, sampling, and task structure simultaneously. In Study 2 annotators are shown the LLM's spans and local labels; the paper's own analysis shows the global label is predictable from local labels in most cases (83.55% majority match), so the agreement gain may be anchoring, not verification. The paper honestly admits a learning effect in the Limitations, but it does not control for it.\n\nWhat is genuinely good: the taxonomy is new and adoptable; the ablation on prompt randomness is a plus; the authors are transparent about limitations; and the code is public. The student distillation experiments report modest scores and the discussion of label overlap is sensible.\n\nSoft spots: the human-LLM Cohen's kappa of 0.8438 is inflated because cases without a 2/3 human majority get a random LLM prediction imputed as the human label, so it partially measures LLM self-agreement. The student models are trained and tested on the teacher's labels, making their F1 scores teacher-student consistency rather than detection quality. No significance tests are reported anywhere. These issues do not destroy the paper's value as a methods proposal, but they do undercut the causal claims.\n\nThis paper is for people building annotation pipelines for propaganda or other fine-grained social media tasks, not for readers needing a confirmed effect size. It deserves a serious referee—there is enough real method here—but the referee should push for a matched head-to-head study with the same annotators, same sampling, and an independent gold set. I would not cite it for the quantitative claims, but I might borrow the taxonomy.","headline":"The three-way propaganda taxonomy and LLM span pre-annotation are useful, but the uncontrolled Study 2 design does not establish that LLM assistance caused the agreement and time gains.","tokens_in":15291,"tokens_out":2402,"would_cite":false,"duration_ms":22810,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper shows that LLM-generated pre-annotations with human verification raise fine-grained propaganda-label agreement and cut annotation time.","keywords":["propaganda detection","LLM-assisted annotation","human-in-the-loop","inter-annotator agreement","span extraction","knowledge distillation","hierarchical taxonomy","social media disinformation"],"falsifier":"A controlled study using the same annotators on the same tweet sample under three conditions—manual labeling, LLM pre-annotation with correct spans, and LLM pre-annotation with randomly generated spans—would settle the claim. If the random-span condition shows the same agreement and time gains, the effect comes from task simplification or anchoring, not from the LLM's content; if only the correct-span condition improves, the central claim is supported.","tokens_in":14175,"feed_emoji":"🤖","tokens_out":7654,"duration_ms":68765,"temperature":0.7,"pith_summary":"This paper tries to establish that a hybrid annotation workflow—a large language model first extracts propagandistic spans, writes short explanations, and assigns fine-grained technique labels, then human annotators verify by choosing coarse and fine labels for the whole tweet—produces more consistent and faster propaganda annotations than fully manual labeling. In the unassisted study, three annotators fully agreed on only 7.61% of fine-grained labels; in the assisted study, full three-way agreement rose to 47.89%, Krippendorff's alpha went from 0.1233 to 0.5941, and average annotation time fell from 151.70 to 41.14 seconds per tweet. The paper also groups the 14 propaganda techniques into three coarse categories intended to ease annotator decisions, and it distills the LLM's structured outputs into smaller open models that can detect spans without human-labeled training data. If the claimed improvement holds, it points to a scalable way to build larger, more consistent propaganda-detection datasets for social media.","feed_headline":"LLM pre-annotations make propaganda labeling consistent and 73% faster","feed_subtitle":"Fine-grained agreement jumped from 7.6% to 47.9% while time per tweet fell from 152 to 41 seconds.","key_machinery":"The load-bearing mechanism is the LLM pre-annotation pipeline: a large open model (LLaMA3-70B-Instruct) extracts propagandistic spans, explains why each is propagandistic, assigns a local fine-grained label to each span, and then selects a global label for the tweet, with outputs forced into a structured JSON format. The global label is hidden from human annotators, who verify by selecting coarse and fine labels for the whole tweet with the spans and local labels visible. That design converts the hardest part of the task—finding and naming the manipulative segment from scratch—into a verification task, which is what carries the reported gains in agreement and speed.","core_discovery":"The paper's central claim is that LLM-generated pre-annotations, when combined with human verification, substantially improve both consistency and efficiency in fine-grained propaganda annotation. Presented with the LLM's extracted spans and local labels but not its global prediction, human annotators reached full three-way agreement on 47.89% of tweets, up from 7.61% with unassisted labeling, while Krippendorff's alpha improved from 0.1233 to 0.5941. The paper also reports that the human majority-vote global label agreed strongly with the LLM's hidden global label (Cohen's kappa 0.8438), and that annotation time dropped by roughly 73%. On tweets where all annotators agreed on the coarse category, fine-grained 2/3 agreement reached 99.55%, which the paper reads as evidence that the three-category taxonomy and the span-level pre-annotation jointly make fine-grained decisions tractable.","pith_inferences":["The reported gains are not cleanly attributable to the LLM pre-annotations, because the second study also replaced the most experienced annotator, switched from random to stratified sampling, and changed the task from free labeling to verification; a controlled comparison holding annotators and sampling fixed would be needed to separate causes.","If the agreement gain comes mainly from anchoring on the displayed spans rather than their correctness, then even wrong pre-annotations could raise agreement, which would inflate confidence in LLM-assisted pipelines; a study with deliberately incorrect spans would reveal this.","The alignment between majority local labels and global labels (83.55% when a majority exists) could be used as a model-confidence signal to route only uncertain tweets to human verification, making the pipeline cheaper without losing consistency.","Because the student models inherit the teacher's training-data biases, the distilled pipeline should be audited on text from opposing political directions; the paper flags this but does not test it."],"forward_implications":["Fine-grained propaganda annotation can be scaled beyond fully manual effort: verification takes about 41 seconds per tweet versus 152 seconds for unassisted labeling, so larger corpora become feasible at similar cost.","Global-label-only annotation is insufficient, because most propagandistic tweets contain multiple spans; keeping span extraction and local labels as the primary output preserves information that a single label discards.","Small open models distilled on LLM outputs can find propaganda spans (fuzzy-span F1 of 0.60–0.67) but remain weak at assigning local techniques (0.21–0.29 F1), so human verification is most valuable at the label-assignment step.","The coarse taxonomy acts as a useful intermediate layer: fine-grained 2/3 agreement reaches 99.55% on tweets where all annotators agree on the coarse category, suggesting hierarchical labeling reduces confusion among fine-grained techniques."],"supporting_citations":[{"why":"Defines the 14 fine-grained propaganda techniques used as local and global labels throughout both annotation studies.","marker":"Martino et al. (2020)"},{"why":"Provides the HQP dataset of propagandistic tweets that both Study 1 and Study 2 sample from.","marker":"Maarouf et al. (2023)"},{"why":"Documents low inter-annotator agreement for fine-grained propaganda labels, the challenge Study 1 replicates.","marker":"Hasanain et al. (2024)"},{"why":"Supplies the LLM-as-annotator approach and the fuzzy-span F1 metric the paper adopts for span evaluation.","marker":"Hasanain et al. (2023)"},{"why":"Establishes GPT-3/GPT-4 multi-label propaganda classification as the baseline the current LLM pipeline extends.","marker":"Sprenkamp et al. (2023)"},{"why":"Supports the premise that LLMs can reduce annotation time and cost across text-annotation tasks.","marker":"Alizadeh et al. (2025)"}],"fun_headline_variants":["LLM pre-annotations make propaganda labeling 73% faster, 6x more consistent","Hybrid LLM-human annotation: propaganda labels 73% faster, agreement 6x higher","LLM pre-annotation + human check: propaganda labeling 73% faster, 6x more reliable","Human-LLM hybrid labeling boosts propaganda detection agreement to 48%","Propaganda labeling: LLM pre-annotation cuts time 73%, lifts agreement to 48%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that Study 2's higher agreement and lower annotation time are caused by the LLM pre-annotations rather than by the simultaneous change of annotators, the switch to stratified sampling, the shift from free labeling to verification, or the annotators' growing familiarity with the task.","fun_headline_variants_meta":{"raw":{"variants":["LLM pre-annotations make propaganda labeling 73% faster, 6x more consistent","Hybrid LLM-human annotation: propaganda labels 73% faster, agreement 6x higher","LLM pre-annotation + human check: propaganda labeling 73% faster, 6x more reliable","Human-LLM hybrid labeling boosts propaganda detection agreement to 48%","Propaganda labeling: LLM pre-annotation cuts time 73%, lifts agreement to 48%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001451,"raw_usage":{"total_tokens":5848,"prompt_tokens":957,"completion_tokens":4891,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":573,"completion_tokens_details":{"reasoning_tokens":4771}},"tokens_in":573,"tokens_out":4891,"duration_ms":37568,"temperature":1.0,"reasoning_tokens":4771,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:13:57.643305+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A controlled study using the same annotators on the same tweet sample under three conditions—manual labeling, LLM pre-annotation with correct spans, and LLM pre-annotation with randomly generated spans—would settle the claim. If the random-span condition shows the same agreement and time gains, the effect comes from task simplification or anchoring, not from the LLM's content; if only the correct-span condition improves, the central claim is supported.","supporting_citations":[{"cited_title":"HQP: A Human-Annotated Dataset for Detecting Online Propaganda","cited_arxiv_id":"2304.14931","evidence_quote":"Provides the HQP dataset of propagandistic tweets that both Study 1 and Study 2 sample from."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents low inter-annotator agreement for fine-grained propaganda labels, the challenge Study 1 replicates."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supports the premise that LLMs can reduce annotation time and cost across text-annotation tasks."}],"review_version":2}