Pith. sign in

REVIEW 3 major objections 6 minor 17 references

Comparing LLM Text Annotation Skills: A Study on Human Rights Violations in Social Media Data

T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read GPT-4.0 is the most reliable off-the-shelf labeler of human rights violation references in Russian and Ukrainian Telegram posts, reaching an F1 of 0.84 in zero-shot English prompting.

desk verdict Useful domain-specific benchmark, but the GPT-4 ranking is not statistically supported and the paper contradicts itself on prompt-language effects. read the letter →

arxiv 2505.10260 v1 pith:753XPMNZ submitted 2025-05-15 cs.CL cs.AI

Poli Apollinaire Nemkova , Solomon Ubani , Mark V. Albert This is my paper · ORCID
classification cs.CLcs.AI
keywords humanrightsviolationslargelanguagemodelstextannotationzero-shotpromptingfew-shotmultilingualsocialmediaRussianandUkrainianTelegram
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether off-the-shelf large language models can label social media posts in Russian and Ukrainian for references to human rights violations as reliably as trained human annotators. It compares five models under zero-shot and few-shot prompting in English and Russian against 1000 human-labeled Telegram posts. The central finding is that GPT-4.0 performs best, reaching an F1 of 0.84 and accuracy of 0.82 with zero-shot English prompts, and that Russian prompts generally improve all models. Open-source models lag behind but gain substantially from few-shot examples, though their precision remains too low for unmonitored use. The upshot is a practical ranking of models and prompting strategies for a sensitive, multilingual classification task.

What carries the argument

The central object is the human-adjudicated gold-standard dataset: 1000 Telegram posts from public channels covering the Russia-Ukraine conflict, each carrying a binary label for whether it references a human rights violation. The evaluation machinery is a pair of prompting configurations, zero-shot and few-shot, each run in English and Russian, scored by precision, recall, F1, and accuracy, with an ablation study on the 816 posts where human annotators fully agreed and the 184 posts requiring adjudication.

What would settle it

Re-annotate the 184 adjudicated posts with several independent expert annotators and measure each model's F1 against each annotator and against the adjudicated labels; if the ranking of GPT-4.0 changes materially, or if the adjudicated labels themselves show low expert agreement, the paper's performance claims are benchmark-dependent.

Watch

Extended reading notes

Core claim

The paper claims that GPT-4.0 is the strongest off-the-shelf annotator for references to human rights violations in Russian and Ukrainian social media, achieving its top F1 of 0.84 and accuracy of 0.82 in zero-shot English prompting, with a precision of 0.78 and recall of 0.92. It further claims that prompting in Russian consistently improves performance across model architectures, and that few-shot prompting mainly helps open-source models such as LLaMA-3 and Mistral-7B, which still remain below the reliability needed for autonomous deployment in sensitive applications. The benchmark is a sample of 1000 Telegram posts, 517 labeled positive, double-annotated by native speakers with disagreements resolved by a senior adjudicator.

Load-bearing premise

The entire comparison rests on treating the human double-annotated labels, with an initial Kappa of 0.63 and 184 disagreements resolved by a single adjudicator, as correct ground truth for all 1000 posts; if those labels are biased or noisy, every model ranking and error analysis is measured against an unreliable benchmark.

Editorial extensions

If this is right

  • If GPT-4.0's F1 of 0.84 generalizes beyond this sample, human rights monitors could use it to pre-screen large volumes of Telegram posts and reserve human review for flagged cases.
  • In settings where recall matters more than precision, GPT-3.5 with Russian few-shot prompts (recall 0.92) offers a cheaper screening option that catches most true positives.
  • Organizations using open-source models should treat their annotations as draft labels requiring manual validation, since LLaMA-3's Russian few-shot run reached recall 0.99 but precision 0.52.
  • Prompt language should match the data language regardless of model choice, because Russian prompts outperformed English prompts across tested models.
  • The disagreement ablation suggests that models, like humans, find ambiguous cases harder, so deployment should include an explicit review step for low-confidence predictions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper treats the adjudicated human labels as ground truth; an editorial extension is that the absolute F1 numbers therefore measure agreement with one particular human judgment, not true detection accuracy, and should be read as upper bounds on real-world performance.
  • Because models show larger precision drops on the 184 disputed posts, a natural next step is to compare models against each annotator's original label rather than only the adjudicated label, which could reveal whether models align with one annotator's bias.
  • A testable extension is to replace binary gold labels with soft or probabilistic labels reflecting annotator disagreement; models that reproduce human uncertainty may be more useful for triage than models that mimic a single consensus label.
  • The same protocol could be applied to other conflict-reporting languages, but performance differences will likely depend on how well each model's training data covers that language.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper presents a comparative evaluation of five large language models—GPT-4.0, GPT-3.5, LLaMA-3 (described as LLaMA-3.2-1B in the method), Mistral-7B, and Claude-2—on the binary classification task of identifying references to human rights violations in a 1000-post sample of Russian and Ukrainian social media content from Telegram. The models are tested under zero-shot and few-shot prompting with prompts written in English and Russian, and their outputs are compared against a human double-annotated gold standard with adjudication. The authors report that GPT-4.0 achieves the best overall performance, with an F1 score of 0.84 and accuracy of 0.82 in zero-shot English prompting, that Russian prompts generally improve performance, that open-source models benefit more from few-shot prompting, and that models struggle on ambiguous cases, mirroring human difficulty. The paper also includes an error analysis and an ablation study on the subset of samples where human annotators disagreed.

Significance. If the reported ranking is statistically robust, the paper provides practically useful evidence on LLM suitability for high-stakes, multilingual annotation tasks in the human-rights domain. Its strengths include a realistic and difficult dataset, double-annotation with adjudication, a clear experimental matrix (two prompt types, two prompt languages, five models), fixed random seeds, and prompts made available in a GitHub repository. The error analysis and the agreement/disagreement ablation are informative. However, the central performance claims are supported only by point estimates, with no uncertainty quantification or significance testing, and several internal inconsistencies about the effect of prompt language undermine the general conclusions.

major comments (3)
  1. [Section 4 and Figure 1] The central claim that GPT-4.0 is the best model, with an F1 score of 0.84 in zero-shot English prompting, rests entirely on point estimates without confidence intervals, standard errors, or significance tests. With n=1000, the approximate binomial 95% confidence interval for accuracy is about ±3 percentage points, and the F1 confidence interval is wider because it combines precision and recall. Moreover, the top score is selected from 20 conditions (5 models × 2 prompt types × 2 languages), which inflates the observed maximum even under the null hypothesis. To support the ranking, the authors should report bootstrap confidence intervals for F1 and accuracy and perform paired tests (e.g., McNemar) for the GPT-4.0 vs. GPT-3.5 comparison and for the prompt-language comparisons.
  2. [Sections 3.2, 4, 5.1, and Key Findings] The paper makes contradictory claims about the effect of prompt language. Section 3.2 states that 'Russian prompts generally outperformed English prompts across models,' and the Key Findings section asserts that 'Models consistently performed better with prompts in Russian.' However, Section 4 reports GPT-4.0's best result is in zero-shot English, and Section 5.1 explicitly notes that Claude-2 and LLaMA-3 'generally performing worse on Russian prompts.' These statements cannot all be true as written. The authors should present a per-model and per-condition breakdown of the language effect and temper or qualify the general claim accordingly.
  3. [Section 3.1] The gold standard reliability is a load-bearing assumption. With an initial Cohen's Kappa of 0.63 and 184 of 1000 labels resolved by a single adjudicator, the reference labels themselves carry substantial uncertainty. Treating these labels as ground truth without a sensitivity analysis can overstate model performance and affect the ranking. The ablation in Figure 2 examines only GPT-4.0 and LLaMA-3 on the agreement/disagreement subsets; the authors should either extend this analysis to all models or explicitly discuss how potential adjudicator bias would affect the reported rankings.
minor comments (6)
  1. [Section 3.2 and Abstract] The model is referred to as both 'LLaMA-3.2-1B' in the method and 'LLaMA-3' in the abstract and figures; please use a single consistent name and specify the exact model version.
  2. [Section 3.2] The few-shot setting does not state the number of examples used or how they were selected; please specify k and the selection procedure for reproducibility.
  3. [Section 3.2] The statement that all experiments used fixed random seeds is ambiguous for API-based models; report the sampling parameters (temperature, top-p, etc.) actually used for each model.
  4. [Figures 1 and 2] The figures are referenced but not included in the provided text; ensure they are legible and include all reported metrics, and consider adding error bars or per-cell sample counts.
  5. [References] The reference list contains Törnberg (2023) twice; please deduplicate the entries.
  6. [Section 5.1] The sentence referencing 'Mistral-7B Few-Shot (Russian)' provides no numerical results; consider adding a summary table of all experimental conditions to make the paper self-contained.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the reported model rankings are direct measurements against a human-adjudicated gold standard, not derived from the models' own outputs.

full rationale

This paper does not contain a derivation chain in which a predicted quantity is equivalent to an input by construction. The central claim—that GPT-4.0 achieves the highest F1 of 0.84 in zero-shot English prompting (Section 4)—is an empirical comparison of LLM outputs against a human double-annotated gold standard described in Section 3.1. The gold-standard labels were produced by native-speaker annotators with disagreements resolved by a senior adjudicator; they were not fitted from or derived from the LLM outputs. No parameter is fitted on a subset and then renamed a prediction, no uniqueness theorem or ansatz is invoked, and no known result is merely relabeled. The only self-citation, Nemkova et al. (2023) for the HRV dataset in Section 3.1, is data provenance rather than a load-bearing argument: the prior dataset provides the input texts and human labels, but it does not by itself force any particular GPT-4 ranking. Concerns about missing confidence intervals, post hoc selection among 20 conditions, and the internal inconsistency about Russian versus English prompts are important statistical and reporting issues, but they are not circularity. The evaluation is self-contained in the sense that any model could in principle score differently on the same annotated sample.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claim rests on the validity of the human gold standard, the representativeness of the 1000-sample subset, and the reproducibility of closed-source model outputs. No free parameters are fitted; no new entities are postulated.

assumptions (3)
  • domain assumption Human-adjudicated labels are valid ground truth for human rights violation references despite initial Cohen's Kappa of 0.63.
    The paper uses these labels as the gold standard for all model comparisons in Section 3.1, without quantifying adjudicator reliability or handling label noise.
  • domain assumption The 1000 post sample is representative of the larger HRV dataset.
    The study draws conclusions about model performance in this domain from a random sample of 1000 posts, with 966 Russian and 34 Ukrainian posts (Section 3.1).
  • domain assumption Fixed random seeds ensure reproducibility of API model outputs.
    The paper claims reproducibility via fixed seeds (Section 3.2), but closed API models are non-deterministic and prompt details are not fully included in the manuscript.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Comparing LLM Text Annotation Skills: A Study on Human Rights Violations in Social Media Data." pith.science (2026). https://pith.science/paper/753XPMNZ

@misc{pith2026250510260,
  author       = {Pith},
  title        = {Pith review of: Comparing LLM Text Annotation Skills: A Study on Human Rights Violations in Social Media Data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/753XPMNZ}},
  note         = {Machine review of arXiv:2505.10260}
}
read the original abstract

In the era of increasingly sophisticated natural language processing (NLP) systems, large language models (LLMs) have demonstrated remarkable potential for diverse applications, including tasks requiring nuanced textual understanding and contextual reasoning. This study investigates the capabilities of multiple state-of-the-art LLMs - GPT-3.5, GPT-4, LLAMA3, Mistral 7B, and Claude-2 - for zero-shot and few-shot annotation of a complex textual dataset comprising social media posts in Russian and Ukrainian. Specifically, the focus is on the binary classification task of identifying references to human rights violations within the dataset. To evaluate the effectiveness of these models, their annotations are compared against a gold standard set of human double-annotated labels across 1000 samples. The analysis includes assessing annotation performance under different prompting conditions, with prompts provided in both English and Russian. Additionally, the study explores the unique patterns of errors and disagreements exhibited by each model, offering insights into their strengths, limitations, and cross-linguistic adaptability. By juxtaposing LLM outputs with human annotations, this research contributes to understanding the reliability and applicability of LLMs for sensitive, domain-specific tasks in multilingual contexts. It also sheds light on how language models handle inherently subjective and context-dependent judgments, a critical consideration for their deployment in real-world scenarios.

Figures

Figures reproduced from arXiv: 2505.10260 by the authors.

Figure 1
Figure 1. Among the evaluated models, GPT-4.0 demon￾strated the highest overall performance, particularly ex￾celling in zero-shot prompting with English prompts, where it achieved the top F1 score of 0.84 and an accuracy of 0.82. This performance highlighted its strong balance between precision (0.78) and recall (0.92). Similarly, GPT-3.5 per￾formed well, particularly in few-shot prompting with Rus￾sian prompts, achieving an … view at source ↗
Figure 1
Figure 1. Performance metrics for GPT 3.5, GPT 4.0, LLaMA-3, Mistral-7B, and Claude-2 in Few-Shot and Zero-Shot Prompt [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Performance metrics for LLaMA-3 and GPT-4.0 on the agreement and disagreement sets in Few-Shot and Zero-Shot [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 8 canonical work pages

  1. [3]

    arXiv preprint arXiv:2108.07258

    On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258. Brown, T.; and Others

    arXiv

  2. [6]

    LLMs Accelerate Annotation for Medical Information Extraction

    LLMs Accelerate Annotation for Medical Information Extraction. ArXiv, abs/2312.02296. Hasanbeig, H.; Sharma, H.; Betthauser, L.; Frujeri, F.; and Momennejad, I

    work page Pith review arXiv

  3. [7]

    ALLURE: Auditing and Improving LLM-based Evaluation of Text using Iterative In-Context-Learning

    ALLURE: Auditing and Improving LLM-based Evaluation of Text using Iterative In-Context- Learning. ArXiv, abs/2309.13701. Japkowicz, N.; and Stephen, S

    work page Pith review arXiv

  4. [8]

    arXiv preprint arXiv:2310.06825

    Mistral 7B. arXiv preprint arXiv:2310.06825. Li, M.; Shi, T.; Ziems, C.; Kan, M.-Y .; Chen, N. F.; Liu, Z.; and Yang, D

    arXiv

  5. [9]

    arXiv preprint arXiv:2310.15638

    Coannotating: Uncertainty-guided work allocation between human and large language models for data annotation. arXiv preprint arXiv:2310.15638. Lu, Y .; Yao, B.; Zhang, S.; Wang, Y .; Zhang, P.; Lu, T.; Li, T.; and Wang, D

    arXiv

  6. [10]

    ArXiv, abs/2311.09825

    Human Still Wins over LLM: An Em- pirical Study of Active Learning on Domain-Specific Anno- tation Tasks. ArXiv, abs/2311.09825. Mistral AI

    arXiv

  7. [11]

    Detecting Human Rights Violations on Social Media during Russia-Ukraine War

    Detecting Human Rights Violations on Social Media during Russia-Ukraine War. ArXiv, abs/2306.05370. OpenAI

    work page Pith review arXiv

  8. [12]

    ArXiv, abs/2306.00176

    Auto- mated Annotation with Generative AI Requires Validation. ArXiv, abs/2306.00176. Pineau, J.; LeBlanc, A.; Raffel, C.; Maddison, C.; Sinha, K.; Roelofs, R.; Cheung, V .; Dodge, J.; Kuznetsova, A.; and Pro- gram, N. R

    arXiv

Show all 17 references
  1. [13]

    Communications of the ACM, 64(4): 76–84

    Improving Reproducibility in Machine Learning Research: A Report from the NeurIPS 2019 Re- producibility Program. Communications of the ACM, 64(4): 76–84. Plaza-del Arco, F. M.; Nozza, D.; and Hovy, D

    2019

  2. [14]

    arXiv preprint arXiv:2307.12973

    Lever- aging label variation in large language models for zero-shot text classification. arXiv preprint arXiv:2307.12973. Raffel, C.; Shazeer, N.; Roberts, A.; Lee, K.; Narang, S.; Matena, M.; Zhou, Y .; Li, W.; and Liu, P. J

    arXiv

  3. [15]

    In Webber, B.; Cohn, T.; He, Y .; and Liu, Y ., eds.,Proceed- ings of the 2020 Conference on Empirical Methods in Nat- ural Language Processing (EMNLP) , 4222–4235

    AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts. In Webber, B.; Cohn, T.; He, Y .; and Liu, Y ., eds.,Proceed- ings of the 2020 Conference on Empirical Methods in Nat- ural Language Processing (EMNLP) , 4222–4235. Online: Association f...

    2020

  4. [17]

    arXiv preprint arXiv:2302.13971

    LLaMA: Open and Efficient Foundation Language Models. arXiv preprint arXiv:2302.13971. Available at https://arxiv.org/abs/2302. 13971. T¨ornberg, P

    arXiv

  5. [18]

    ArXiv, abs/2304.06588

    ChatGPT-4 Outperforms Experts and Crowd Workers in Annotating Political Twitter Messages with Zero-Shot Learning. ArXiv, abs/2304.06588. Wong, E

    arXiv

  6. [2020]

    Advances in Neural Information Processing Systems, 33: 1877–1901

    Language Models Are Few- Shot Learners. Advances in Neural Information Processing Systems, 33: 1877–1901. Cohen, J

    1901

  7. [2021]

    Bommasani, R.; Hudson, D

    On the dangers of stochastic par- rots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, 610–623. Bommasani, R.; Hudson, D. A.; Adeli, E.; Altman, R.; Arora, S.; von Arx, S.; Bernstein, M. S.; Bohg, J.; Bo...

    2021

  8. [2023]

    Human Annotators: A Comprehensive Analysis of ChatGPT for Text Annotation

    ChatGPT vs. Human Annotators: A Comprehensive Analysis of ChatGPT for Text Annotation. In 2023 International Conference on Machine Learning and Applications (ICMLA), 602–609. Anthropic

    2023

  9. [2024]

    arXiv preprint arXiv:2410.06415

    Biased ai can influence political decision-making. arXiv preprint arXiv:2410.06415. Goel, A.; Gueta, A.; Gilon, O.; Liu, C.; Erell, S.; Nguyen, L.; Hao, X.; Jaber, B.; Reddy, S.; Kartha, R.; Steiner, J.; Laish, I.; and Feder, A

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.