Scaling Public Health Text Annotation: Zero-Shot Learning vs. Crowdsourcing for Improved Efficiency and Labeling Accuracy

Kamyar Kazari; Yong Chen; Zahra Shakeri

REVIEW 4 major objections 4 minor 15 references

Scaling Public Health Text Annotation: Zero-Shot Learning vs. Crowdsourcing for Improved Efficiency and Labeling Accuracy

T0 review · 4 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Zero-shot GPT-4 Turbo labeling matches crowdworker accuracy on clear-cut public health tweets, while costing roughly $6 and one hour instead of $120–200 and one to two weeks.

desk verdict Useful cost/time data, but the accuracy comparison is undermined by class imbalance: GPT-4 says 'No' on 92.5% of tweets, so the 'rivalry' claim needs per-class balanced metrics. read the letter →

arxiv 2502.06150 v1 pith:WQNNWH35 submitted 2025-02-10 cs.CL

Kamyar Kazari , Yong Chen , Zahra Shakeri This is my paper · ORCID

classification cs.CL

keywords zero-shotlearninglargelanguagemodelstextannotationcrowdsourcingpublichealthsurveillanceTwitter/XclassificationpromptengineeringGPT-4

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether a zero-shot large language model can replace paid crowdworkers for labeling public health tweets, and answers with a head-to-head comparison on 12,000 posts about physical activity, sedentary behavior, and sleep. It reports that GPT-4 Turbo matches or beats Amazon Mechanical Turk accuracy on the two straightforward categories (81.0% vs. 79.1% for physical activity; 73.3% vs. 64.7% for sedentary behavior) while cutting the cost from $120–200 to roughly $6 and the turnaround from one to two weeks to about an hour. On sleep-problem tweets, however, the model falls behind crowdworkers (56.1% vs. 69.8%), because those posts often encode domain knowledge in medication names or indirect references. The paper argues for a hybrid workflow: let the LLM handle clear-cut cases and reserve human expertise for ambiguous, domain-heavy items.

What carries the argument

The comparison rests on a three-source labeling design: three Amazon Mechanical Turk workers produce a majority-vote label $y^{(AMT)}$, three domain experts produce the gold-standard label $y^{(EXP)}$ by majority vote, and GPT-4 Turbo, queried through its API with category-specific prompts engineered iteratively and optimized by an automated prompt-refinement tool, produces $y^{(LLM)}$. Accuracy is measured as agreement with $y^{(EXP)}$, alongside time and cost. The pipeline partitions tweets into batches of 800–1000, resends requests until the output parses cleanly, and reintroduces emojis that preprocessing had stripped, since emojis carry context. Prompt engineering, temperature adjustments, and repeated labeling with majority voting are all tested; none closes the sleep-problem gap, which the paper attributes to the model's tendency to avoid 'Unclear' and to its unfamiliarity with medication names and indirect references.

What would settle it

Recompute both systems' accuracy on a sample of sleep-problem tweets using an adjudicated gold standard (a fourth expert resolving disagreements among the three, or a panel discussion) and check whether GPT-4 Turbo's 56.1% still trails AMT's 69.8%; if the ranking flips, the claimed domain-knowledge gap is an artifact of the expert-label proxy. A cheaper falsifier is to report Cohen's kappa among the three experts: low agreement would undermine the accuracy comparisons themselves.

Watch

Extended reading notes

Core claim

The central claim is that zero-shot GPT-4 Turbo labeling is a viable substitute for crowd-sourced annotation in straightforward public health text classification, but not in tasks that require specialized domain knowledge. Measuring every label against a gold standard of three domain experts, the paper finds GPT-4 Turbo accuracy of 81.0% on physical activity and 73.3% on sedentary behavior, exceeding the AMT majority vote at 79.1% and 64.7%, while rarely using the 'Unclear' label. For sleep problems, GPT-4 Turbo drops to 56.1% against AMT's 69.8%; the confusion matrices show the model misreading medication names and partial clues as definite positive or negative signals. The cost and time advantages are stark: about $6 and one hour for all 12,000 tweets versus $120–200 and one to two weeks for AMT. The paper concludes that LLM-based labeling can be integrated into public health surveillance pipelines for unambiguous categories, with human oversight reserved for nuanced cases.

Load-bearing premise

The entire comparison stands on the premise that the majority vote of three domain experts is a reliable gold standard for inherently ambiguous tweets, and the paper does not quantify inter-expert agreement.

Editorial extensions

If this is right

Public health projects can replace AMT with GPT-4 Turbo for clear-cut tweet categories, cutting labeling budgets by roughly 95% and turnaround from weeks to one hour.
The LLM's underuse of the 'Unclear' label points to a need for confidence thresholds or abstention mechanisms in automated annotation pipelines.
Sleep-related annotation should remain with human experts, or be augmented with domain-adapted fine-tuning, until language models reliably recognize medication names and indirect references.
A hybrid workflow—automated labeling for common cases and expert review for the rest—offers a practical design for large-scale social media surveillance, including outbreak and hazard monitoring.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A testable extension: the sleep-problem accuracy gap may be more a calibration problem than a knowledge gap, since GPT-4 Turbo almost never chooses 'Unclear'; a model with an explicit abstain option might recover much of the difference.
This editor infers that the reported cost comparison omits the engineering time spent on prompt iteration and output validation, so the LLM's economic advantage is smaller for one-off projects but still large for repeated or streaming annotation.
The natural next experiment is to apply the same pipeline to a reasoning-oriented language model or a small fine-tuned adapter (the paper names LoRA as future work), using the sleep category as the primary test bed for whether domain-specific tuning closes the gap.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: the paper's own Table II undercuts its central accuracy claim. GPT-4 Turbo says 'No' on 11,101 of 12,000 tweets (92.5%), so a classifier that always says 'No' gets 72.0% overall accuracy—slightly above GPT's 71.5% and about equal to AMT's 70.2%. The 'rivalry' headline is therefore not established by the accuracy numbers as reported.

That said, the paper does something useful: it measures cost and throughput on a real 12k public-health tweet set, with GPT-4 Turbo at roughly $6 and about one hour versus $120–200 and one to two weeks for AMT, and it shows sleep-related tweets that hinge on medication names or hidden context are where the LLM collapses. The limitations section is honest, and the authors do not oversell.

The main soft spots are real and mostly in the evaluation. There are no per-category confusion matrices, so we cannot tell whether GPT's edge on physical activity and sedentary behavior is genuine or just majority-class behavior. There are no confidence intervals or significance tests, and the expert gold standard is taken as ground truth without reporting inter-annotator agreement (e.g., Cohen's kappa). Saying 'three experts majority vote' is not enough when the task is inherently ambiguous. There is also a numerical inconsistency: Table I gives AMT overall accuracy 70.21%, but the diagonal of Table II gives 69.75%. That is minor, but it should be fixed. The prompt selection on a 100-tweet sample is a potential overfitting point, and the 'GitHub Repository' link in the manuscript is not an actual URL.

The stress-test note is right on the merits. The underuse of 'Unclear' is not a secondary quirk; it is the reason the accuracy comparison misleads. A reader cannot conclude from this paper that GPT-4 rivals crowdworkers at annotation; the correct conclusion is that a cheap 'No'-heavy heuristic would look about as good on this dataset, and that per-class balanced metrics are essential before comparing.

I would send this to peer review, not desk-reject, because the empirical setup—real dataset, concrete cost and time measurements, clear task—is worth refereeing. But I would require a reanalysis with per-category balanced accuracy, majority baselines, confidence intervals, and the missing reproducibility details. For a public health group weighing whether to spend $200 or $6, the paper is a useful caution even if its headline is not proven.

Referee Report

4 major / 4 minor

Summary. The paper compares zero-shot GPT-4 Turbo labeling against Amazon Mechanical Turk (AMT) annotations for 12,000 tweets across three public-health categories (physical activity, sedentary behavior, sleep problems), using expert majority vote as the gold standard. It reports category-level and overall accuracies, finding that GPT-4 Turbo rivals or exceeds AMT on the 'straightforward' categories (physical activity, sedentary behavior) but lags on sleep problems, while reducing cost to about $6 and time to about one hour versus $120-200 and one to two weeks for AMT. The authors also experiment with prompt engineering, temperature adjustment, and repeated labeling, concluding that a hybrid LLM-human workflow is promising.

Significance. If the accuracy results were properly established, the paper would be a useful contribution to the growing literature on LLM-assisted annotation for public health surveillance. Its strengths include a transparent experimental pipeline, publicly available source code, a non-trivial real-world dataset, and explicit attention to cost and time metrics. The main scientific claims, however, rest on accuracy numbers that are not robust to the severe class imbalance in the data and lack statistical validation; the reported overall GPT-4 accuracy is actually below a trivial majority-class baseline. The paper also does not quantify the reliability of its expert gold standard. These issues are fixable through reanalysis and additional metrics, so the work has potential, but the current evidence is insufficient to support the central 'rivalry' claim.

major comments (4)

[Section III, Table II; Section IV] The overall accuracy comparison is undermined by severe class imbalance. From Table II, the expert gold standard y_EXP contains 8,642 'No' (72.0%), 2,187 'Yes' (18.2%), and 1,171 'Unclear' (9.8%), while GPT-4 predicts 'No' on 11,101 of 12,000 tweets (92.5%) and 'Unclear' on only 11 tweets. A constant-'No' classifier would achieve 72.0% accuracy, which is higher than the reported GPT-4 overall accuracy of 71.5% and essentially equal to the AMT overall accuracy. The paper's conclusion that LLM labeling 'can rival human performance' therefore needs to be supported by per-category balanced accuracy, macro-F1, per-class recall/precision, and comparisons to majority-class baselines; otherwise the accuracy figures do not measure labeling competence in this imbalanced setting.
[Section II.A and Section IV (Limitations)] The gold standard y_EXP is defined as the majority vote of three domain experts, but the paper does not report any inter-expert agreement statistic (e.g., Cohen's kappa or Fleiss' kappa) on the 12,000 tweets. Since the Limitations section acknowledges that 'consensus remains only a proxy for absolute truth', the reliability of the gold standard is load-bearing for every accuracy number in the paper. The authors should quantify agreement and, if feasible, report accuracy on the subset of tweets where experts agree unanimously, to check robustness of the GPT-4 vs. AMT comparison.
[Section III, Table I] The category-level accuracies in Table I are reported without per-category label distributions or confusion matrices. Given the overall imbalance, the apparent advantage of GPT-4 in the 'physical activity' (81.0% vs. 79.1%) and 'sedentary behavior' (73.3% vs. 64.7%) categories may be driven by majority-class prediction rather than genuine labeling ability. Per-category baseline accuracies, balanced accuracy, and recall per class are needed to substantiate the paper's central 'straightforward vs. sleep' contrast.
[Section III, Tables I and II] There is an internal inconsistency: Table I reports AMT overall accuracy as 70.21%, but the diagonal of the AMT confusion matrix in Table II sums to (1471+6869+30)/12000 = 69.75%. The authors should reconcile these numbers and report which is correct; the discrepancy may affect the comparison.

minor comments (4)

[Section I] The introduction says 'Section V discusses challenges...' but the paper has no Section V; the discussion and conclusions are presented in Section IV. Please correct the cross-reference.
[Table I] The row labelled 'Sedentary Activity' in Table I should be 'Sedentary Behavior' to match the terminology used throughout the text.
[Section II.C] The $6 GPT-4 cost is reported as including multiple requests for format mismatches, but no breakdown of retry costs is given; a short cost table would improve transparency.
[Section III] No confidence intervals or significance tests (e.g., McNemar's test) are reported for the accuracy differences between GPT-4 and AMT; given the large sample size, the authors should at least note the statistical uncertainty around the point estimates.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation: the comparisons are empirical benchmarks against pre-existing labels; author-overlap citations and prompt tuning are not load-bearing circularity.

full rationale

The paper's central claims are empirical measurements rather than derivations. GPT-4 Turbo outputs y^(LLM)_i are generated by the API and compared with fixed y^(EXP)_i and y^(AMT)_i labels taken from a previously published repository [11]; no equation defines the LLM labels in terms of the gold labels, and the gold labels are not derived from the LLM outputs. The author-overlap citations ([2], [11]) supply the dataset and contextual background, but they are not used as evidence that the LLM performs well; the reported accuracies are determined by the API's actual outputs against pre-existing labels. The gold-standard subjectivity acknowledged in the Limitations and the class-imbalance concerns raised by skeptics are threats to validity, not circularity. The only mild concern is prompt selection on a 100-tweet gold-labeled sample (Section II.C) and the reuse of the same instructional document used by AMT workers, which slightly compromises the label 'zero-shot' and could introduce optimistic bias; however, the reported full-dataset accuracy is independently measured and is not statistically forced by that selection, so this is a methodological weakness rather than a circular reduction. Therefore no step in the paper reduces to its own inputs by definition or by self-citation; the score reflects minor, non-load-bearing self-citations rather than any substantive circularity.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claim rests on treating expert majority labels as ground truth, on the rehydrated tweets faithfully representing the original posts, and on the comparability of the old AMT labels with the new GPT labels. The only free parameter of note is the per-category prompt variant selected on a 100-tweet sample. No new entities are introduced.

free parameters (2)

Per-category prompt variant = Best accuracy on a 100-tweet sample
Section II.C: each category's prompt was optimized by PromptPerfect over five runs and the variant with highest labeling accuracy on a 100-tweet sample was chosen. This is a small model-selection step on the gold standard and can modestly inflate reported accuracy.
GPT-4 Turbo temperature = Not specified (decreased from default)
Section II.C and III: the paper states temperature was decreased to reduce randomness but does not report the value. The central accuracy comparison is not very sensitive to this, but it is a hand-chosen setting.

assumptions (3)

domain assumption Expert majority vote y_EXP is treated as the gold standard for accuracy.
Section II.A defines y_EXP as the reference for computing alpha, tau, and kappa. The paper acknowledges in Limitations that expert consensus is only a proxy for truth and does not quantify inter-expert agreement.
domain assumption Rehydrated tweets faithfully represent the original posts used in the repository.
Section II.B describes rehydrating tweets from Tweet IDs after collecting from a public repository. Deleted or unavailable tweets could alter the dataset, and emojis were removed then partially reintroduced.
domain assumption AMT labels from reference [11] were produced under instructions comparable to those given to GPT-4.
Section II.C states the same instructional document used by AMT workers was supplied to GPT-4 Turbo to maintain consistency. Any mismatch in instruction comprehension could confound the comparison.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Scaling Public Health Text Annotation: Zero-Shot Learning vs. Crowdsourcing for Improved Efficiency and Labeling Accuracy." pith.science (2026). https://pith.science/paper/WQNNWH35

@misc{pith2026250206150,
  author       = {Pith},
  title        = {Pith review of: Scaling Public Health Text Annotation: Zero-Shot Learning vs. Crowdsourcing for Improved Efficiency and Labeling Accuracy},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WQNNWH35}},
  note         = {Machine review of arXiv:2502.06150}
}

read the original abstract

Public health researchers are increasingly interested in using social media data to study health-related behaviors, but manually labeling this data can be labor-intensive and costly. This study explores whether zero-shot labeling using large language models (LLMs) can match or surpass conventional crowd-sourced annotation for Twitter posts related to sleep disorders, physical activity, and sedentary behavior. Multiple annotation pipelines were designed to compare labels produced by domain experts, crowd workers, and LLM-driven approaches under varied prompt-engineering strategies. Our findings indicate that LLMs can rival human performance in straightforward classification tasks and significantly reduce labeling time, yet their accuracy diminishes for tasks requiring more nuanced domain knowledge. These results clarify the trade-offs between automated scalability and human expertise, demonstrating conditions under which LLM-based labeling can be efficiently integrated into public health research without undermining label quality.

Figures

Figures reproduced from arXiv: 2502.06150 by the authors.

**Figure 1.** Overview of the LLM-based tweet annotation pipeline implemented in this paper. The dataset X is partitioned into smaller chunks xi,k to fit GPT-4's context window. Annotation guidelines are extracted from the instruction PDF and structured into a prompt P = g(D). GPT-4 generates tweet labels y (LLM) i , which undergo validation and correction in the output check stage. The final structured labels are evaluated by co… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

15 extracted references · 7 canonical work pages

[1]

A contrastive topic-aware attentive framework with label encodings for post-disaster resource classification,

S. S. Dar, M. K. Karandikar, M. Z. U. Rehman, S. Bansal, and N. Kumar, “A contrastive topic-aware attentive framework with label encodings for post-disaster resource classification,” Knowledge- Based Systems , vol. 304, p. 112 526, 2024, ISSN : 0950-7051

work page 2024
[2]

Crowdsourcing for machine learning in public health surveillance: Lessons learned from amazon mechanical turk,

Z. Shakeri Hossein Abad, G. P. Butler, W. Thompson, and J. Lee, “Crowdsourcing for machine learning in public health surveillance: Lessons learned from amazon mechanical turk,” Journal of Medical Internet Research, vol. 24, no. 1, e28749, 2022

work page 2022
[3]

Who broke amazon mechanical turk? an analysis of crowdsourcing data quality over time,

C. C. Marshall, P. S. Goguladinne, M. Maheshwari, A. Sathe, and F. M. Shipman, “Who broke amazon mechanical turk? an analysis of crowdsourcing data quality over time,” in Proceedings of the 15th ACM Web Science Conference 2023 , 2023, pp. 335–345

work page 2023
[4]

The perils of using me- chanical turk to evaluate open-ended text generation,

M. Karpinska, N. Akoury, and M. Iyyer, “The perils of using me- chanical turk to evaluate open-ended text generation,” arXiv preprint arXiv:2109.06835, 2021

arXiv 2021
[5]

Language models are few-shot learners,

T. B. Brown, “Language models are few-shot learners,” arXiv preprint arXiv:2005.14165, 2020

arXiv 2005
[6]

Language models are unsupervised multitask learners,

A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al., “Language models are unsupervised multitask learners,” OpenAI blog, vol. 1, no. 8, p. 9, 2019

2019
[7]

C. Qin, A. Zhang, Z. Zhang, J. Chen, M. Yasunaga, and D. Yang, Is chatgpt a general-purpose natural language processing task solver?

work page
[8]

T ¨ornberg, Chatgpt-4 outperforms experts and crowd workers in annotating political twitter messages with zero-shot learning , 2023

P. T ¨ornberg, Chatgpt-4 outperforms experts and crowd workers in annotating political twitter messages with zero-shot learning , 2023. arXiv: 2304.06588 [cs.CL]

arXiv 2023

Show all 15 references

[9]

Y . Zhu, P. Zhang, E.-U. Haq, P. Hui, and G. Tyson, Can chatgpt reproduce human-generated labels? a study of social computing tasks, 2023. arXiv: 2304.10145 [cs.AI]

2023 arXiv
[10]

Chatgpt outperforms crowd workers for text-annotation tasks,

F. Gilardi, M. Alizadeh, and M. Kubli, “Chatgpt outperforms crowd workers for text-annotation tasks,” Proceedings of the National Academy of Sciences , vol. 120, no. 30, Jul. 2023, ISSN : 1091-6490

2023
[11]

Physical activity, sedentary behavior, and sleep on twitter: Multicountry and fully labeled public data set for digital public health surveillance research,

Z. Shakeri Hossein Abad, G. P. Butler, W. Thompson, and J. Lee, “Physical activity, sedentary behavior, and sleep on twitter: Multicountry and fully labeled public data set for digital public health surveillance research,” JMIR Public Health and Surveillance , vol. 8, no. 2, e...

2022
[12]

[Online]

PromptPerfect - AI Prompt Generator and Optimizer . [Online]. Available: https://promptperfect.jina.ai/
[13]

Evaluation of openai o1: Opportuni- ties and challenges of agi,

T. Zhong, Z. Liu, Y . Pan, et al., “Evaluation of openai o1: Opportuni- ties and challenges of agi,” arXiv preprint arXiv:2409.18486 , 2024

2024
[14]

E. J. Hu, Y . Shen, P. Wallis, et al. , Lora: Low-rank adaptation of large language models , 2021. arXiv: 2106.09685 [cs.CL]

2021 arXiv
[2023]

arXiv: 2302.06476 [cs.CL]

arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.

[1] [1]

A contrastive topic-aware attentive framework with label encodings for post-disaster resource classification,

S. S. Dar, M. K. Karandikar, M. Z. U. Rehman, S. Bansal, and N. Kumar, “A contrastive topic-aware attentive framework with label encodings for post-disaster resource classification,” Knowledge- Based Systems , vol. 304, p. 112 526, 2024, ISSN : 0950-7051

work page 2024

[2] [2]

Crowdsourcing for machine learning in public health surveillance: Lessons learned from amazon mechanical turk,

Z. Shakeri Hossein Abad, G. P. Butler, W. Thompson, and J. Lee, “Crowdsourcing for machine learning in public health surveillance: Lessons learned from amazon mechanical turk,” Journal of Medical Internet Research, vol. 24, no. 1, e28749, 2022

work page 2022

[3] [3]

Who broke amazon mechanical turk? an analysis of crowdsourcing data quality over time,

C. C. Marshall, P. S. Goguladinne, M. Maheshwari, A. Sathe, and F. M. Shipman, “Who broke amazon mechanical turk? an analysis of crowdsourcing data quality over time,” in Proceedings of the 15th ACM Web Science Conference 2023 , 2023, pp. 335–345

work page 2023

[4] [4]

The perils of using me- chanical turk to evaluate open-ended text generation,

M. Karpinska, N. Akoury, and M. Iyyer, “The perils of using me- chanical turk to evaluate open-ended text generation,” arXiv preprint arXiv:2109.06835, 2021

arXiv 2021

[5] [5]

Language models are few-shot learners,

T. B. Brown, “Language models are few-shot learners,” arXiv preprint arXiv:2005.14165, 2020

arXiv 2005

[6] [6]

Language models are unsupervised multitask learners,

A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al., “Language models are unsupervised multitask learners,” OpenAI blog, vol. 1, no. 8, p. 9, 2019

2019

[7] [7]

C. Qin, A. Zhang, Z. Zhang, J. Chen, M. Yasunaga, and D. Yang, Is chatgpt a general-purpose natural language processing task solver?

work page

[8] [8]

T ¨ornberg, Chatgpt-4 outperforms experts and crowd workers in annotating political twitter messages with zero-shot learning , 2023

P. T ¨ornberg, Chatgpt-4 outperforms experts and crowd workers in annotating political twitter messages with zero-shot learning , 2023. arXiv: 2304.06588 [cs.CL]

arXiv 2023