REVIEW 8 cited by
Can ChatGPT Reproduce Human-Generated Labels? A Study of Social Computing Tasks
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The release of ChatGPT has uncovered a range of possibilities whereby large language models (LLMs) can substitute human intelligence. In this paper, we seek to understand whether ChatGPT has the potential to reproduce human-generated label annotations in social computing tasks. Such an achievement could significantly reduce the cost and complexity of social computing research. As such, we use ChatGPT to relabel five seminal datasets covering stance detection (2x), sentiment analysis, hate speech, and bot detection. Our results highlight that ChatGPT does have the potential to handle these data annotation tasks, although a number of challenges remain. ChatGPT obtains an average accuracy 0.609. Performance is highest for the sentiment analysis dataset, with ChatGPT correctly annotating 64.9% of tweets. Yet, we show that performance varies substantially across individual labels. We believe this work can open up new lines of analysis and act as a basis for future research into the exploitation of ChatGPT for human annotation tasks.
Forward citations
Cited by 8 Pith papers
-
Evaluating Large Language Models as Expert Annotators
Material Fingerprinting recovers the form and parameters of hyperelastic material models by nearest-neighbor matching of test data against a simulated fingerprint database: exact at zero noise, degrading under 5% noise.
-
Calibrating Pre-trained Language Classifiers on LLM-generated Noisy Labels via Iterative Refinement
SiDyP improves classifiers trained on LLM-generated noisy labels by retrieving likely true labels from embedding-space neighbors and iteratively refining them with a simplex diffusion model, reporting average gains of...
-
Deconstructing Depression Stigma: Integrating AI-driven Data Collection and Analysis with Causal Knowledge Graphs
A chatbot interview pipeline with AI-assisted coding and causal knowledge graphs can elicit and decompose depression stigma at scale.
-
Specializing General-purpose LLM Embeddings for Implicit Hate Speech Detection across Datasets
Fine-tuning large general-purpose text embeddings with a simple instruction yields state-of-the-art implicit hate speech detection, with up to 20.35 point cross-dataset F1 gains.
-
ACD-CLIP: Decoupling Representation and Dynamic Fusion for Zero-Shot Anomaly Detection
ACD-CLIP improves zero-shot anomaly detection by co-designing a convolutional low-rank adapter with a dynamic fusion gateway that modulates text prompts from visual context.
-
Conversation Kernels: A Flexible Mechanism to Learn Relevant Context for Online Conversation Understanding
Conversation Kernels retrieve small tree-neighborhood windows around a post and feed them to RoBERTa, improving Slashdot comment label prediction over standard text-only baselines.
-
Neither Valid nor Reliable? Investigating the Use of LLMs as Judges
An argument, grounded in social-science measurement theory, that LLM-as-judge adoption has outpaced validity and reliability testing, with an analysis of four underlying assumptions.
-
Scaling Public Health Text Annotation: Zero-Shot Learning vs. Crowdsourcing for Improved Efficiency and Labeling Accuracy
On a 12,000-tweet public health dataset, zero-shot GPT-4 Turbo matched overall crowdworker accuracy but missed many positive cases, especially sleep-related tweets, while being far faster and cheaper.
Discussion (0). Continue with ORCID to comment.