Pith. sign in

REVIEW 8 cited by

AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotators

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2303.16854 v2 pith:SIX2PNGL submitted 2023-03-29 cs.CL

Xingwei He , Zhenghao Lin , Yeyun Gong , A-Long Jin , Hang Zhang , Chen Lin , Jian Jiao , Siu Ming Yiu
show 2 more authors
Nan Duan Weizhu Chen
This is my paper · ORCID
classification cs.CL
keywords modelsannollmdatallmscrowdsourceddatasetfirstlanguage
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Many natural language processing (NLP) tasks rely on labeled data to train machine learning models with high performance. However, data annotation is time-consuming and expensive, especially when the task involves a large amount of data or requires specialized domains. Recently, GPT-3.5 series models have demonstrated remarkable few-shot and zero-shot ability across various NLP tasks. In this paper, we first claim that large language models (LLMs), such as GPT-3.5, can serve as an excellent crowdsourced annotator when provided with sufficient guidance and demonstrated examples. Accordingly, we propose AnnoLLM, an annotation system powered by LLMs, which adopts a two-step approach, explain-then-annotate. Concretely, we first prompt LLMs to provide explanations for why the specific ground truth answer/label was assigned for a given example. Then, we construct the few-shot chain-of-thought prompt with the self-generated explanation and employ it to annotate the unlabeled data with LLMs. Our experiment results on three tasks, including user input and keyword relevance assessment, BoolQ, and WiC, demonstrate that AnnoLLM surpasses or performs on par with crowdsourced annotators. Furthermore, we build the first conversation-based information retrieval dataset employing AnnoLLM. This dataset is designed to facilitate the development of retrieval models capable of retrieving pertinent documents for conversational text. Human evaluation has validated the dataset's high quality.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. OpenAlex reports about 34 citations worldwide. Full citation record

  1. SERUM: State Extraction and Refinement for User Modeling

    cs.LG 2026-07 conditional novelty 6.0 of 10

    A multi-pass VLM annotation pipeline extracts action and intent state machines from egocentric video, then fits Markov models that improve on frequency baselines by a few points after normalization (activity top-1 48....

  2. Evaluating Large Language Models as Expert Annotators

    cs.CL 2025-08 unverdicted novelty 6.0 of 10

    Material Fingerprinting recovers the form and parameters of hyperelastic material models by nearest-neighbor matching of test data against a simulated fingerprint database: exact at zero noise, degrading under 5% noise.

  3. ACD-CLIP: Decoupling Representation and Dynamic Fusion for Zero-Shot Anomaly Detection

    cs.CV 2025-08 unverdicted novelty 5.0 of 10

    ACD-CLIP improves zero-shot anomaly detection by co-designing a convolutional low-rank adapter with a dynamic fusion gateway that modulates text prompts from visual context.

  4. MetaWriter: Personalized Handwritten Text Recognition Using Meta-Learned Prompt Tuning

    cs.CV 2025-05 conditional novelty 5.0 of 10

    MetaWriter uses meta-learned prompt tuning with an image-reconstruction auxiliary task to adapt a handwritten text recognizer to new writers from unlabeled examples, reporting lower error rates on IAM and RIMES with l...

  5. Large language model as user daily behavior data generator: balancing population diversity and individual personality

    cs.LG 2025-05 conditional novelty 5.0 of 10

    LLM-generated synthetic behavior data improves mobility and smartphone-use prediction models by up to 18.9% and captures roughly 60 to 88% of the gains from real-data fine-tuning.

  6. Are Large Language Models the future crowd workers of Linguistics?

    cs.CL 2025-02 conditional novelty 5.0 of 10

    In two replicated linguistics experiments, GPT-4o-mini's zero-shot responses matched or beat published human performance, but the study lacks statistical validation and relies on only two tasks.

  7. Few-shot LLM Synthetic Data with Distribution Matching

    cs.CL 2025-02 conditional novelty 5.0 of 10

    SynAlign generates LLM synthetic text from diversity-guided demonstrations, then reweights it by MMD-based distribution matching, improving downstream classification accuracy.

  8. Reliable Annotations with Less Effort: Evaluating LLM-Human Collaboration in Search Clarifications

    cs.IR 2025-07 reject novelty 4.0 of 10

    LLMs alone annotate search clarifications unreliably; adding confidence-based selective human review cuts effort 24-45% in simulation, but the evaluation is partly built from the ground truth it predicts.

Pith tools