Pith. sign in

REVIEW 6 cited by

The Parrot Dilemma: Human-Labeled vs. LLM-augmented Data in Classification Tasks

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2304.13861 v2 pith:UFDBOHJB submitted 2023-04-26 cs.CL cs.CYphysics.soc-ph

Anders Giovanni M{o}ller , Jacob Aarup Dalsgaard , Arianna Pera , Luca Maria Aiello This is my paper · ORCID
classification cs.CLcs.CYphysics.soc-ph
keywords dataperformanceclassificationhuman-labeledtaskschallengescomparedgpt-4
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

In the realm of Computational Social Science (CSS), practitioners often navigate complex, low-resource domains and face the costly and time-intensive challenges of acquiring and annotating data. We aim to establish a set of guidelines to address such challenges, comparing the use of human-labeled data with synthetically generated data from GPT-4 and Llama-2 in ten distinct CSS classification tasks of varying complexity. Additionally, we examine the impact of training data sizes on performance. Our findings reveal that models trained on human-labeled data consistently exhibit superior or comparable performance compared to their synthetically augmented counterparts. Nevertheless, synthetic augmentation proves beneficial, particularly in improving performance on rare classes within multi-class tasks. Furthermore, we leverage GPT-4 and Llama-2 for zero-shot classification and find that, while they generally display strong performance, they often fall short when compared to specialized classifiers trained on moderately sized training sets.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. OccScene: Semantic Occupancy-based Cross-task Mutual Learning for 3D Scene Generation

    cs.CV 2024-12 conditional novelty 7.0 of 10

    A joint diffusion framework trains a Stable Diffusion generator and a semantic occupancy perception model together, so each task improves the other, producing text-conditional RGB-occupancy pairs.

  2. Backtranslation and paraphrasing in the LLM era? Comparing data augmentation methods for emotion classification

    cs.CL 2025-07 conditional novelty 6.0 of 10

    Backtranslation and paraphrasing produce competitive or better classification gains than zero-shot and few-shot generation when augmenting a low-resource emotion dataset.

  3. Automatic Labelling with Open-source LLMs using Dynamic Label Schema Integration

    cs.CL 2025-01 conditional novelty 6.0 of 10

    RAC ranks label descriptions by semantic similarity, then runs iterative binary LLM checks one label at a time, improving zero-shot labelling accuracy and enabling a precision-coverage trade-off.

  4. Scaling Up Occupancy-centric Driving Scene Generation: Dataset and Method

    cs.CV 2025-10 conditional novelty 5.0 of 10

    UniScenev2 scales occupancy-centric driving-scene generation to NuPlan scale, releasing a 3.6M-frame semantic-occupancy dataset and jointly generating occupancy, video, and LiDAR that beats published baselines on its ...

  5. Automated Extraction of Acronym-Expansion Pairs from Scientific Papers

    cs.CL 2024-12 conditional novelty 5.0 of 10

    A regex-plus-GPT-4 pipeline extracts acronym-expansion pairs from scientific papers more fully than either component alone, though precision is not measured.

  6. Know Your RAG: Dataset Taxonomy and Generation Strategies for Evaluating RAG Systems

    cs.IR 2024-11 conditional novelty 5.0 of 10

    A label taxonomy and answer-first generation strategies help RAG developers build evaluation datasets whose question mix matches real usage.

Pith tools