REVIEW 4 cited by
Are Expert-Level Language Models Expert-Level Annotators?
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Data annotation refers to the labeling or tagging of textual data with relevant information. A large body of works have reported positive results on leveraging LLMs as an alternative to human annotators. However, existing studies focus on classic NLP tasks, and the extent to which LLMs as data annotators perform in domains requiring expert knowledge remains underexplored. In this work, we investigate comprehensive approaches across three highly specialized domains and discuss practical suggestions from a cost-effectiveness perspective. To the best of our knowledge, we present the first systematic evaluation of LLMs as expert-level data annotators.
Forward citations
Cited by 4 Pith papers
-
Evaluating Large Language Models as Expert Annotators
Material Fingerprinting recovers the form and parameters of hyperelastic material models by nearest-neighbor matching of test data against a simulated fingerprint database: exact at zero noise, degrading under 5% noise.
-
Exploring the Potential of LLMs for Serendipity Evaluation in Recommender Systems
Basic and multi-model LLM prompts can evaluate recommendation serendipity as well as or better than standard proxy formulas, reaching 21.5% Pearson correlation with user-study ratings.
-
How Significant Are the Real Performance Gains? An Unbiased Evaluation Framework for GraphRAG
A new GraphRAG evaluation framework using graph-grounded questions and bias-correction yields much smaller win rates than earlier reports, casting doubt on reported GraphRAG gains.
-
ACD-CLIP: Decoupling Representation and Dynamic Fusion for Zero-Shot Anomaly Detection
ACD-CLIP improves zero-shot anomaly detection by co-designing a convolutional low-rank adapter with a dynamic fusion gateway that modulates text prompts from visual context.
Discussion (0). Continue with ORCID to comment.