REVIEW 6 cited by
LLMaAA: Making Large Language Models as Active Annotators
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Prevalent supervised learning methods in natural language processing (NLP) are notoriously data-hungry, which demand large amounts of high-quality annotated data. In practice, acquiring such data is a costly endeavor. Recently, the superior few-shot performance of large language models (LLMs) has propelled the development of dataset generation, where the training data are solely synthesized from LLMs. However, such an approach usually suffers from low-quality issues, and requires orders of magnitude more labeled data to achieve satisfactory performance. To fully exploit the potential of LLMs and make use of massive unlabeled data, we propose LLMaAA, which takes LLMs as annotators and puts them into an active learning loop to determine what to annotate efficiently. To learn robustly with pseudo labels, we optimize both the annotation and training processes: (1) we draw k-NN examples from a small demonstration pool as in-context examples, and (2) we adopt the example reweighting technique to assign training samples with learnable weights. Compared with previous approaches, LLMaAA features both efficiency and reliability. We conduct experiments and analysis on two classic NLP tasks, named entity recognition and relation extraction. With LLMaAA, task-specific models trained from LLM-generated labels can outperform the teacher within only hundreds of annotated examples, which is much more cost-effective than other baselines.
Forward citations
Cited by 6 Pith papers
-
A Patient Simulation Framework for Risk Assessment of Conversational Healthcare AI: Evaluation of an Antidepressant Decision Aid
A patient simulator integrating medical, linguistic, and behavioral profiles exposes a monotonic performance decline in an antidepressant decision aid as simulated health literacy decreases.
-
PuzzleClone: A DSL-Powered Framework for Synthesizing Verifiable Data
A DSL plus SMT solver generates and validates 83,657 logic puzzles, and fine-tuning on them improves a 7B model's scores on several reasoning benchmarks.
-
Evaluating Large Language Models as Expert Annotators
Material Fingerprinting recovers the form and parameters of hyperelastic material models by nearest-neighbor matching of test data against a simulated fingerprint database: exact at zero noise, degrading under 5% noise.
-
DE-NER : Zero-shot Named Entity Recognition via Dialogue Elicitation of Large Language Models
A self-play dialogue framework with a self-trained questioner improves zero-shot named entity recognition over basic prompting, but not consistently over the strongest existing methods.
-
Active Domain Knowledge Acquisition with 100-Dollar Budget: Enhancing LLMs via Cost-Efficient, Expert-Involved Interaction in Sensitive Domains
A budget-aware framework (PU-ADKA) selects which domain expert an LLM should query under a fixed $100 budget, improving specialized-domain answers at low cost.
-
ACD-CLIP: Decoupling Representation and Dynamic Fusion for Zero-Shot Anomaly Detection
ACD-CLIP improves zero-shot anomaly detection by co-designing a convolutional low-rank adapter with a dynamic fusion gateway that modulates text prompts from visual context.
Discussion (0). Continue with ORCID to comment.