REVIEW 11 cited by
Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
We introduce Generalized Instruction Tuning (called GLAN), a general and scalable method for instruction tuning of Large Language Models (LLMs). Unlike prior work that relies on seed examples or existing datasets to construct instruction tuning data, GLAN exclusively utilizes a pre-curated taxonomy of human knowledge and capabilities as input and generates large-scale synthetic instruction data across all disciplines. Specifically, inspired by the systematic structure in human education system, we build the taxonomy by decomposing human knowledge and capabilities to various fields, sub-fields and ultimately, distinct disciplines semi-automatically, facilitated by LLMs. Subsequently, we generate a comprehensive list of subjects for every discipline and proceed to design a syllabus tailored to each subject, again utilizing LLMs. With the fine-grained key concepts detailed in every class session of the syllabus, we are able to generate diverse instructions with a broad coverage across the entire spectrum of human knowledge and skills. Extensive experiments on large language models (e.g., Mistral) demonstrate that GLAN excels in multiple dimensions from mathematical reasoning, coding, academic exams, logical reasoning to general instruction following without using task-specific training data of these tasks. In addition, GLAN allows for easy customization and new fields or skills can be added by simply incorporating a new node into our taxonomy.
Forward citations
Cited by 11 Pith papers
-
Towards Efficient and Effective Alignment of Large Language Models
A thesis presenting Lion, WebR, LTE, BMC, and FollowBench, five empirical methods that together address LLM alignment data, training, and evaluation.
-
HSS-Synth: Humanities and Social Sciences Data Synthesis for LLMs
HSS-Synth generates 230k instruction-tuning samples for 14 humanities/social-science fields and reports state-of-the-art fine-tuning results on 16 benchmarks.
-
Search Hardness-Aware LLM-Based Problem Formulation for Expensive Simulation-Driven Design
SHA-PF uses initial simulation data to select a 'hard but promising' anchor satisfaction state and evolves LLM-generated formulations that prioritize it, reaching target designs with fewer expensive simulations on ant...
-
NexForge: Scaling Agent Capabilities through Requirement-Driven Task Synthesis for LLMs
Requirement-driven synthesis generates executable agent training tasks and trajectories from a capability description, improving a 35B model from 22.5% to 52.0% on Terminal-Bench 2.0 and from 813 to 1338 GDPval Elo.
-
GLAN-QnA-KR: A Seedless Taxonomy-Driven Korean Instruction Corpus
A 303,581-row Korean instruction corpus generated seedlessly from a 1,084-discipline taxonomy, with near-zero duplicates and low measured overlap with KMMLU, KoBEST, and HAE-RAE-Bench.
-
ChemActor: Enhancing Automated Extraction of Chemical Synthesis Actions with LLM-Generated Data
A fine-tuned LLaMA-2-7B model trained with selected LLM-generated data improves extraction of chemical synthesis actions from experimental text.
-
A Survey of LLM $\times$ DATA
A comprehensive survey of the bidirectional links between LLMs and data management, organized as DATA4LLM and LLM4DATA with a new 'IaaS' data-quality framework.
-
From Tens of Hours to Tens of Thousands: Scaling Back-Translation for Speech Recognition
Fine-tuning TTS models on tens of hours of real audio enables generation of 500,000 hours of synthetic speech that reduces ASR error rates by over 30% on Whisper-large-v3.
-
LongMagpie: A Self-synthesis Method for Generating Large-scale Long-context Instructions
By prompting an aligned LLM with a document and the special token that precedes a user query, LongMagpie synthesizes long-context instruction data that outperforms prior datasets when used to fine-tune Llama-3-8B.
-
Two Experts Are All You Need for Steering Thinking: Reinforcing Cognitive Effort in MoE Reasoning Models Without Additional Training
Reinforcing the two experts most correlated with thinking tokens improves reasoning accuracy and efficiency in MoE large reasoning models, with gains of up to 10 points on AIME benchmarks.
-
A Penalty Goes a Long Way: Measuring Lexical Diversity in Synthetic Texts Under Prompt-Influenced Length Variations
PATTR adds a target-length penalty to the Type-Token Ratio, producing a lexical diversity score with tunable, reduced short-text bias for LLM synthetic data.
Discussion (0). Continue with ORCID to comment.