REVIEW 7 cited by
GenQA: Generating Millions of Instructions from a Handful of Prompts
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Most public instruction finetuning datasets are relatively small compared to the closed source datasets used to train industry models. To study questions about finetuning at scale, such as curricula and learning rate cooldown schedules, there is a need for industrial-scale datasets. However, this scale necessitates a data generation process that is almost entirely automated. In this work, we study methods for generating large instruction datasets from a single prompt. With little human oversight, we get LLMs to write diverse sets of instruction examples ranging from simple completion tasks to complex multi-turn dialogs across a variety of subject areas. When finetuning a Llama-3 8B base model, our dataset meets or exceeds both WizardLM and Ultrachat on both knowledge-intensive leaderboard tasks as well as conversational evaluations. We release our dataset, the "generator" prompts that created it, and our finetuned model checkpoints.
Forward citations
Cited by 7 Pith papers
-
Zero-Shot Vision Encoder Grafting via LLM Surrogates
Training a vision encoder against a small surrogate made from a target LLM's early layers lets the encoder be grafted into the full LLM with no fine-tuning, matching some full-training results at roughly half the cost.
-
Less is Enough: Synthesizing Diverse Data in LLM Feature Space with Sparse Autoencoders
Coverage of sparse-autoencoder-identified task features predicts post-training performance and can guide synthesis of small, high-impact datasets (2,000 vs. 300,000 samples).
-
One Prompt, Many Sounds: Modeling Listener Variability in LLM-Based Equalization
LLMs using in-context learning and fine-tuning on listener experiment data generate equalization settings that align better with population preferences than random sampling or static presets.
-
FarSkip-Collective: Unhobbling Blocking Communication in Mixture of Experts Models
FarSkip-Collective makes Mixture-of-Experts communication non-blocking by feeding later layers already-available activations, and self-distillation keeps converted 16B–109B models within ~1–2.5% of the original on average.
-
RAD: Redundancy-Aware Distillation for Hybrid Models via Self-Speculative Decoding
RAD selects redundant attention layers via self-speculative decoding throughput, replaces them with SSM blocks, and distills them into hybrid models that beat the original on GSM8K and CRUX.
-
Adaptive Depth Sparse Framework: Similarity-Driven Resource Allocation for Pre-Trained LLMs
AdaDSF uses per-layer cosine similarity to decide which tokens skip which layers, then distills the sparse model back toward the dense one, cutting FLOPs while holding accuracy close to dense.
-
Improved Supervised Fine-Tuning for Large Language Models to Mitigate Catastrophic Forgetting
A self-generated and committee-filtered synthetic rehearsal dataset lets a fine-tuned LLM nearly preserve its general benchmark scores while adding domain data.
Discussion (0). Continue with ORCID to comment.