REVIEW 6 cited by
Smaller, Weaker, Yet Better: Training LLM Reasoners via Compute-Optimal Sampling
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Training on high-quality synthetic data from strong language models (LMs) is a common strategy to improve the reasoning performance of LMs. In this work, we revisit whether this strategy is compute-optimal under a fixed inference budget (e.g., FLOPs). To do so, we investigate the trade-offs between generating synthetic data using a stronger but more expensive (SE) model versus a weaker but cheaper (WC) model. We evaluate the generated data across three key metrics: coverage, diversity, and false positive rate, and show that the data from WC models may have higher coverage and diversity, but also exhibit higher false positive rates. We then finetune LMs on data from SE and WC models in different settings: knowledge distillation, self-improvement, and a novel weak-to-strong improvement setup where a weaker LM teaches reasoning to a stronger LM. Our findings reveal that models finetuned on WC-generated data consistently outperform those trained on SE-generated data across multiple benchmarks and multiple choices of WC and SE models. These results challenge the prevailing practice of relying on SE models for synthetic data generation, suggesting that WC may be the compute-optimal approach for training advanced LM reasoners.
Forward citations
Cited by 6 Pith papers
-
A Task-Centric Theory for Iterative Self-Improvement with Easy-to-Hard Curricula
Iterative self-improvement provably keeps improving only when initial performance lies in a moderate difficulty interval, and easy-to-hard curricula beat fixed mixtures under moderate difficulty separation and suffici...
-
What Matters in LLM-generated Data: Diversity and Its Effect on Model Fine-Tuning
Moderately diverse LLM-generated data can improve fine-tuned model performance in low-data settings when distribution shift is minimal, while high diversity or large distribution shift hurts.
-
Using Scaling Laws for Data Source Utility Estimation in Domain-Specific Pre-Training
Running multiple short annealing runs at different token scales can reveal per-source utility scaling curves that change data-source rankings compared with single point estimates.
-
Beyond Explainable AI (XAI): An Overdue Paradigm Shift and Post-XAI Research Directions
Current XAI methods for DNNs and LLMs rest on paradoxes and false assumptions that demand a paradigm shift to verification protocols, scientific foundations, context-aware design, and faithful model analysis rather th...
-
Enhancing Reasoning Capabilities in SLMs with Reward Guided Dataset Distillation
AdvDistill uses group relative advantages computed from rule-based rewards to weight teacher responses during distillation, reportedly improving a 1.5B student on math tasks beyond its 7B teacher.
-
A Survey on Large Language Models for Mathematical Reasoning
Recent advances in LLM mathematical reasoning are organized into comprehension and generation phases, covering methods from prompting to test-time scaling.
Discussion (0). Sign in to comment.