REVIEW 27 cited by
Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
Fine-tuning language models~(LMs) on human-generated data remains a prevalent practice. However, the performance of such models is often limited by the quantity and diversity of high-quality human data. In this paper, we explore whether we can go beyond human data on tasks where we have access to scalar feedback, for example, on math problems where one can verify correctness. To do so, we investigate a simple self-training method based on expectation-maximization, which we call ReST$^{EM}$, where we (1) generate samples from the model and filter them using binary feedback, (2) fine-tune the model on these samples, and (3) repeat this process a few times. Testing on advanced MATH reasoning and APPS coding benchmarks using PaLM-2 models, we find that ReST$^{EM}$ scales favorably with model size and significantly surpasses fine-tuning only on human data. Overall, our findings suggest self-training with feedback can substantially reduce dependence on human-generated data.
Forward citations
Cited by 27 Pith papers
-
Bridging Compute- and Data-Optimal Pretraining
Pretraining loss obeys a single law in which repeated or paraphrased tokens count as η(N, data-per-parameter, expansion-ratio) fresh tokens, with total effective data saturating as derived tokens grow.
-
Turbo Connection: Reasoning as Information Flow from Higher to Lower Layers
An architecture that routes hidden states from higher to lower layers between consecutive tokens improves reasoning accuracy and length generalization in fine-tuned LLMs.
-
The Verifier is the Curriculum: Execution-Gated Self-Distillation for Cross-Family Game Generation
Strict-launch-gated rejection-sampling self-distillation compounds cross-family clean Godot generation from 8.8% to 42.2% and full best-of-K coverage, while gold duplication and a lenient BUILD filter erase the gain.
-
Recursive Synthesis for Long-Horizon Terminal Tasks
A recursive verified synthesis pipeline generated 37,484 progressively harder terminal-agent tasks and used them to improve agent performance on three benchmarks.
-
From Scoring to Acting: Outcome-Verified Comparative Self-Distillation for LLM Agents
Outcome-verified teacher continuations from student failure prefixes, plus divergence-local comparison and suffix distillation, raise skill-free agent success over scoring-based self-distillation.
-
Off-Context GRPO: Learning to Reason on Hard Problems using Privileged Information
OC-GRPO reweights GRPO gradients with an importance ratio so that hints used during rollout generation still optimize the original unguided objective, delivering a 13.8% relative Pass@1 gain over vanilla GRPO.
-
Answer-Conditioned Chains of Thought Degrade Verifiable-Reasoning Distillation in Large Language Models
Answer-conditioned chains of thought are harmful supervision for reasoning distillation—causing up to ~27-point accuracy drops—and the damage is invisible to correctness filters.
-
Post-Training Shifts Confidence: A Three-Stage Analysis of How SFT, RL, and OPD Shape CoT Calibration
Post-training changes where confidence is trustworthy in reasoning traces: OPD best gauges difficulty before thinking, SFT best supports early stopping, RL best selects final answers, and confidence should be read onl...
-
UNIBROWSE: A Data-to-Agent Framework for Multimodal BrowseComp
A unified KG-plus-live-web data pipeline covering all three multimodal BrowseComp information-flow patterns, plus an exploration-degree filter, yields a 35B agent at 54.4 avg accuracy.
-
Multi-Turn On-Policy Distillation with Prefix Replay
ReOPD offline-distills multi-turn agentic LLMs via teacher-prefix replay plus step-decay sampling, matching online OPD accuracy at ≥4× speed with zero tool calls.
-
FormulaSPIN: Self-Play Fine-Tuning for Natural Language to Spreadsheet Formula Generation
FormulaSPIN applies execution-aware self-play with an adaptive curriculum to spreadsheet formula generation, improving exact match from 68.2% to 74.9% on NL2Formula-70K without additional data.
-
A Task-Centric Theory for Iterative Self-Improvement with Easy-to-Hard Curricula
Iterative self-improvement provably keeps improving only when initial performance lies in a moderate difficulty interval, and easy-to-hard curricula beat fixed mixtures under moderate difficulty separation and suffici...
-
rePIRL: Learn PRM with Inverse RL for LLM Reasoning
rePIRL learns token-level process rewards for LLM reasoning via a guided-cost-learning-style IRL objective, and shows these rewards improve reasoning policies on math/coding benchmarks.
-
Bridging the Capability Gap: Joint Alignment Tuning for Harmonizing LLM-based Multi-Agent Systems
MOAT alternately aligns a planning and a grounding LLM agent via perplexity-based DPO and self-generated SFT, reporting small but consistent gains over independently tuned baselines.
-
Learn from What We HAVE: History-Aware VErifier that Reasons about Past Interactions Online
A history-aware verifier that scores candidate actions using past interactions cuts failure rates in ambiguous robot manipulation tasks compared to using the generator alone.
-
Utilizing Training Data to Improve LLM Reasoning for Tabular Understanding
LRTab retrieves error-avoiding prompt conditions learned from incorrect chain-of-thought traces on training tables to improve LLM tabular reasoning, achieving modest gains on WikiTQ and TabFact.
-
SyncLoop: A Multimodal Dual-Loop Framework for Self-Improving Mathematical Reasoning
SyncLoop jointly evolves multimodal training data and model capability through alternating SFT and RL, selecting error-prone samples to improve geometry reasoning.
-
SPARQ: Synthetic Problem Generation for Reasoning via Quality-Diversity Algorithms
Filtering self-generated math problems by a model's own solve-rate improves that model's MATH accuracy from 38% to 47% and helps out-of-distribution generalization when data is diverse.
-
HardTests: Synthesizing High-Quality Test Cases for LLM Coding
HardTestGen generates higher-precision, higher-recall test suites for 47,136 competitive programming problems, improving test precision by 11.3 points and recall by 17.5 points over TACO and CodeContests when judging ...
-
ReST-RL: Achieving Accurate Code Reasoning of LLMs with Optimized Self-Training and Decoding
ReST-RL pairs reward-variance-filtered self-training (ReST-GRPO) with value-model-guided MCTS decoding (VM-MCTS) to improve LLM code reasoning.
-
Spectra 1.1: Scaling Laws and Efficient Inference for Ternary Language Models
Ternary language models trained on 1.2 trillion tokens continue to improve, and a new GPU kernel speeds up their inference up to 5x end-to-end.
-
Maximizing Confidence Alone Improves Reasoning
Reinforcement learning with a reward equal to the negative entropy of the model's own token distribution improves benchmark accuracy without any labeled answers.
-
SLearnLLM: A Self-Learning Framework for Efficient Domain-Specific Adaptation of Large Language Models
Fine-tuning only on questions a model answers incorrectly reaches nearly the same performance as full-data fine-tuning while substantially reducing training time.
-
Exploring the Limit of Outcome Reward for Learning Mathematical Reasoning
OREAL shows that outcome-reward RL with best-of-N positive behavior cloning, negative reward shaping, and token-level reweighting reaches state-of-the-art MATH-500 accuracy at 7B and 32B scale.
-
CoTu at EXACT 2026: Neuro-Symbolic Reasoning for Transparent Educational QA
A 4B LLM that generates solver-runnable code (Z3 for logic, Python for physics) achieved a perfect physics score and the highest technical score (13.44/15) at EXACT 2026, with premise selection as the main remaining e...
-
CIIR@LiveRAG 2025: Optimizing Multi-Agent Retrieval Augmented Generation through Self-Training
A multi-agent RAG framework trained by self-supervision on high-reward interaction trajectories outperforms a vanilla RAG baseline on DataMorgana-generated questions and places 7th in the LiveRAG 2025 competition.
-
Large Language Models for Planning: A Comprehensive and Systematic Survey
A structured survey of LLM planning methods, benchmarks, and interpretability work, organized around a three-way taxonomy.
Discussion (0). Continue with ORCID to comment.