REVIEW 18 cited by
Rho-1: Not All Tokens Are What You Need
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Previous language model pre-training methods have uniformly applied a next-token prediction loss to all training tokens. Challenging this norm, we posit that "9l training". Our initial analysis examines token-level training dynamics of language model, revealing distinct loss patterns for different tokens. Leveraging these insights, we introduce a new language model called Rho-1. Unlike traditional LMs that learn to predict every next token in a corpus, Rho-1 employs Selective Language Modeling (SLM), which selectively trains on useful tokens that aligned with the desired distribution. This approach involves scoring pretraining tokens using a reference model, and then training the language model with a focused loss on tokens with higher scores. When continual pretraining on 15B OpenWebMath corpus, Rho-1 yields an absolute improvement in few-shot accuracy of up to 30% in 9 math tasks. After fine-tuning, Rho-1-1B and 7B achieved state-of-the-art results of 40.6% and 51.8% on MATH dataset, respectively - matching DeepSeekMath with only 3% of the pretraining tokens. Furthermore, when continual pretraining on 80B general tokens, Rho-1 achieves 6.8% average enhancement across 15 diverse tasks, increasing both efficiency and performance of the language model pre-training.
Forward citations
Cited by 18 Pith papers
-
Bridging Compute- and Data-Optimal Pretraining
Pretraining loss obeys a single law in which repeated or paraphrased tokens count as η(N, data-per-parameter, expansion-ratio) fresh tokens, with total effective data saturating as derived tokens grow.
-
Understanding LoRA as Knowledge Memory: An Empirical Analysis
LoRA modules function as composable knowledge memories for LLMs with measurable storage capacity, internalization efficiency, and advantages in multi-module long-context reasoning.
-
GradAlign: Gradient-Aligned Data Selection for LLM Reinforcement Learning
GradAlign selects RL training problems whose policy gradients align with validation-set gradients, claiming more stable and better-performing LLM post-training.
-
Hardness-Aware Dynamic Curriculum Learning for Robust Multimodal Emotion Recognition with Missing Modalities
HARDY-MER is a retrieval-augmented curriculum training method that gives harder samples more retrieved support samples, improving missing-modality emotion recognition on average across IEMOCAP and CMU-MOSEI.
-
Language Models Improve When Pretraining Data Matches Target Tasks
Ranking pretraining documents by similarity to benchmark training examples (BETR) yields consistent benchmark gains and a 2.1x compute multiplier over DCLM-Baseline.
-
Demystifying Reasoning Dynamics with Mutual Information: Thinking Tokens are Information Peaks in LLM Reasoning
Reasoning tokens like 'Hmm' and 'Wait' mark steps where a model's internal state carries unusually high dependence with the correct answer, and suppressing them hurts accuracy.
-
Probability-Consistent Preference Optimization for Enhanced LLM Reasoning
PCPO selects preference pairs by combining correct-answer status with token-level probability consistency, then trains with a weighted DPO+NLL loss, yielding small and partly inconsistent gains over outcome-only metho...
-
One Example Shown, Many Concepts Known! Counterexample-Driven Conceptual Reasoning in Mathematical LLMs
LLMs score poorly on CounterMATH, a new counterexample-based university math benchmark, and a 1,025-sample counterexample fine-tune yields small and partly inconsistent gains.
-
DomainPilot: Domain-Level Loss-Guided Two-Stage Data Mixture Optimization for Efficient Language Model Fine-Tuning
Reallocating fine-tuning data proportions according to fitted per-domain loss curves improved benchmark scores by 1.8–3.8 points on Qwen3-1.7B, in a single unseeded run; the planned second stage is not yet implemented.
-
Not All Preferences are What You Need for Post-Training: Selective Alignment Strategy for Preference Optimization
Selective-DPO applies DPO only to the top 40% of tokens ranked by policy-reference log-probability difference, reporting improved benchmark scores but with tuned hyperparameters on test sets and omitted negative results.
-
RefineX: Learning to Refine Pre-training Data at Scale from Expert-Guided Programs
A deletion-only program refiner, trained on expert end-to-end edits converted via minimum edit distance, improves LLM pretraining data and downstream accuracy.
-
Capability Salience Vector: Fine-grained Alignment of Loss and Capabilities for Downstream Task Scaling Law
A fitted token-level loss weighting, optimized against known model accuracies, predicts held-out downstream task performance more accurately than mean validation loss on five of six benchmarks.
-
Not Every Token Needs Forgetting: Selective Unlearning to Limit Change in Utility in Large Language Model Unlearning
SU uses two assistant models trained on different data splits to score tokens, then unlearns only tokens whose scores diverge, claiming better retain-set utility with comparable forget quality.
-
ESLM: Risk-Averse Selective Language Modeling for Efficient Pretraining
ESLM keeps only high-loss or high-entropy tokens in each batch via a value-at-risk threshold, cutting pretraining FLOPs by about 6% while roughly matching perplexity and downstream accuracy.
-
Masked Diffusion Language Models with Frequency-Informed Training
Masked diffusion language models trained on 100M words match a hybrid GPT-BERT baseline on BabyLM tests, with a rare-word-focused masking variant.
-
Read Quietly, Think Aloud: Decoupling Comprehension and Reasoning in LLMs
Masking loss on early training tokens and injecting embeddings from an auxiliary reader improves LLM accuracy on several reasoning benchmarks.
-
SFT-GO: Supervised Fine-Tuning with Group Optimization for Large Language Models
SFT-GO retrains LLMs by focusing on the worst-performing group of important or unimportant tokens, yielding modest average benchmark improvements over standard supervised fine-tuning.
-
A Survey on Large Language Models for Mathematical Reasoning
Recent advances in LLM mathematical reasoning are organized into comprehension and generation phases, covering methods from prompting to test-time scaling.
Discussion (0). Continue with ORCID to comment.