HERMES provides a reusable hierarchical labeling substrate for pre-training data that reveals granularity-specific effects in data mixing rules during model training.
Unsupervised topic models are data mixers for pre-training language models
3 Pith papers cite this work. Polarity classification is still indexing.
3
Pith papers citing it
years
2026 3representative citing papers
HDS uses Soft Actor-Critic RL with a multi-objective reward (data quality, inter-domain loss influence, weight norms) for online data mixing in LLM pre-training, reaching target perplexity with 44% fewer iterations and 7.2% MMLU gain on The Pile.
A survey that taxonomizes data mixing strategies for LLM pretraining into static rule-based, learning-based, and dynamic adaptive families while highlighting transferability challenges and evaluation gaps.
citing papers explorer
-
HERMES: A Multi-Granularity Labeling Substrate for Pre-training Data Mixtures
HERMES provides a reusable hierarchical labeling substrate for pre-training data that reveals granularity-specific effects in data mixing rules during model training.
-
Holistic Data Scheduler for LLM Pre-training via Multi-Objective Reinforcement Learning
HDS uses Soft Actor-Critic RL with a multi-objective reward (data quality, inter-domain loss influence, weight norms) for online data mixing in LLM pre-training, reaching target perplexity with 44% fewer iterations and 7.2% MMLU gain on The Pile.
-
Data Mixing for Large Language Models Pretraining: A Survey and Outlook
A survey that taxonomizes data mixing strategies for LLM pretraining into static rule-based, learning-based, and dynamic adaptive families while highlighting transferability challenges and evaluation gaps.