REVIEW 10 cited by
SlimPajama-DC: Understanding Data Combinations for LLM Training
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
This paper aims to understand the impacts of various data combinations (e.g., web text, Wikipedia, GitHub, books) on the pretraining of large language models using SlimPajama. SlimPajama is a rigorously deduplicated, multi-source dataset, which has been refined and further deduplicated to 627B tokens from the extensive 1.2T token RedPajama dataset contributed by Together. We have termed our research as SlimPajama-DC, an empirical analysis designed to uncover fundamental characteristics and best practices associated with employing SlimPajama in the training of large language models. During our research with SlimPajama, two pivotal observations emerged: (1) Global deduplication vs. local deduplication. We analyze and discuss how global (across different sources of datasets) and local (within the single source of dataset) deduplications affect the performance of trained models. (2) Proportions of highly-deduplicated multi-source datasets in the combination. To study this, we construct six configurations on SlimPajama dataset and train individual ones using 1.3B Cerebras-GPT model with Alibi and SwiGLU. Our best configuration outperforms the 1.3B model trained on RedPajama using the same number of training tokens by a significant margin. All our 1.3B models are trained on Cerebras 16$\times$ CS-2 cluster with a total of 80 PFLOP/s in bf16 mixed precision. We further extend our discoveries (such as increasing data diversity is crucial after global deduplication) on a 7B model with large batch-size training. Our SlimPajama-DC models are available at: https://huggingface.co/MBZUAI-LLM/SlimPajama-DC and the separate SlimPajama-DC datasets are available at: https://huggingface.co/datasets/MBZUAI-LLM/SlimPajama-627B-DC.
Forward citations
Cited by 10 Pith papers
-
Do LLMs Know Their Vulnerable Scenarios?
Scenario jailbreaks suppress refusal via internal concept directions; Concept2Scenario attributes those concepts with SAEs and turns them into transferable natural-language attack scenarios.
-
CuraWeb: Joint Optimization of Quality, Redundancy, and Diversity for Web-Scale Pretraining Data
CuraWeb jointly optimizes cleaning, deduplication, and sampling to build a 2T-token pretraining corpus that outperforms FineWeb-Edu, DCLM, Nemotron-CC, and Dolma3 by 1.82% average at 200B tokens.
-
Fix the Structural Bottleneck: Context Compression via Explicit Information Transmission
Soft context compression via explicit depth/width information transmission over frozen LLM hidden states beats prior gist-token compressors on six QA benchmarks.
-
Essential-Web v1.0: 24T tokens of organized web data
A 24T-token web corpus with automatic document-level taxonomy labels enables competitive domain-specific datasets via simple filters.
-
Chameleon: A Flexible Data-mixing Framework for Language Model Pretraining and Finetuning
Chameleon uses kernel ridge leverage scores on domain embeddings to set LLM training-mixture weights, matching DoGE-level pretraining quality at roughly one fifth the compute and improving finetuning perplexity.
-
Merge to Mix: Mixing Datasets via Model Merging
Merge to Mix shows that the performance of a parameter-averaged model predicts the performance of a model fine-tuned on any dataset mixture, enabling fast and accurate dataset mixture selection.
-
HuggingGraph: Understanding the Supply Chain of LLM Ecosystem
A directed heterogeneous graph of 402,654 Hugging Face models and datasets is constructed and analyzed to reveal supply-chain dependencies and structural patterns such as a connected core and heavy-tailed reuse.
-
Spectra 1.1: Scaling Laws and Efficient Inference for Ternary Language Models
Ternary language models trained on 1.2 trillion tokens continue to improve, and a new GPU kernel speeds up their inference up to 5x end-to-end.
-
On-the-Fly Adaptive Distillation of Transformer to Dual-State Linear Attention
On-the-fly distillation of Transformer layers to dual-state linear attention produces about 2.3x faster simulated LLM serving than Llama2-7B with roughly comparable benchmark accuracy.
-
Rethinking Data Mixture for Large Language Models: A Comprehensive Survey and New Perspectives
A survey organizing LLM data mixture methods into offline and online families, with a fine-grained taxonomy based on optimization frameworks.
Discussion (0). Continue with ORCID to comment.