REVIEW 11 cited by
RedPajama: an Open Dataset for Training Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large language models are increasingly becoming a cornerstone technology in artificial intelligence, the sciences, and society as a whole, yet the optimal strategies for dataset composition and filtering remain largely elusive. Many of the top-performing models lack transparency in their dataset curation and model development processes, posing an obstacle to the development of fully open language models. In this paper, we identify three core data-related challenges that must be addressed to advance open-source language models. These include (1) transparency in model development, including the data curation process, (2) access to large quantities of high-quality data, and (3) availability of artifacts and metadata for dataset curation and analysis. To address these challenges, we release RedPajama-V1, an open reproduction of the LLaMA training dataset. In addition, we release RedPajama-V2, a massive web-only dataset consisting of raw, unfiltered text data together with quality signals and metadata. Together, the RedPajama datasets comprise over 100 trillion tokens spanning multiple domains and with their quality signals facilitate the filtering of data, aiming to inspire the development of numerous new datasets. To date, these datasets have already been used in the training of strong language models used in production, such as Snowflake Arctic, Salesforce's XGen and AI2's OLMo. To provide insight into the quality of RedPajama, we present a series of analyses and ablation studies with decoder-only language models with up to 1.6B parameters. Our findings demonstrate how quality signals for web data can be effectively leveraged to curate high-quality subsets of the dataset, underscoring the potential of RedPajama to advance the development of transparent and high-performing language models at scale.
Forward citations
Cited by 11 Pith papers
-
A First-Principles Theory of Slow Thinking and Active Perception
Active lifting of data distributions via latent-sequence sampling and max-rate uncertainty reduction formally derives slow-thinking LLMs and places them on representation and sampler hierarchies that can be climbed.
-
Small edits, large models: How Wikipedia advocacy shapes LLM values
Wikipedia edits by animal welfare advocates measurably influence LLM outputs on animal welfare topics, shown via retrieval and gradient attribution plus fine-tuning experiments.
-
OLMoASR: Open Models and Data for Training Robust Speech Recognition Models
An open 1M-hour English speech dataset plus Whisper-architecture models trained on it match Whisper's word error rates on short and long-form benchmarks.
-
Libra: Taming Attention Workload Skew in Long-Context LLM Training with Bounded Sequence Pool
Libra bounds attention load-balancing to fixed-size sequence pools and combines variance-reduced sequence placement with tiled attention pooling, achieving up to 2.54x end-to-end throughput over Ulysses on 256K- and 1...
-
The Effect of Scripts and Formats on LLM Numeracy
LLM arithmetic accuracy falls sharply when numerals leave the familiar Hindu–Arabic format, and few-shot prompting with examples narrows most of that gap.
-
Essential-Web v1.0: 24T tokens of organized web data
A 24T-token web corpus with automatic document-level taxonomy labels enables competitive domain-specific datasets via simple filters.
-
Domain2Vec: Vectorizing Datasets to Find the Optimal Data Mixture without Training
Domain2Vec finds better LLM pretraining data mixtures by aligning, in a training-free way, the meta-domain distribution of the training set with the validation set's distribution.
-
The Common Pile v0.1: An 8TB Dataset of Public Domain and Openly Licensed Text
A new 8TB openly-licensed text corpus trains 7B LLMs that are competitive with Llama 1/2, showing that performant models need not depend on unlicensed web data.
-
A Survey on Evaluating Quality and Trustworthiness in LLM-Generated Data
A metric-oriented survey that classifies intrinsic quality and trustworthiness metrics for LLM-generated data across six modalities and documents systematic evaluation gaps in the current literature.
-
Router Upcycling: Leveraging Mixture-of-Routers in Mixture-of-Experts Upcycling
Router Upcycling initializes MoE routers from the dense model's attention heads, achieving about 2% higher average zero-shot accuracy than vanilla upcycling on small Qwen 8x0.5B models.
-
From Data to Device: ELMOD An Efficient German-First 2.7B Language Model for Mobile Inference
A 2.7B German-first LLM trained cheaply on public data with language-specific quality filtering matches larger 7B models on German reasoning benchmarks and runs on-device.
Discussion (0). Sign in to comment.