REVIEW 7 cited by
The MiniPile Challenge for Data-Efficient Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
The ever-growing diversity of pre-training text corpora has equipped language models with generalization capabilities across various downstream tasks. However, such diverse datasets are often too large for academic budgets; hence, most research on Transformer architectures, training procedures, optimizers, etc. gets conducted on smaller, homogeneous datasets. To this end, we present The MiniPile Challenge, where one pre-trains a language model on a diverse text corpus containing at most 1M documents. MiniPile is a 6GB subset of the deduplicated 825GB The Pile corpus. To curate MiniPile, we perform a simple, three-step data filtering process: we (1) infer embeddings for all documents of the Pile, (2) cluster the embedding space using $k$-means, and (3) filter out low-quality clusters. To verify MiniPile's suitability for language model pre-training, we use it to pre-train a BERT and T5 model, yielding a performance drop of only $1.9\%$/$2.5\%$ on the GLUE and SNI benchmarks compared to the original pre-trained checkpoints trained on $2.6$x/$745$x the amount of data. MiniPile is available at https://huggingface.co/datasets/JeanKaddour/minipile.
Forward citations
Cited by 7 Pith papers
-
Causal Estimation of Tokenisation Bias
Using regression discontinuity, the paper shows that adding a subword to a tokenizer's vocabulary can raise the model's probability for that string by up to about 17 times in small models.
-
Joint Optimization for Greedy Longest-match Tokenization
JOLT trains subword vocabularies by directly optimizing token counts under greedy longest-match decoding, certifies via an LP bound that BPE is within 1–2% of the achievable optimum, and recovers most of that small ga...
-
Dataset Ownership Verification for Pre-trained Masked Models
DOV4MM detects whether a masked pre-trained model was trained on a given dataset via relative embedding reconstruction difficulty, reporting p<0.05 in tests on ImageNet-1K and WikiText-103.
-
KV-Latent: Dimensional-level KV Cache Reduction with Frequency-aware Rotary Positional Embedding
By downsampling key and value head dimensions and retraining with distillation, KV-Latent cuts KV cache memory by about 50% on 7-8B LLMs while keeping average benchmark scores within about 1 point of the base model.
-
ByteSpan: Information-Driven Subword Tokenisation
A tokeniser that groups predictable bytes using a byte-level LM's surprisal or entropy achieves higher morphological alignment than BPE without loss of compression.
-
Sparsified State-Space Models are Efficient Highway Networks
Simba prunes tokens hierarchically in pre-trained SSMs, creating sparse upper layers that act as highways, improving the accuracy-FLOPs trade-off and long-context perplexity.
-
Enabling Flexible Multi-LLM Integration for Scalable Knowledge Aggregation
Adaptive selection and dynamic weighted fusion of source LLMs reduces knowledge interference and improves target model accuracy compared to FuseLLM.
Discussion (0). Continue with ORCID to comment.