REVIEW 8 cited by
The MiniPile Challenge for Data-Efficient Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
The ever-growing diversity of pre-training text corpora has equipped language models with generalization capabilities across various downstream tasks. However, such diverse datasets are often too large for academic budgets; hence, most research on Transformer architectures, training procedures, optimizers, etc. gets conducted on smaller, homogeneous datasets. To this end, we present The MiniPile Challenge, where one pre-trains a language model on a diverse text corpus containing at most 1M documents. MiniPile is a 6GB subset of the deduplicated 825GB The Pile corpus. To curate MiniPile, we perform a simple, three-step data filtering process: we (1) infer embeddings for all documents of the Pile, (2) cluster the embedding space using $k$-means, and (3) filter out low-quality clusters. To verify MiniPile's suitability for language model pre-training, we use it to pre-train a BERT and T5 model, yielding a performance drop of only $1.9\%$/$2.5\%$ on the GLUE and SNI benchmarks compared to the original pre-trained checkpoints trained on $2.6$x/$745$x the amount of data. MiniPile is available at https://huggingface.co/datasets/JeanKaddour/minipile.
Forward citations
Cited by 8 Pith papers
-
Causal Estimation of Tokenisation Bias
Using regression discontinuity, the paper shows that adding a subword to a tokenizer's vocabulary can raise the model's probability for that string by up to about 17 times in small models.
-
Joint Optimization for Greedy Longest-match Tokenization
JOLT trains subword vocabularies by directly optimizing token counts under greedy longest-match decoding, certifies via an LP bound that BPE is within 1–2% of the achievable optimum, and recovers most of that small ga...
-
Dataset Ownership Verification for Pre-trained Masked Models
DOV4MM detects whether a masked pre-trained model was trained on a given dataset via relative embedding reconstruction difficulty, reporting p<0.05 in tests on ImageNet-1K and WikiText-103.
-
KV-Latent: Dimensional-level KV Cache Reduction with Frequency-aware Rotary Positional Embedding
By downsampling key and value head dimensions and retraining with distillation, KV-Latent cuts KV cache memory by about 50% on 7-8B LLMs while keeping average benchmark scores within about 1 point of the base model.
-
ByteSpan: Information-Driven Subword Tokenisation
A tokeniser that groups predictable bytes using a byte-level LM's surprisal or entropy achieves higher morphological alignment than BPE without loss of compression.
-
Sparsified State-Space Models are Efficient Highway Networks
Simba prunes tokens hierarchically in pre-trained SSMs, creating sparse upper layers that act as highways, improving the accuracy-FLOPs trade-off and long-context perplexity.
-
Enabling Flexible Multi-LLM Integration for Scalable Knowledge Aggregation
Adaptive selection and dynamic weighted fusion of source LLMs reduces knowledge interference and improves target model accuracy compared to FuseLLM.
-
MoETuner: Optimized Mixture of Expert Serving with Balanced Expert Placement and Token Routing
MoETuner formulates MoE expert placement as two ILPs to balance token loads and minimize inter-GPU communication, reporting 9.3% and 17.5% end-to-end inference speedups on Mixtral-8x7B.
Discussion (0). Continue with ORCID to comment.