Pith. sign in

REVIEW 9 cited by

Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2208.03306 v1 pith:AB76BDVV submitted 2022-08-05 cs.CL

Margaret Li , Suchin Gururangan , Tim Dettmers , Mike Lewis , Tim Althoff , Noah A. Smith , Luke Zettlemoyer This is my paper
classification cs.CL
keywords dataelmstrainingdifferentdomaindomainsexpertllms
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We present Branch-Train-Merge (BTM), a communication-efficient algorithm for embarrassingly parallel training of large language models (LLMs). We show it is possible to independently train subparts of a new class of LLMs on different subsets of the data, eliminating the massive multi-node synchronization currently required to train LLMs. BTM learns a set of independent expert LMs (ELMs), each specialized to a different textual domain, such as scientific or legal text. These ELMs can be added and removed to update data coverage, ensembled to generalize to new domains, or averaged to collapse back to a single LM for efficient inference. New ELMs are learned by branching from (mixtures of) ELMs in the current set, further training the parameters on data for the new domain, and then merging the resulting model back into the set for future use. Experiments show that BTM improves in- and out-of-domain perplexities as compared to GPT-style Transformer LMs, when controlling for training cost. Through extensive analysis, we show that these results are robust to different ELM initialization schemes, but require expert domain specialization; LM ensembles with random data splits do not perform well. We also present a study of scaling BTM into a new corpus of 64 domains (192B whitespace-separated tokens in total); the resulting LM (22.4B total parameters) performs as well as a Transformer LM trained with 2.5 times more compute. These gains grow with the number of domains, suggesting more aggressive parallelism could be used to efficiently train larger models in future work.

Discussion (0). Sign in to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. OpenAlex reports about 26 citations worldwide. Full citation record

  1. FlexOlmo: Open Language Models for Flexible Data Use

    cs.CL 2025-07 conditional novelty 7.0 of 10

    FlexOlmo merges independently trained language-model experts, trained on private data, into a single mixture-of-experts model without joint training.

  2. Decentralised AI Training and Inference with BlockTrain

    cs.AI 2026-06 unverdicted novelty 6.0 of 10

    BlockTrain partitions models into blocks trained on local objectives, reaching CE 1.359 on WikiText within 0.04 of end-to-end baseline while enabling distributed training and inference over TCP for up to 75B-parameter models.

  3. Automatic Expert Discovery in LLM Upcycling via Sparse Interpolated Mixture-of-Experts

    cs.LG 2025-06 conditional novelty 6.0 of 10

    Sparse Interpolated Mixture-of-Experts learns neuron-level masks and a per-prompt router to upcycle a dense LLM into a sparse expert model, improving instruction-tuning benchmarks with lower inference cost.

  4. Group then Scale: Dynamic Mixture-of-Experts Multilingual Language Model

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A multilingual LLM training method that groups similar languages, converts high-deviation layers into mixture-of-experts layers, and assigns one expert per language group improves perplexity across 18 to 128 languages.

  5. NoLoCo: No-all-reduce Low Communication Training Method for Large Models

    cs.LG 2025-06 conditional novelty 6.0 of 10

    NoLoCo trains large language models without any all-to-all synchronization by using pairwise weight averaging and random pipeline routing, matching or slightly beating DiLoCo in experiments.

  6. Local Mixtures of Experts: Essentially Free Test-Time Training via Model Merging

    cs.LG 2025-05 conditional novelty 6.0 of 10

    TTMM pre-trains many local LoRA experts on data clusters and merges the most relevant few at test time, approximating test-time training with a 100x speedup and near-TTT perplexity.

  7. Modular Foundation Models for Time-Series Perception in Digital Twins

    cs.LG 2026-07 conditional novelty 5.0 of 10

    A gated bank of frozen self-supervised time-series encoders, aligned and aggregated by a Transformer, supports competitive multi-task perception for digital twins and hydro-generator virtual sensing.

  8. CHERRY: Compressed Hierarchical Experts with Recurrent Representational Yield

    cs.CL 2026-06 unverdicted novelty 5.0 of 10

    Selective-pivot-token training plus layer-averaging-with-recurrence reportedly gives 2.5x parameter compression on a small Korean LLM, but the efficiency claim lacks its decisive controls and the abstract advertises r...

  9. CLUES: Collaborative High-Quality Data Selection for LLMs via Training Dynamics

    cs.CL 2025-07 conditional novelty 4.0 of 10

    A collaborative data-selection method that scores each private sample's influence on a public anchor set and filters by a global threshold before federated learning or model merging.

Pith tools