Pith. sign in

REVIEW 12 cited by

DoGE: Domain Reweighting with Generalization Estimation

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2310.15393 v2 pith:BYULAFPQ submitted 2023-10-23 cs.LG cs.AIcs.CL

Simin Fan , Matteo Pagliardini , Martin Jaggi This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords domaindogegeneralizationmodelbasetargettrainingweights
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

The coverage and composition of the pretraining data significantly impacts the generalization ability of Large Language Models (LLMs). Despite its importance, recent LLMs still rely on heuristics and trial and error to increase or reduce the influence of data-domains. We propose DOmain reweighting with Generalization Estimation (DoGE), which optimizes the probability of sampling from each domain (domain weights) in a principled way. Our approach is a two-stage process consisting of (i) training a proxy model to obtain domain weights using a bi-level optimization algorithm; (ii) training a larger base model by sampling training domains according to the learned domain weights. In our experiments, we extensively show how DoGE improves the generalization of the base model to any target data mixture. On the SlimPajama dataset, our base model gets better perplexity and few-shot reasoning accuracies across $6$ tasks compared to baseline methods. Moreover, aiming to generalize to out-of-domain target tasks, which is unseen in the pretraining corpus (OOD domain), DoGE can effectively identify inter-domain dependencies, and consistently achieves better test perplexity on the target domain.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 12 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Multi-Task GRPO: Reliable LLM Reasoning Across Tasks

    cs.CL 2026-02 conditional novelty 6.0 of 10

    MT-GRPO reweights tasks by reward and improvement and enforces those weights after zero-gradient filtering, improving worst-task accuracy by 6–28% over GRPO/DAPO baselines on 3- and 9-task setups.

  2. BLISS: A Lightweight Bilevel Influence Scoring Method for Data Selection in Language Model Pretraining

    cs.LG 2025-10 conditional novelty 6.0 of 10

    A bilevel optimization method ranks pretraining data by training a small proxy model on weighted samples, yielding modest downstream-task gains without external pretrained models.

  3. Language Models Improve When Pretraining Data Matches Target Tasks

    cs.CL 2025-07 conditional novelty 6.0 of 10

    Ranking pretraining documents by similarity to benchmark training examples (BETR) yields consistent benchmark gains and a 2.1x compute multiplier over DCLM-Baseline.

  4. Transformers Pretrained on Procedural Data Contain Modular Structures for Algorithmic Reasoning

    cs.LG 2025-05 conditional novelty 6.0 of 10

    Different procedural pretraining tasks create complementary, transferable structures in a transformer's attention and MLP weights, and structures from different tasks can be combined into one initialization.

  5. GRAPE: Optimize Data Mixture for Group Robust Multi-target Adaptive Pretraining

    cs.LG 2025-05 conditional novelty 6.0 of 10

    GRAPE uses a minimax group-DRO scheme to reweight both source domains and target tasks during pretraining, improving multi-task reasoning and low-resource language modeling.

  6. Merge to Mix: Mixing Datasets via Model Merging

    cs.LG 2025-05 conditional novelty 6.0 of 10

    Merge to Mix shows that the performance of a parameter-averaged model predicts the performance of a model fine-tuned on any dataset mixture, enabling fast and accurate dataset mixture selection.

  7. Procedural Pretraining: Warming Up Language Models with Abstract Data

    cs.CL 2026-01 conditional novelty 5.0 of 10

    A short warm-up on procedural data (brackets, sorting, sets) makes language models more accurate and more data-efficient on language, code, and informal math.

  8. Assessing the Role of Data Quality in Training Bilingual Language Models

    cs.CL 2025-06 conditional novelty 5.0 of 10

    A quality filter trained only on English labels can select better French, German, and Chinese pretraining data, improving bilingual model performance and cutting the monolingual-bilingual gap to about 1%.

  9. ESLM: Risk-Averse Selective Language Modeling for Efficient Pretraining

    cs.LG 2025-05 conditional novelty 5.0 of 10

    ESLM keeps only high-loss or high-entropy tokens in each batch via a value-at-risk threshold, cutting pretraining FLOPs by about 6% while roughly matching perplexity and downstream accuracy.

  10. A Survey of LLM $\times$ DATA

    cs.DB 2025-05 conditional novelty 5.0 of 10

    A comprehensive survey of the bidirectional links between LLMs and data management, organized as DATA4LLM and LLM4DATA with a new 'IaaS' data-quality framework.

  11. Salamandra Technical Report

    cs.CL 2025-02 conditional novelty 5.0 of 10

    Salamandra is an open, from-scratch multilingual LLM family with 2B, 7B, and 40B checkpoints, instruction-tuned variants, a vision proof-of-concept, and detailed evaluations across Iberian and European languages.

  12. AgentScope 1.0: A Developer-Centric Framework for Building Agentic Applications

    cs.AI 2025-08 unverdicted novelty 4.0 of 10

    AgentScope 1.0 packages the components needed to build, evaluate, and deploy LLM agent applications into one developer framework.

Pith tools