Pith. sign in

REVIEW 9 cited by

Continual Pre-training of Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2302.03241 v4 pith:HQ3PBZTH submitted 2023-02-07 cs.CL cs.AIcs.LGcs.NE

Zixuan Ke , Yijia Shao , Haowei Lin , Tatsuya Konishi , Gyuhak Kim , Bing Liu This is my paper
classification cs.CLcs.AIcs.LGcs.NE
keywords knowledgedomaincontinualmethodpre-trainingend-taskimprovelanguage
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Language models (LMs) have been instrumental for the rapid advance of natural language processing. This paper studies continual pre-training of LMs, in particular, continual domain-adaptive pre-training (or continual DAP-training). Existing research has shown that further pre-training an LM using a domain corpus to adapt the LM to the domain can improve the end-task performance in the domain. This paper proposes a novel method to continually DAP-train an LM with a sequence of unlabeled domain corpora to adapt the LM to these domains to improve their end-task performances. The key novelty of our method is a soft-masking mechanism that directly controls the update to the LM. A novel proxy is also proposed to preserve the general knowledge in the original LM. Additionally, it contrasts the representations of the previously learned domain knowledge (including the general knowledge in the pre-trained LM) and the knowledge from the current full network to achieve knowledge integration. The method not only overcomes catastrophic forgetting, but also achieves knowledge transfer to improve end-task performances. Empirical evaluation demonstrates the effectiveness of the proposed method.

Discussion (0). Sign in to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Learning to Prepare Molecular Ground States with Transformer Models

    quant-ph 2026-07 conditional novelty 6.0 of 10

    Transformers trained on ADAPT-VQE data generate imipramine ground-state circuits in seconds at roughly reference accuracy — and beat the training data after reinforcement learning — though real-hardware energies still...

  2. AI-Assisted Fixes to Code Review Comments at Scale

    cs.SE 2025-07 conditional novelty 6.0 of 10

    Fine-tuned Llama models generate exact-match patches for 68% of internal code review comments, and a safety trial shows AI suggestions slow reviewers unless hidden from them.

  3. Bisecle: Binding and Separation in Continual Learning for Video Language Understanding

    cs.CV 2025-07 conditional novelty 6.0 of 10

    Bisecle reduces catastrophic forgetting and improves accuracy in sequential VideoQA learning using multi-directional auxiliary losses and contrastive prompt regularization.

  4. Scaling Point-in-Time Language Models

    cs.CL 2026-04 conditional novelty 5.5 of 10

    Scaling point-in-time LLMs to 4B parameters and 1T temporally filtered tokens narrows the gap to unrestricted models to about 8–11 average points and yields positive out-of-sample Sharpe ratios from news embeddings.

  5. LinearARD: Linear-Memory Attention Distillation for RoPE Restoration

    cs.CL 2026-03 conditional novelty 5.0 of 10

    LinearARD restores RoPE-scaled LLMs by exact linear-memory KL distillation of Q/Q, K/K, and V/V self-relations, reaching ~95% short-text recovery with 4.25M tokens.

  6. Data Doping or True Intelligence? Evaluating the Transferability of Injected Knowledge in LLMs

    cs.CL 2025-05 conditional novelty 5.0 of 10

    LLMs fine-tuned on comprehension tasks like question answering retain injected facts at more than double the rate of models fine-tuned on translation or JSON mapping, but all models struggle to apply the facts in new ...

  7. Continual Learning for Generative AI: From LLMs to MLLMs and Beyond

    cs.LG 2025-06 conditional novelty 4.0 of 10

    A survey that categorizes continual learning methods for generative models into architecture-based, regularization-based, and replay-based paradigms across four model families.

  8. Unified Game Moderation: Soft-Prompting and LLM-Assisted Label Transfer for Resource-Efficient Toxicity Detection

    cs.CL 2025-06 conditional novelty 4.0 of 10

    A single BERT-scale model with a game-context token and LLM-assisted label transfer achieves toxicity detection comparable to per-game models while extending to seven languages.

  9. Beyond Distillation: Pushing the Limits of Medical LLM Reasoning with Minimalist Rule-Based RL

    cs.CL 2025-05 conditional novelty 4.0 of 10

    Medical QA accuracy improves substantially through reinforcement learning with a binary correct-answer reward alone, without supervised fine-tuning on distilled reasoning traces.

Pith tools