REVIEW 8 cited by
To Code, or Not To Code? Exploring Impact of Code in Pre-training
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Including code in the pre-training data mixture, even for models not specifically designed for code, has become a common practice in LLMs pre-training. While there has been anecdotal consensus among practitioners that code data plays a vital role in general LLMs' performance, there is only limited work analyzing the precise impact of code on non-code tasks. In this work, we systematically investigate the impact of code data on general performance. We ask "what is the impact of code data used in pre-training on a large variety of downstream tasks beyond code generation". We conduct extensive ablations and evaluate across a broad range of natural language reasoning tasks, world knowledge tasks, code benchmarks, and LLM-as-a-judge win-rates for models with sizes ranging from 470M to 2.8B parameters. Across settings, we find a consistent results that code is a critical building block for generalization far beyond coding tasks and improvements to code quality have an outsized impact across all tasks. In particular, compared to text-only pre-training, the addition of code results in up to relative increase of 8.2% in natural language (NL) reasoning, 4.2% in world knowledge, 6.6% improvement in generative win-rates, and a 12x boost in code performance respectively. Our work suggests investments in code quality and preserving code during pre-training have positive impacts.
Forward citations
Cited by 8 Pith papers
-
Can Transformers Really Do It All? On the Compatibility of Inductive Biases Across Tasks
Learned replacement non-linearities show transformers are rarely optimal for algorithmic tasks, with benefits that are task-specific, while language/code gains are smaller and more transferable.
-
Domain-Aware Scaling Laws Uncover Data Synergy
Domain-aware scaling laws with fitted γ and σ synergy terms recover stable code-math interactions from observational LLM mixtures and correctly predict mixture rankings in controlled small-scale trainings.
-
Can Large Language Models Generalize Procedures Across Representations?
Post-training on graph or code versions of a planning task does not transfer to natural-language versions, but a symbolic-then-natural-language RL curriculum achieves strong transfer.
-
The Common Pile v0.1: An 8TB Dataset of Public Domain and Openly Licensed Text
A new 8TB openly-licensed text corpus trains 7B LLMs that are competitive with Llama 1/2, showing that performant models need not depend on unlicensed web data.
-
How Programming Concepts and Neurons Are Shared in Code Language Models
In Llama-based code models, programming languages are represented through an English-like intermediate token space, with language-specific neurons concentrated in bottom layers and exclusive PL neurons in top layers; ...
-
Transformers Pretrained on Procedural Data Contain Modular Structures for Algorithmic Reasoning
Different procedural pretraining tasks create complementary, transferable structures in a transformer's attention and MLP weights, and structures from different tasks can be combined into one initialization.
-
What I cannot execute, I do not understand: Training and Evaluating LLMs on Program Execution Traces
Training an 8B LLM on Python execution traces improves output prediction on CruxEval and MBPP to about 80%, with dynamic scratchpads extending accurate predictions to executions of over ten thousand steps.
-
Procedural Pretraining: Warming Up Language Models with Abstract Data
A short warm-up on procedural data (brackets, sorting, sets) makes language models more accurate and more data-efficient on language, code, and informal math.
Discussion (0). Continue with ORCID to comment.