REVIEW 13 cited by
The Stack: 3 TB of permissively licensed source code
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large Language Models (LLMs) play an ever-increasing role in the field of Artificial Intelligence (AI)--not only for natural language processing but also for code understanding and generation. To stimulate open and responsible research on LLMs for code, we introduce The Stack, a 3.1 TB dataset consisting of permissively licensed source code in 30 programming languages. We describe how we collect the full dataset, construct a permissively licensed subset, present a data governance plan, discuss limitations, and show promising results on text2code benchmarks by training 350M-parameter decoders on different Python subsets. We find that (1) near-deduplicating the data significantly boosts performance across all experiments, and (2) it is possible to match previously reported HumanEval and MBPP performance using only permissively licensed data. We make the dataset available at https://hf.co/BigCode, provide a tool called "Am I in The Stack" (https://hf.co/spaces/bigcode/in-the-stack) for developers to search The Stack for copies of their code, and provide a process for code to be removed from the dataset by following the instructions at https://www.bigcode-project.org/docs/about/the-stack/.
Forward citations
Cited by 13 Pith papers
-
Through the Bottleneck: How Multi-head Latent Attention Separates Content from Position in Language Models
In a small MLA language model, the compressed KV cache carries content/entity information but little positional information, and all five induction heads appear in one layer.
-
Basis Transformers for Multi-Task Tabular Regression
Basis transformers beat fine-tuned LLMs on 34 multi-task tabular regression datasets while using five times fewer parameters and no data preprocessing.
-
RTLCurator: Label-Efficient Data Curation for RTL Generation
Keeping 80% of a synthetic RTL corpus selected by a behavior-aware alignment score plus coverage and structural balancing outperforms training on the full corpus, with only 10% of pairs validated.
-
GVR-Coder: A Visual-Feedback Framework for Structured SVG Generation in Complex Document and Meeting Scenarios
A curriculum-plus-RL-plus-repair SVG agent, trained on a new 100K office-diagram dataset, beats strong LLM baselines on layout-correct logical diagrams from long professional text.
-
SimulS2ST-Omni: Data-Efficient Streaming Speech-to-Speech Translation via Explicit Trajectory Supervision
A joint text-code trajectory supervision recipe lets a two-stream speech LM achieve competitive long-form streaming S2ST with ~2k hours of paired speech.
-
EmbeddedKittens: An Evaluation of Code Embeddings for Scratch
Structure-aware embeddings such as GGNN transfer to Scratch and can support sprite naming and, with weaker evidence, correctness and progress prediction.
-
Targeted Recovery of Weight-Space Mechanisms From Neural Networks
A targeted decomposition method recovers the weight-space mechanisms behind specific inputs at low FLOPs, enabling focused ablation and rewiring of a 12-block transformer.
-
Investigating Training Data Detection in AI Coders
Most existing training-data detection methods perform poorly on code, while prefix-relative method ReCaLL consistently scores highest, though all degrade under code mutations.
-
SCGAgent: Recreating the Benefits of Reasoning Models for Secure Code Generation with Agentic Workflows
An agentic workflow with security guidelines and LLM-generated unit tests improves secure code generation on CWEval C tasks from 61% to 76% Func-Sec@1 with Sonnet-3.7, at roughly 98% of its original functionality.
-
Chameleon: A Flexible Data-mixing Framework for Language Model Pretraining and Finetuning
Chameleon uses kernel ridge leverage scores on domain embeddings to set LLM training-mixture weights, matching DoGE-level pretraining quality at roughly one fifth the compute and improving finetuning perplexity.
-
Eradicating the Unseen: Detecting, Exploiting, and Remediating a Path Traversal Vulnerability across GitHub
A single vulnerable Node.js path traversal pattern was found in 1,756 GitHub projects, most rated critical, and the authors' automated pipeline produced patches, disclosures, and evidence that LLMs have learned the pattern.
-
CIDRe: A Reference-Free Multi-Aspect Criterion for Code Comment Quality Measurement
CIDRe is a reference-free, four-part quality score for structured code comments, validated on 840 manually labeled Russian comments and reported to improve comment generation after dataset filtering.
-
Applying the Chinese Wall Reverse Engineering Technique to Large Language Model Code Editing
Using Gemini 2.5 Pro to annotate code with edit instructions improved Comma v0.1 1T's CanItEdit pass@20 from 20.00 to 33.33 and Starcoder2 Instruct's pass@1 from 35.10 to 42.05.
Discussion (0). Continue with ORCID to comment.