pith. sign in

arxiv: 2310.19341 · v1 · pith:SZKLXFLNnew · submitted 2023-10-30 · 💻 cs.CL · cs.AI

Skywork: A More Open Bilingual Foundation Model

Tianwen Wei , Liang Zhao , Lichang Zhang , Bo Zhu , Lijie Wang , Haihua Yang , Biye Li , Cheng Cheng
show 22 more authors
Weiwei L\"u Rui Hu Chenxia Li Liu Yang Xilin Luo Xuejie Wu Lunan Liu Wenjun Cheng Peng Cheng Jianhao Zhang Xiaoyu Zhang Lei Lin Xiaokun Wang Yutuan Ma Chuanhai Dong Yanqi Sun Yifu Chen Yongyi Peng Xiaojuan Liang Shuicheng Yan Han Fang Yahui Zhou
This is my paper
classification 💻 cs.CL cs.AI
keywords corpustrainingchinesellmsmodelopenskywork-13bbilingual
0
0 comments X
read the original abstract

In this technical report, we present Skywork-13B, a family of large language models (LLMs) trained on a corpus of over 3.2 trillion tokens drawn from both English and Chinese texts. This bilingual foundation model is the most extensively trained and openly published LLMs of comparable size to date. We introduce a two-stage training methodology using a segmented corpus, targeting general purpose training and then domain-specific enhancement training, respectively. We show that our model not only excels on popular benchmarks, but also achieves \emph{state of the art} performance in Chinese language modeling on diverse domains. Furthermore, we propose a novel leakage detection method, demonstrating that test data contamination is a pressing issue warranting further investigation by the LLM community. To spur future research, we release Skywork-13B along with checkpoints obtained during intermediate stages of the training process. We are also releasing part of our SkyPile corpus, a collection of over 150 billion tokens of web text, which is the largest high quality open Chinese pre-training corpus to date. We hope Skywork-13B and our open corpus will serve as a valuable open-source resource to democratize access to high-quality LLMs.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. TokAlign++: Advancing Vocabulary Adaptation via Better Token Alignment

    cs.CL 2026-05 unverdicted novelty 7.0

    TokAlign++ learns token alignments between LLM vocabularies from monolingual representations to enable faster adaptation, better text compression, and effective token-level distillation across 15 languages with minimal steps.

  2. Frankentext: Stitching random text fragments into long-form narratives

    cs.CL 2025-05 conditional novelty 7.0

    Frankentexts force LLMs to compose coherent long-form stories by copying 90% of tokens verbatim from random human snippets, yielding better quality and originality than vanilla generation while evading detectors.

  3. Exploring Code Analysis: Zero-Shot Insights on Syntax and Semantics with LLMs

    cs.SE 2023-05 unverdicted novelty 7.0

    LLMs achieve strong results on syntax parsing tasks but show limited and variable performance on dynamic reasoning, with a clear performance hierarchy across model scales.

  4. NiuTrans.LMT: Toward Inclusive and Scalable Multilingual Machine Translation with LLMs

    cs.CL 2025-11 conditional novelty 6.0

    NiuTrans.LMT introduces Strategic Downsampling and Parallel Multilingual Prompting to mitigate Directional Degeneration in multilingual MT and releases competitive open models for 60 languages and 234 directions.

  5. InternLM2 Technical Report

    cs.CL 2024-03 unverdicted novelty 5.0

    InternLM2 is a new open-source LLM that outperforms prior versions on 30 benchmarks and long-context tasks through scaled pre-training to 32k tokens and a conditional online RLHF alignment strategy.

  6. InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model

    cs.CV 2024-01 unverdicted novelty 5.0

    InternLM-XComposer2 introduces Partial LoRA on InternLM2-7B to enable high-quality free-form text-image composition while matching or exceeding GPT-4V on select vision-language benchmarks.

  7. InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks

    cs.CV 2023-12 unverdicted novelty 5.0

    InternVL scales a vision model to 6B parameters and aligns it with LLMs using web data to achieve state-of-the-art results on 32 visual-linguistic benchmarks.

  8. How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites

    cs.CV 2024-04 unverdicted novelty 4.0

    InternVL 1.5 narrows the performance gap to proprietary multimodal models via a stronger transferable vision encoder, dynamic high-resolution tiling, and curated English-Chinese training data.

  9. TinyLlama: An Open-Source Small Language Model

    cs.CL 2024-01 accept novelty 4.0

    TinyLlama is a 1.1B-parameter open-source language model pretrained on 1 trillion tokens that outperforms other open-source models of similar size on downstream tasks.

  10. A Survey of Large Language Models

    cs.CL 2023-03 accept novelty 3.0

    This survey reviews the background, key techniques, and evaluation methods for large language models, emphasizing emergent abilities that appear at large scales.