Pith. sign in

REVIEW 3 major objections 5 minor 62 references

Steel-LLM:From Scratch to Open Source -- A Personal Journey in Building a Chinese-Centric LLM

T0 review · 3 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A 1-billion-parameter Chinese LLM built on 8 GPUs posts 41.9 C-Eval.

desk verdict A genuinely useful small-model training report with real open assets, but the '1T tokens in 30 days on 8 GPUs' claim conflicts with its own speed table and needs verification before the efficiency story stands. read the letter →

arxiv 2502.06635 v2 pith:KSLM7ZGF submitted 2025-02-10 cs.CL cs.AI

Qingshui Gu , Shu Li , Tianyu Zheng , Zhaoxiang Zhang This is my paper · ORCID
classification cs.CLcs.AI
keywords Chinese-centricLLMfrom-scratchpretraining1B-parameterlanguagemodelSoftMixtureofExpertssupervisedfine-tuningDirectPreferenceOptimizationC-EvalCMMLU
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Steel-LLM is a 1-billion-parameter language model trained from scratch with a Chinese-first corpus and released with checkpoints and training scripts. The paper reports that after supervised fine-tuning the chat model reaches 41.90 on C-Eval and 36.08 on CMMLU, and that a DPO variant reaches 42.04 and 36.04, beating models of similar or larger size from large labs. The intended contribution is a fully transparent recipe: 30 days of pretraining on 8 GPUs, a Soft Mixture-of-Experts feed-forward block, and careful SFT data mixing. The point of the exercise is to show that a small team with modest hardware can produce a usable Chinese LLM without hiding the details.

What carries the argument

Soft Mixture of Experts (Soft MoE) is the central architectural device. Instead of routing each token to a few sparse experts, Soft MoE forms dispatch weights that mix all input tokens into per-expert slots, applies the expert function, then combines output slots back into tokens, so every expert is fully trained and no expert-imbalance problem arises. Steel-LLM pairs this with an enhanced feed-forward network that applies SwiGLU activation to both MLP layers rather than one. The rest of the load is carried by the training recipe: FlashAttention, FSDP, bf16 mixed precision, fused RoPE and cross-entropy, and an SFT data mixture that mirrors pretraining domain proportions.

What would settle it

Multiply the reported throughput (13,000-15,000 tokens/s/GPU) by 8 GPUs and 30 days: the result is roughly 280-320 billion tokens, not one trillion, so checking the released training logs or checkpoint step counts against token counts would settle whether the data-scale claim is correct.

Watch

Extended reading notes

Core claim

The central claim is that a fully open Chinese-centric LLM can be built from scratch with one billion parameters and only eight GPUs while still scoring competitively on Chinese benchmarks. The strongest result is Steel-LLM-Chat's 41.90 C-Eval and 36.08 CMMLU after SFT, and 42.04/36.04 after DPO, which the paper argues outperforms TinyLlama-1.1B (25.02/24.03) and Gemma-2b-it (32.30/33.07). The paper also argues that fine-tuning data distribution matters: matching the 80% Chinese / 20% English split of pretraining improves Chinese scores, and adding English multiple-choice questions raises MMLU without hurting Chinese performance. The authors position this as evidence that transparency and resource efficiency need not come at the cost of quality.

Load-bearing premise

The load-bearing premise is that the model was really trained from scratch on roughly one trillion tokens in 30 days on eight GPUs as claimed; if the actual token count was much smaller, the benchmark comparisons and the resource-efficiency story would need to be re-read.

Editorial extensions

If this is right

  • If the reported pipeline is real at scale, a 1B-parameter model can be pretrained on about a trillion Chinese-focused tokens in 30 days on eight GPUs, making such projects feasible for small labs.
  • The SFT ablation shows that keeping fine-tuning data in the same Chinese/English proportion as pretraining yields higher C-Eval and CMMLU than using all Chinese instructions alone.
  • Adding English multiple-choice data improves MMLU by several points without lowering Chinese benchmark scores.
  • Using soft rather than sparse MoE trains all expert parameters at 1B scale, which is a plausible reason the model outperforms similar-size dense baselines.
  • All checkpoints and training scripts are released, so the exact recipe can be reproduced or extended rather than only described.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is to pretrain with a verified token budget and report per-GPU throughput logs; those two numbers would determine whether the '1 trillion tokens in 30 days on 8 GPUs' claim transfers to other 8-GPU setups.
  • The same Soft MoE plus enhanced FFN recipe could be tried on other low-resource languages; the paper's claim is Chinese-specific, but nothing in the architecture is language-bound.
  • DPO gave a small gain over SFT on C-Eval (41.90 to 42.04) but a slight loss on CMMLU (36.08 to 36.04); whether preference tuning helps at this scale is an open question the paper leaves implicit.
  • The authors' implicit guide for small labs is that data composition and transparency matter more than architecture novelty; one could test this by ablating Soft MoE against a dense FFN at the same token budget.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper reports Steel-LLM, a 1-billion-parameter Chinese-centric language model pretrained from scratch on approximately one trillion tokens in 30 days using only 8 A100/H800 GPUs, then post-trained with SFT and DPO. The authors describe their data collection and filtering pipeline, a Soft-MoE-enhanced FFN architecture, training-framework engineering (model loading, progress restoration, and online data appending), SFT data-composition ablations, and final benchmark results on C-Eval (41.90) and CMMLU (36.08). They position the work as a fully open, reproducible, resource-efficient alternative to larger transparency-focused projects, and they release checkpoints and training code.

Significance. If the claims are substantiated, the paper would be a useful data point for small-scale LLM development: the released checkpoints and scripts are concrete assets, the SFT data-composition ablations in Section 6.2 are informative, and the engineering details around training restoration and data appending are practical. The headline resource-efficiency claim is, however, not yet supported by the paper's own speed measurements, and the benchmark comparison lacks evaluation protocol details. The central value of the paper therefore depends on correcting and independently verifying the reported training scale.

major comments (3)
  1. [Section 5, Pretraining, and Table 2] The reported training scale is internally inconsistent with the reported hardware throughput. With sequence length 2048, a per-GPU micro-batch of 8, 8 gradient-accumulation steps, and 8 GPUs, each optimizer step sees 8 × 8 × 8 × 2048 = 1,048,576 tokens; over 1.07M steps this gives approximately 1.12T tokens. To complete that in 30 days (2.592M seconds) requires roughly 432K tokens/s aggregate, or about 54K tokens/s/GPU. Table 2, however, reports 10,500–15,000 tokens/s/GPU on a 1.8B model on A100, a factor of 3.6–5.1 below the required rate. Steel-LLM is 1B and used H800 for most of training, which partly closes the gap, but the paper does not report the actual measured throughput of the Steel-LLM run. The authors must either provide the real throughput and step-time logs for the reported run or correct the token count, step count, and hardware assumptions; as written, the "~1T tokens in 30 days on 8 GPUs" claim cannot be verified from the paper's own data.
  2. [Section 6.2, Tables 3 and 4] The benchmark evaluation is under-specified, which makes the main performance comparison non-reproducible. The paper does not state which evaluation harness was used for C-Eval, CMMLU, and MMLU, what prompt format or few-shot settings were employed, or whether the reported numbers are single runs or averages. This matters because the final SFT recipe was selected after five ablation runs (Table 3); the "best" numbers in Table 4 are therefore a selected result, not an independent prediction, and without variance estimates they cannot be distinguished from noise. The authors should report the exact evaluation protocol and, ideally, multiple seeds or a confidence interval.
  3. [Figure 2 and Appendix D] The transparency contribution is undermined by corrupted data presentation. Figure 2, which is supposed to show the pretraining data distribution, appears in the manuscript as raw JavaScript/HTML code rather than as a rendered chart, so the actual data mixture is not visible. Appendix D also contains implausible filter ranges (e.g., "word repetition filter [10, 0.357]" and "words num filter [20, 6640]"), which appear to be swapped or mistyped. These issues should be fixed before the paper can claim to provide a usable account of the data pipeline.
minor comments (5)
  1. [Section 5, Pretraining] The term "batch size" in "we set the maximum sequence length to 2048, the batch size to 8, and the number of gradient accumulation steps to 8" should be explicitly defined as the per-GPU micro-batch size; otherwise the global batch size and resulting token-per-step count are ambiguous.
  2. [Throughout] The benchmark name is inconsistently written as "CEV AL" in the abstract, Section 6, Tables 3 and 4, and the conclusion; it should be "C-Eval".
  3. [References] The BAAI Infinity-Instruct reference contains a placeholder ID ("arXiv:2406.XXXX") and should be updated to the actual technical report or dataset URL.
  4. [Section 6.1, SFT] The statement that the model was fine-tuned for "approximately 4 epochs" with a global batch size of 256 is inconsistent with Figure 4, whose loss curve extends to 70,000 steps; at a global batch of 256 that corresponds to about 17.9M samples, which is many more than 4 epochs over the described SFT corpus. The actual number of steps and data samples seen should be reported.
  5. [Appendix D] Several filter ranges in the code-processing table appear erroneous: the "word repetition filter" is listed as [10, 0.357], which is not a valid interval for a ratio, and the "words num filter" upper bound appears to be missing a decimal point. These should be corrected or clarified.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Steel-LLM's claims are empirical training and evaluation results against external benchmarks, with no load-bearing derivation reducing to its own inputs.

full rationale

The paper contains no derived claim that collapses into its own assumptions. The architecture is assembled from externally cited components (Soft MoE from Puigcerver et al., FlashAttention, RoPE, RMSNorm, Qwen-style blocks), and the only modifications are described as empirical choices rather than theorems. The pretraining token count is arithmetically consistent with the reported 1.07M steps, sequence length 2048, batch size 8, gradient accumulation 8, and 8 GPUs, so the 'approximately one trillion tokens' statement is a computation from stated hyperparameters rather than a self-defined prediction. The SFT section reports five explicit ablation experiments whose final configuration was selected after inspecting benchmark results; this is model selection, not a fitted parameter disguised as a prediction, and the paper does not claim the chosen composition was derived from first principles. The C-Eval and CMMLU comparisons are against external benchmark scores from other published models, so the central performance claim is not equivalent to an input of the paper. Self-citations are limited to comparative entries such as CT-LLM and OpenCodeInterpreter and are not load-bearing premises; no uniqueness theorem or ansatz is imported from the authors' prior work. The apparent inconsistency between Table 2 throughput and the 30-day training schedule is a verification/reproducibility concern about whether the stated scale occurred, not a circularity in the derivation chain. Overall, the paper's reasoning is self-contained in the relevant sense: empirical results are reported as empirical results, and no 'prediction' is forced by definition or by a self-citation chain.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

No new theoretical entities are introduced. The model architecture is a combination of known components, and the main crafted inputs are the SFT/DPO data mixtures and hyperparameters, which are post-hoc empirical choices rather than scientifically fitted parameters.

free parameters (5)
  • Soft MoE slots per expert = 1
    Chosen as a hyperparameter in Table 1 with no ablation or sensitivity analysis.
  • Number of experts = 6
    Chosen in Table 1 with no ablation or sensitivity analysis.
  • SFT data composition = 80% Chinese / 20% English + English MCQ
    Selected after five ablation runs in Table 3, not independently predicted.
  • DPO hyperparameters = LR 5e-6, beta 0.1, 3 epochs
    Reported in Section 6.3 with no ablation or sensitivity analysis.
  • Pretraining schedule = 1.07M steps, LR 3e-4, 2000 warmup, cosine to 0
    Reported in Section 5 following TinyLlama conventions, with no sweep.
assumptions (3)
  • domain assumption Open-source Chinese corpora (SkyPile-150B, Wanjuan, Wikipedia-cn, etc.) filtered with Data-Juicer adequately represent Chinese language and knowledge for pretraining.
    Section 5 and Appendix A; filtering operators may introduce corpus-specific biases, but this is standard practice for LLM training.
  • domain assumption C-Eval, CMMLU, and MMLU scores measure language-model quality in a way that transfers to practical Chinese applications.
    Section 6.4 and Table 4; no contamination checks or evaluation details are provided.
  • domain assumption The training framework faithfully implements the described Soft MoE, SwiGLU, FlashAttention, and FSDP pipeline so that the trained weights correspond to the reported benchmark scores.
    Sections 3-4; no code is audited in the paper, only claimed on GitHub.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Steel-LLM:From Scratch to Open Source -- A Personal Journey in Building a Chinese-Centric LLM." pith.science (2026). https://pith.science/paper/KSLM7ZGF

@misc{pith2026250206635,
  author       = {Pith},
  title        = {Pith review of: Steel-LLM:From Scratch to Open Source -- A Personal Journey in Building a Chinese-Centric LLM},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KSLM7ZGF}},
  note         = {Machine review of arXiv:2502.06635}
}
read the original abstract

Steel-LLM is a Chinese-centric language model developed from scratch with the goal of creating a high-quality, open-source model despite limited computational resources. Launched in March 2024, the project aimed to train a 1-billion-parameter model on a large-scale dataset, prioritizing transparency and the sharing of practical insights to assist others in the community. The training process primarily focused on Chinese data, with a small proportion of English data included, addressing gaps in existing open-source LLMs by providing a more detailed and practical account of the model-building journey. Steel-LLM has demonstrated competitive performance on benchmarks such as CEVAL and CMMLU, outperforming early models from larger institutions. This paper provides a comprehensive summary of the project's key contributions, including data collection, model design, training methodologies, and the challenges encountered along the way, offering a valuable resource for researchers and practitioners looking to develop their own LLMs. The model checkpoints and training script are available at https://github.com/zhanshijinwat/Steel-LLM.

Figures

Figures reproduced from arXiv: 2502.06635 by the authors.

Figure 1
Figure 1. Token 1 Token 2 Token 3 …… Token N Self- Attention Token 1 Token 2 Token 3 …… Token N Soft MOE Dispatch Weights Slot 1 Slot 2…… Slot S Expert 1 Expert 2 …… Expert E Soft MOE Combine Weights Token 1 Token 2 Token 3 …… Token N Slot MLP MLP Swish MLP MLP Swish Slot Enhance FFN [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 3
Figure 3. Pre-training loss curve for Steel-LLM F SUPERVISED FINETUNING LOSS CURVE 0 10000 20000 30000 40000 50000 60000 70000 Steps 1.0 1.2 1.4 1.6 1.8 2.0 Loss Supervised Fine-tuning Loss Curve SFT Loss [PITH_FULL_IMAGE:figures/full_fig_p020_3.png] view at source ↗
Figure 4
Figure 4. Supervised Fine-tuning loss curve for Steel-LLM 20 [PITH_FULL_IMAGE:figures/full_fig_p020_4.png] view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: Direct Preference Optimization loss curve for [PITH_FULL_IMAGE:figures/full_fig_p021_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

62 extracted references · 18 canonical work pages

  1. [1]

    In OpenBMB Blog, 2024

    Minicpm: Unveiling the potential of end-side large language models. In OpenBMB Blog, 2024

    work page 2024

  2. [2]

    Phi-2: The surprising power of small language models

    Marah Abdin, Jyoti Aneja, Sebastien Bubeck, Caio César Teodoro Mendes, Weizhu Chen, Allie Del Giorno, Ronen Eldan, Sivakanth Gopi, Suriya Gunasekar, Mojan Javaheripi, Piero Kauffmann, Yin Tat Lee, Yuanzhi Li, Anh Nguyen, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Michael Santacroce, Harkirat Singh Behl, Adam Taumann Kalai, Xin Wang, Rachel ...

    work page 2023

  3. [3]

    01. AI, :, Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Yang, Tao Yu, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yuchi Xu, Yudong Liu, Yue Wang, Yuxuan Cai, Zhenyu Gu, Zhiyuan Liu, and...

    arXiv 2024

  4. [4]

    Infinity instruct

    BAAI . Infinity instruct. arXiv preprint, arXiv:2406.XXXX, 2024

    work page 2024

  5. [6]

    Qwen technical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng X...

    arXiv 2023

  6. [7]

    Belle: Be everyone's large language model engine

    BELLEGroup. Belle: Be everyone's large language model engine. https://github.com/LianjiaTech/BELLE, 2023

    2023

  7. [8]

    Internlm2 technical report, 2024

    Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, Xiaoyi Dong, Haodong Duan, Qi Fan, Zhaoye Fei, Yang Gao, Jiaye Ge, Chenya Gu, Yuzhe Gu, Tao Gui, Aijia Guo, Qipeng Guo, Conghui He, Yingfan Hu, Ting Huang, Tao Jiang, Penglong Jiao, Zhenjiang Jin, Zhikai Lei, Jiaxing Li, Jingwen Li, Linyang Li, S...

    2024

  8. [9]

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradb...

    arXiv 2022

Show all 62 references
  1. [10]

    Damai Dai, Chengqi Deng, Chenggang Zhao, R. X. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y. Wu, Zhenda Xie, Y. K. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, and Wenfeng Liang. Deepseekmoe: Towards ultimate expert specialization in mixture-of-ex...

    2024 arXiv

  2. [11]

    Fu, Stefano Ermon, Atri Rudra, and Christopher Ré

    Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness, 2022. URL https://arxiv.org/abs/2205.14135

    2022 arXiv

  3. [12]

    DeepSeek-AI, :, Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, Huazuo Gao, Kaige Gao, Wenjun Gao, Ruiqi Ge, Kang Guan, Daya Guo, Jianzhong Guo, Guangbo Hao, Zhewen Hao, Ying He, Wenjie Hu, Panpan Huang, Er...

    2024

  4. [13]

    Zhang, Hanwei Xu, Hao Yang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J

    DeepSeek-AI, Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Hanwei Xu, Hao Yang, Haowei...

    2024 arXiv

  5. [14]

    Chinese tiny llm: Pretraining a chinese-centric large language model

    Xinrun Du, Zhouliang Yu, Songyang Gao, Ding Pan, Yuyang Cheng, Ziyang Ma, Ruibin Yuan, Xingwei Qu, Jiaheng Liu, Tianyu Zheng, Xinchen Luo, Guorui Zhou, Binhang Yuan, Wenhu Chen, Jie Fu, and Ge Zhang. Chinese tiny llm: Pretraining a chinese-centric large language model. ArXiv, ...

    2024 arXiv

  6. [15]

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2022

    William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2022. URL https://arxiv.org/abs/2101.03961

    2022 arXiv

  7. [16]

    Thomas Mesnard Gemma Team, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, and et al. Gemma. 2024. doi:10.34740/KAGGLE/M/3301. URL https://www.kaggle.com/m/3301

    2024 doi

  8. [17]

    Chatglm: A family of large language models from glm-130b to glm-4 all tools, 2024

    Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, Hao Yu, Hongning Wang, Jiadai Sun, Jiajie Zhang, Jiale Cheng, Jiayi Gui, Jie Tang, Jing Zhang, Juanzi Li, Lei Zhao, Lindong Wu, Lucen Zhong, Mingdao Liu, Minlie H...

    2024

  9. [18]

    Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, A. Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Raghavi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, T...

    2024

  10. [19]

    Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel,...

    2024

  11. [20]

    Wanjuan: A comprehensive multimodal dataset for advancing english and chinese large models, 2023

    Conghui He, Zhenjiang Jin, Chao Xu, Jiantao Qiu, Bin Wang, Wei Li, Hang Yan, Jiaqi Wang, and Dahua Lin. Wanjuan: A comprehensive multimodal dataset for advancing english and chinese large models, 2023

    2023

  12. [21]

    Jacobs, Michael I

    Robert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. Adaptive mixtures of local experts. Neural Computation, 3 0 (1): 0 79--87, 1991. doi:10.1162/neco.1991.3.1.79

    1991 doi

  13. [22]

    Exploring the impact of instruction data scaling on large language models: An empirical study on real-world use cases, 2023

    Yunjie Ji, Yong Deng, Yan Gong, Yiping Peng, Qiang Niu, Lei Zhang, Baochang Ma, and Xiangang Li. Exploring the impact of instruction data scaling on large language models: An empirical study on real-world use cases, 2023. URL https://arxiv.org/abs/2303.14742

    2023 arXiv

  14. [23]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    2023

  15. [24]

    A study of bfloat16 for deep learning training, 2019

    Dhiraj Kalamkar, Dheevatsa Mudigere, Naveen Mellempudi, Dipankar Das, Kunal Banerjee, Sasikanth Avancha, Dharma Teja Vooturi, Nataraj Jammalamadaka, Jianyu Huang, Hector Yuen, Jiyan Yang, Jongsoo Park, Alexander Heinecke, Evangelos Georganas, Sudarshan Srinivasan, Abhisek Kund...

    2019 arXiv

  16. [25]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017. URL https://arxiv.org/abs/1412.6980

    2017 arXiv

  17. [26]

    \ GS \ hard: Scaling giant models with conditional computation and automatic sharding

    Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. \ GS \ hard: Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations, ...

    2021

  18. [27]

    Starcoder: may the source be with you! 2023

    Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, ...

    2023

  19. [28]

    Llm360: Towards fully transparent open-source llms

    Zhengzhong Liu, Aurick Qiao, Willie Neiswanger, Hongyi Wang, Bowen Tan, Tianhua Tao, Junbo Li, Yuqi Wang, Suqi Sun, Omkar Pangarkar, et al. Llm360: Towards fully transparent open-source llms. arXiv preprint arXiv:2312.06550, 2023

    2023 arXiv

  20. [29]

    Modeling task relationships in multi-task learning with multi-gate mixture-of-experts

    Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H Chi. Modeling task relationships in multi-task learning with multi-gate mixture-of-experts. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pp.\ 1930--1939, 2018

    1930

  21. [30]

    Olmoe: Open mixture-of-experts language models

    Niklas Muennighoff, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Jacob Morrison, Sewon Min, Weijia Shi, Pete Walsh, Oyvind Tafjord, Nathan Lambert, et al. Olmoe: Open mixture-of-experts language models. arXiv preprint arXiv:2409.02060, 2024

    2024 arXiv

  22. [31]

    2 olmo 2 furious

    Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, et al. 2 olmo 2 furious. arXiv preprint arXiv:2501.00656, 2024

    2024 arXiv

  23. [32]

    From sparse to soft mixtures of experts, 2024

    Joan Puigcerver, Carlos Riquelme, Basil Mustafa, and Neil Houlsby. From sparse to soft mixtures of experts, 2024. URL https://arxiv.org/abs/2308.00951

    2024 arXiv

  24. [33]

    Manning, and Chelsea Finn

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model, 2024. URL https://arxiv.org/abs/2305.18290

    2024 arXiv

  25. [34]

    R. Rivest. The md5 message-digest algorithm. RFC 1321, 4 1992

    1992

  26. [35]

    Better ruozhiba

    Misdirection Ruozhiba, FunnySaltyFish. Better ruozhiba. https://github.com/FunnySaltyFish/Better-Ruozhiba, 2024

    2024

  27. [36]

    Glu variants improve transformer, 2020

    Noam Shazeer. Glu variants improve transformer, 2020. URL https://arxiv.org/abs/2002.05202

    2020 arXiv

  28. [37]

    Roformer: Enhanced transformer with rotary position embedding, 2023

    Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, 2023. URL https://arxiv.org/abs/2104.09864

    2023 arXiv

  29. [38]

    Moss: An open conversational large language model

    Tianxiang Sun, Xiaotian Zhang, Zhengfu He, Peng Li, Qinyuan Cheng, Xiangyang Liu, Hang Yan, Yunfan Shao, Qiong Tang, Shiduo Zhang, Xingjian Zhao, Ke Chen, Yining Zheng, Zhejian Zhou, Ruixiao Li, Jun Zhan, Yunhua Zhou, Linyang Li, Xiaogui Yang, Lingling Wu, Zhangyue Yin, Xuanji...

    2024

  30. [39]

    Llm360 k2-65b: Scaling up fully transparent open-source llms

    Bowen Tan, Hongyi Wang37, Willie Neiswanger, Tianhua Tao, Haonan Li, Fajri Koto, Yuqi Wang, Suqi Sun, Omkar Pangarkar, Richard Fan, et al. Llm360 k2-65b: Scaling up fully transparent open-source llms. 2024

    2024

  31. [40]

    Emollm: Reinventing mental health support with large language models

    EmoLLM Team. Emollm: Reinventing mental health support with large language models. https://github.com/SmartFlowAI/EmoLLM, 2024 a

    2024

  32. [41]

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex...

    2024 arXiv

  33. [42]

    Introducing qwen1.5, February 2024 b

    Qwen Team. Introducing qwen1.5, February 2024 b . URL https://qwenlm.github.io/blog/qwen1.5/

    2024

  34. [43]

    Qwen1.5-moe: Matching 7b model performance with 1/3 activated parameters", February 2024 c

    Qwen Team. Qwen1.5-moe: Matching 7b model performance with 1/3 activated parameters", February 2024 c . URL https://qwenlm.github.io/blog/qwen-moe/

    2024

  35. [44]

    Openhermes 2.5: An open dataset of synthetic data for generalist llm assistants, 2023

    Teknium. Openhermes 2.5: An open dataset of synthetic data for generalist llm assistants, 2023. URL https://huggingface.co/datasets/teknium/OpenHermes-2.5

    2023

  36. [45]

    Philippe Tillet, H. T. Kung, and David Cox. Triton: an intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, MAPL 2019, pp.\ 10–19, New York, NY, USA...

    2019

  37. [46]

    Llama: Open and efficient foundation language models, 2023 a

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language...

    2023 arXiv

  38. [47]

    Llama 2: Open foundation and fine-tuned chat models, 2023 b

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, W...

    2023 arXiv

  39. [48]

    Skywork: A more open bilingual foundation model, 2023

    Tianwen Wei, Liang Zhao, Lichang Zhang, Bo Zhu, Lijie Wang, Haihua Yang, Biye Li, Cheng Cheng, Weiwei Lü, Rui Hu, Chenxia Li, Liu Yang, Xilin Luo, Xuejie Wu, Lunan Liu, Wenjun Cheng, Peng Cheng, Jianhao Zhang, Xiaoyu Zhang, Lei Lin, Xiaokun Wang, Yutuan Ma, Chuanhai Dong, Yanq...

    2023

  40. [49]

    Transformers: State-of-the-art natural language processing

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mari...

    2020

  41. [50]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  42. [51]

    Firefly (flowing fireflies): Chinese conversational large language model

    Jianxin Yang. Firefly (flowing fireflies): Chinese conversational large language model. https://github.com/yangjianxin1/Firefly, 2023

    2023

  43. [52]

    Mammoth2: Scaling instructions from the web

    Xiang Yue, Tuney Zheng, Ge Zhang, and Wenhu Chen. Mammoth2: Scaling instructions from the web. Advances in Neural Information Processing Systems, 2024

    2024

  44. [53]

    Root mean square layer normalization, 2019

    Biao Zhang and Rico Sennrich. Root mean square layer normalization, 2019. URL https://arxiv.org/abs/1910.07467

    2019 arXiv

  45. [55]

    Map-neo: Highly capable and transparent bilingual large language model series

    Ge Zhang, Scott Qu, Jiaheng Liu, Chenchen Zhang, Chenghua Lin, Chou Leuang Yu, Danny Pan, Esther Cheng, Jie Liu, Qunshu Lin, et al. Map-neo: Highly capable and transparent bilingual large language model series. arXiv preprint arXiv:2405.19327, 2024 b

    2024 arXiv

  46. [56]

    Tinyllama: An open-source small language model, 2024 c

    Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model, 2024 c . URL https://arxiv.org/abs/2401.02385

    2024 arXiv

  47. [57]

    Pytorch fsdp: Experiences on scaling fully sharded data parallel, 2023

    Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, Alban Desmaison, Can Balioglu, Pritam Damania, Bernard Nguyen, Geeta Chauhan, Yuchen Hao, Ajit Mathews, and Shen Li. Pytorch fsdp: Experiences on sc...

    2023 arXiv

  48. [58]

    Opencodeinterpreter: Integrating code generation with execution and refinement, 2025

    Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and Xiang Yue. Opencodeinterpreter: Integrating code generation with execution and refinement, 2025. URL https://arxiv.org/abs/2402.14658

    2025 arXiv

  49. [59]

    Lima: Less is more for alignment, 2023

    Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. Lima: Less is more for alignment, 2023. URL https://arxiv.org/abs/2305.11206

    2023 arXiv

  50. [60]

    Designing effective sparse expert models

    Barret Zoph. Designing effective sparse expert models. In 2022 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), pp.\ 1044--1044, 2022. doi:10.1109/IPDPSW55747.2022.00171

    2022

  51. [61]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  52. [62]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  53. [63]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  54. [64]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.