pith. sign in

arxiv: 2606.15079 · v1 · pith:5TJKCSB7new · submitted 2026-06-13 · 💻 cs.CL · cs.AI

Ling and Ring 2.6 Technical Report: Efficient and Instant Agentic Intelligence at Trillion-Parameter Scale

Ang Li , Ben Liu , Bin Han , Bin Hu , Bin Jing , Binbin Hu , Bing Li , Cai Chen
show 210 more authors
Caizhi Tang Changxin Tian Chao Huang Chao Zhang Chen Liang Chen Qian Chengfu Tang Chengyao Wen Chilin Fu Chunwei Wu Cong Zhang Cunyin Peng Daixin Wang Dalong Zhang Deng Zhao Dingnan Jin Dingyuan Zhu Donghao Zhang Fan Yuan Fangzheng Zhao Fanzhuang Meng Feifan Wu Feng Xu Fengbin Fang Gangshan Wang Guodong Yang Hailin Zhao Haitao Wang Haitao Zhang Hanxiao Zhang Hanzi Wang Hao Dai Hao Liu Hao Qian Hao Wu Haoxiong Liu Haoyu Xu Heng Zhang Hong Liu Hongliang Zhang Hongrui Liu Hongxun Li Hongzhi Ruan Huaidong Xiong Huihuang Zheng Huikang Tang Jia Guo Jia Li Jia Liu Jiameng Wang Jiaming Liu Jiannan Shi Jianping Wei Jiaolong Yang Jiapeng Wang Jie Gao Jie Wang Jiewei Wu Jin Yang Jinjin Li Jinjing Huang Jinquan Sun Jinyao Chen Juanhui Tu Jun Liu Jun Mei Jun Xu Jun Zhou Junjie Ou Junnan Sipan Junpeng Fang Kaihong Zhang Kaiqin Hu Ke Shi Kuan Xu Kun Tang Kunlong Chen Lanyin Mei Lei Chen Lei Liang Lei Xu Li Tang Liang Jiang Liangcheng Fu Lihui Zhang Linfeng Shi Lintao Ma Liyuan Liu Longfei Li Longfei Zheng Lu Liu Lu Yu Man Li Meiqi Zhu Meng Li Mengjie Gao Mengshu Sun Mingming Yin Mingyang Zhang Mingyuan Fan Nuo Xu Pan Tang Peijie Jiang Peilong Zhao Peng Lin Pingping Liu Qi Zuo Qian Zhao Qiang Cheng Qianggang Cao Qiaoben Bao Qing Cui Qingyuan Yang Qitao Shi Qiyin Huang Qizheng Zhou Quan Wan Runyuan Zhao Shaomian Zheng Shaowei Wei Shengnan Zhang Shuaicheng Li Shujie Li Shuo Zhang Sikang Bian Tianchu Yao Tiange Xu Tianshu Wang Ting Guo Tinghao Wang Tingwei Huang Tong Zhao Tongkai Yang Wang Hong Wanli Gu Wei Lu Weichang Wu Weiguang Han Weiquan Li Wenbo Shen Wenjing Fang Wenzhi Tang Xiang Shu Xiao Shi Xiaodong Yan Xiaolu Zhang Xiaopei Wan Xiaqing Sun Xin Zhao Xingyu Lu Xinxing Yang Xinyao Tang Xinyu Kong Xinyu Liu Xiong Xu Xuan Sun Xudong Han Xudong Wang Xujie Shen Yalin Zhang Yangyang Hou Yankun Ren Yao Zhao Ye Chen Yeyang Chen Yibo Cao Yifan Zuo Yijie Chen Ying Li Yingjie Song Yingxue Li Yiqi Wang Yixuan Sun Yizhu Xiao Yongfei Xu Yu Liu Yuchen Fang Yue Gao Yue Yu Yue Zhang Yuqi Zhang Yuxiao He Yuxiao Lu Yuxin Tian Yuxuan Li Yuzhuo Fu Zhankai Xu Zhaoxin Huan Zhenduo Zhang Zhengke Gui Zhengyu Huang Zhenjun Ma Zhenxuan Pan Zheping Qu Zhibo Zhu Zhidong Fan Zhigang Huangfu Zhihao Wang Zhiqiang Zhang Zhizhen Liu Zhuyan Zhou Zibin Lin Zihang Zeng Zihao Wang Zilong Wang Ziqi Liu Zitao Xuan Zixuan Cheng Zujie Wen Zuoli Tang
This is my paper

Pith reviewed 2026-07-02 22:08 UTC · model grok-4.3

classification 💻 cs.CL cs.AI
keywords agentic intelligencelarge language modelshybrid linear attentionreinforcement learningmodel optimizationtrillion-parameter scaleopen-source modelsefficient serving
0
0 comments X

The pith

Ling-2.6 and Ring-2.6 upgrade the Ling-2.0 base through architectural migration and targeted post-training to deliver efficient instant responses and advanced agentic reasoning at trillion-parameter scale.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents Ling-2.6 and Ring-2.6 as models built from the Ling-2.0 base via architectural migration pre-training and large-scale post-training. It aims to establish that a single co-design process spanning model architecture, optimization objectives, serving systems, and agent training environments can produce both low-latency generation and strong performance on agentic tasks without training new models from scratch. A hybrid linear attention design combines Lightning Attention with MLA to ease long-context work, while Evolutionary Chain-of-Thought, Linguistic Unit Policy Optimization, bidirectional preference alignment, and shortest-correct-response distillation raise capability per output token. KPop supplies a reinforcement learning method that trains Ring-2.6-1T stably on large environment-grounded data through asynchronous scheduling across coding, search, tool use, and workflow tasks. Readers would care because the report claims this route yields practical, scalable, and open agentic systems at very large sizes.

Core claim

By upgrading the Ling-2.0 base model with architectural migration pre-training and large-scale post-training under a unified co-design of architecture, objectives, serving, and agent environments, Ling-2.6 achieves instant response generation and high capability per output token while Ring-2.6 supports deeper reasoning and advanced agentic workflows, all at trillion-parameter scale.

What carries the argument

Unified co-design of model architecture, optimization objectives, serving systems, and agent training environments, which incorporates a hybrid linear attention design and the KPop reinforcement learning framework.

If this is right

  • Ling-2.6 produces instant responses with higher capability per output token than the base.
  • Ring-2.6 handles deeper reasoning and more advanced agentic workflows.
  • KPop enables stable reinforcement learning on large-scale environment-grounded data through asynchronous scheduling across multiple task types.
  • The overall approach improves both model capability and deployment efficiency at trillion-parameter scale.
  • Open-sourcing the 2.6 family checkpoints supports further development of practical agentic systems.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Similar migration methods could shorten the path from existing base models to new specialized agent variants without full retraining.
  • The focus on serving systems inside the co-design may produce better real-world latency than architecture changes alone.
  • Open release of the checkpoints could let external groups test the KPop framework on their own agent environments.

Load-bearing premise

The architectural migration pre-training and large-scale post-training applied to the Ling-2.0 base model, together with the listed techniques, will deliver the stated gains in capability per token and agentic performance.

What would settle it

A side-by-side evaluation in which Ling-2.6 or Ring-2.6 shows no improvement in response latency, capability per token, or success rate on agent-environment tasks compared with the unmodified Ling-2.0 base would falsify the central claim.

read the original abstract

Efficient and scalable agentic intelligence requires models that can deliver both low-latency responses and strong reasoning capabilities while remaining practical to train, serve, and deploy. In this report, we present Ling-2.6 and Ring-2.6, a family of models designed to address this challenge at scale. Ling-2.6 is optimized for instant response generation and high capability per output token, whereas Ring-2.6 is tailored for deeper reasoning and more advanced agentic workflows. Instead of training from scratch, we upgrade the Ling-2.0 base model through architectural migration pre-training and large-scale post-training. This upgrade is guided by a unified co-design of model architecture, optimization objectives, serving systems, and agent training environments, enabling improvements in both model capability and deployment efficiency. At the architectural level, we introduce a hybrid linear attention design that integrates Lightning Attention with MLA, improving the efficiency of long-context training and decoding. To further enhance token efficiency, we optimize capability per output token through Evolutionary Chain-of-Thought, Linguistic Unit Policy Optimization, bidirectional preference alignment, and shortest-correct-response distillation. For agentic capabilities, we propose KPop, a reinforcement learning framework designed to support stable training of Ring-2.6-1T on large-scale environment-grounded data. KPop improves training efficiency through asynchronous scheduling across coding, search, tool use, and workflow execution, enabling scalable learning from complex agent-environment interactions. Together, Ling-2.6 and Ring-2.6 provide a practical pathway toward efficient, scalable, and open agentic systems. We open-source all checkpoints in the 2.6 family to support further research and development in practical agentic intelligence.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 0 minor

Summary. The manuscript presents Ling-2.6 and Ring-2.6 as upgrades to the Ling-2.0 base model via architectural migration pre-training and large-scale post-training. It describes a hybrid Lightning+MLA attention mechanism for long-context efficiency, token-efficiency methods (Evolutionary Chain-of-Thought, Linguistic Unit Policy Optimization, bidirectional preference alignment, shortest-correct-response distillation), and the KPop RL framework for stable training on agent-environment interactions. The central claim is that this unified co-design yields efficient, scalable agentic intelligence at trillion-parameter scale, with all 2.6-family checkpoints open-sourced.

Significance. If the claimed gains in capability per token and agentic performance are substantiated, the work would offer a notable contribution by demonstrating co-design across architecture, objectives, serving, and training environments for practical large-scale agentic systems. The open-sourcing of checkpoints is an explicit strength that directly supports reproducibility and further research.

major comments (1)
  1. [Abstract] Abstract and technical description sections: the manuscript asserts that the listed techniques produce measurable improvements in capability per output token and agentic performance, yet supplies no benchmark numbers, baseline comparisons against Ling-2.0, latency measurements, ablation results, or error bars to establish any causal link between the techniques and the asserted gains.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the constructive feedback highlighting the need for stronger quantitative grounding of our claims. We will revise the manuscript to address this directly.

read point-by-point responses

  1. Referee: [Abstract] Abstract and technical description sections: the manuscript asserts that the listed techniques produce measurable improvements in capability per output token and agentic performance, yet supplies no benchmark numbers, baseline comparisons against Ling-2.0, latency measurements, ablation results, or error bars to establish any causal link between the techniques and the asserted gains.

    Authors: We agree that the abstract and technical overview sections would be strengthened by including explicit quantitative evidence. The full experimental sections contain benchmark comparisons to Ling-2.0, latency measurements, and ablation studies; however, these were not summarized in the abstract or high-level descriptions. In the revised version we will add concise benchmark numbers, baseline deltas, latency figures, and error-bar summaries to the abstract and technical description sections, with pointers to the detailed tables and ablations. revision: yes

Circularity Check

0 steps flagged

No derivation chain or equations present; claims are descriptive assertions without mathematical reduction or self-referential inputs.

full rationale

The paper is a technical report describing upgrades to the Ling-2.0 base model via architectural migration, hybrid Lightning+MLA attention, Evolutionary Chain-of-Thought, Linguistic Unit Policy Optimization, bidirectional preference alignment, shortest-correct-response distillation, and the KPop RL framework. No equations, formal derivations, fitted parameters, or load-bearing self-citations appear in the abstract or described content. Central claims about capability gains and agentic performance are asserted without quantitative modeling, benchmarks, or reductions to prior inputs by construction. This is the most common honest finding for descriptive reports lacking a derivation chain.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract introduces named techniques (hybrid linear attention, KPop, Evolutionary Chain-of-Thought) but supplies no free parameters, axioms, or invented entities with independent evidence.

pith-pipeline@v0.9.1-grok · 6713 in / 1031 out tokens · 23987 ms · 2026-07-02T22:08:45.534973+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

237 extracted references · 128 canonical work pages · 62 internal anchors

  1. [1]

    Skywork Open Reasoner 1 Technical Report

    Skywork Open Reasoner 1 Technical Report , author=. arXiv preprint arXiv:2505.22312 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  2. [2]

    2025 , eprint=

    GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models , author=. 2025 , eprint=

    2025

  3. [4]

    Understanding R1-Zero-Like Training: A Critical Perspective

    Understanding r1-zero-like training: A critical perspective , author=. arXiv preprint arXiv:2503.20783 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  4. [5]

    2023 , eprint=

    Adding Conditional Control to Text-to-Image Diffusion Models , author=. 2023 , eprint=

    2023

  5. [6]

    2021 , url=

    Wang, Yue and Wang, Weishi and Joty, Shafiq and Hoi, Steven CH , booktitle=. 2021 , url=

    2021

  6. [7]

    The Open Images Dataset V4 , journal=

    Alina Kuznetsova and Hassan Rom and Neil Alldrin and Jasper Uijlings and Ivan Krasin and Jordi Pont-Tuset and Shahab Kamali and Stefan Popov and Matteo Malloci and Alexander Kolesnikov and Tom Duerig and Vittorio Ferrari , url=. The Open Images Dataset V4 , journal=. 2020 , volume=

    2020

  7. [8]

    GPQA: A Graduate-Level Google-Proof Q&A Benchmark

    GPQA: A Graduate-Level Google-Proof Q&A Benchmark , author=. arXiv preprint arXiv:2311.12022 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  8. [9]

    The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore , year=

    LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code , author=. The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore , year=

    2025

  9. [10]

    2024 , url=

    Jimenez, Carlos E and Yang, John and Wettig, Alexander and Yao, Shunyu and Pei, Kexin and Press, Ofir and Narasimhan, Karthik R , booktitle=. 2024 , url=

    2024

  10. [11]

    SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?

    SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks? , author=. arXiv preprint arXiv:2509.16941 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  11. [12]

    Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving

    Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving , author=. arXiv preprint arXiv:2504.02605 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  12. [13]

    OpenHands: An Open Platform for AI Software Developers as Generalist Agents

    Wang, Xingyao and Jiang, Boxuan and Lu, Yufan and Shao, Frank F. and Liu, Jiaju and Iong, Io Kei and Zhang, Ruisheng and Shi, Tianjia and Nikoli. OpenHands: An Open Platform for. arXiv preprint arXiv:2407.16741 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  13. [14]

    ARC-AGI-2: A New Challenge for Frontier

    Chollet, Fran. ARC-AGI-2: A New Challenge for Frontier. CoRR , volume=

  14. [15]

    GAIA: a benchmark for General AI Assistants

    Mialon, Gr. GAIA: A Benchmark for General. arXiv preprint arXiv:2311.12983 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  15. [16]

    $\tau$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains

    -bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains , author=. arXiv preprint arXiv:2406.12045 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  16. [17]

    $\tau^2$-Bench: Evaluating Conversational Agents in a Dual-Control Environment

    ^2 -bench: Evaluating Conversational Agents in a Dual-Control Environment , author=. arXiv preprint arXiv:2506.07982 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  17. [18]

    AIME 2026 Problems and Solutions , howpublished=

    2026

  18. [19]

    Kimi K2.5: Visual Agentic Intelligence

    Kimi K2.5: Visual Agentic Intelligence , author=. arXiv preprint arXiv:2602.02276 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  19. [20]

    Kimi K2: Open Agentic Intelligence

    Kimi K2: Open Agentic Intelligence , author=. arXiv preprint arXiv:2507.20534 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  20. [21]

    OpenAI GPT-5 System Card

    OpenAI GPT-5 System Card , author=. arXiv preprint arXiv:2601.03267 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  21. [22]

    YaRN: Efficient Context Window Extension of Large Language Models

    YaRN: Efficient Context Window Extension of Large Language Models , author=. arXiv preprint arXiv:2309.00071 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  22. [23]

    MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark

    MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark , author=. arXiv preprint arXiv:2406.01574 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  23. [24]

    Advances in Neural Information Processing Systems , volume=

    C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models , author=. Advances in Neural Information Processing Systems , volume=

  24. [25]

    arXiv preprint arXiv:2412.04604 , year=

    ARC Prize 2024: Technical Report , author=. arXiv preprint arXiv:2412.04604 , year=

    work page arXiv 2024

  25. [26]

    Every step evolves: Scaling reinforcement learning for trillion-scale thinking model.arXiv preprint arXiv:2510.18855,

    Every Step Evolves: Scaling Reinforcement Learning for Trillion-Scale Thinking Model , author =. arXiv preprint arXiv:2510.18855 , year =

    work page arXiv

  26. [27]

    AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning

    AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning , author =. arXiv preprint arXiv:2505.24298 , year =

    work page internal anchor Pith review Pith/arXiv arXiv

  27. [28]

    arXiv preprint arXiv:2404.07353 , year=

    Addressing the abstraction and reasoning corpus via procedural example generation , author=. arXiv preprint arXiv:2404.07353 , year=

    work page arXiv

  28. [29]

    InternBootcamp Technical Report: Boosting LLM Reasoning with Verifiable Task Scaling

    Internbootcamp technical report: Boosting LLM reasoning with verifiable task scaling , author=. arXiv preprint arXiv:2508.08636 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  29. [30]

    arXiv preprint arXiv:2505.19641 , year=

    Synlogic: Synthesizing verifiable reasoning data at scale for learning logical reasoning and beyond , author=. arXiv preprint arXiv:2505.19641 , year=

    work page arXiv

  30. [31]

    2025 , journal=

    Enigmata: A synthetic puzzle generation and verification framework , author=. 2025 , journal=. 2505.19914 , archivePrefix=

    work page arXiv 2025

  31. [32]

    2025 , howpublished=

    NVARC , author=. 2025 , howpublished=

    2025

  32. [33]

    KPop: Taming Training–Inference Mismatch in Reinforcement Learning with Adaptive Masking Regions , url =

    Jia Guo and Yan Sun and Zhenyu Huang and Zihao Wang and Zujie Wen and Zhiqiang Zhang and Jun Zhou and Stanley Kok , year =. KPop: Taming Training–Inference Mismatch in Reinforcement Learning with Adaptive Masking Regions , url =

  33. [34]

    2024 , eprint=

    MARIO: MAth Reasoning with code Interpreter Output -- A Reproducible Pipeline , author=. 2024 , eprint=

    2024

  34. [35]

    Junlong Li and Daya Guo and Dejian Yang and Runxin Xu and Yu Wu and Junxian He , booktitle=. Code. 2025 , url=

    2025

  35. [36]

    ArXiv , year=

    Efficient Training of Language Models to Fill in the Middle , author=. ArXiv , year=

  36. [37]

    Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities , author=. arXiv preprint arXiv:2507.06261 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  37. [38]

    2026 , eprint=

    Focal Reward: Balanced Reinforcement Learning under Rubric-Based Rewards , author=. 2026 , eprint=

    2026

  38. [39]

    2026 , eprint=

    ReportLogic: Evaluating Logical Quality in Deep Research Reports , author=. 2026 , eprint=

    2026

  39. [40]

    2025 , eprint=

    Every Activation Boosted: Scaling General Reasoner to 1 Trillion Open Language Foundation , author=. 2025 , eprint=

    2025

  40. [41]

    2025 , eprint=

    Every Step Evolves: Scaling Reinforcement Learning for Trillion-Scale Thinking Model , author=. 2025 , eprint=

    2025

  41. [42]

    Kimi k1.5: Scaling Reinforcement Learning with LLMs

    Kimi k1. 5: Scaling reinforcement learning with llms , author=. arXiv preprint arXiv:2501.12599 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  42. [43]

    2025 , month =

    OpenAI , title =. 2025 , month =

    2025

  43. [44]

    1999 , publisher=

    Byte pair encoding: A text compression scheme that accelerates pattern matching , author=. 1999 , publisher=

    1999

  44. [45]

    2023 , eprint=

    L-Eval: Instituting Standardized Evaluation for Long Context Language Models , author=. 2023 , eprint=

    2023

  45. [46]

    The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants

    Bandarkar, Lucas and Liang, Davis and Muller, Benjamin and Artetxe, Mikel and Shukla, Satya Narayan and Husa, Donald and Goyal, Naman and Krishnan, Abhinandan and Zettlemoyer, Luke and Khabsa, Madian. The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants. Proceedings of the 62nd Annual Meeting of the Association for Com...

    work page doi:10.18653/v1/2024.acl-long.44 2024

  46. [47]

    LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding

    Longbench: A bilingual, multitask benchmark for long context understanding , author=. arXiv preprint arXiv:2308.14508 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  47. [48]

    arXiv preprint arXiv:2409.12640 , year=

    Michelangelo: Long context evaluations beyond haystacks via latent structure queries , author=. arXiv preprint arXiv:2409.12640 , year=

    work page arXiv

  48. [49]

    L ong B ench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks

    Bai, Yushi and Tu, Shangqing and Zhang, Jiajie and Peng, Hao and Wang, Xiaozhi and Lv, Xin and Cao, Shulin and Xu, Jiazheng and Hou, Lei and Dong, Yuxiao and Tang, Jie and Li, Juanzi. L ong B ench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks. Proceedings of the 63rd Annual Meeting of the Association for Computational...

    work page doi:10.18653/v1/2025.acl-long.183 2025

  49. [50]

    arXiv preprint arXiv:2508.18824 , year=

    Arrows of Math Reasoning Data Synthesis for Large Language Models: Diversity, Complexity and Correctness , author=. arXiv preprint arXiv:2508.18824 , year=

    work page arXiv

  50. [51]

    2025 , eprint=

    MaP: A Unified Framework for Reliable Evaluation of Pre-training Dynamics , author=. 2025 , eprint=

    2025

  51. [52]

    arXiv preprint arXiv:2503.00812 , year=

    BOSE: A Systematic Evaluation Method Optimized for Base Models , author=. arXiv preprint arXiv:2503.00812 , year=

    work page arXiv

  52. [53]

    Group Sequence Policy Optimization

    Group sequence policy optimization , author=. arXiv preprint arXiv:2507.18071 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  53. [54]

    DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models , author=. arXiv preprint arXiv:2402.03300 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  54. [55]

    Muon is Scalable for LLM Training

    Muon is scalable for LLM training , author=. arXiv preprint arXiv:2502.16982 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  55. [56]

    arXiv preprint arXiv:2010.04245 , year=

    Query-key normalization for transformers , author=. arXiv preprint arXiv:2010.04245 , year=

    work page arXiv 2010

  56. [57]

    Better & Faster Large Language Models via Multi-token Prediction

    Better & faster large language models via multi-token prediction , author=. arXiv preprint arXiv:2404.19737 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  57. [58]

    Deep Learning Scaling is Predictable, Empirically

    Deep learning scaling is predictable, empirically , author=. arXiv preprint arXiv:1712.00409 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  58. [59]

    Scaling Laws for Neural Language Models

    Scaling laws for neural language models , author=. arXiv preprint arXiv:2001.08361 , year=

    work page internal anchor Pith review Pith/arXiv arXiv 2001

  59. [60]

    Training Compute-Optimal Large Language Models

    Training compute-optimal large language models , author=. arXiv preprint arXiv:2203.15556 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  60. [61]

    DeepSeek LLM: Scaling Open-Source Language Models with Longtermism

    Deepseek llm: Scaling open-source language models with longtermism , author=. arXiv preprint arXiv:2401.02954 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  61. [62]

    The Llama 3 Herd of Models

    The llama 3 herd of models , author=. arXiv preprint arXiv:2407.21783 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  62. [63]

    arXiv preprint arXiv:2403.08540 , year=

    Language models scale reliably with over-training and on downstream tasks , author=. arXiv preprint arXiv:2403.08540 , year=

    work page arXiv

  63. [64]

    arXiv preprint arXiv:2406.12907 , year=

    Reconciling kaplan and chinchilla scaling laws , author=. arXiv preprint arXiv:2406.12907 , year=

    work page arXiv

  64. [65]

    arXiv preprint arXiv:2503.04715 , year=

    Predictable Scale: Part I--Optimal Hyperparameter Scaling Law in Large Language Model Pretraining , author=. arXiv preprint arXiv:2503.04715 , year=

    work page arXiv

  65. [66]

    arXiv preprint arXiv:2410.08527 , year=

    Scaling laws for predicting downstream performance in LLMs , author=. arXiv preprint arXiv:2410.08527 , year=

    work page arXiv

  66. [67]

    arXiv preprint arXiv:2405.10938 , year=

    Observational scaling laws and the predictability of language model performance , author=. arXiv preprint arXiv:2405.10938 , year=

    work page arXiv

  67. [68]

    arXiv preprint arXiv:2401.00448 , year=

    Beyond chinchilla-optimal: Accounting for inference in language model scaling laws , author=. arXiv preprint arXiv:2401.00448 , year=

    work page arXiv

  68. [69]

    arXiv preprint arXiv:2404.09937 , year=

    Compression represents intelligence linearly , author=. arXiv preprint arXiv:2404.09937 , year=

    work page arXiv

  69. [70]

    The Thirteenth International Conference on Learning Representations , year=

    Scaling laws for downstream task performance in machine translation , author=. The Thirteenth International Conference on Learning Representations , year=

  70. [71]

    arXiv preprint arXiv:2310.03262 , year=

    Predicting Emergent Abilities with Infinite Resolution Evaluation , author=. arXiv preprint arXiv:2310.03262 , year=

    work page arXiv

  71. [72]

    Journal of Machine Learning Research , volume=

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity , author=. Journal of Machine Learning Research , volume=

  72. [73]

    GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding

    Gshard: Scaling giant models with conditional computation and automatic sharding , author=. arXiv preprint arXiv:2006.16668 , year=

    work page internal anchor Pith review Pith/arXiv arXiv 2006

  73. [74]

    Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer , author=. arXiv preprint arXiv:1701.06538 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  74. [75]

    International conference on machine learning , pages=

    Unified scaling laws for routed language models , author=. International conference on machine learning , pages=. 2022 , organization=

    2022

  75. [76]

    Forty-first International Conference on Machine Learning , year=

    Scaling laws for fine-grained mixture of experts , author=. Forty-first International Conference on Machine Learning , year=

  76. [77]

    arXiv preprint arXiv:2410.05661 , year=

    Scaling laws across model architectures: A comparative analysis of dense and MoE models in large language models , author=. arXiv preprint arXiv:2410.05661 , year=

    work page arXiv

  77. [78]

    arXiv preprint arXiv:2501.12370 , year=

    Parameters vs flops: Scaling laws for optimal sparsity for mixture-of-experts language models , author=. arXiv preprint arXiv:2501.12370 , year=

    work page arXiv

  78. [79]

    arXiv preprint arXiv:2410.19034 , year=

    Mixture of parrots: Experts improve memorization more than reasoning , author=. arXiv preprint arXiv:2410.19034 , year=

    work page arXiv

  79. [80]

    arXiv preprint arXiv:2404.02852 , year=

    Toward inference-optimal mixture-of-expert large language models , author=. arXiv preprint arXiv:2404.02852 , year=

    work page arXiv

  80. [81]

    DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models

    Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models , author=. arXiv preprint arXiv:2401.06066 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

Showing first 80 references.