Pith. sign in

REVIEW 3 major objections 5 minor 142 references

DeepSeek: Paradigm Shifts and Technical Evolution in Large AI Models

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A review argues that DeepSeek-V3 and R1 mark a paradigm shift in LLM research from raw scale to algorithmic and data-driven efficiency.

desk verdict A faithful but uncritical survey of DeepSeek that reproduces the key equations, yet repeats vendor claims without caveats and misstates the model license. read the letter →

arxiv 2507.09955 v1 pith:QEZ36GER submitted 2025-07-14 cs.AI

Luolin Xiong , Haofen Wang , Xi Chen , Lu Sheng , Yun Xiong , Jingping Liu , Yanghua Xiao , Huajun Chen
show 2 more authors
Qing-Long Han Yang Tang
This is my paper
classification cs.AI
keywords DeepSeeklargelanguagemodelsmixtureofexpertsmulti-headlatentattentionmulti-tokenpredictiongrouprelativepolicyoptimizationreinforcementlearningreasoning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the release of DeepSeek-V3 and DeepSeek-R1 marks a paradigm shift in large-language-model research: progress is now driven by algorithmic innovation and data quality rather than primarily by increasing parameters and compute. It makes the case by reviewing four DeepSeek techniques—MLA, DeepSeekMoE, MTP, and GRPO—and by describing engineering optimizations that cut training and inference costs. A fair reader would care because the claim, if true, means open-source and resource-constrained teams can reach frontier-level reasoning without trillion-parameter-scale budgets, rebalancing competitive pressure toward algorithms, data, and systems. The paper also highlights DeepSeek-R1-Zero as the first model to attempt improving reasoning through pure large-scale reinforcement learning, without supervised fine-tuning or model-based rewards.

What carries the argument

Four named mechanisms carry the argument. Multi-head Latent Attention (MLA) compresses keys and values into a low-rank latent vector so only compressed key-value pairs are cached, reducing inference memory without the accuracy loss associated with multi-query and grouped-query attention. DeepSeekMoE replaces dense feed-forward layers with fine-grained shared experts, always active, plus sparsely routed experts, aiming for specialization without knowledge redundancy and with auxiliary-loss-free load balancing. Multi-Token Prediction (MTP) predicts several future tokens sequentially from each position, with ground-truth tokens fed into the next module to preserve the causal chain and increase the density of training signals. Group Relative Policy Optimization (GRPO) samples a group of outputs per question, normalizes rewards within the group, and optimizes the policy with a direct KL regularizer, eliminating the large value model that PPO requires. These four mechanisms are what make the claimed combination of frontier-level performance and low cost possible.

What would settle it

One concrete test is to train a mid-size open base model with pure GRPO—no supervised fine-tuning and no model-based rewards—on verifiable math and code tasks, and check whether long chain-of-thought reasoning emerges as R1-Zero's report describes. Another is to have an independent team audit the reported training cost and benchmark numbers for DeepSeek-V3; if reproducing the stated results requires substantially more compute or yields materially lower scores, the efficiency-based paradigm claim weakens.

Watch

Extended reading notes

Core claim

On the paper's own account, the central discovery is that DeepSeek-V3, a 671-billion-parameter mixture-of-experts model with 37 billion active parameters per token, reaches performance that surpasses other open-source models and matches leading closed-source models, while DeepSeek-R1 is roughly on par with the closed-source o1 reasoning series. These results are taken as evidence for a new LLM paradigm centered on efficiency: low-rank latent attention (MLA) shrinks the KV cache, fine-grained shared and routed experts in DeepSeekMoE control compute, multi-token prediction (MTP) densifies training signal, and GRPO removes the value model from policy optimization. The paper further claims that DeepSeek-R1-Zero is the first demonstration that reasoning ability can be improved solely through large-scale GRPO reinforcement learning, without supervised fine-tuning or model-based rewards, with the model spontaneously learning to spend more time re-evaluating its approach.

Load-bearing premise

The load-bearing premise is that DeepSeek's own technical reports accurately and completely describe the architectures, training costs, and benchmark results; if those vendor self-reports overstate performance or understate cost, the paradigm-shift claim loses its factual foundation.

Editorial extensions

If this is right

  • If DeepSeek-V3's reported costs hold, training a frontier-level open-weights MoE model becomes feasible for organizations far smaller than the largest AI labs.
  • If R1-Zero's pure-RL result is real, reasoning skill can be incentivized rather than taught, and SFT may not be a prerequisite for strong reasoning.
  • If MLA and fine-grained MoE deliver their stated efficiency, long-context and long-chain-of-thought inference will shift from memory-bound to algorithm-bound, reshaping hardware and serving-system priorities.
  • If the paradigm shift is genuine, competitive advantage in LLMs moves from raw parameter scale toward data curation, synthetic data, and reinforcement-learning recipe design.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the paper's five-stage periodization implies a testable prediction—most near-term LLM gains will come from training and inference algorithms and data pipelines rather than from simply scaling parameters, so new SOTA results should arrive with flat or modest compute growth.
  • Editorial extension: R1-Zero's pure-RL result suggests a research program the paper only gestures at: studying when reward-signal design alone can elicit emergent behaviors, and whether the approach transfers to domains without verifiable accuracy rewards, such as open-ended dialogue or scientific discovery.
  • Editorial extension: the paper's reliance on vendor reports implies that independent, reproducible evaluations of DeepSeek's benchmark claims would either solidify or undercut the paradigm-shift narrative; that is the cheapest way for the community to test the paper's main assertion.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This manuscript is a survey-style review of the DeepSeek-V3 and DeepSeek-R1 model families, organized around a claimed paradigm shift in large-language-model research from raw scale toward algorithm optimization and data quality. The paper reviews the historical evolution of LLMs, describes the DeepSeek training pipeline, reproduces the four headline algorithms (MLA, DeepSeekMoE, MTP, and GRPO), surveys engineering optimizations for training and inference, and discusses the competitive and societal impact of DeepSeek models. The central claims are that DeepSeek-V3/R1 achieve performance comparable to leading closed-source models at a fraction of the cost, that R1-Zero demonstrates pure reinforcement learning without SFT, and that these results are redirecting the field toward efficiency-oriented research.

Significance. If its factual premises are accurate, this is a useful and broadly competent exposition of an important recent development in applied LLM research. The paper's reproductions of the core equations for MLA, DeepSeekMoE, MTP, and GRPO are largely faithful to DeepSeek's technical reports, and the structured comparison of training pipelines is informative for readers outside the immediate field. The manuscript derives no new results of its own, so circularity is not a concern; its value is descriptive and organizational. However, the significance of the paradigm-shift claim rests almost entirely on performance and cost figures taken from DeepSeek's own technical reports, which the paper presents without explicit provenance caveats or independent verification. A concrete, checkable error in Section V.A regarding the license terms of the released weights further underscores that the paper does not consistently distinguish vendor claims from independently verifiable facts.

major comments (3)
  1. [Section V.A] The sentence claiming that DeepSeek "fully open-sourced its model weights and inference parameters ... under the MIT license, granting global developers unrestricted rights" is factually incorrect. DeepSeek's code repositories are MIT-licensed, but the model weights are released under DeepSeek's own custom model license, which is not MIT and carries additional conditions. Because the democratization argument in this subsection rests on the unrestricted-rights framing, the claim must be corrected and the distinction between code and weights stated explicitly.
  2. [Sections II.C, III, IV] Performance and cost statements such as "DeepSeek-V3 surpasses other open-source models" and "DeepSeek-R1-Zero performs on par with the closed-source LLM series o1" are presented as established facts, but their cited sources are DeepSeek's own technical reports. The paradigm-shift argument in Section I depends on these comparisons, so the authors should clearly label all such figures as vendor-reported, state which numbers have independent third-party confirmation, and ideally include a short table or discussion of third-party evaluations and their limitations.
  3. [Table II] The five-stage periodization of LLM paradigms is asserted without a stated methodology for choosing the stages, their starting dates, or the assignment of specific models to stages. For example, GPT-2 is placed in the "Transformer and Pre-training Models" stage while GPT-3 is placed in "Scaling-Up of LLMs", and DeepSeek-V3 is listed in the scaling stage even though Section I frames DeepSeek as a departure from scaling-centric research. The authors should either provide explicit periodization criteria or soften the paradigm-shift framing to reflect that the boundaries are illustrative.
minor comments (5)
  1. [Section III.A, Eq. (1)] The notation in Eq. (1) is confusing: "q_t = W_Q h_t → q_t = W_UQ(W_DQ h_t)" uses an arrow where a clear replacement or conditional definition is needed, and the same symbol q_t is defined twice. Please rewrite to distinguish the standard MHA formulation from the MLA reformulation.
  2. [Section III.D, Eq. (12)] The expression labeled D_KL(πθ||πref) in Eq. (12) is a sample-based estimator of the KL divergence used in GRPO, not the KL divergence itself; the manuscript should state this explicitly so readers are not misled about the mathematical definition.
  3. [Table I and Section II.C] There are several typos: "actived" in Table I should be "activated", "perfroming" in Section II.C should be "performing", and "serval" in Section VI.A should be "several".
  4. [Throughout] Minor inconsistencies include "Open AI" instead of "OpenAI" and "LlaMa" instead of "Llama"; these should be standardized.
  5. [Section IV.B] The claim that FP8 relative error stays "under 0.25%" is reported without specifying which baseline model sizes or training setups this was measured on; a brief qualifier or a reference to the specific experiment in the DeepSeek-V3 report would improve traceability.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is an expository survey of DeepSeek's primary technical reports, with no self-derived predictions or fitted inputs.

full rationale

The paper's central claims, architecture descriptions, benchmark comparisons, and cost figures are all attributed to DeepSeek's own technical reports and public repositories, which are external primary sources authored by DeepSeek rather than by the present authors. The survey derives nothing of its own: there are no fitted parameters, no equations whose outputs are defined in terms of their inputs, and no prediction that is statistically forced. The descriptions of MLA, DeepSeekMoE, MTP, and GRPO are restatements of the cited reports' equations and results, not new derivations that could reduce to their inputs. The periodization in Table II is asserted without a stated methodology, but that is a methodological transparency issue rather than a circular derivation. The inaccurate statement in Section V.A that DeepSeek released model weights under the MIT license is a checkable factual error, not circularity. The paper's dependence on vendor self-reported benchmark and cost data is a legitimate concern about independent verification, but it does not make any of the paper's claims circular, because the paper does not construct a result out of its own assumptions and then present it as a finding. The only self-citation in the reference list, [6], is about ChatGPT background and is not load-bearing for any of the paper's central claims. Under the rule that non-circularity is the default and that unverified external sources constitute correctness risk rather than circularity, the appropriate score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

This is a review paper, so the ledger is nearly empty: no free parameters are fit and no entities are invented. The paper's dependencies are external: the accuracy of DeepSeek's technical reports and the fidelity of the reproduced equations. The periodization taxonomy in Table II is an asserted interpretive frame rather than a derived result.

assumptions (3)
  • domain assumption DeepSeek's technical reports [1], [38], [40], [41], [42] accurately report the architectures, training pipelines, benchmark results, and cost figures reproduced in this paper.
    Sections II.C, III, and IV build the entire survey on these vendor sources. The paper performs no independent verification and does not flag the self-report status of the numbers.
  • standard math The equations reproduced in Section III (MLA compression, MoE gating, MTP loss, GRPO objective and KL estimator) are correct transcriptions of the actual mechanisms in DeepSeek-V3/R1 and obey standard linear algebra and probability rules.
    The technical exposition assumes these functional forms are the mechanisms deployed in DeepSeek-V3/R1 and that standard operator identities hold as written.
  • domain assumption The five-stage periodization of LLM history (Table II) with the listed boundary dates is a valid taxonomy of the field.
    Section II.A asserts this periodization without a stated methodology (criteria for stage boundaries, corpus of surveyed models, or selection rule). The 'paradigm shift' framing depends on this taxonomy.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DeepSeek: Paradigm Shifts and Technical Evolution in Large AI Models." pith.science (2026). https://pith.science/paper/QEZ36GER

@misc{pith2026250709955,
  author       = {Pith},
  title        = {Pith review of: DeepSeek: Paradigm Shifts and Technical Evolution in Large AI Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QEZ36GER}},
  note         = {Machine review of arXiv:2507.09955}
}
read the original abstract

DeepSeek, a Chinese Artificial Intelligence (AI) startup, has released their V3 and R1 series models, which attracted global attention due to their low cost, high performance, and open-source advantages. This paper begins by reviewing the evolution of large AI models focusing on paradigm shifts, the mainstream Large Language Model (LLM) paradigm, and the DeepSeek paradigm. Subsequently, the paper highlights novel algorithms introduced by DeepSeek, including Multi-head Latent Attention (MLA), Mixture-of-Experts (MoE), Multi-Token Prediction (MTP), and Group Relative Policy Optimization (GRPO). The paper then explores DeepSeek engineering breakthroughs in LLM scaling, training, inference, and system-level optimization architecture. Moreover, the impact of DeepSeek models on the competitive AI landscape is analyzed, comparing them to mainstream LLMs across various fields. Finally, the paper reflects on the insights gained from DeepSeek innovations and discusses future trends in the technical and engineering development of large AI models, particularly in data, training, and reasoning.

Figures

Figures reproduced from arXiv: 2507.09955 by the authors.

Figure 1
Figure 1. The landscape of large AI models. and public attention to the field of AI. As the capabilities of LLMs like ChatGPT continue to evolve [9], [30], [31], researchers have begun to explore the potential of Large Multimodal Models (LMMs), which extend the scope of AI by integrating and processing multiple modalities of data, such as text, images, and audio. For example, GPT-4 has introduced the ability to process both t… view at source ↗
Figure 2
Figure 2. The training pipeline of the DeepSeek (DS) series models. [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Illustration of Multi-Token Prediction implementation [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Comparison between RLHF and GRPO used in DeepSeek models [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Influence on LLM Landscape and Future Trends. [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

142 extracted references · 16 canonical work pages

  1. [1]

    Deepseek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning,

    D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Biet al., “Deepseek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning,”arXiv preprint arXiv:2501.12948, 2025

    arXiv 2025

  2. [2]

    Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer,

    C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y . Zhou, W. Li, and P. Liu, “Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer,”Journal of Machine Learning Research, vol. 21, no. 140, pp. 1–67, 2020

    2020

  3. [3]

    China’s cheap, open AI model DeepSeek thrills scientists,

    E. Gibney, “China’s cheap, open AI model DeepSeek thrills scientists,” Nature, vol. 638, no. 8049, pp. 13–14, 2025

    2025

  4. [4]

    What to know about DeepSeek and how it is up- ending A.I. - The New York Times,

    C. Metz, “What to know about DeepSeek and how it is up- ending A.I. - The New York Times,” https://www.nytimes.com/ 2025/01/27/technology/what-is-deepseek-china-ai.html, (Accessed on 01/27/2025). 14 IEEE/CAA JOURNAL OF AUTOMATICA SINICA, VOL. X, NO. X, X X

    2025

  5. [5]

    What is DeepSeek, and why is it causing Nvidia and other stocks to slump? - CBS News,

    A. Picchi, “What is DeepSeek, and why is it causing Nvidia and other stocks to slump? - CBS News,” https://www.cbsnews.com/news/ what-is-deepseek-ai-china-stock-nvidia-nvda-asml/, (Accessed on 01/28/2025)

    2025

  6. [6]

    A brief overview of ChatGPT: The history, status quo and potential future development,

    T. Wu, S. He, J. Liu, S. Sun, K. Liu, Q.-L. Han, and Y . Tang, “A brief overview of ChatGPT: The history, status quo and potential future development,”IEEE/CAA Journal of Automatica Sinica, vol. 10, no. 5, pp. 1122–1136, 2023

    2023

  7. [7]

    BERT: Pre- training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre- training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2018

    arXiv 2018

  8. [8]

    Training language models to follow instructions with human feedback,

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Rayet al., “Training language models to follow instructions with human feedback,”Advances in neural information processing systems, vol. 35, pp. 27 730–27 744, 2022

    2022

Show all 142 references
  1. [9]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clarket al., “Learning transferable visual models from natural language supervision,” inProceedings of International Conference on Machine Learning (ICML). PMLR, 2021, pp. 8748–8763

    2021

  2. [10]

    Pre-trained language models for text generation: A survey,

    J. Li, T. Tang, W. X. Zhao, J.-Y . Nie, and J.-R. Wen, “Pre-trained language models for text generation: A survey,”ACM Computing Surveys, vol. 56, no. 9, pp. 1–39, 2024

    2024

  3. [11]

    Recent advances in natural language processing via large pre-trained language models: A survey,

    B. Min, H. Ross, E. Sulem, A. P. B. Veyseh, T. H. Nguyen, O. Sainz, E. Agirre, I. Heintz, and D. Roth, “Recent advances in natural language processing via large pre-trained language models: A survey,”ACM Computing Surveys, vol. 56, no. 2, pp. 1–40, 2023

    2023

  4. [12]

    Large language models versus natural language under- standing and generation,

    N. Karanikolas, E. Manga, N. Samaridi, E. Tousidou, and M. Vassi- lakopoulos, “Large language models versus natural language under- standing and generation,”in Proceedings of the 27th Pan-Hellenic Conference on Progress in Computing and Informatics, pp. 278–290, 2023

    2023

  5. [13]

    Application of deep belief networks for natural language understanding,

    R. Sarikaya, G. E. Hinton, and A. Deoras, “Application of deep belief networks for natural language understanding,”IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 22, no. 4, pp. 778– 784, 2014

    2014

  6. [14]

    PaLM: Scal- ing language modeling with pathways,

    A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmannet al., “PaLM: Scal- ing language modeling with pathways,”Journal of Machine Learning Research, vol. 24, no. 240, pp. 1–113, 2023

    2023

  7. [15]

    Vision-enabled large language and deep learning models for image-based emotion recognition,

    M. Nadeem, S. S. Sohail, L. Javed, F. Anwer, A. K. J. Saudagar, and K. Muhammad, “Vision-enabled large language and deep learning models for image-based emotion recognition,”Cognitive Computation, vol. 16, no. 5, pp. 2566–2579, 2024

    2024

  8. [16]

    A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets,

    K. Bayoudh, R. Knani, F. Hamdaoui, and A. Mtibaa, “A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets,”The Visual Computer, vol. 38, no. 8, pp. 2939–2970, 2022

    2022

  9. [17]

    Deep learning models for digital image processing: A review,

    R. Archana and P. S. E. Jeevaraj, “Deep learning models for digital image processing: A review,”Artificial Intelligence Review, vol. 57, no. 1, p. 11, 2024

    2024

  10. [18]

    AI Action Summit (10 and 11 february 2025),

    “AI Action Summit (10 and 11 february 2025),” https://onu. delegfrance.org/ai-action-summit-10-and-11-february-2025

    2025

  11. [19]

    Can ChatGPT replace traditional KBQA models? An in-depth analysis of the question answering performance of the GPT LLM family,

    Y . Tan, D. Min, Y . Li, W. Li, N. Hu, Y . Chen, and G. Qi, “Can ChatGPT replace traditional KBQA models? An in-depth analysis of the question answering performance of the GPT LLM family,”in Proceedings of International Semantic Web Conference, pp. 348–367, 2023

    2023

  12. [20]

    Reasoning with large language models for medical question answering,

    M. M. Lucas, J. Yang, J. K. Pomeroy, and C. C. Yang, “Reasoning with large language models for medical question answering,”Journal of the American Medical Informatics Association, vol. 31, no. 9, pp. 1964–1975, 2024

    1964

  13. [21]

    Proactive conversational agents in the post-ChatGPT world,

    L. Liao, G. H. Yang, and C. Shah, “Proactive conversational agents in the post-ChatGPT world,”in Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 3452–3455, 2023

    2023

  14. [22]

    Unlock life with a chat GPT: Integrating conversational AI with large language models into everyday lives of autistic individuals,

    D. Choi, S. Lee, S.-I. Kim, K. Lee, H. J. Yoo, S. Lee, and H. Hong, “Unlock life with a chat GPT: Integrating conversational AI with large language models into everyday lives of autistic individuals,” in Proceedings of the 2024 CHI Conference on Human Factors in Computing Syst...

    2024

  15. [23]

    A contemporary review on chatbots, AI-powered virtual conversational agents, ChatGPT: Applications, open challenges and future research directions,

    A. Casheekar, A. Lahiri, K. Rath, K. S. Prabhakar, and K. Srinivasan, “A contemporary review on chatbots, AI-powered virtual conversational agents, ChatGPT: Applications, open challenges and future research directions,”Computer Science Review, vol. 52, p. 100632, 2024

    2024

  16. [24]

    Self-collaboration code gener- ation via ChatGPT,

    Y . Dong, X. Jiang, Z. Jin, and G. Li, “Self-collaboration code gener- ation via ChatGPT,”ACM Transactions on Software Engineering and Methodology, vol. 33, no. 7, pp. 1–38, 2024

    2024

  17. [25]

    Is your code generated by ChatGPT really correct? rigorous evaluation of large language models for code generation,

    J. Liu, C. S. Xia, Y . Wang, and L. Zhang, “Is your code generated by ChatGPT really correct? rigorous evaluation of large language models for code generation,” inAdvances in Neural Information Processing Systems, vol. 36, 2023, pp. 21 558–21 572

    2023

  18. [26]

    Gem- ini: a family of highly capable multimodal models,

    G. Team, R. Anil, S. Borgeaud, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millicanet al., “Gem- ini: a family of highly capable multimodal models,”arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  19. [27]

    Introducing Claude 2.1,

    “Introducing Claude 2.1,” https://www.anthropic.com/news/claude-2-1

  20. [28]

    Introducing llama 3.1: Our most capable models to date,

    A. Meta, “Introducing llama 3.1: Our most capable models to date,” Meta AI Blog, vol. 12, 2024

    2024

  21. [29]

    Mistral 7B,

    A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. l. Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnieret al., “Mistral 7B,”arXiv preprint arXiv:2310.06825, 2023

    2023 arXiv

  22. [30]

    A comprehensive survey on transfer learning,

    F. Zhuang, Z. Qi, K. Duan, D. Xi, Y . Zhu, H. Zhu, H. Xiong, and Q. He, “A comprehensive survey on transfer learning,”in Proceedings of the IEEE, vol. 109, no. 1, pp. 43–76, 2020

    2020

  23. [31]

    Multimodal learning with transform- ers: A survey,

    P. Xu, X. Zhu, and D. A. Clifton, “Multimodal learning with transform- ers: A survey,”IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 10, pp. 12 113–12 132, 2023

    2023

  24. [32]

    GPT-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkatet al., “GPT-4 technical report,”arXiv preprint arXiv:2303.08774, 2023

    2023 arXiv

  25. [33]

    DALL·E 2,

    “DALL·E 2,” https://openai.com/dall-e-2/

  26. [34]

    Janus-pro: Unified multimodal understanding and generation with data and model scaling,

    X. Chen, Z. Wu, X. Liu, Z. Pan, W. Liu, Z. Xie, X. Yu, and C. Ruan, “Janus-pro: Unified multimodal understanding and generation with data and model scaling,”arXiv preprint arXiv:2501.17811, 2025

    2025 arXiv

  27. [35]

    Introducing OpenAI o1,

    “Introducing OpenAI o1,” https://openai.com/o1/

  28. [36]

    Math-shepherd: Verify and reinforce llms step-by-step without human annotations,

    P. Wang, L. Li, Z. Shao, R. Xu, D. Dai, Y . Li, D. Chen, Y . Wu, and Z. Sui, “Math-shepherd: Verify and reinforce llms step-by-step without human annotations,” inin Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),...

    2024

  29. [37]

    Grok 3 beta — the age of reasoning agents,

    “Grok 3 beta — the age of reasoning agents,” https://x.ai/blog/grok-3

  30. [38]

    Deepseek-V3 technical report,

    A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruanet al., “Deepseek-V3 technical report,”arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  31. [39]

    Deepseek LLM: Scaling open-source language models with longtermism,

    X. Bi, D. Chen, G. Chen, S. Chen, D. Dai, C. Deng, H. Ding, K. Dong, Q. Du, Z. Fuet al., “Deepseek LLM: Scaling open-source language models with longtermism,”arXiv preprint arXiv:2401.02954, 2024

    2024 arXiv

  32. [40]

    DeepseekMoE: Towards ultimate expert spe- cialization in mixture-of-experts language models,

    D. Dai, C. Deng, C. Zhao, R. Xu, H. Gao, D. Chen, J. Li, W. Zeng, X. Yu, Y . Wuet al., “DeepseekMoE: Towards ultimate expert spe- cialization in mixture-of-experts language models,”arXiv preprint arXiv:2401.06066, 2024

    2024 arXiv

  33. [41]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models,

    Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y . Li, Y . Wuet al., “Deepseekmath: Pushing the limits of mathematical reasoning in open language models,”arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  34. [42]

    Deepseek-V2: A strong, econom- ical, and efficient mixture-of-experts language model,

    A. Liu, B. Feng, B. Wang, B. Wang, B. Liu, C. Zhao, C. Dengr, C. Ruan, D. Dai, D. Guoet al., “Deepseek-V2: A strong, econom- ical, and efficient mixture-of-experts language model,”arXiv preprint arXiv:2405.04434, 2024

    2024 arXiv

  35. [43]

    Hidden markov models,

    P. Blunsom, “Hidden markov models,”Lecture notes, August, vol. 15, no. 18-19, p. 48, 2004

    2004

  36. [44]

    Large language models in machine translation,

    T. Brants, A. Popat, P. Xu, F. J. Och, and J. Dean, “Large language models in machine translation,” inProceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), 2007, pp. 858–867

    2007

  37. [45]

    Dis- tributed representations of words and phrases and their composition- ality,

    T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean, “Dis- tributed representations of words and phrases and their composition- ality,” inProceedings of Advances in Neural Information Processing Systems (NIPS), vol. 26, 2013

    2013

  38. [46]

    Glove: Global vectors for word representation,

    J. Pennington, R. Socher, and C. D. Manning, “Glove: Global vectors for word representation,” inProceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), 2014, pp. 1532–1543

    2014

  39. [47]

    Fundamentals of recurrent neural network (RNN) and long short-term memory (LSTM) network,

    A. Sherstinsky, “Fundamentals of recurrent neural network (RNN) and long short-term memory (LSTM) network,”Physica D: Nonlinear Phenomena, vol. 404, p. 132306, 2020

    2020

  40. [48]

    Long short-term memory network for learning sentences similarity using deep contextual embeddings,

    S. Meshram and M. Anand Kumar, “Long short-term memory network for learning sentences similarity using deep contextual embeddings,” International Journal of Information Technology, vol. 13, no. 4, pp. 1633–1641, 2021. SHELLet al.: BARE DEMO OF IEEETRAN.CLS FOR JOURNALS 15

    2021

  41. [49]

    Roberta: A robustly optimized bert pretraining approach,

    Y . Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V . Stoyanov, “Roberta: A robustly optimized bert pretraining approach,”arXiv preprint arXiv:1907.11692, 2019

    1907 arXiv

  42. [50]

    DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter,

    V . Sanh, L. Debut, J. Chaumond, and T. Wolf, “DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter,”arXiv preprint arXiv:1910.01108, 2019

    1910 arXiv

  43. [51]

    BioBERT: a pre-trained biomedical language representation model for biomedical text mining,

    J. Lee, W. Yoon, S. Kim, D. Kim, S. Kim, C. H. So, and J. Kang, “BioBERT: a pre-trained biomedical language representation model for biomedical text mining,”Bioinformatics, vol. 36, no. 4, pp. 1234–1240, 2020

    2020

  44. [52]

    ALBERT: A lite BERT for self-supervised learning of language representations,

    Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut, “ALBERT: A lite BERT for self-supervised learning of language representations,”arXiv preprint arXiv:1909.11942, 2019

    1909 arXiv

  45. [53]

    A general language assistant as a laboratory for alignment,

    A. Askell, Y . Bai, A. Chen, D. Drain, D. Ganguli, T. Henighan, A. Jones, N. Joseph, B. Mann, N. DasSarmaet al., “A general language assistant as a laboratory for alignment,”arXiv preprint arXiv:2112.00861, 2021

    2021 arXiv

  46. [54]

    Ernie 3.0: Large-scale knowledge enhanced pre-training for language understanding and generation,

    Y . Sun, S. Wang, S. Feng, S. Ding, C. Pang, J. Shang, J. Liu, X. Chen, Y . Zhao, Y . Luet al., “Ernie 3.0: Large-scale knowledge enhanced pre-training for language understanding and generation,”arXiv preprint arXiv:2107.02137, 2021

    2021 arXiv

  47. [55]

    Multitask prompted training enables zero-shot task generalization,

    V . Sanh, A. Webson, C. Raffel, S. H. Bach, L. Sutawika, Z. Alyafeai, A. Chaffin, A. Stiegler, T. L. Scao, A. Rajaet al., “Multitask prompted training enables zero-shot task generalization,”arXiv preprint arXiv:2110.08207, 2021

    arXiv

  48. [56]

    Language models are unsupervised multitask learners,

    A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskeveret al., “Language models are unsupervised multitask learners,”OpenAI blog, vol. 1, no. 8, p. 9, 2019

    2019

  49. [57]

    Evaluating large language models trained on code,

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockmanet al., “Evaluating large language models trained on code,”arXiv preprint arXiv:2107.03374, 2021

    2021 arXiv

  50. [58]

    CodeGeeX: A pre-trained model for code generation with multilingual benchmarking on HumanEval-X,

    Q. Zheng, X. Xia, X. Zou, Y . Dong, S. Wang, Y . Xue, L. Shen, Z. Wang, A. Wang, Y . Liet al., “CodeGeeX: A pre-trained model for code generation with multilingual benchmarking on HumanEval-X,” inProceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Min...

    2023

  51. [59]

    Llama: Open and efficient foundation language models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample, “Llama: Open and efficient foundation language models,” 2023

    2023

  52. [60]

    Pythia: A suite for analyzing large language models across training and scaling,

    S. Biderman, H. Schoelkopf, Q. G. Anthony, H. Bradley, K. O’Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raffet al., “Pythia: A suite for analyzing large language models across training and scaling,” inProceedings of International Conference on Machine Learn...

    2023

  53. [61]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context,

    G. Team, P. Georgiev, V . I. Lei, R. Burnell, L. Bai, A. Gulati, G. Tanzer, D. Vincent, Z. Pan, S. Wanget al., “Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context,”arXiv preprint arXiv:2403.05530, 2024

    2024 arXiv

  54. [62]

    Language models are few-shot learners,

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askellet al., “Language models are few-shot learners,” inProceedings of Advances in Neural Information Processing Systems (NIPS), vol. 33, 2020, pp. 1877–1901

    2020

  55. [63]

    Scaling language models: Methods, analysis & insights from training gopher,

    J. W. Rae, S. Borgeaud, T. Cai, K. Millican, J. Hoffmann, F. Song, J. Aslanides, S. Henderson, R. Ring, S. Younget al., “Scaling language models: Methods, analysis & insights from training gopher,”arXiv preprint arXiv:2112.11446, 2021

    2021 arXiv

  56. [64]

    Bloom: A 176B- parameter open-access multilingual language model,

    T. L. Scao, A. Fan, C. Akiki, E. Pavlick, S. Ili ´c, D. Hesslow, R. Castagn´e, A. S. Luccioni, F. Yvon, M. Gall´eet al., “Bloom: A 176B- parameter open-access multilingual language model,”arXiv preprint arXiv:2211.05100, 2022

    2022 arXiv

  57. [65]

    Training compute-optimal large language models,

    J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hendricks, J. Welbl, A. Clarket al., “Training compute-optimal large language models,”arXiv preprint arXiv:2203.15556, 2022

    2022 arXiv

  58. [66]

    GLaM: Efficient scaling of lan- guage models with mixture-of-experts,

    N. Du, Y . Huang, A. M. Dai, S. Tong, D. Lepikhin, Y . Xu, M. Krikun, Y . Zhou, A. W. Yu, O. Firatet al., “GLaM: Efficient scaling of lan- guage models with mixture-of-experts,” inProceedings of International Conference on Machine Learning. PMLR, 2022, pp. 5547–5569

    2022

  59. [67]

    Lamda: Language models for dialog applications,

    R. Thoppilan, D. De Freitas, J. Hall, N. Shazeer, A. Kulshreshtha, H.-T. Cheng, A. Jin, T. Bos, L. Baker, Y . Duet al., “Lamda: Language models for dialog applications,”arXiv preprint arXiv:2201.08239, 2022

    2022 arXiv

  60. [68]

    Using DeepSpeed and megatron to train megatron-turing NLG 530B, a large-scale generative language model,

    S. Smith, M. Patwary, B. Norick, P. LeGresley, S. Rajbhandari, J. Casper, Z. Liu, S. Prabhumoye, G. Zerveas, V . Korthikanti et al., “Using DeepSpeed and megatron to train megatron-turing NLG 530B, a large-scale generative language model,”arXiv preprint arXiv:2201.11990, 2022

    2022 arXiv

  61. [69]

    OPT: Open pre-trained transformer language models,

    S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V . Linet al., “OPT: Open pre-trained transformer language models,”arXiv preprint arXiv:2205.01068, 2022

    2022 arXiv

  62. [70]

    BloombergGPT: A large language model for finance,

    S. Wu, O. Irsoy, S. Lu, V . Dabravolski, M. Dredze, S. Gehrmann, P. Kambadur, D. Rosenberg, and G. Mann, “BloombergGPT: A large language model for finance,”arXiv preprint arXiv:2303.17564, 2023

    2023 arXiv

  63. [71]

    The llama 3 herd of models,

    A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fanet al., “The llama 3 herd of models,”arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  64. [72]

    Qwen technical report,

    J. Bai, S. Bai, Y . Chu, Z. Cui, K. Dang, X. Deng, Y . Fan, W. Ge, Y . Han, F. Huanget al., “Qwen technical report,”arXiv preprint arXiv:2309.16609, 2023

    2023 arXiv

  65. [73]

    ChatGLM: A family of large language models from GLM-130B to GLM-4 all tools,

    T. GLM, A. Zeng, B. Xu, B. Wang, C. Zhang, D. Yin, D. Zhang, D. Rojas, G. Feng, H. Zhaoet al., “ChatGLM: A family of large language models from GLM-130B to GLM-4 all tools,”arXiv preprint arXiv:2406.12793, 2024

    2024 arXiv

  66. [74]

    Kimi k1. 5: Scaling reinforcement learning with LLMs,

    K. Team, A. Du, B. Gao, B. Xing, C. Jiang, C. Chen, C. Li, C. Xiao, C. Du, C. Liaoet al., “Kimi k1. 5: Scaling reinforcement learning with LLMs,”arXiv preprint arXiv:2501.12599, 2025

    2025 arXiv

  67. [75]

    Gemma: Open models based on gemini research and technology,

    G. Team, T. Mesnard, C. Hardin, R. Dadashi, S. Bhupatiraju, S. Pathak, L. Sifre, M. Rivi `ere, M. S. Kale, J. Loveet al., “Gemma: Open models based on gemini research and technology,”arXiv preprint arXiv:2403.08295, 2024

    2024 arXiv

  68. [76]

    Grok-2 beta release,

    “Grok-2 beta release,” https://x.ai/blog/grok-2

  69. [77]

    Llama 3.1: An in-depth analysis of the next-generation large language model,

    R. Vavekanand and K. Sam, “Llama 3.1: An in-depth analysis of the next-generation large language model,” 2024

    2024

  70. [78]

    PredRNN: A recurrent neural network for spatiotemporal predictive learning,

    Y . Wang, H. Wu, J. Zhang, Z. Gao, J. Wang, S. Y . Philip, and M. Long, “PredRNN: A recurrent neural network for spatiotemporal predictive learning,”IEEE Transactions on Pattern Analysis and Machine Intel- ligence, vol. 45, no. 2, pp. 2208–2225, 2022

    2022

  71. [79]

    Efficient and effective training of sparse recurrent neural networks,

    S. Liu, I. Ni’mah, V . Menkovski, D. C. Mocanu, and M. Pechenizkiy, “Efficient and effective training of sparse recurrent neural networks,” Neural Computing and Applications, vol. 33, pp. 9625–9636, 2021

    2021

  72. [80]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems, vol. 30, 2017

    2017

  73. [81]

    A tutorial on LLM reasoning: Relevant methods behind ChatGPT o1,

    J. Wang, “A tutorial on LLM reasoning: Relevant methods behind ChatGPT o1,”arXiv preprint arXiv:2502.10867, 2025

    2025 arXiv

  74. [82]

    Star: Bootstrapping reasoning with reasoning,

    E. Zelikman, Y . Wu, J. Mu, and N. Goodman, “Star: Bootstrapping reasoning with reasoning,”Advances in Neural Information Processing Systems, vol. 35, pp. 15 476–15 488, 2022

    2022

  75. [83]

    Alphazero-like tree-search can guide large language model decoding and training,

    X. Feng, Z. Wan, M. Wen, S. M. McAleer, Y . Wen, W. Zhang, and J. Wang, “Alphazero-like tree-search can guide large language model decoding and training,”arXiv preprint arXiv:2309.17179, 2023

    2023 arXiv

  76. [84]

    Improve mathematical reasoning in lan- guage models by automated process supervision,

    L. Luo, Y . Liu, R. Liu, S. Phatale, H. Lara, Y . Li, L. Shu, Y . Zhu, L. Meng, J. Sunet al., “Improve mathematical reasoning in lan- guage models by automated process supervision,”arXiv preprint arXiv:2406.06592, vol. 2, 2024

    2024 arXiv

  77. [85]

    Mak- ing large language models better reasoners with step-aware verifier,

    Y . Li, Z. Lin, S. Zhang, Q. Fu, B. Chen, J.-G. Lou, and W. Chen, “Mak- ing large language models better reasoners with step-aware verifier,” arXiv preprint arXiv:2206.02336, 2022

    2022 arXiv

  78. [86]

    An empirical analysis of compute-optimal inference for problem-solving with language models,

    Y . Wu, Z. Sun, S. Li, S. Welleck, and Y . Yang, “An empirical analysis of compute-optimal inference for problem-solving with language models,” arXiv e-prints, pp. arXiv–2408, 2024

    2024

  79. [87]

    Scaling LLM test-time compute optimally can be more effective than scaling model parameters,

    C. Snell, J. Lee, K. Xu, and A. Kumar, “Scaling LLM test-time compute optimally can be more effective than scaling model parameters,”arXiv preprint arXiv:2408.03314, 2024

    2024 arXiv

  80. [88]

    Dynamic programming and stochastic control processes,

    R. Bellman, “Dynamic programming and stochastic control processes,” Information and control, vol. 1, no. 3, pp. 228–239, 1958

    1958

  81. [89]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V . Le, D. Zhouet al., “Chain-of-thought prompting elicits reasoning in large language models,”Advances in neural information processing systems, vol. 35, pp. 24 824–24 837, 2022

    2022

  82. [90]

    Pangu-agent: A fine-tunable generalist agent with structured reasoning,

    F. Christianos, G. Papoudakis, M. Zimmer, T. Coste, Z. Wu, J. Chen, K. Khandelwal, J. Doran, X. Feng, J. Liuet al., “Pangu-agent: A fine-tunable generalist agent with structured reasoning,”arXiv preprint arXiv:2312.14878, 2023

    2023 arXiv

  83. [91]

    Qwen2.5: A party of foundation models,

    Q. Team, “Qwen2.5: A party of foundation models,” September 2024. [Online]. Available: https://qwenlm.github.io/blog/qwen2.5/

    2024

  84. [92]

    Deepseek-VL2: Mixture-of-experts vision- language models for advanced multimodal understanding,

    Z. Wu, X. Chen, Z. Pan, X. Liu, W. Liu, D. Dai, H. Gao, Y . Ma, C. Wu, B. Wanget al., “Deepseek-VL2: Mixture-of-experts vision- language models for advanced multimodal understanding,”arXiv preprint arXiv:2412.10302, 2024

    2024 arXiv

  85. [93]

    Visual instruction tuning,

    H. Liu, C. Li, Q. Wu, and Y . J. Lee, “Visual instruction tuning,” Advances in neural information processing systems, vol. 36, 2024. 16 IEEE/CAA JOURNAL OF AUTOMATICA SINICA, VOL. X, NO. X, X X

    2024

  86. [94]

    Sigmoid loss for language image pre-training,

    X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer, “Sigmoid loss for language image pre-training,” inProceedings of the IEEE/CVF Inter- national Conference on Computer Vision, 2023, pp. 11 975–11 986

    2023

  87. [95]

    Janus: Decoupling visual encoding for unified multimodal understanding and generation,

    C. Wu, X. Chen, Z. Wu, Y . Ma, X. Liu, Z. Pan, W. Liu, Z. Xie, X. Yu, C. Ruanet al., “Janus: Decoupling visual encoding for unified multimodal understanding and generation,”arXiv preprint arXiv:2410.13848, 2024

    2024 arXiv

  88. [96]

    Roformer: En- hanced transformer with rotary position embedding,

    J. Su, M. Ahmed, Y . Lu, S. Pan, W. Bo, and Y . Liu, “Roformer: En- hanced transformer with rotary position embedding,”Neurocomputing, vol. 568, p. 127063, 2024

    2024

  89. [97]

    MHA-Net: Multipath hybrid attention network for building footprint extraction from high-resolution remote sensing im- agery,

    J. Cai and Y . Chen, “MHA-Net: Multipath hybrid attention network for building footprint extraction from high-resolution remote sensing im- agery,”IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 14, pp. 5807–5817, 2021

    2021

  90. [98]

    Improving transformers with dynamically composable multi-head attention,

    D. Xiao, Q. Meng, S. Li, and X. Yuan, “Improving transformers with dynamically composable multi-head attention,”arXiv preprint arXiv:2405.08553, 2024

    2024 arXiv

  91. [99]

    Keep the cost down: A review on methods to optimize LLM’s KV-cache consumption,

    L. Shi, H. Zhang, Y . Yao, Z. Li, and H. Zhao, “Keep the cost down: A review on methods to optimize LLM’s KV-cache consumption,”arXiv preprint arXiv:2407.18003, 2024

    2024 arXiv

  92. [100]

    Fast transformer decoding: One write-head is all you need,

    N. Shazeer, “Fast transformer decoding: One write-head is all you need,”arXiv preprint arXiv:1911.02150, 2019

    1911 arXiv

  93. [101]

    Reducing transformer key-value cache size with cross-layer attention,

    W. Brandon, M. Mishra, A. Nrusimha, R. Panda, and J. Ragan-Kelley, “Reducing transformer key-value cache size with cross-layer attention,” Advances in Neural Information Processing Systems, vol. 37, pp. 86 927–86 957, 2025

    2025

  94. [102]

    GQA: Training generalized multi-query transformer mod- els from multi-head checkpoints,

    J. Ainslie, J. Lee-Thorp, M. de Jong, Y . Zemlyanskiy, F. Lebr ´on, and S. Sanghai, “GQA: Training generalized multi-query transformer mod- els from multi-head checkpoints,”arXiv preprint arXiv:2305.13245, 2023

    2023 arXiv

  95. [103]

    QCQA: Quality and capacity-aware grouped query attention,

    V . Joshi, P. Laddha, S. Sinha, O. J. Omer, and S. Subramoney, “QCQA: Quality and capacity-aware grouped query attention,”arXiv preprint arXiv:2406.10247, 2024

    2024 arXiv

  96. [104]

    MoEfica- tion: Transformer feed-forward layers are mixtures of experts,

    Z. Zhang, Y . Lin, Z. Liu, P. Li, M. Sun, and J. Zhou, “MoEfica- tion: Transformer feed-forward layers are mixtures of experts,”arXiv preprint arXiv:2110.01786, 2021

    2021 arXiv

  97. [105]

    Auxiliary-loss- free load balancing strategy for mixture-of-experts,

    L. Wang, H. Gao, C. Zhao, X. Sun, and D. Dai, “Auxiliary-loss- free load balancing strategy for mixture-of-experts,”arXiv preprint arXiv:2408.15664, 2024

    2024 arXiv

  98. [106]

    ScheMoE: An extensible mixture-of-experts distributed training system with task scheduling,

    S. Shi, X. Pan, Q. Wang, C. Liu, X. Ren, Z. Hu, Y . Yang, B. Li, and X. Chu, “ScheMoE: An extensible mixture-of-experts distributed training system with task scheduling,”in Proceedings of the Nineteenth European Conference on Computer Systems, pp. 236–249, 2024

    2024

  99. [107]

    Are decoder-only language models better than encoder-only language models in understanding word meaning?

    M. Qorib, G. Moon, and H. T. Ng, “Are decoder-only language models better than encoder-only language models in understanding word meaning?”Findings of the Association for Computational Linguistics ACL 2024, pp. 16 339–16 347, 2024

    2024

  100. [108]

    DeTiME: Diffusion- enhanced topic modeling using encoder-decoder based LLM,

    W. Xu, W. Hu, F. Wu, and S. Sengamedu, “DeTiME: Diffusion- enhanced topic modeling using encoder-decoder based LLM,”arXiv preprint arXiv:2310.15296, 2023

    2023 arXiv

  101. [109]

    When linear attention meets autoregressive decoding: Towards more effec- tive and efficient linearized large language models,

    H. You, Y . Fu, Z. Wang, A. Yazdanbakhsh, and Y . C. Lin, “When linear attention meets autoregressive decoding: Towards more effec- tive and efficient linearized large language models,”arXiv preprint arXiv:2406.07368, 2024

    2024 arXiv

  102. [110]

    Large language models with adaptive token fusion: A novel approach to reducing hallucinations and improving inference efficiency,

    L. Guo, Y . Fang, F. Chen, P. Liu, and S. Xu, “Large language models with adaptive token fusion: A novel approach to reducing hallucinations and improving inference efficiency,”Authorea, pp. 1–10, 2024

    2024

  103. [111]

    Object recognition as next token prediction,

    K. Yue, B.-C. Chen, J. Geiping, H. Li, T. Goldstein, and S.-N. Lim, “Object recognition as next token prediction,”in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16 645–16 656, 2024

    2024

  104. [112]

    Better & faster large language models via multi-token prediction,

    F. Gloeckle, B. Y . Idrissi, B. Rozi `ere, D. Lopez-Paz, and G. Synnaeve, “Better & faster large language models via multi-token prediction,” arXiv preprint arXiv:2404.19737, 2024

    2024 arXiv

  105. [113]

    The pitfalls of next-token prediction,

    G. Bachmann and V . Nagarajan, “The pitfalls of next-token prediction,” arXiv preprint arXiv:2403.06963, 2024

    2024 arXiv

  106. [114]

    DynaMo: Accelerating language model inference with dynamic multi- token sampling,

    S. Tuli, C.-H. Lin, Y .-C. Hsu, N. K. Jha, Y . Shen, and H. Jin, “DynaMo: Accelerating language model inference with dynamic multi- token sampling,”arXiv preprint arXiv:2405.00888, 2024

    2024 arXiv

  107. [115]

    Proximal policy optimization via enhanced exploration efficiency,

    J. Zhang, Z. Zhang, S. Han, and S. L ¨u, “Proximal policy optimization via enhanced exploration efficiency,”Information Sciences, vol. 609, pp. 750–765, 2022

    2022

  108. [116]

    Prox- imal policy optimization algorithms,

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Prox- imal policy optimization algorithms,”arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  109. [117]

    Truly proximal policy optimization,

    Y . Wang, H. He, and X. Tan, “Truly proximal policy optimization,” inProceedings of The 35th Uncertainty in Artificial Intelligence Conference. PMLR, 2020, pp. 113–122

    2020

  110. [118]

    Proximal policy opti- mization with policy feedback,

    Y . Gu, Y . Cheng, C. P. Chen, and X. Wang, “Proximal policy opti- mization with policy feedback,”IEEE Transactions on Systems, Man, and Cybernetics: Systems, vol. 52, no. 7, pp. 4600–4610, 2021

    2021

  111. [119]

    Group robust preference optimization in reward-free RLHF,

    S. S. Ramesh, Y . Hu, I. Chaimalas, V . Mehta, P. G. Sessa, H. B. Ammar, and I. Bogunovic, “Group robust preference optimization in reward-free RLHF,”arXiv preprint arXiv:2405.20304, 2024

    2024 arXiv

  112. [120]

    Process reinforcement through implicit rewards,

    G. Cui, L. Yuan, Z. Wang, H. Wang, W. Li, B. He, Y . Fan, T. Yu, Q. Xu, W. Chenet al., “Process reinforcement through implicit rewards,”arXiv preprint arXiv:2502.01456, 2025

    2025 arXiv

  113. [121]

    Hybrid group relative policy optimization: A multi- sample approach to enhancing policy optimization,

    S. Sane, “Hybrid group relative policy optimization: A multi- sample approach to enhancing policy optimization,”arXiv preprint arXiv:2502.01652, 2025

    2025 arXiv

  114. [122]

    Scaling laws for neural language models,

    J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei, “Scaling laws for neural language models,”arXiv preprint arXiv:2001.08361, 2020

    2001 arXiv

  115. [123]

    GPT3.int8(): 8-bit matrix multiplication for transformers at scale,

    T. Dettmers, M. Lewis, Y . Belkada, and L. Zettlemoyer, “GPT3.int8(): 8-bit matrix multiplication for transformers at scale,”Advances in Neural Information Processing Systems, vol. 35, pp. 30 318–30 332, 2022

    2022

  116. [124]

    8- bit numerical formats for deep neural networks,

    B. Noune, P. Jones, D. Justus, D. Masters, and C. Luschi, “8- bit numerical formats for deep neural networks,”arXiv preprint arXiv:2206.02915, 2022

    2022 arXiv

  117. [125]

    FP8-LM: Training FP8 large language models,

    H. Peng, K. Wu, Y . Wei, G. Zhao, Y . Yang, Z. Liu, Y . Xiong, Z. Yang, B. Ni, J. Huet al., “FP8-LM: Training FP8 large language models,” arXiv preprint arXiv:2310.18313, 2023

    2023 arXiv

  118. [126]

    GPTQ: Accu- rate post-training quantization for generative pre-trained transformers,

    E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh, “GPTQ: Accu- rate post-training quantization for generative pre-trained transformers,” arXiv preprint arXiv:2210.17323, 2022

    2022 arXiv

  119. [127]

    SmoothQuant: Accurate and efficient post-training quantization for large language models,

    G. Xiao, J. Lin, M. Seznec, H. Wu, J. Demouth, and S. Han, “SmoothQuant: Accurate and efficient post-training quantization for large language models,” inProceedings of International Conference on Machine Learning. PMLR, 2023, pp. 38 087–38 099

    2023

  120. [128]

    Scaling FP8 training to trillion-token LLMs,

    M. Fishman, B. Chmiel, R. Banner, and D. Soudry, “Scaling FP8 training to trillion-token LLMs,”arXiv preprint arXiv:2409.12517, 2024

    2024 arXiv

  121. [129]

    Understanding and minimising outlier features in transformer training,

    B. He, L. Noci, D. Paliotta, I. Schlag, and T. Hofmann, “Understanding and minimising outlier features in transformer training,” inProceed- ings of The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  122. [130]

    Massive activations in large language models,

    M. Sun, X. Chen, J. Z. Kolter, and Z. Liu, “Massive activations in large language models,”arXiv preprint arXiv:2402.17762, 2024

    2024 arXiv

  123. [131]

    DeepGEMM: clean and efficient FP8 GEMM kernels with fine-grained scaling,

    C. Zhao, L. Zhao, J. Li, and Z. Xu, “DeepGEMM: clean and efficient FP8 GEMM kernels with fine-grained scaling,” https://github.com/ deepseek-ai/DeepGEMM, 2025

    2025

  124. [132]

    FlashMLA: Efficient MLA decoding kernel,

    J. Li, “FlashMLA: Efficient MLA decoding kernel,” https://github.com/ deepseek-ai/FlashMLA, 2025

    2025

  125. [133]

    GShard: Scaling giant models with conditional computation and automatic sharding,

    D. Lepikhin, H. Lee, Y . Xu, D. Chen, O. Firat, Y . Huang, M. Krikun, N. Shazeer, and Z. Chen, “GShard: Scaling giant models with conditional computation and automatic sharding,”arXiv preprint arXiv:2006.16668, 2020

    2006 arXiv

  126. [134]

    The sparsely-gated mixture-of-experts layer,

    N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean, “The sparsely-gated mixture-of-experts layer,”Outrageously large neural networks, 2017

    2017

  127. [135]

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity,

    W. Fedus, B. Zoph, and N. Shazeer, “Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity,”Journal of Machine Learning Research, vol. 23, no. 120, pp. 1–39, 2022

    2022

  128. [136]

    Zero bubble pipeline paral- lelism,

    P. Qi, X. Wan, G. Huang, and M. Lin, “Zero bubble pipeline paral- lelism,”arXiv preprint arXiv:2401.10241, 2023

    2023 arXiv

  129. [137]

    DeepEP: an efficient expert-parallel communication library,

    C. Zhao, S. Zhou, L. Zhang, C. Deng, Z. Xu, Y . Liu, K. Yu, J. Li, and L. Zhao, “DeepEP: an efficient expert-parallel communication library,” https://github.com/deepseek-ai/DeepEP, 2025

    2025

  130. [138]

    Training large language models to reason in a continuous latent space,

    S. Hao, S. Sukhbaatar, D. Su, X. Li, Z. Hu, J. Weston, and Y . Tian, “Training large language models to reason in a continuous latent space,” arXiv preprint arXiv:2412.06769, 2024

    2024 arXiv

  131. [139]

    SafeChain: Safety of language models with long chain-of- thought reasoning capabilities,

    F. Jiang, Z. Xu, Y . Li, L. Niu, Z. Xiang, B. Li, B. Y . Lin, and R. Pooven- dran, “SafeChain: Safety of language models with long chain-of- thought reasoning capabilities,”arXiv preprint arXiv:2502.12025, 2025

    2025 arXiv

  132. [140]

    Reasoning-augmented conversation for multi-turn jailbreak attacks on large language models,

    Z. Ying, D. Zhang, Z. Jing, Y . Xiao, Q. Zou, A. Liu, S. Liang, X. Zhang, X. Liu, and D. Tao, “Reasoning-augmented conversation for multi-turn jailbreak attacks on large language models,”arXiv preprint arXiv:2502.11054, 2025

    2025 arXiv

  133. [141]

    OpenAI’s’ deep research’tool: is it useful for scientists?

    N. Jones, “OpenAI’s’ deep research’tool: is it useful for scientists?” Nature. SHELLet al.: BARE DEMO OF IEEETRAN.CLS FOR JOURNALS 17

  134. [142]

    Introducing operator,

    “Introducing operator,” https://openai.com/index/introducing-operator/

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.