Pith. sign in

REVIEW 4 major objections 4 minor 80 references

Motif 3: Technical Report

T0 review · 4 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Motif 3, a 314B-parameter Mixture-of-Experts model activating only 13.2B parameters per token, matches or surpasses much larger open-weight models on long-horizon agentic tasks while staying competitive on math, science, and…

desk verdict A serious 314B MoE technical report with a genuinely new attention variant; the architecture and engineering are detailed and coherent, but the headline performance claims lean on uncontrolled leaderboard comparisons. read the letter →

arxiv 2608.09119 v1 pith:HRAYVZNU submitted 2026-08-10 cs.AI

Junghwan Lim , Joon Son Chung , Sungmin Lee , Wai Ting Cheung , Gihun Cho , Minsu Ha , Sangho Kang , Beomgyu Kim
show 19 more authors
Dongseok Kim Jangwoong Kim Taehyun Kim Taewhan Kim Jeesoo Lee Jeongdoo Lee Junhyeok Lee Dongpin Oh Hyeyeon Cho Dahye Choi Jaeheui Her Hanbin Jung Changjin Kang Minjae Kim Youngrok Kim Hyukjin Kweon Hongjoo Lee Yeongjae Park Bokki Ryu
This is my paper · ORCID
classification cs.AI
keywords Mixture-of-ExpertsGroupedDifferentialLatentAttentionfine-grainedexpertroutingmulti-teacheron-policydistillationagentictooluselong-contextpretrainingopen-weightlanguagemodel12.5trilliontokens
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The report introduces Motif 3, a Mixture-of-Experts language model with 314 billion total parameters but only 13.2 billion activated per token, and argues that this fine-grained sparsity, combined with a new attention mechanism and a multi-teacher distillation pipeline, can make a relatively small active-parameter model competitive with the strongest open-weight systems. The paper's central claim is that careful architecture and training choices substitute for raw scale: it reports top scores on several long-horizon agentic benchmarks, such as τ3-Banking (35.3) and Terminal-Bench 2.1 (74.9), and strong results on SWE-bench Verified (76.2), mathematical reasoning, scientific knowledge, and hallucination-sensitive evaluation. If this claim holds, it would mean that a 13B-active model can outperform models that activate many more parameters per token on economically valuable agentic tasks, and that specialized capabilities can be folded into a single deployable model without losing generality.

What carries the argument

The central object is Grouped Differential Latent Attention (GDLA), which combines the noise-subtraction idea of differential attention with the compressed key-value representation of Multi-head Latent Attention: 64 signal heads and 16 noise heads share a single expanded latent KV state, and a token-dependent coefficient λ scales the repeated noise-head output before subtraction, with an additional element-wise query-dependent output gate. Around this sit the fine-grained MoE (384 experts, top-8 routing, one shared expert, Expert-Specific PolyNorm activations), modified manifold-constrained hyper-connections whose post-mapping scale anneals from 2 to 1 to suppress activation outliers, and a multi-token prediction head for self-speculative decoding. The post-training machinery is Multi-teacher On-Policy Distillation, which routes each on-policy student trajectory to a domain specialist and trains the student to match that teacher's token log-probabilities, omitting environment rewards entirely.

What would settle it

Running the Table 6 benchmarks for Motif 3 and the listed baselines under a single shared harness with identical sampling settings, agent scaffolds, step limits, and timeouts would settle the competitive claim; if Motif 3's margins on τ3-Banking, Terminal-Bench 2.1, and SWE-bench Verified shrink or reverse under such a head-to-head comparison, the central claim would be falsified.

Watch

Extended reading notes

Core claim

This report claims that Motif 3 achieves competitive performance against leading open-weight models across a broad suite, with its clearest strengths in agentic tool use and terminal-based problem solving. The supporting numbers include 35.3 on τ3-Banking, 74.9 on Terminal-Bench 2.1, 76.2 on SWE-bench Verified, 94.7 on τ2-Bench Telecom, 83.2 on IMO-AnswerBench, and a non-hallucination score of 71.6 on AA-Omniscience, with the top result among listed models on the public ITBench-AA subset. The paper attributes this performance to four design pillars: Grouped Differential Latent Attention that suppresses attention noise over a compressed key-value latent, a fine-grained MoE with 384 routed experts per layer and only eight selected per token, architectural stabilizers including modified manifold-constrained hyper-connections and Expert-Specific PolyNorm, and a post-training recipe in which six RL-trained specialist teachers and one software-engineering teacher are distilled into a single student via Multi-teacher On-Policy Distillation.

Load-bearing premise

The claim that Motif 3 is competitive with leading open-weight models assumes that the published leaderboard scores for those models were produced under conditions comparable to Motif 3's evaluation protocol (same harness, temperature 1.0, top-p 0.95, and benchmark-specific agent scaffolds), which the report does not verify.

Editorial extensions

If this is right

  • A 13-billion-active-parameter model can match or beat models that activate 23B–49B parameters per token on long-horizon agentic benchmarks, suggesting that expert capacity and routing quality matter more than raw activated size for tool-use tasks.
  • Fine-grained MoE with 384 experts per layer can be trained stably to 12.5 trillion tokens using the layered balancing and stabilization strategies described, without routing collapse or specialization death.
  • Multi-teacher On-Policy Distillation can consolidate RL-trained specialists in agentic work, professional work, software engineering, math, code-and-science, and chat into a single student while retaining broad general capability.
  • GDLA reaches a pretraining loss of 3.2 with 9.2% fewer training tokens than a standard latent-attention baseline in controlled 10B-parameter comparisons, implying a direct training-cost reduction if the effect transfers to scale.
  • The hybrid full/sliding-window attention schedule together with window-aware context parallelism permits efficient 256K-token training, so the architecture's efficiency gains extend to long-context scenarios.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The competitive-performance claim depends on leaderboard scores for baselines that may have been measured under different harnesses, temperature, and sampling settings; a direct head-to-head evaluation under a single protocol would cleanly separate architecture-driven gains from evaluation-driven artifacts.
  • The 10B-parameter controlled experiments used to justify GDLA and Expert-Specific PolyNorm could be extended by running the same ablations at the 314B scale on a short training budget, providing a direct test of whether the efficiency and specialization findings survive scale-up.
  • If GDLA's token-efficiency result holds at scale, it implies a roughly 9% reduction in pretraining compute for a given loss, which for a 12.5-trillion-token run translates into a substantial reduction in data and energy costs.
  • Because GDLA and the MoE router are text-agnostic, the architecture could be carried into multimodal settings by adding vision encoders, potentially preserving the agentic strengths on visually grounded tasks that the current text-only model cannot address.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The manuscript reports Motif 3, a 314B-parameter Mixture-of-Experts language model with 13.2B activated parameters per token, built on Grouped Differential Latent Attention (GDLA), modified manifold-constrained hyper-connections, Expert-Specific PolyNorm, and multi-token prediction. It describes a 12.5T-token pretraining pipeline, a staged post-training pipeline of SFT, six GRPO-trained specialist teachers plus one SFT software-engineering teacher, and Multi-teacher On-Policy Distillation (MOPD). The paper claims competitive performance against leading open-weight models across agentic, coding, reasoning, knowledge, long-context, and instruction-following benchmarks, with particular strengths in agentic tool use such as τ3-Banking 35.3, Terminal-Bench 2.1 74.9, and SWE-bench Verified 76.2, and reports architectural diagnostics from controlled ~10B-parameter experiments.

Significance. If the architecture and pipeline claims hold, the paper provides a detailed recipe for fine-grained MoE training at scale: GDLA offers a plausible improvement over MLA and GDA with compact KV cache, and the MOPD post-training scheme is a technically substantive method for consolidating specialist capabilities without deploying separate models. The paper is unusually detailed on systems-level optimization: MXFP8 dispatch, gradient synchronization, QK-Clip monitoring via FlashAttention-4, window-aware context parallelization, and the empirical workload-rebalancing heuristic are concrete and potentially reproducible. Strength in reproducibility is partially offset by the fact that the training-example repository is referenced but not fully described in the manuscript; nevertheless, the derivation of communication volumes in Appendix B and the explicit tokenizer patterns in Appendix A are useful and checkable. The central quantitative comparative claim, however, is not yet established because the evaluation protocol is not matched against baselines.

major comments (4)
  1. [§5.3.1, Table 6, Abstract] The headline claim of 'competitive performance against leading open-weight models' rests on leaderboard scores for MiniMax-3, GLM-5.1, Kimi-K2.6, Qwen-3.7-Max, and DS-v4-Pro taken from 'the corresponding benchmark leaderboards' without a shared harness, sampling temperature, maximum sequence length, or agent configuration. The manuscript states only Motif 3's settings (temperature=1.0, top-p=0.95, Appendix C), and Appendix C reveals harness-specific details for Motif 3 (Terminus 2, mini-SWE-agent, a Qwen3 user simulator) that need not match the baselines. Since agentic and coding scores are known to move several points with harness, episode budget, and environment version, the claimed advantages in τ2-Bench and Terminal-Bench 2.1 are not currently established.
  2. [§5.3.2, Table 6, ITBench-AA row] The ITBench-AA entry is explicitly labeled as evaluated on the public subset only (asterisk in Table 6), while the comparison scores are presumably full-set. The text then describes 51.5 as 'the highest among the available results listed in Table 6.' Comparing a public-subset score to full-set baseline scores is not a valid basis for claiming superiority; the row should either be reported on the identical full set or be marked as non-comparable in both the table and the discussion.
  3. [§5.3, Table 6] No error bars, variance estimates, or repeated-run statistics are provided for any of the reported point estimates. Several key comparisons are extremely close (Terminal-Bench 2.1: 74.9 vs 75.0 for Qwen-3.7-Max; SWE-bench Verified: 76.2 vs 76.2 for Kimi-K2.6 and 80.4 for Qwen-3.7-Max), so a few points of harness or decoding variation could flip the ordering. Without a shared protocol or uncertainty quantification, the paper's sentence 'the highest score among the models and results listed in Table 6' is stronger than the evidence supports.
  4. [§2.2, Figure 2] The core architectural claims for GDLA—lower loss than GDA and MLA, and reaching loss 3.2 with 9.2% fewer training tokens—are based on controlled experiments with approximately 10B-parameter models, as explicitly stated in §2.2. The manuscript does not provide evidence that this diagnostic advantage transfers to the 314B model with 53 layers, hybrid full/sliding-window attention, mHC, and MTP. Figure 2 is presented as a general validation of the architecture, but the link from the 10B setting to the production 314B setting is an assumption that should be stated as such and ideally tested with at least an intermediate-scale run.
minor comments (4)
  1. [Abstract] The word 'foreach' in 'selected foreach token' is a typo and should read 'for each token.'
  2. [§2.3, Eq. (23)] The notation 's t : 2− →1' is ambiguous; it should be written as a function or schedule, e.g., 's t monotonically decreases from 2 to 1 over pretraining', to match the surrounding text.
  3. [§4.2, Eq. (29)] The cosine decay schedule would be clearer if the training-step variable t and the noise period T_noise were defined immediately before the equation; currently the symbols are only explained in the following paragraph.
  4. [§5.3.1, Appendix C] The evaluation details in Appendix C are reported for only three benchmarks; the settings for the remaining agentic, reasoning, and knowledge benchmarks (e.g., GDPval-AA v2, IMO-AnswerBench, GPQA Diamond, AA-Omniscience) are absent, which makes the protocol difficult to reproduce.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central architecture and post-training claims are supported by new controlled experiments and external benchmarks, and the self-citations are not load-bearing.

full rationale

The report's central claims are empirical and self-contained against external benchmarks. GDLA is specified by explicit equations (Section 2.2) and supported by a new 10B-parameter controlled loss comparison (Figure 2) rather than by re-deriving the result from a cited theorem; reference [63] (the authors' own GDA paper) supplies background but is not the sole support. The MOPD pipeline is described with its own objective (Eq. 37), and environment rewards are explicitly withheld from the distillation loss, so the reported benchmark scores are not constructed from the evaluation targets. Self-citations to Motif 2 (dynamic data-mixture scheduling, Parallel Muon) are implementation references, not uniqueness claims or fitted parameters. Evaluation-protocol concerns raised by the reader—leaderboard baselines without a shared harness and the ITBench-AA public-subset comparison—are comparability/validity issues, not circularity: no step reduces to its inputs by construction. The paper therefore exhibits no self-definitional, fitted-input, or author-imported-uniqueness circularity.

Assumptions & free parameters 8 free parameters · 6 assumptions · 0 invented entities

The paper's claims rest mainly on empirical engineering choices, including hyperparameters, data mixtures, and stabilization thresholds, rather than on a derivation. The most load-bearing assumptions are that small-scale 10B ablations transfer to the 314B model and that external leaderboard scores are comparable. No new physical entities are postulated.

free parameters (8)
  • Grouped ratio g = 4 (Table 1)
    Sets the signal/noise head split (64 signal, 16 noise heads) in GDLA; chosen by design and central to the claimed attention improvement.
  • Sliding window size W = 128 tokens
    Determines the hybrid attention schedule and the long-context communication design; a hand-set hyperparameter.
  • Routed experts per token (top-k) = 8
    Number of experts selected per token from 384; central to the sparsity and capacity claims.
  • Number of parallel residual streams n = 4
    Number of parallel streams in modified mHC; chosen architecture hyperparameter.
  • mHC post-mapping scale s(t) = annealed from 2 to 1 during pretraining
    Time-dependent scaling modification introduced to suppress activation outliers; the schedule is chosen by hand.
  • QK-Clip threshold tau = 100 (pretrain), 200 (long-context)
    Threshold for attention-logit clipping under Muon; chosen to control logit growth.
  • FFN magnitude regularizer threshold tau = 128 (most layers), 1024 (final layer)
    Threshold in the soft FFN-output magnitude penalty; chosen after observing activation outliers.
  • Router noise decay schedule = start and minimum noise scales and decay period not fully disclosed
    Early-stage Gaussian router noise is used to balance expert exploration; exact schedule values are not reported.
assumptions (6)
  • domain assumption Differential attention improves selectivity by subtracting a noise attention distribution from a signal distribution.
    Inherited from Differential Transformer [70] and the authors' GDA [63]; invoked in Section 2.2 as the basis for GDLA.
  • domain assumption MLA's low-rank KV compression preserves enough information for attention.
    Invoked in Section 2.2 to justify sharing one compressed KV latent across signal and noise query paths.
  • domain assumption Controlled 10B-parameter training comparisons are representative of 314B-scale behavior.
    Figures 2 and 6 use approximately 10B-parameter models; the report assumes these results transfer to the final model.
  • domain assumption Leaderboard scores from different models and harnesses are comparable.
    Section 5.3.1 states comparisons use leaderboard scores; no controlled re-evaluation of baseline models is provided.
  • domain assumption The pretraining data mixture and tokenizer probe are representative enough for the claimed capabilities.
    Section 4.1 gives a partial, non-exhaustive accounting of data; exact mixture and filtering details are not released.
  • standard math Sinkhorn-Knopp normalization converges to the Birkhoff polytope as used in mHC.
    Standard iterative projection used in Eq. 20; assumed to be numerically stable at 314B scale.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Motif 3: Technical Report." pith.science (2026). https://pith.science/paper/HRAYVZNU

@misc{pith2026260809119,
  author       = {Pith},
  title        = {Pith review of: Motif 3: Technical Report},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HRAYVZNU}},
  note         = {Machine review of arXiv:2608.09119}
}
read the original abstract

We introduce Motif 3, a decoder-only Mixture-of-Experts language model with 314 billion total parameters and 13.2 billion activated per token. Each sparse MoE layer contains 384 routed experts, with eight selected per token. This fine-grained sparsity provides substantial expert capacity while limiting computation. Motif 3 is built around Grouped Differential Latent Attention (GDLA), which integrates grouped differential attention with the compressed key-value representation of Multi-head Latent Attention. The architecture further incorporates modified manifold-constrained hyper-connections, Expert Specific PolyNorm activations, and multi-token prediction to improve optimization stability, expert specialization, and inference efficiency. We pretrain Motif 3 on approximately 12.5 trillion tokens spanning web documents, STEM, code, mathematics, multilingual content, and domain-specialized corpora. Expert-balancing and numerical-stabilization techniques support stable training at scale, while selective MXFP8 computation and communication, memory-efficient fused kernels, and window-aware context parallelism enable training with context lengths up to 256K tokens. Our post-training pipeline combines general supervised fine-tuning, six specialist teachers trained with reinforcement learning, a software-engineering teacher trained with supervised fine-tuning, and Multi-teacher On-Policy Distillation. The resulting unified model consolidates complementary capabilities in reasoning, coding, tool use, professional work, long-context understanding, calibrated abstention, and instruction following. Across a broad evaluation suite, Motif 3 demonstrates competitive performance against leading open weight models, including strong results on long-horizon agentic tasks, mathematical reasoning, scientific knowledge, and hallucination-sensitive evaluation.

Figures

Figures reproduced from arXiv: 2608.09119 by the authors.

Figure 1
Figure 1. Illustration of the GDLA (Grouped Differential Latent Attention) architecture of Motif 3. Left: each Transformer block pairs a GDLA token-mixing layer with a sparse MoE channel-mixing layer and uses mHC to mix n parallel residual streams. Bottom right: the query and KV paths use low-rank projections. The complete KV latent c KV is RMS-normalized and expanded once by WKV b into 16 KV heads shared by the signal and no… view at source ↗
Figure 2
Figure 2. Attention training-loss comparison. GDLA achieves lower loss than GDA and MLA and reaches [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Overall low-precision training recipe. Color encodes numerical precision: teal for MXFP8, blue for BF16, and peach for FP32. Dashed borders mark collectives that cross ranks. Only the row-wise MXFP8 weight is All￾Gathered; the column-wise copy required by Dgrad is produced locally by a fused row-to-column transcode kernel. Expert activations are quantized once before EP dispatch, allowing the dispatch itself to use … view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Selecting the context-parallel algorithm per attention layer. (a) A packed sequence sampled from the training data and containing 473 documents, shown on the attention score matrix QK⊤. (b) Rank￾average forward-pass communication volume per layer. The marker above the …
Figure 5
Figure 5. Figure 5: Ring Attention communication under full and sliding-window attention. The block structure of QK⊤ for P = 8 context-parallel ranks, shaded from the perspective of rank 5: query shards on the left, key/value shards on top, score blocks in between. Only the key and value …
Figure 6
Figure 6. Figure 6: MoE component training comparisons from controlled experiments using models with approxi [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 7
Figure 7. Figure 7: Mean reward over cumulative RL compute for the six GRPO-trained specialist teachers. The [PITH_FULL_IMAGE:figures/full_fig_p024_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

80 extracted references · 28 canonical work pages

  1. [1]

    Open-swe-traces: Advancing dual-mode multilingual distillation for software engineering agents.arXiv preprint arXiv:2606.16038, 2026

    Wasi Uddin Ahmad, Nikolai Ludwig, Somshubra Majumdar, and Boris Ginsburg. Open-swe-traces: Advancing dual-mode multilingual distillation for software engineering agents.arXiv preprint arXiv:2606.16038, 2026. URLhttps://arxiv.org/abs/2606.16038

    arXiv 2026

  2. [2]

    GQA: Training generalized multi-query transformer models from multi-head checkpoints

    Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebr´on, and Sumit Sanghai. GQA: Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint arXiv:2305.13245, 2023

    arXiv 2023

  3. [3]

    Artificial analysis long context reasoning benchmark (AA-LCR)

    Artificial Analysis. Artificial analysis long context reasoning benchmark (AA-LCR). On- line benchmark and leaderboard, 2025. URL https://artificialanalysis.ai/evaluations/ artificial-analysis-long-context-reasoning. Accessed August 9, 2026

    work page 2025

  4. [4]

    GDPval-AA v2 leaderboard

    Artificial Analysis. GDPval-AA v2 leaderboard. Online benchmark and leaderboard, 2026. URL https://artificialanalysis.ai/evaluations/gdpval-aa. Accessed August 9, 2026

    work page 2026

  5. [5]

    ITBench-AA benchmark leaderboard

    Artificial Analysis. ITBench-AA benchmark leaderboard. Online benchmark and leaderboard, 2026. URLhttps://artificialanalysis.ai/evaluations/itbench-aa. Accessed August 9, 2026. 26

    work page 2026

  6. [6]

    Online benchmark and leaderboard, 2026

    Artificial Analysis.τ 3-banking benchmark leaderboard. Online benchmark and leaderboard, 2026. URLhttps://artificialanalysis.ai/evaluations/tau3-banking. Accessed August 9, 2026

    work page 2026

  7. [7]

    Program synthesis with large language models.arXiv preprint arXiv:2108.07732, 2021

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, et al. Program synthesis with large language models.arXiv preprint arXiv:2108.07732, 2021

    arXiv 2021

  8. [8]

    τ2-bench: Evaluating conversational agents in a dual-control environment.arXiv preprint arXiv:2506.07982, 2025

    Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan. τ2-bench: Evaluating conversational agents in a dual-control environment.arXiv preprint arXiv:2506.07982, 2025

    arXiv 2025

Show all 80 references
  1. [9]

    PIQA: Reasoning about physical commonsense in natural language.Proceedings of the AAAI Conference on Artificial Intelligence, 2020

    Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. PIQA: Reasoning about physical commonsense in natural language.Proceedings of the AAAI Conference on Artificial Intelligence, 2020

    2020

  2. [10]

    Striped attention: Faster ring attention for causal transformers.arXiv preprint arXiv:2311.09431, 2023

    William Brandon, Aniruddha Nrusimha, Kevin Qian, Zachary Ankner, Tian Jin, Zhiye Song, and Jonathan Ragan-Kelley. Striped attention: Faster ring attention for causal transformers.arXiv preprint arXiv:2311.09431, 2023

    2023 arXiv

  3. [11]

    Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374, 2021

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, et al. Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374, 2021

    2021 arXiv

  4. [12]

    Think you have solved question answering? try ARC, the AI2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try ARC, the AI2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018

    2018 arXiv

  5. [13]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, et al. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    2021 arXiv

  6. [14]

    DeepSeek-V2: A strong, economical, and efficient mixture-of-experts language model

    DeepSeek-AI. DeepSeek-V2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434, 2024. doi: 10.48550/arXiv.2405.04434

    2024 doi arXiv

  7. [15]

    DeepSeek-V3 technical report.arXiv preprint arXiv:2412.19437, 2024

    DeepSeek-AI. DeepSeek-V3 technical report.arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  8. [16]

    DeepEP: Hybrid expert-parallel communication

    DeepSeek-AI. DeepEP: Hybrid expert-parallel communication. https://github.com/deepseek-ai/ DeepEP/tree/hybrid-ep, 2025. HybridEP branch of the DeepEP repository

    2025

  9. [17]

    DeepSeek-V4: Towards highly efficient million-token context intelligence.arXiv preprint arXiv:2606.19348, June 2026

    DeepSeek-AI. DeepSeek-V4: Towards highly efficient million-token context intelligence.arXiv preprint arXiv:2606.19348, June 2026. doi: 10.48550/arXiv.2606.19348

    2026 doi

  10. [18]

    Beyond benchmarks: Matharena as an evaluation platform for mathematics with llms.arXiv preprint arXiv:2605.00674, 2026

    Jasper Dekoninck, Nikola Jovanovi´c, Tim Gehrunger, K´ari R¨ognvaldsson, Ivo Petrov, Chenhao Sun, and Martin Vechev. Beyond benchmarks: Matharena as an evaluation platform for mathematics with llms.arXiv preprint arXiv:2605.00674, 2026

    2026 arXiv

  11. [19]

    GLM-5: From vibe coding to agentic engineering.arXiv preprint arXiv:2602.15763, February 2026

    GLM-5 Team. GLM-5: From vibe coding to agentic engineering.arXiv preprint arXiv:2602.15763, February 2026. doi: 10.48550/arXiv.2602.15763

    2026 doi arXiv

  12. [20]

    Measuring massive multitask language understanding.International Conference on Learning Representations, 2021

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding.International Conference on Learning Representations, 2021

    2021

  13. [21]

    Measuring mathematical problem solving with the MATH dataset.NeurIPS Datasets and Benchmarks, 2021

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset.NeurIPS Datasets and Benchmarks, 2021

    2021

  14. [22]

    Liger kernel: Efficient triton kernels for LLM training.arXiv preprint arXiv:2410.10989, 2024

    Pin-Lun Hsu, Yun Dai, Vignesh Kothapalli, Qingquan Song, Shao Tang, Siyu Zhu, Steven Shimizu, Shivam Sahni, Haowen Ning, and Yanning Chen. Liger kernel: Efficient triton kernels for LLM training.arXiv preprint arXiv:2410.10989, 2024

    arXiv

  15. [23]

    Aa-omniscience: Evaluat- ing cross-domain knowledge reliability in large language models.arXiv preprint arXiv:2511.13029, 2025

    Declan Jackson, William Keating, George Cameron, and Micah Hill-Smith. Aa-omniscience: Evaluat- ing cross-domain knowledge reliability in large language models.arXiv preprint arXiv:2511.13029, 2025

    2025

  16. [24]

    DeepSpeed Ulysses: System optimizations for enabling training of extreme long sequence transformer models.arXiv preprint arXiv:2309.14509, 2023

    Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Shuaiwen Leon Song, Samyam Rajbhandari, and Yuxiong He. DeepSpeed Ulysses: System optimizations for enabling training of extreme long sequence transformer models.arXiv preprint arXiv:2309.14509, 2023

    2023 arXiv

  17. [25]

    Itbench: Evaluating ai agents across diverse real-world it automation tasks.arXiv preprint arXiv:2502.05352, 2025

    Saurabh Jha, Rohan Arora, Yuji Watanabe, Takumi Yanagawa, Yinfang Chen, Jackson Clark, Bhavya Bhavya, Mudit Verma, Harshit Kumar, Hirokuni Kitahara, et al. Itbench: Evaluating ai agents across diverse real-world it automation tasks.arXiv preprint arXiv:2502.05352, 2025. 27

    2025 arXiv

  18. [26]

    DCP: Addressing input dynamism in long-context training via dynamic context parallelism

    Chenyu Jiang, Zhenkun Cai, Ye Tian, Zhen Jia, Yida Wang, and Chuan Wu. DCP: Addressing input dynamism in long-context training via dynamic context parallelism. InProceedings of the 31st ACM Symposium on Operating Systems Principles, 2025. doi: 10.1145/3731569.3764849

    2025

  19. [27]

    Swe-bench: Can language models resolve real-world github issues? InInternational Conference on Learning Representations, volume 2024, pages 54107–54157, 2024

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? InInternational Conference on Learning Representations, volume 2024, pages 54107–54157, 2024

    2024

  20. [28]

    Muon: An optimizer for hidden layers in neural networks

    Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks. Online technical report, 2024. URLhttps://kellerjordan.github.io/posts/muon/

    2024

  21. [29]

    Kimi K2: Open agentic intelligence.arXiv preprint arXiv:2507.20534, 2025

    Kimi Team. Kimi K2: Open agentic intelligence.arXiv preprint arXiv:2507.20534, 2025

    2025 arXiv

  22. [30]

    Kimi K2.5: Visual agentic intelligence.arXiv preprint arXiv:2602.02276, February 2026

    Kimi Team. Kimi K2.5: Visual agentic intelligence.arXiv preprint arXiv:2602.02276, February 2026. doi: 10.48550/arXiv.2602.02276

    2026 doi arXiv

  23. [31]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with PagedAttention. InProceedings of the 29th ACM Symposium on Operating Systems Princip...

    2023

  24. [32]

    TorchTitan: One-stop PyTorch-native solution for production-ready LLM pre-training

    Wanchao Liang, Tianyu Liu, Less Wright, Will Constable, Andrew Gu, Chien-Chin Huang, Iris Zhang, Wei Feng, Howard Huang, Junjie Wang, Sanket Purandare, Gokul Nadathur, and Stratos Idreos. TorchTitan: One-stop PyTorch-native solution for production-ready LLM pre-training. In In...

    2025 doi arXiv

  25. [33]

    Motif 2 12.7b technical report.arXiv preprint arXiv:2511.07464, 2025

    Junghwan Lim, Sungmin Lee, Dongseok Kim, Taehyun Kim, Eunhwan Park, et al. Motif 2 12.7b technical report.arXiv preprint arXiv:2511.07464, 2025

    2025

  26. [34]

    Every step evolves: Scaling reinforcement learning for trillion-scale thinking model.arXiv preprint arXiv:2510.18855, 2025

    Ling Team, Anqi Shen, Baihui Li, Bin Hu, Bin Jing, Cai Chen, Chao Huang, Chao Zhang, Chaokun Yang, Cheng Lin, et al. Every step evolves: Scaling reinforcement learning for trillion-scale thinking model.arXiv preprint arXiv:2510.18855, 2025. doi: 10.48550/arXiv.2510.18855

    doi

  27. [35]

    Smith, and Yejin Choi

    Alisa Liu, Jonathan Hayase, Valentin Hofmann, Sewoong Oh, Noah A. Smith, and Yejin Choi. SuperBPE: Space travel for language models.arXiv preprint arXiv:2503.13423, 2025

    2025 arXiv

  28. [36]

    Ring attention with blockwise transformers for near- infinite context.arXiv preprint arXiv:2310.01889, 2023

    Hao Liu, Matei Zaharia, and Pieter Abbeel. Ring attention with blockwise transformers for near- infinite context.arXiv preprint arXiv:2310.01889, 2023

    2023 arXiv

  29. [37]

    Muon is scalable for LLM training.arXiv preprint arXiv:2502.16982, 2025

    Jingyuan Liu, Jianlin Su, Xingcheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du, Yidao Qin, Weixin Xu, Enzhe Lu, Junjie Yan, et al. Muon is scalable for LLM training.arXiv preprint arXiv:2502.16982, 2025

    2025 arXiv

  30. [38]

    Towards robust mathematical reasoning

    Minh-Thang Luong, Dawsen Hwang, Hoang H Nguyen, Golnaz Ghiasi, Yuri Chervonyi, Insuk Seo, Junsu Kim, Garrett Bingham, Jonathan Lee, Swaroop Mishra, et al. Towards robust mathematical reasoning. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Proce...

    2025

  31. [39]

    Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces

    Mike Merrill, Alexander Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Shin, Thomas Walshe, E Kelly Buchanan, et al. Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces. InInternational Conference on Learning Re...

    2026

  32. [40]

    Recipes for pre-training LLMs with MXFP8.arXiv preprint arXiv:2506.08027, 2025

    Asit Mishra, Dusan Stosic, Simon Layton, and Paulius Micikevicius. Recipes for pre-training LLMs with MXFP8.arXiv preprint arXiv:2506.08027, 2025

    2025 arXiv

  33. [41]

    Jordan, and Ion Stoica

    Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang, William Paul, Michael I. Jordan, and Ion Stoica. Ray: A distributed framework for emerging AI applications. In13th USENIX Symposium on Operating Systems Des...

    2018

  34. [42]

    Motif 3 training example

    Motif Technologies. Motif 3 training example. https://github.com/MotifTechnologies/ motif3-training-example, 2026. Reference implementation of the training-system optimizations

    2026

  35. [43]

    Scalable training of mixture-of-experts models with megatron core.arXiv preprint arXiv:2603.07685, 2026

    NVIDIA. Scalable training of mixture-of-experts models with megatron core.arXiv preprint arXiv:2603.07685, 2026. 28

    2026

  36. [44]

    Nemotron post-training v3

    NVIDIA. Nemotron post-training v3. Hugging Face dataset collection, 2026. URL https:// huggingface.co/collections/nvidia/nemotron-post-training-v3. Accessed August 7, 2026

    2026

  37. [45]

    Nemotron pre-training datasets

    NVIDIA. Nemotron pre-training datasets. Hugging Face dataset collection, 2026. URL https: //huggingface.co/collections/nvidia/nemotron-pre-training-datasets . Accessed August 5, 2026

    2026

  38. [46]

    NeMo Gym: Environments for evaluating and improving models and agents

    NVIDIA. NeMo Gym: Environments for evaluating and improving models and agents. https: //github.com/NVIDIA-NeMo/Gym, 2026. Open-source environment and verifier library

    2026

  39. [47]

    NeMo RL: A scalable and efficient post-training library

    NVIDIA. NeMo RL: A scalable and efficient post-training library. https://github.com/ NVIDIA-NeMo/RL, 2026. Open-source reinforcement-learning and post-training library

    2026

  40. [48]

    Nemotron 3 Ultra: Open, efficient mixture-of-experts hybrid mamba-transformer model for agentic reasoning.arXiv preprint arXiv:2606.15007, 2026

    NVIDIA. Nemotron 3 Ultra: Open, efficient mixture-of-experts hybrid mamba-transformer model for agentic reasoning.arXiv preprint arXiv:2606.15007, 2026. doi: 10.48550/arXiv.2606.15007

    2026 doi

  41. [49]

    gpt-oss: Reference implementations for gpt-oss-120b and gpt-oss-20b

    OpenAI. gpt-oss: Reference implementations for gpt-oss-120b and gpt-oss-20b. GitHub repository,

  42. [50]

    TorchAO: PyTorch-native training-to-serving model optimization.ICML Workshop on Championing Open-source Development, 2025

    Andrew Or, Apurva Jain, Daniel Vega-Myhre, Jesse Cai, Charles David Hernandez, Zhenrui Zheng, Driss Guessous, Vasiliy Kuznetsov, Christian Puhrsch, Mark Saroufim, Supriya Rao, Thien Tran, and Aleksandar Samardˇzi´c. TorchAO: PyTorch-native training-to-serving model optimizatio...

    2025

  43. [51]

    Gdpval: Evaluating ai model performance on real-world economically valuable tasks

    Tejal Patwardhan, Rachel Dias, Elizabeth Proehl, Grace Kim, Michele Wang, Olivia Watkins, Simon Fishman, Marwan Aljubeh, Phoebe Thacker, Laurance Fauconnet, et al. Gdpval: Evaluating ai model performance on real-world economically valuable tasks. InInternational Conference on ...

    2026

  44. [52]

    Humanity’s last exam.arXiv preprint arXiv:2501.14249, 2025

    Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, et al. Humanity’s last exam.arXiv preprint arXiv:2501.14249, 2025

    2025 arXiv

  45. [53]

    On data engineering for scaling llm terminal capabilities, 2026

    Renjie Pi, Grace Lam, Mohammad Shoeybi, Pooya Jannaty, Bryan Catanzaro, and Wei Ping. On data engineering for scaling llm terminal capabilities, 2026. URLhttps://arxiv.org/abs/2602.21193

    2026

  46. [54]

    Generalizing verifiable instruction following.Advances in Neural Information Processing Systems, 38, 2025

    Valentina Pyatkin, Saumya Malik, Victoria Graf, Hamish Ivison, Shengyi Huang, Pradeep Dasigi, Nathan Lambert, and Hanna Hajishirzi. Generalizing verifiable instruction following.Advances in Neural Information Processing Systems, 38, 2025

    2025

  47. [55]

    Gated attention for large language models: Non-linearity, sparsity, and attention-sink-free.arXiv preprint arXiv:2505.06708,

    Zihan Qiu, Zekun Wang, Bo Zheng, Zeyu Huang, Kaiyue Wen, Songlin Yang, Rui Men, Le Yu, Fei Huang, Suozhi Huang, Dayiheng Liu, Jingren Zhou, and Junyang Lin. Gated attention for large language models: Non-linearity, sparsity, and attention-sink-free.arXiv preprint arXiv:2505.06708,

    arXiv

  48. [56]

    A unified view of attention and residual sinks: Outlier-driven rescaling is essential for transformer training.arXiv preprint arXiv:2601.22966, 2026

    Zihan Qiu, Zeyu Huang, Kaiyue Wen, Peng Jin, Bo Zheng, Yuxin Zhou, et al. A unified view of attention and residual sinks: Outlier-driven rescaling is essential for transformer training.arXiv preprint arXiv:2601.22966, 2026

    2026

  49. [57]

    doi: 10.48550/arXiv.2505.06708

    doi arXiv

  50. [58]

    Gpqa: A graduate-level google-proof q&a benchmark.arXiv preprint arXiv:2311.12022, 2023

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark.arXiv preprint arXiv:2311.12022, 2023

    2023 arXiv

  51. [59]

    Qwen3-Next: Towards ultimate training and inference efficiency.https://qwen.ai/ blog?id=4074cca80393150c248e508aa62983f9cb7d27cd, September 2025

    Qwen Team. Qwen3-Next: Towards ultimate training and inference efficiency.https://qwen.ai/ blog?id=4074cca80393150c248e508aa62983f9cb7d27cd, September 2025. Official Qwen technical blog

    2025

  52. [60]

    WinoGrande: An adver- sarial winograd schema challenge at scale.Proceedings of the AAAI Conference on Artificial Intelligence, 2020

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WinoGrande: An adver- sarial winograd schema challenge at scale.Proceedings of the AAAI Conference on Artificial Intelligence, 2020

    2020

  53. [61]

    Microscaling data formats for deep learning.arXiv preprint arXiv:2310.10537, 2023

    Bita Darvish Rouhani, Ritchie Zhao, Ankit More, Mathew Hall, Alireza Khodamoradi, Summer Deng, Dhruv Choudhary, Marius Cornea, et al. Microscaling data formats for deep learning.arXiv preprint arXiv:2310.10537, 2023. doi: 10.48550/arXiv.2310.10537

    2023 doi arXiv

  54. [62]

    RoFormer: Enhanced transformer with rotary position embedding.arXiv preprint arXiv:2104.09864, 2021

    Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. RoFormer: Enhanced transformer with rotary position embedding.arXiv preprint arXiv:2104.09864, 2021

    2021 arXiv

  55. [63]

    τ-knowledge: Evaluating conversational agents over unstructured knowledge.arXiv preprint arXiv:2603.04370,

    Quan Shi, Alexandra Zytek, Pedram Razavi, Karthik Narasimhan, and Victor Barres. τ-knowledge: Evaluating conversational agents over unstructured knowledge.arXiv preprint arXiv:2603.04370,

  56. [64]

    Scicode: A research coding benchmark curated by scientists

    Minyang Tian, Luyu Gao, Shizhuo D Zhang, Xinan Chen, Cunwei Fan, Xuefei Guo, Roland Haas, Pan Ji, Kittithat Krongchon, Yao Li, et al. Scicode: A research coding benchmark curated by scientists. Advances in Neural Information Processing Systems, 37:30624–30650, 2024

    2024

  57. [65]

    Auxiliary-loss-free load balancing strategy for mixture-of-experts.arXiv preprint arXiv:2408.15664, 2024

    Lean Wang, Huazuo Gao, Chenggang Zhao, Xu Sun, and Damai Dai. Auxiliary-loss-free load balancing strategy for mixture-of-experts.arXiv preprint arXiv:2408.15664, 2024

    2024 arXiv

  58. [66]

    Grouped differential attention.arXiv preprint arXiv:2510.06949, 2025

    Motif Technologies. Grouped differential attention.arXiv preprint arXiv:2510.06949, 2025

    2025

  59. [67]

    MiMo-V2-Flash technical report.arXiv preprint arXiv:2601.02780, 2026

    Xiaomi LLM-Core Team. MiMo-V2-Flash technical report.arXiv preprint arXiv:2601.02780, 2026. doi: 10.48550/arXiv.2601.02780

    2026 doi arXiv

  60. [68]

    mHC: Manifold-constrained hyper-connections.arXiv preprint arXiv:2512.24880, 2025

    Zhenda Xie, Yixuan Wei, Huanqi Cao, Chenggang Zhao, Chengqi Deng, Jiashi Li, Damai Dai, Huazuo Gao, Jiang Chang, Kuai Yu, Liang Zhao, Shangyan Zhou, Zhean Xu, Zhengyan Zhang, Wangding Zeng, Shengding Hu, Yuqing Wang, Jingyang Yuan, Lean Wang, and Wenfeng Liang. mHC: Manifold-c...

    2025 doi arXiv

  61. [69]

    MMLU-Pro: A more robust and challenging multi-task language understanding benchmark.arXiv preprint arXiv:2406.01574, 2024

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, et al. MMLU-Pro: A more robust and challenging multi-task language understanding benchmark.arXiv preprint arXiv:2406.01574, 2024

    2024 arXiv

  62. [70]

    Differential transformer.arXiv preprint arXiv:2410.05258, 2024

    Tianzhu Ye, Li Dong, Yuqing Xia, Yutao Sun, Yi Zhu, Gao Huang, and Furu Wei. Differential transformer.arXiv preprint arXiv:2410.05258, 2024. doi: 10.48550/arXiv.2410.05258

    2024 doi arXiv

  63. [71]

    DAPO: An open-source LLM reinforcement learning system at scale.arXiv preprint arXiv:2503.14476, 2025

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. DAPO: An open-source LLM reinforcement learning system at scale.arXiv preprint arXiv:2503.14476, 2025. doi: 10.48550/arXiv.2503.14476

    2025 doi arXiv

  64. [72]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    2025 arXiv

  65. [73]

    HellaSwag: Can a machine really finish your sentence?Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence?Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019

    2019

  66. [74]

    Root mean square layer normalization.arXiv preprint arXiv:1910.07467, 2019

    Biao Zhang and Rico Sennrich. Root mean square layer normalization.arXiv preprint arXiv:1910.07467, 2019

    1910 arXiv

  67. [75]

    FlashAttention-4: Algorithm and kernel pipelining co-design for asymmetric hardware scaling

    Ted Zadouri, Markus Hoehnerbach, Jay Shah, Timmy Liu, Vijay Thakkar, and Tri Dao. FlashAttention-4: Algorithm and kernel pipelining co-design for asymmetric hardware scaling. arXiv preprint arXiv:2603.05451, 2026

    2026

  68. [76]

    Probing the critical point (critpt) of ai reasoning: a frontier physics research benchmark.arXiv preprint arXiv:2509.26574, 2025

    Minhui Zhu, Minyang Tian, Xiaocheng Yang, Tianci Zhou, Lifan Yuan, Penghao Zhu, Eli Chertkov, Shengyan Liu, Yufeng Du, Ziming Ji, et al. Probing the critical point (critpt) of ai reasoning: a frontier physics research benchmark.arXiv preprint arXiv:2509.26574, 2025

    2025 arXiv

  69. [77]

    [ˆ\r\n\p{L}\p{N}]?[\p{Lu}\p{Lt}\p{Lm}\p{Lo}\p{M}]*[\p{Ll}\p{Lm}\p{Lo}\p{M}]+

    Zilin Zhu. Zigzag ring attention. GitHub repository, ring-flash-attention, 2024. 30 A Tokenizer A.1 Pre-tokenization Regexes The two tokenizer training stages use different pre-tokenization patterns. Both patterns are given below using regex syntax with Unicode property classe...

    2024

  70. [78]

    Gonzalez, Clark Barrett, and Ying Sheng

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. SGLang: Efficient execution of structured language model programs. InAdvances in Neural Information P...

    2024

  71. [2025]

    Reference PyTorch implementation

  72. [2026]

    doi: 10.48550/arXiv.2603.04370. 29

    doi

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.