Pith. sign in

REVIEW 5 major objections 8 minor 2 cited by

Pangu Pro MoE: Mixture of Grouped Experts for Efficient Sparsity

T0 review · 5 major / 8 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Pangu Pro MoE claims that routing each token to an equal number of experts per predefined group makes device-level load imbalance in MoE vanish by construction, and that the resulting 72B/16B model reaches 1148 tokens/s per card on Ascend…

desk verdict MoGE is a simple, correct routing constraint, but the throughput claims are vendor-reported until a matched MoE ablation isolates the effect. read the letter →

arxiv 2505.21411 v2 pith:FBOWBOYJ submitted 2025-05-27 cs.CL

Yehui Tang , Xiaosong Li , Fangcheng Liu , Wei Guo , Hang Zhou , Yaoyuan Wang , Kai Han , Xianzhi Yu
show 14 more authors
Jinpeng Li Hui Zang Fei Mi Xiaojun Meng Zhicheng Liu Hanting Chen Binfan Zheng Can Chen Youliang Yan Ruiming Tang Peifeng Qin Xinghao Chen Dacheng Tao Yunhe Wang (and Other Contributors)
This is my paper · ORCID
classification cs.CL
keywords MixtureofGroupedExpertsloadbalancingexpertparallelismsparseactivationinferencethroughputAscendNPUspeculativedecoding
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Mixture-of-experts models activate only a fraction of their parameters per token, but the fraction varies wildly: popular experts get overloaded and their devices stall the whole pipeline. This paper claims the imbalance can be removed at the architectural level rather than patched with heuristics. Its Mixture of Grouped Experts (MoGE) partitions the expert pool into groups, assigns each group to a device, and forces every token to activate the same number of experts from each group, making the load-imbalance score exactly zero for every token and every batch. The authors build Pangu Pro MoE on this design — 72B total parameters, 16.5B activated per token, trained on 13 trillion tokens — and report 1148 tokens/s per card within a 100 ms latency budget on Ascend 800I A2 hardware, with benchmark scores that match or beat 32B-scale open models while activating roughly half their parameters. A sympathetic reader would care because the paper makes a structural promise: balanced execution comes from the routing rule itself, not from hoping the router learns to behave, which would let MoE deployments skip load-balancing heuristics and over-provisioning.

What carries the argument

The load-bearing object is the MoGE routing rule: a global softmax over all $N$ experts followed by an independent Top-$K'$ selection inside each of $M$ predefined expert groups, with the per-group selections concatenated into the final gating vector (Eq. 6). Because each group is mapped to a distinct device, this rule converts load balance from an emergent property into a deterministic constraint — every device handles exactly $K'$ expert computations per token, so the imbalance score is identically zero. Pangu Pro MoE runs the sharpest version of the rule with $K' = 1$: 64 routed experts arranged in 8 groups of 8, one expert drawn from each group, plus 4 shared experts, for 16.5B activated parameters. An auxiliary load-balancing loss is retained, but it is computed from the global softmax scores rather than per-group, so it only shapes usage within groups; inter-group balance no longer depends on the loss at all.

What would settle it

Deploy Pangu Pro MoE's exact 64-expert, 8-group configuration and a standard global Top-K MoE with identical total parameters, active-expert count, width, depth, data, and hardware, then measure per-card decode throughput and the wall-clock spread across the four accelerators for one decoding step. MoGE predicts near-zero inter-device spread on the expert operators for every batch, while the Top-K baseline predicts a spread that grows as batch size shrinks; if the standard MoE matches the throughput under the same 100 ms budget, or the measured MoGE spread is not dramatically smaller, the IS=0 guarantee is not what drives the reported speedups. A cheaper check already sits in the paper's data: the reported >50% reduction in maximum execution-time disparity for the permute and gmm_up operators is an operator-level, not end-to-end, demonstration.

Watch

Extended reading notes

Core claim

The paper's central claim is that group-balanced routing suffices to eliminate device-level load imbalance in distributed MoE execution. Formally, with $N$ experts partitioned into $M$ groups and $K' = K/M$ experts activated per group per token, the Imbalance Score $IS(X) = (\max_i T_i(X) - \min_i T_i(X))/|X|$ is zero for every batch, whereas Monte Carlo simulation of standard global Top-K routing shows nearly a probability-one chance of nonzero imbalance at small batch sizes. The authors establish the point by construction and then demonstrate it empirically: expert activation in the first MoE layer of Pangu Pro MoE sits near the theoretical 12.5% per expert, against up to 30% for the busiest DeepSeek-V2 expert, and the maximum execution-time disparity of the permute and gmm_up operators drops by over 50%. On the strength of this balance they report that Pangu Pro MoE, a 72B-parameter model activating 16.5B per token, reaches 1148 tokens/s per card in high-concurrency decode on Ascend 800I A2 (1528 tokens/s with multi-token-prediction decoding) and outperforms dense 32B and 72B models under the same latency constraints, while matching or beating open 32B-scale models on many benchmarks.

Load-bearing premise

The reported speedups assume expert computation is the dominant and roughly uniform cost across the devices that hold the experts, so that forcing equal per-device expert counts forces equal wall-clock time; the paper never compares MoGE against a standard MoE at the same scale to verify that the routing rule itself causes the gains, and its own profiling puts attention, KV-cache traffic, and communication at large shares of inference time (Sections 4.2.3–4.2.4).

Editorial extensions

If this is right

  • In any deployment that pins expert groups to devices, every device receives exactly the same number of expert computations per token, so the straggler bottleneck disappears at every batch size, including single-token batches.
  • Pangu Pro MoE reaches 1148 tokens/s per card (1528 tokens/s with multi-token-prediction decoding) on Ascend 800I A2 within a 100 ms latency budget, exceeding 32B and 72B dense models by 18% and 97% respectively on the same hardware.
  • On the training side, combining MoGE with hierarchical expert-parallel communication and pipeline overlap raises Model FLOPs Utilization by 35% over the baseline configuration.
  • Expert utilization becomes nearly uniform — around 12.5% of tokens per expert in the first layer versus up to 30% for the busiest DeepSeek-V2 expert — so the model makes fuller use of its expert capacity.
  • On the cheaper Ascend 300I Duo, the same model sustains 201 tokens/s per card (321 tokens/s with MTP) within 100 ms latency, which the authors point to as a favorable cost-to-performance ratio for sub-100B inference.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the IS=0 guarantee matters most exactly where Top-K imbalance is worst — small batches and skewed routing distributions — which is also the regime where the paper's own profiling shows attention and weight transfer dominating wall-clock time, so the routing rule is a necessary but not sufficient condition for the reported speedups.
  • Editorial inference: because the paper never ablates MoGE against a standard Top-K MoE at matched scale, the causal link from the group constraint to the measured throughput is not isolated; a fair test would hold total parameters, active parameters, data, and hardware fixed and vary only the routing rule.
  • Editorial inference: MoGE trades routing freedom for a hard balance guarantee — when a token's best experts concentrate in one group, it is forced to take weaker experts elsewhere — so the design implies a quality-versus-balance trade-off curve that the paper does not map, especially at larger $K'$.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 8 minor

Summary. The paper introduces Mixture of Grouped Experts (MoGE), an MoE variant that partitions experts into groups and forces each token to activate a fixed number of experts from every group, thereby guaranteeing an Imbalance Score (IS) of 0 by construction. The authors build Pangu Pro MoE, a 72B-total/16B-activated MoE trained on 13T tokens and optimized for Ascend NPUs, and report inference throughput of 1148 tokens/s per card on Ascend 800I A2. The paper also describes system-level optimizations (H2P parallelism, MulAttention, SwiftGMM, quantization) and reports benchmark results against dense and MoE baselines. The central claim is that MoGE's group-balanced routing produces better load balance and significantly higher training and inference throughput than conventional MoE.

Significance. If the causal efficiency claim were established, MoGE would be a valuable architectural contribution: it is simple, requires no change to the expert computation itself, and its IS=0 guarantee is mathematically true by construction. The paper also contains substantial engineering content, including a hierarchical hybrid parallel inference scheme, fused kernels, and expert-aware quantization, and it reports a trained 72B model with extensive benchmark numbers. These strengths are real. However, the paper does not yet demonstrate that the MoGE routing constraint is the cause of the reported throughput gains; all efficiency comparisons are against dense models, and the paper's own profiling suggests that expert-count balance is only one component of end-to-end latency. The benchmark claims are also stronger than the data support. The significance is therefore conditional on a controlled ablation and a more careful causal analysis.

major comments (5)
  1. [Section 2.2, Eq. (3)-(7); Section 5.3; Section 4.2.3] The IS=0 guarantee is a statement about expert activation counts per device, not about wall-clock execution time. Section 4.2.3 reports that attention can consume 30%-50% of total inference time and that GMM accounts for over 50% of end-to-end latency in high-concurrency settings, while Section 5.3 reports that weight transfer is only 29% of decode latency. Under these conditions, equal per-device expert counts do not necessarily equalize device finish times, because per-expert token counts, grouped-GEMM shapes, attention, and communication all vary. The paper should provide a measured per-device time breakdown for a MoGE deployment versus a Top-K MoE deployment, or otherwise directly demonstrate that the MoGE constraint is the causal source of the 97%/18% decode throughput advantages reported in Table 6.
  2. [Section 5.3, Table 6] All inference efficiency comparisons are against 32B and 72B dense models that differ in hidden size, layer count, KV-cache size, communication volume, quantization, and custom kernels. No conventional Top-K MoE with matched total and activated parameter counts is evaluated under the same system optimizations. Without such a baseline, the reported throughput advantages cannot be attributed to MoGE; they could arise from the smaller activated parameter count, the smaller hidden dimension, or the custom Ascend kernels. A same-scale Top-K MoE baseline, ideally with the same H2P parallelism and kernels, is required to support the central efficiency claim.
  3. [Figure 4 caption; Section 4.2.3] The claim that MoGE mitigates computational load imbalance 'by over 50%' is based on reduced maximum disparity in execution time for the permute and gmm_up operators only. The comparison baseline is not described: no configuration, routing scheme, batch size, or measurement procedure is given. This component-level measurement does not establish end-to-end throughput improvement, especially because the same section says attention and GMM dominate different regimes. The paper should either report full end-to-end time distributions for MoGE versus Top-K MoE or temper the claim to the specific operators measured.
  4. [Section 4.2.1 (H2P parallelism) versus Section 2.2] The MoGE argument in Section 2.2 assumes that each expert group is mapped to a distinct device, so equal per-group activation counts imply equal per-device counts. The inference deployment, however, uses TP2+EP4 hybrid parallelism for expert modules, which splits expert matrices across devices and changes the mapping between groups and physical devices. The paper does not explain how the per-group constraint translates into per-device balance under this hybrid parallel scheme. Without this clarification, the theoretical IS=0 guarantee does not directly apply to the reported inference configuration.
  5. [Abstract; Section 5.2.2, Table 4] The abstract states that Pangu Pro MoE outperforms Qwen3-32B and GLM-Z1-32B, but Table 4 shows Pangu Pro MoE trailing Qwen3-32B on LiveCodeBench (59.6 vs. 62.6), MBPP+ (80.2 vs. 82.0), AIME2024 (79.2 vs. 80.4), AIME2025 (68.1 vs. 70.9), and MMLU-Redux (81.5 vs. 83.2). The claim of overall outperformance is therefore not supported by the presented numbers. The abstract and conclusion should be revised to describe the results as competitive rather than uniformly superior.
minor comments (8)
  1. [Abstract] The word 'surgence' should be 'surge' or 'resurgence'.
  2. [Section 5.1.1, 5.1.2, 5.2.1] The headings contain a typo: 'Evalutaion' should be 'Evaluation'.
  3. [Table 3] The baseline name appears as 'Llama-4-Scout-baset'; the final 't' appears to be a typo for 'base'.
  4. [Eq. (8) and Eq. (10)] The symbol α is used for both the auxiliary load-balancing loss weight in Eq. (8) and the quantization migration strength in Eq. (10). These are unrelated hyperparameters and should use different symbols.
  5. [Section 3.1.2] The training hardware is described as 'Ascend 800T A2', while inference experiments use 'Ascend 800I A2'. The relationship between these platforms should be clarified.
  6. [Figure 2(b)] The caption refers to 'right, blue bars' for the MoE distribution, but the figure shows MoGE as a yellow bar and MoE as blue bars; the color-description mapping should be corrected.
  7. [Table 2] The MFU improvement is reported only as a relative 35% increase with no absolute MFU values or wall-clock training throughput. Absolute numbers would make the claim more interpretable.
  8. [Figure 3] Simulation results are normalized relative to a randomly selected candidate, which makes it difficult to assess absolute throughput or to reproduce the selection of the final configuration. Reporting absolute simulated throughput values would help.

Circularity Check

1 steps flagged · score 2.0 of 10

No significant circularity: the only definitional identity is MoGE's guaranteed IS=0, which is transparent and not load-bearing for the empirical throughput claims.

  1. self definitional [Section 2.2, Eqs. (3) and (6), and the sentence 'This guarantees an Imbalance Score (IS) of 0 by design'.]
    "The core idea is to ensure that for each token, an equal number of expert computations are distributed to each of the devices. This guarantees an Imbalance Score (IS) of 0 by design ... IS(X) = 1/|X| (max_i T_i(X) - min_i T_i(X)) ... G'(h) = (TopK(S_1, K'), ..., TopK(S_M, K'))."

    The IS metric in Eq. (3) is defined as the normalized max-minus-min spread of per-device expert-computation counts T_i. MoGE's routing rule in Eq. (6) fixes exactly K' = K/M experts per group per token, and the groups are 'typically assigned to reside on a specific computational device.' Therefore, equal per-device counts, and hence IS=0, is a restatement of the routing constraint rather than an independent prediction. The paper is transparent about this ('by design'), and this identity does not by itself establish wall-clock load balance, since IS counts expert activations rather than execution time.

full rationale

The only candidate circular step is the load-balance guarantee: MoGE's group-balanced routing is exactly the constraint that makes the paper's own Imbalance Score zero. This is a legitimate design identity rather than a fitted or predicted result, and the paper does not disguise it as empirical evidence. The central throughput and efficiency claims are system measurements reported in Tables 5-7 against dense baselines, and the model-quality claims in Tables 3 and 4 use external benchmarks. No fitted parameter is renamed as a prediction, and no load-bearing argument relies on a self-citation or on a uniqueness theorem imported from the authors' prior work. The self-citations to Pangu Ultra MoE [34] and Pangu Ultra [47] are contextual and not load-bearing. The main validity risk is experimental confounding, because MoGE is never ablated against a matched Top-K MoE at the same scale; that is a correctness concern, not circularity.

Assumptions & free parameters 3 free parameters · 2 assumptions · 1 invented entities

The central load-balance claim rests on a definitional identity, not on fitted parameters; the main free parameters are undisclosed hyperparameters. The throughput claim further depends on the unverified assumption that expert compute dominates and is uniform enough for per-group counts to determine device latency.

free parameters (3)
  • Auxiliary load-balancing loss weight α = not specified in the paper
    Equation in Section 2.2 defines ℓaux = α Σ f_i p_i but the chosen value of α is never reported; the loss directly shapes routing behavior.
  • Quantization smoothing migration strength α (Eq. 10) = not specified
    The expert-aware quantization formula depends on a migration strength α that is not given.
  • Checkpoint merging weights λk (Eq. 9) = not specified
    The two-layer checkpoint merge uses per-group weights λk that are not reported.
assumptions (2)
  • domain assumption Expert execution time is roughly uniform and dominates device latency, so equal expert counts per device imply balanced wall-clock load.
    Section 2.1 motivates imbalance via expert counts; Section 4.2.3 shows attention is 30-50% of latency, so this is not always true.
  • domain assumption The benchmark evaluations of baseline models are conducted under identical settings and fair conditions.
    Section 5.2.1 states standardized settings but no raw logs or reproducibility artifacts are provided.
invented entities (1)
  • MoGE group-balanced routing constraint
    purpose: Guarantee each device handles an equal number of expert calls per token
    Introduced as the paper's key design; the only evidence is internal experiments, and no external implementation confirms the mechanism's benefits in isolation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Pangu Pro MoE: Mixture of Grouped Experts for Efficient Sparsity." pith.science (2026). https://pith.science/paper/FBOWBOYJ

@misc{pith2026250521411,
  author       = {Pith},
  title        = {Pith review of: Pangu Pro MoE: Mixture of Grouped Experts for Efficient Sparsity},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FBOWBOYJ}},
  note         = {Machine review of arXiv:2505.21411}
}
read the original abstract

The surgence of Mixture of Experts (MoE) in Large Language Models promises a small price of execution cost for a much larger model parameter count and learning capacity, because only a small fraction of parameters are activated for each input token. However, it is commonly observed that some experts are activated far more often than others, leading to system inefficiency when running the experts on different devices in parallel. Therefore, we introduce Mixture of Grouped Experts (MoGE), which groups the experts during selection and balances the expert workload better than MoE in nature. It constrains tokens to activate an equal number of experts within each predefined expert group. When a model execution is distributed on multiple devices, this architectural design ensures a balanced computational load across devices, significantly enhancing throughput, particularly for the inference phase. Further, we build Pangu Pro MoE on Ascend NPUs, a sparse model based on MoGE with 72 billion total parameters, 16 billion of which are activated for each token. The configuration of Pangu Pro MoE is optimized for Ascend 300I Duo and 800I A2 through extensive system simulation studies. Our experiments indicate that MoGE indeed leads to better expert load balancing and more efficient execution for both model training and inference on Ascend NPUs. The inference performance of Pangu Pro MoE achieves 1148 tokens/s per card and can be further improved to 1528 tokens/s per card by speculative acceleration, outperforming comparable 32B and 72B Dense models. Furthermore, we achieve an excellent cost-to-performance ratio for model inference on Ascend 300I Duo. Our studies show that Ascend NPUs are capable of training Pangu Pro MoE with massive parallelization to make it a leading model within the sub-100B total parameter class, outperforming prominent open-source models like GLM-Z1-32B and Qwen3-32B.

Figures

Figures reproduced from arXiv: 2505.21411 by the authors.

Figure 1
Figure 1. Illustration of the Mixture of Grouped Experts (MoGE) architecture. The [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Comparison of expert activation patterns and load imbalance between conventional Top-K routing in [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Simulation results of candidate model configurations. Throughput under 100 ms and 50 ms TPOT [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Overview of the inference system optimization. A [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Computation flow of the MulAttention operator. A large-packet KV transfer strategy is adopted to [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 6
Figure 6. Figure 6: Overview of SwiftGMM. (a) A tiling cache strategy leverages historical profiling to predict [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: Expert specialization in Pangu Pro MoE . Each subplot illustrates the token-to-expert distribution [PITH_FULL_IMAGE:figures/full_fig_p019_7.png]
Figure 8
Figure 8. Figure 8: Expert co-activation across three layers (shallow, middle, and deep), evaluated on a random 0.5 [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]
Figure 9
Figure 9. Figure 9: Intra-group expert distribution in Pangu Pro MoE . the observed token distributions closely align [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 10
Figure 10. Figure 10: Global expert distribution on the first MoE layer, evaluated using a random 0.5 [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SCOPE and SCION: A Benchmark and an Auditable Reference Pipeline for Schema Induction and Fusion from Text

    cs.AI 2026-05 conditional novelty 6.0 of 10

    A 24-dataset benchmark for inducing schema graphs from raw text, plus an auditable LLM-based pipeline that reports the highest scores on the benchmark's four schema-similarity metrics.

  2. Spectral Signatures of Large Language Models

    cs.CL 2026-07 accept novelty 5.0 of 10

    Layer-wise PL_Alpha_Hill spectral shapes of LLM weight matrices form a robust, data-free signature that tracks lineage, clusters families, and proxies broad performance.

Reference graph

Works this paper leans on

60 extracted references · 28 canonical work pages · cited by 2 Pith papers

  1. [1]

    The llama 4 herd: The beginning of a new era of natively multimodal ai innovation

    Meta AI. The llama 4 herd: The beginning of a new era of natively multimodal ai innovation. https: //ai.meta.com/blog/llama-4-multimodal-intelligence/, 2025. Accessed: 2025-04-05

    work page 2025

  2. [2]

    Cai, Michael Terry, Quoc V

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V . Le, and Charles Sutton. Program synthesis with large language models.ArXiv, abs/2108.07732, 2021

    arXiv 2021

  3. [3]

    Piqa: Reasoning about physical commonsense in natural language

    Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. InAAAI Conference on Artificial Intelligence, 2019

    work page 2019

  4. [4]

    A survey on mixture of experts.arXiv preprint arXiv:2407.06204, 2024

    Weilin Cai, Juyong Jiang, Fan Wang, Jing Tang, Sunghun Kim, and Jiayi Huang. A survey on mixture of experts.arXiv preprint arXiv:2407.06204, 2024

    arXiv 2024

  5. [5]

    Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-V oss, William H

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé, Jared Kaplan, Harrison Ed- wards, Yura Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mo Bavarian, Clemens Winter, P...

    arXiv 2021

  6. [6]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    arXiv

  7. [7]

    A span-extraction dataset for Chinese machine reading comprehension

    Yiming Cui, Ting Liu, Wanxiang Che, Li Xiao, Zhipeng Chen, Wentao Ma, Shijin Wang, and Guoping Hu. A span-extraction dataset for Chinese machine reading comprehension. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors,Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conf...

    work page 2019

  8. [8]

    Glam: Efficient scaling of language models with mixture-of-experts

    Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. Glam: Efficient scaling of language models with mixture-of-experts. InInternational Conference on Machine Learning, pages 5547–5569. PMLR, 2022

    2022

Show all 60 references
  1. [9]

    Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs

    Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. InNorth American Chapter of the Association for Computational Linguistics, 2019

    2019

  2. [10]

    Model tells you what to discard: Adaptive kv cache compression for llms.arXiv preprint arXiv:2310.01801, 2023

    Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. Model tells you what to discard: Adaptive kv cache compression for llms.arXiv preprint arXiv:2310.01801, 2023

    2023 arXiv

  3. [11]

    Chatglm: A family of large language models from glm-130b to glm-4 all tools, 2024

    Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, Hao Yu, Hongning Wang, Jiadai Sun, Jiajie Zhang, Jiale Cheng, Jiayi Gui, Jie Tang, Jing Zhang, Juanzi Li, Lei Zhao, Lindong Wu, Lucen Zhong, Mingdao Liu, Minlie H...

    2024

  4. [12]

    Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300, 2020

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300, 2020

    2009 arXiv

  5. [13]

    Measuring mathematical problem solving with the math dataset.arXiv preprint arXiv:2103.03874, 2021

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset.arXiv preprint arXiv:2103.03874, 2021

    2021 arXiv

  6. [14]

    C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models.ArXiv, abs/2305.08322, 2023

    Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, Fanchao Qi, Yao Fu, Maosong Sun, and Junxian He. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models.ArXiv, abs/...

    2023 arXiv

  7. [15]

    Livecodebench: Holistic and contamination free evaluation of large language models for code.arXiv preprint arXiv:2403.07974, 2024

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code.arXiv preprint arXiv:2403.07974, 2024

    2024 arXiv

  8. [16]

    Mixtral of experts.arXiv preprint arXiv:2401.04088, 2024

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts.arXiv preprint arXiv:2401.04088, 2024

    2024 arXiv

  9. [17]

    Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard H. Hovy. Race: Large-scale reading comprehension dataset from examinations.ArXiv, abs/1704.04683, 2017

    2017 arXiv

  10. [18]

    {GS}hard: Scaling giant models with conditional computation and automatic sharding

    Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. {GS}hard: Scaling giant models with conditional computation and automatic sharding. InInternational Conference on Learning Representations, 2021

    2021

  11. [19]

    Cmmlu: Measuring massive multitask language understanding in chinese.arXiv preprint arXiv:2306.09212, 2023

    Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. Cmmlu: Measuring massive multitask language understanding in chinese.arXiv preprint arXiv:2306.09212, 2023

    2023 arXiv

  12. [20]

    Gonzalez, and Ion Stoica

    Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica. From live data to high-quality benchmarks: The arena-hard pipeline, April 2024

    2024

  13. [21]

    Ccpm: A chinese classical poetry matching dataset, 2021

    Wenhao Li, Fanchao Qi, Maosong Sun, Xiaoyuan Yi, and Jiarui Zhang. Ccpm: A chinese classical poetry matching dataset, 2021

    2021

  14. [22]

    Kvtuner: Sensitivity-aware layer-wise mixed precision kv cache quantization for efficient and nearly lossless llm inference, 2025

    Xing Li, Zeyu Xing, Yiming Li, Linping Qu, Hui-Ling Zhen, Wulong Liu, Yiwu Yao, Sinno Jialin Pan, and Mingxuan Yuan. Kvtuner: Sensitivity-aware layer-wise mixed precision kv cache quantization for efficient and nearly lossless llm inference, 2025

    2025

  15. [23]

    Davinci: A scalable architecture for neural network computing

    Heng Liao, Jiajin Tu, Jing Xia, and Xiping Zhou. Davinci: A scalable architecture for neural network computing. In2019 IEEE Hot Chips 31 Symposium (HCS), pages 1–44. IEEE Computer Society, 2019

    2019

  16. [24]

    Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model.arXiv preprint arXiv:2405.04434, 2024

    Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model.arXiv preprint arXiv:2405.04434, 2024

    2024 arXiv

  17. [25]

    Kivi: A tuning-free asymmetric 2bit quantization for kv cache.arXiv preprint arXiv:2402.02750, 2024

    Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. Kivi: A tuning-free asymmetric 2bit quantization for kv cache.arXiv preprint arXiv:2402.02750, 2024

    2024 arXiv

  18. [26]

    Codeforces

    MAA. Codeforces. American Invitational Mathematics Examination - AIME 2024, 2024. https:// maa.org/math-competitions/american-invitational-mathematics-examination-aime

    2024

  19. [27]

    Codeforces

    MAA. Codeforces. American Invitational Mathematics Examination - AIME 2025, 2025. https:// maa.org/math-competitions/american-invitational-mathematics-examination-aime

    2025

  20. [28]

    Gpqa: A graduate-level google-proof q&a benchmark

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. InFirst Conference on Language Modeling, 2024

    2024

  21. [29]

    Winogrande: An adversarial winograd schema challenge at scale, 2019

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale, 2019

    2019

  22. [30]

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.arXiv preprint arXiv:1701.06538, 2017

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.arXiv preprint arXiv:1701.06538, 2017

    2017 arXiv

  23. [31]

    Language models are multilingual chain-of-thought reasoners

    Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush V osoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. Language models are multilingual chain-of-thought reasoners. InThe Eleventh International Conference on Le...

    2023

  24. [32]

    Investigating prior knowledge for challenging chinese machine reading comprehension, 2019

    Kai Sun, Dian Yu, Dong Yu, and Claire Cardie. Investigating prior knowledge for challenging chinese machine reading comprehension, 2019

    2019

  25. [33]

    Le, Ed H

    Mirac Suzgun, Nathan Scales, Nathanael Scharli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V . Le, Ed H. Chi, Denny Zhou, and Jason Wei. Challenging big-bench tasks and whether chain-of-thought can solve them. InAnnual Meeting of the Association for...

    2022

  26. [34]

    Pangu ultra moe: How to train your big moe on ascend npus, 2025

    Yehui Tang, Yichun Yin, Yaoyuan Wang, Hang Zhou, Yu Pan, Wei Guo, Ziyang Zhang, Miao Rang, Fangcheng Liu, Naifu Zhang, Binghan Li, Yonghan Dong, Xiaojun Meng, Yasheng Wang, Dong Li, Yin Li, Dandan Tu, Can Chen, Youliang Yan, Fisher Yu, Ruiming Tang, Yunhe Wang, Botian Huang, B...

    2025

  27. [35]

    Gemma 3 technical report.arXiv preprint arXiv:2503.19786, 2025

    Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report.arXiv preprint arXiv:2503.19786, 2025

    2025 arXiv

  28. [36]

    Supergpqa: Scaling llm evaluation across 285 graduate disciplines, 2025

    M-A-P Team, Xinrun Du, Yifan Yao, Kaijing Ma, Bingli Wang, Tianyu Zheng, Kang Zhu, Minghao Liu, Yiming Liang, Xiaolong Jin, Zhenlin Wei, Chujie Zheng, Kaixing Deng, Shuyue Guo, Shian Jia, Sichao Jiang, Yiyan Liao, Rui Li, Qinrui Li, Sirun Li, Yizhi Li, Yunwen Li, Dehua Ma, Yua...

    2025

  29. [37]

    Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  30. [38]

    Mmlu-pro: A more robust and challenging multi-task language understanding benchmark

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. InThe Thirty-eight Conference on Neural Information Pro...

    2024

  31. [39]

    Pangu- π: Enhancing language model architectures via nonlinearity compensation.arXiv preprint arXiv:2312.17276, 2023

    Yunhe Wang, Hanting Chen, Yehui Tang, Tianyu Guo, Kai Han, Ying Nie, Xutao Wang, Hailin Hu, Zheyuan Bai, Yun Wang, et al. Pangu- π: Enhancing language model architectures via nonlinearity compensation.arXiv preprint arXiv:2312.17276, 2023

    2023 arXiv

  32. [40]

    Cmath: Can your language model pass chinese elementary school math test?, 2023

    Tianwen Wei, Jian Luan, Wei Liu, Shuang Dong, and Bin Wang. Cmath: Can your language model pass chinese elementary school math test?, 2023

    2023

  33. [41]

    Efficient streaming language models with attention sinks.arXiv preprint arXiv:2309.17453, 2023

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks.arXiv preprint arXiv:2309.17453, 2023

    2023 arXiv

  34. [42]

    Clue: A chinese language understanding evaluation benchmark.arXiv preprint arXiv:2004.05986, 2020

    Liang Xu, Hai Hu, Xuanwei Zhang, Lu Li, Chenjie Cao, Yudong Li, Yechen Xu, Kai Sun, Dian Yu, Cong Yu, et al. Clue: A chinese language understanding evaluation benchmark.arXiv preprint arXiv:2004.05986, 2020

    2004 arXiv

  35. [43]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    2025 arXiv

  36. [44]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report.arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  37. [45]

    Attentionpredictor: Temporal pattern matters for efficient llm inference

    Qingyue Yang, Jie Wang, Xing Li, Zhihai Wang, Chen Chen, Lei Chen, Xianzhi Yu, Wulong Liu, Jianye Hao, Mingxuan Yuan, et al. Attentionpredictor: Temporal pattern matters for efficient llm inference. arXiv preprint arXiv:2502.04077, 2025

    2025

  38. [46]

    Entropy law: The story behind data compression and llm performance

    Mingjia Yin, Chuhan Wu, Yufei Wang, Hao Wang, Wei Guo, Yasheng Wang, Yong Liu, Ruiming Tang, Defu Lian, and Enhong Chen. Entropy law: The story behind data compression and llm performance. arXiv preprint arXiv:2407.06645, 2024. 24

    2024 arXiv

  39. [47]

    Pangu ultra: Pushing the limits of dense large language models on ascend npus

    Yichun Yin, Wenyong Huang, Kaikai Song, Yehui Tang, Xue-Fei Wu, Wei Guo, Peng Guo, Yaoyuan Wang, Xiaojun Meng, Yasheng Wang, Dong Li, Can Chen, Dandan Tu, Yin Li, Fisher Yu, Ruiming Tang, Yunhe Wang, Baojun Wang, Bin Wang, Bo Wang, Boxiao Liu, Changzheng Zhang, Duyu Tang, Fei ...

    2025

  40. [48]

    Hellaswag: Can a machine really finish your sentence? InAnnual Meeting of the Association for Computational Linguistics, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? InAnnual Meeting of the Association for Computational Linguistics, 2019

    2019

  41. [49]

    轻解罗裳，独上兰舟

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models.arXiv preprint arXiv:2311.07911, 2023. 25 A Contributions and Acknowledgments Core Contributors Yehui Tang, Xia...

    2023 arXiv

  42. [50]

    Convert the percentage to a decimal: 35% = 0.35

  43. [51]

    Perform the multiplication:82×40 = 3280

  44. [52]

    Substitute the converted percentage and multiplication result back into the expression: 28.97 - 0.35 - 3280 - 58.87

  45. [53]

    如果昨天是明天就好了，那 么今天就是周五了，请问今天周几？

    Perform the subtractions from left to right: - Subtract0.35from28.97: 28.97 - 0.35 = 28.62 - Subtract3280from28.62: 28.62 - 3280 = -3251.38 - Subtract58.87from−3251.38: -3251.38 - 58.87 = -3310.25 Thus, the result of the expression is−3310.25 . 28 Table 11: This is an example ...

  46. [54]

    **条件分析**：题目假设“如果昨天是明天”，即假设中的昨天等于实际中的明天。此时假设中的今天为周五。

  47. [55]

    **建立方程**：假设中的昨天=实际中的明天→h−1 =X+ 1。

  48. [56]

    昨天（假设中的）是明天（实际中 的）

    **求解**：h=X+ 1→X=hΨ。 **验证**：- 若实际今天是周三，实际昨天是周二，实际明天是周四。 - 根据题意，假设 “昨天（假设中的）是明天（实际中 的）”，即假设中的昨天为周四，则假设中的今天为h+ 1 =h，与题目条件一致。 **答案**：今天周三。 29 Table 12: This is an instruction following case. There are two instructions on the question. One is that the answer must contain more than ...

  49. [57]

    左”为尊贵方位（如 “左丞相

    **传统规范** - **左手为尊 **：古代以 “左”为尊贵方位（如 “左丞相 ”高于“右丞相 ”），因此行拱手礼时，左手在外、右手在内，象征对对方的 尊重。 - **手心方向**：双手合抱时，手心应相对或向下（而非向上），以示谦逊和诚意。

  50. [58]

    **性别差异（传统）** - **男性**：通常左手在外，右手在内。 - **女性**：传统上部分场合可能右手在外，左手在内（如某些祭祀礼仪），但这一差异在现代已逐渐淡化。

  51. [59]

    **现代简化** -当代礼仪中，拱手礼更注重形式统一， **普遍采用左手在外 **的规范，性别区分较少。日常使用或非正式场合，也可根据个 人习惯调整，但需保持双手对称、自然。

  52. [60]

    **特殊场合** - **祭祀或正式礼仪**：需严格遵循传统，左手在外，右手在内。 - **国际交往**：可简化为双手合抱、掌心相对，避免过度拘泥细节。 **总结**：现代行拱手礼时，建议以 **左手在外、右手在内 **为标准，既符合传统又便于操作。若遇特殊礼仪场合，可提前了 解具体要求。 31

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.