Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Megrez2 Technical Report

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A 7.5B-parameter Mixture-of-Experts model that activates only 3B parameters per token can match or beat substantially larger language models on understanding, math, and code, thanks to cross-layer expert sharing.

desk verdict A competent technical report with a genuinely trained model, but the key causal claim about cross-layer expert sharing is not experimentally isolated, and the evaluation is too thin to back the 'superior' language. read the letter →

arxiv 2507.17728 v1 pith:MQCGAXLI submitted 2025-07-23 cs.CL

Boxun Li , Yadong Li , Zhiyuan Li , Congyi Liu , Weilin Liu , Guowei Niu , Zheyue Tan , Haiyang Xu
show 7 more authors
Zhuyu Yao Tao Yuan Dong Zhou Yueqing Zhuang Bo Zhao Guohao Dai Yu Wang
This is my paper · ORCID
classification cs.CL
keywords Mixture-of-Expertscross-layerexpertsharingpre-gatedroutinglightweightlanguagemodelon-devicedeploymentparameterefficiencyreinforcementlearningwithverifiablerewardsMegrez2-Preview
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Megrez2 is an architecture for large language models that aims to resolve the trade-off among speed, accuracy, and cost for on-device deployment. Its central move is cross-layer expert sharing: the same pool of mixture-of-experts modules is reused across every three consecutive transformer layers, cutting the stored parameter count by roughly a factor of three while keeping 3B parameters active per token. The paper reports that Megrez2-Preview, the first model built on this architecture and trained on five trillion tokens, matches or beats substantially larger models on Chinese and English understanding, instruction following, mathematical reasoning, and code generation. If correct, the result shows that a small stored footprint need not cost accuracy, which would make high-capacity language models practical on memory-constrained devices.

What carries the argument

The load-bearing object is the cross-layer expert-sharing group: Megrez2 partitions the transformer into groups of $n=3$ consecutive layers, and all layers in a group draw from one shared pool of 64 experts with their own gating networks, so the total parameter count falls by roughly $n$ while activated parameters per token stay constant. Pre-gated routing moves the gating computation to the preceding layer, so selected experts can be loaded into memory before they are used, and within a group an already-loaded expert can be kept in cache across layers. The dense-layer-first layout with a hidden width of 10,944 and an expert width of 1,408 sets the capacity the shared pools are asked to preserve.

What would settle it

Train a MoE with each layer owning its own expert pool (group size 1) on the same 5-trillion-token corpus with the same SFT and RLVR post-training, holding activated parameters equal; if that model matches or beats Megrez2-Preview, cross-layer sharing is not the source of the reported performance.

Watch

Extended reading notes

Core claim

The paper's claim is that a mixture-of-experts language model can shed most of its stored parameters without shrinking its per-token compute by having successive layers reuse the same expert modules. Megrez2-Preview activates only 3B parameters through top-6 routing among 64 shared experts per group of layers, yet stores just 7.5B parameters total; against Qwen2.5-7B, Qwen3-8B, Gemma-3-4B, Phi-4-mini, and GPT-4o-mini it posts higher or tied scores on C-EVAL, MMLU-Pro, IFEval, MATH-500, GSM8K, HumanEval, and MBPP. The paper attributes this balance to the combination of cross-layer expert sharing, pre-gated routing that loads experts one layer ahead, a dense-layer-first 31-layer layout, and a three-stage training pipeline topped by reinforcement learning with verifiable rewards.

Load-bearing premise

The assumption that one shared expert pool serving three consecutive layers keeps most of the model's capacity is never tested against a non-sharing MoE trained on the same data, so the benchmark wins could stem from the 5-trillion-token corpus and post-training recipe rather than from the sharing mechanism.

Editorial extensions

If this is right

  • Device deployment becomes feasible at 7.5B stored parameters with 3B activated, a footprint that fits in a few gigabytes even before quantization.
  • Memory savings scale with group size: choosing larger $n$ cuts stored parameters further, at the cost of asking the same experts to serve more layers.
  • Pre-gated routing lets expert loading overlap with computation, which should translate to lower latency on bandwidth-limited hardware.
  • Competitive scores on math and code suggest the architecture retains the reasoning capacity that larger dense models pay for with far more parameters.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the sharing mechanism is the true driver, the same principle could be applied to other reusable modules, such as attention projections, to compress models further; this is an editorial extension the report does not explore.
  • The reported results could instead reflect the 5-trillion-token corpus, the turn-level loss, or the RLVR recipe; an ablation isolating group size would settle which ingredient carries the gain.
  • A natural testable extension is to vary the group size $n$ from 1 to 6 at fixed data and compute: a performance cliff at larger $n$ would map the capacity-memory trade-off curve, which the report leaves implicit.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents Megrez2, a Mixture-of-Experts (MoE) language model architecture whose main novelty is cross-layer expert sharing: every group of n consecutive transformer layers reuses a single pool of M experts while each layer keeps its own gating network and projection weights, and a pre-gated routing mechanism lets the router of layer i-1 select experts for layer i. The authors instantiate this architecture as Megrez2-Preview, a model with 3B activated and 7.5B stored parameters, pretrained on 5 trillion tokens and then refined with supervised fine-tuning and reinforcement learning with verifiable rewards. Evaluation in Section 5 compares Megrez2-Preview against several dense and MoE baselines on C-EVAL, MMLU-Pro, IFEval, MATH-500, GSM8K, HumanEval, and MBPP. The paper claims that cross-layer expert sharing reduces stored parameters while maintaining capacity, and that pre-gated routing enables memory-efficient expert loading and faster inference, making the model suitable for device-native deployment.

Significance. If the central empirical claim held, a 3B-activated, 7.5B-stored MoE that is competitive with 7B-8B dense models while reusing experts across layers would be a practically valuable result for on-device deployment. The architecture description in Eqs. (1)-(3) is simple, internally consistent, and clearly stated, and the evaluation uses external benchmarks with no evidence that hyperparameters were fitted to those benchmarks. However, the evidence as presented does not establish the unique contribution of cross-layer expert sharing: every comparison is against models with different architectures, training data, and post-training procedures, and the deployment-efficiency claims (latency, memory, throughput) are never measured. The core idea is plausible, but the report currently reads as a model card with architectural ambition rather than a validated architecture study.

major comments (4)
  1. [Sections 3.2 and 3.4, Eq. (2)] The central claim that cross-layer expert sharing 'maintains most of the model's capacity' is not tested in isolation. All Table 1 baselines differ in architecture, pretraining corpus, and post-training recipe; none is a non-sharing MoE trained on the same data with the same activated-parameter budget. The reported results are therefore equally consistent with the hypothesis that the 5-trillion-token corpus, the three-stage pretraining, the SFT/RLVR recipe, or the dense-layer-first design drives performance, and that sharing itself contributes little. An ablation with group size n=1 (per-layer experts) or a same-data comparison against a standard MoE without parameter reuse is necessary to support the architectural attribution currently made throughout the abstract and conclusion.
  2. [Section 5 and Table 1] The evaluation protocol is not reported. The paper does not state the sampling temperature, number of runs, prompt templates, or the exact computation of Pass@1 for HumanEval and MBPP; no error bars or variance estimates are given; and many baseline entries are missing (e.g., Qwen3-4B on MMLU-Pro, GSM8K, and HumanEval; Gemma-3-4B on C-EVAL and MMLU-Pro; GPT-4o-mini on MMLU-Pro and MBPP). This makes the 'competitive or superior' claim difficult to verify, and the presence of missing entries allows selective comparison across benchmarks. The authors should either complete the table or clearly state which baseline numbers were unavailable and why.
  3. [Section 5, prose around Table 1] The summary statements overreach the data. Megrez2-Preview trails Qwen2.5-7B on GSM8K (83.6 vs 91.6) and HumanEval (74.4 vs 84.8), and trails Qwen3-8B on IFEval (80.2 vs 83.0), GSM8K (83.6 vs 93.2), and HumanEval (74.4 vs 85.9). The claim that the model 'matches or even exceeds the performance of substantially larger models' is therefore not supported as a global statement; it should be restricted to the specific benchmarks and baselines where it actually holds, with the trade-offs stated plainly.
  4. [Sections 3.3 and 5] The second headline advantage, pre-gated routing for 'memory-efficient expert loading and faster inference', is never evaluated. There are no latency, peak-memory, or throughput measurements, no device-deployment results, and no comparison against a non-pre-gated variant of the same model. Since deployment efficiency is a core motivation of the paper (the 'Impossible Triangle' in Section 1), the absence of any system-level evaluation is a substantive gap that should be filled with at least basic inference-cost measurements.
minor comments (5)
  1. [Table 1] The table header uses 'Megrez2' rather than 'Megrez2-Preview', which is the name used elsewhere in the paper; please make the naming consistent.
  2. [Section 3.3, Eq. (3)] Equation (3) evaluates the previous layer's router G_{i-1} at h_i, which is the input to layer i (i.e., the output of layer i-1 after gating). If pre-gating means the router of layer i-1 decides for layer i, the argument to G_{i-1} should be specified precisely (e.g., the hidden state before layer i-1's gating, or h_{i-1}), because as written the notation is ambiguous and could be read as a chicken-and-egg dependency.
  3. [Section 3.4] The role of the 4 shared experts is ambiguous: the paper does not state whether the same 4 shared experts are reused across all layers in a group or are per-layer, and Eq. (2) omits them entirely. Please clarify how the shared experts are counted in the 7.5B stored parameter total.
  4. [Section 4.2.2] The RL description would benefit from more detail: the paper says it uses a 'modified version of GRPO' with a PPO loss and GAE, but does not give the PPO clip range, advantage normalization, rollout count, or how the GRPO sampling framework and the PPO loss interact. These details are needed for reproducibility.
  5. [General] There are several minor text and formatting issues: 'C-EVAL' is typeset inconsistently ('C-EVAL', 'C-EV AL', 'C-E VAL'), the phrase 'device-model trilemma' appears mid-sentence without definition, and 'Pass@1' is used without a definition or citation of the standard evaluation methodology.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central performance claims rest on external benchmarks, and no fitted parameter or load-bearing self-citation is disguised as a prediction.

full rationale

Megrez2's central claims are empirical rather than derivational: the stored/activated parameter counts are definitional properties of the architecture, and the quality of the model is assessed with standard external benchmarks (C-EVAL, MMLU-Pro, IFEval, MATH-500, GSM8K, HumanEval, MBPP). No evaluation score is used to fit any model constant, and no equation in the report defines the architecture's capacity in terms of those scores. The factor-n reduction in stored parameters from cross-layer expert sharing (Eq. 2) is an algebraic consequence of reusing one expert pool across n layers, not a fitted result; the separate claim that capacity is 'maintained' is an empirical assertion tested on external baselines. The self-citations to the Megrez-3B series for the tokenizer, RoPE extension, and dataset-curation approach are inherited implementation details documented in the report, not citations used as proof that expert sharing preserves quality. The absence of an ablation training a non-sharing MoE on the same data is a limitation in experimental isolation, but it does not make the reported benchmark results equivalent to their inputs. Under the stated rules, non-load-bearing self-citation is not circularity. Therefore no significant circularity is found.

Assumptions & free parameters 7 free parameters · 4 assumptions · 0 invented entities

The central claim rests on a large set of manually chosen architecture hyperparameters, a 5T-token training corpus whose composition is not disclosed, and standard MoE assumptions adopted from prior work. No parameters are fitted to the benchmark numbers, so the ledger is heavy but not circular.

free parameters (7)
  • expert group size n = 3
    Layers are partitioned into contiguous groups of 3, reducing stored parameters by roughly 3x; the choice is not derived or ablated.
  • number of experts per group M = 64
    Architecture choice in Section 3.4; not derived or ablated.
  • number of shared experts = 4
    Every layer includes 4 always-active shared experts per Section 3.4.
  • top-k routing count = 6
    Top-6 of 64 experts selected per token per Section 3.4.
  • dense layer hidden size = 10944
    Set in Section 3.4.
  • expert hidden dimension = 1408
    Set in Section 3.4.
  • number of transformer layers = 31
    Set in Section 3.4.
assumptions (4)
  • domain assumption Top-k MoE with fine-grained experts is an effective base architecture for language modeling.
    The paper builds on DeepSeekMoE and Qwen MoE designs without re-deriving their effectiveness, Section 3.1.
  • domain assumption Pre-gated routing as introduced in [13] is a valid mechanism for reducing expert-loading memory.
    Adopted from a prior ISCA paper, Section 3.3.
  • domain assumption A 5-trillion-token curated corpus with the Megrez-3B tokenizer is sufficient to train a strong 3B-activated model.
    No analysis links corpus composition to benchmark performance, Section 4.1.
  • domain assumption The selected benchmarks are reliable proxies for general language, instruction-following, math, and coding capability.
    Used as the sole evidence for the central claim, Section 5.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Megrez2 Technical Report." pith.science (2026). https://pith.science/paper/MQCGAXLI

@misc{pith2026250717728,
  author       = {Pith},
  title        = {Pith review of: Megrez2 Technical Report},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MQCGAXLI}},
  note         = {Machine review of arXiv:2507.17728}
}
read the original abstract

We present Megrez2, a novel lightweight and high-performance language model architecture optimized for device native deployment. Megrez2 introduces a novel cross-layer expert sharing mechanism, which significantly reduces total parameter count by reusing expert modules across adjacent transformer layers while maintaining most of the model's capacity. It also incorporates pre-gated routing, enabling memory-efficient expert loading and faster inference. As the first instantiation of the Megrez2 architecture, we introduce the Megrez2-Preview model, which is pre-trained on a 5-trillion-token corpus and further enhanced through supervised fine-tuning and reinforcement learning with verifiable rewards. With only 3B activated and 7.5B stored parameters, Megrez2-Preview demonstrates competitive or superior performance compared to larger models on a wide range of tasks, including language understanding, instruction following, mathematical reasoning, and code generation. These results highlight the effectiveness of the Megrez2 architecture to achieve a balance between accuracy, efficiency, and deployability, making it a strong candidate for real-world, resource-constrained applications.

Figures

Figures reproduced from arXiv: 2507.17728 by the authors.

Figure 1
Figure 1. The Impossible Triangle of Device AI 1 Introduction Large Language Models (LLMs) have progressed at an unprecedented pace, scaling to hundreds of billions of parameters and demonstrating remarkable advances toward Artificial General Intelligence (AGI). Recent foundation models such as GPT-4 [2], GPT-o3 [19], Gemini [23], Llama 4 [17], DeepSeekR1 [10], Kimi-K2 [3], and the Qwen3 series [27] have exhibited strong perf… view at source ↗
Figure 2
Figure 2. Expert Sharing and Pre-gated Routing in Megrez2 Model [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Scaling Latent Reasoning via Looped Language Models

    cs.CL 2025-10 unverdicted novelty 7.0 of 10

    Looped language models with latent iterative computation and entropy-regularized depth allocation achieve performance matching up to 12B standard LLMs through superior knowledge manipulation.

Reference graph

Works this paper leans on

30 extracted references · 9 canonical work pages · cited by 1 Pith paper

  1. [1]

    Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-loras

    Abdelrahman Abouelenin, Atabak Ashfaq, Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao, Alon Benhaim, Martin Cai, Vishrav Chaudhary, Congcong Chen, et al. “Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-loras”. In:arXiv preprint arXiv:2503.01743(2025)

    arXiv 2025

  2. [2]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. “Gpt-4 technical report”. In:arXiv preprint arXiv:2303.08774(2023)

    arXiv 2023

  3. [3]

    https://github.com/MoonshotAI/Kimi-K2

    Moonshot AI.Kimi-K2: Open-Source Models by Moonshot AI. https://github.com/MoonshotAI/Kimi-K2. 2025

    work page 2025

  4. [4]

    Program synthesis with large language models

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. “Program synthesis with large language models”. In:arXiv preprint arXiv:2108.07732(2021)

    arXiv 2021

  5. [5]

    Read-ME: Refactorizing LLMs as Router-Decoupled Mixture of Experts with System Co-Design

    Ruisi Cai, Yeonju Ro, Geon-Woo Kim, Peihao Wang, Babak Ehteshami Bejnordi, Aditya Akella, Zhangyang Wang, et al. “Read-ME: Refactorizing LLMs as Router-Decoupled Mixture of Experts with System Co-Design”. In:Advances in Neural Information Processing Systems37 (2024), pp. 116126–116148

    work page 2024

  6. [6]

    Evaluating Large Language Models Trained on Code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. “Evaluating Large Language Models Trained on Code”. In:arXiv preprint arXiv:2107.03374(2021)

    arXiv 2021

  7. [7]

    Training Verifiers to Solve Math Word Problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. “Training Verifiers to Solve Math Word Problems”. In:arXiv preprint arXiv:2110.14168(2021)

    arXiv 2021

  8. [8]

    Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models

    Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Yu Wu, et al. “Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models”. In:arXiv preprint arXiv:2401.06066(2024)

    arXiv 2024

Show all 30 references
  1. [9]

    Fast inference of mixture-of-experts language models with offloading

    Artyom Eliseev and Denis Mazur. “Fast inference of mixture-of-experts language models with offloading”. In: arXiv preprint arXiv:2312.17238(2023)

    2023 arXiv

  2. [10]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. “Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning”. In:arXiv preprint arXiv:2501.12948(2025)

    2025 arXiv

  3. [11]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. “Measuring mathematical problem solving with the math dataset”. In:arXiv preprint arXiv:2103.03874(2021)

    2021 arXiv

  4. [12]

    C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models

    Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Yao Fu, et al. “C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models”. In:Advances in Neural Information Processing Systems36 (2024)

    2024

  5. [13]

    Pre-gated moe: An algorithm-system co-design for fast and scalable mixture-of-expert inference

    Ranggi Hwang, Jianyu Wei, Shijie Cao, Changho Hwang, Xiaohu Tang, Ting Cao, and Mao Yang. “Pre-gated moe: An algorithm-system co-design for fast and scalable mixture-of-expert inference”. In:2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA). IEE...

    2024

  6. [14]

    Gemma 3 Technical Report

    Gemma Team Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ram’e, Morgane Rivière, Louis Rouillard, et al. “Gemma 3 Technical Report”. In:ArXivabs/2503.19786 (2025)

    2025 arXiv

  7. [15]

    Megrez-omni technical report

    Boxun Li, Yadong Li, Zhiyuan Li, Congyi Liu, Weilin Liu, Guowei Niu, Zheyue Tan, Haiyang Xu, Zhuyu Yao, Tao Yuan, et al. “Megrez-omni technical report”. In:arXiv preprint arXiv:2502.15803(2025)

    2025 arXiv

  8. [16]

    Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model

    Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. “Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model”. In:arXiv preprint arXiv:2405.04434(2024). 7 Megrez2 Technical Report

    2024 arXiv

  9. [17]

    https://ai

    Meta AI.The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation. https://ai. meta.com/blog/llama-4-multimodal-intelligence/. Apr. 2025

    2025

  10. [18]

    OpenAI blog post

    OpenAI.GPT-4o mini: advancing cost-efficient intelligence. OpenAI blog post. Released July 18, 2024. July 2024. URL:https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/

    2024

  11. [19]

    2025.URL: https://openai.com/index/introducing-o3- and-o4-mini/

    OpenAI.Introducing OpenAI o3 and o4-mini. 2025.URL: https://openai.com/index/introducing-o3- and-o4-mini/

    2025

  12. [20]

    High-dimensional continu- ous control using generalized advantage estimation

    John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. “High-dimensional continu- ous control using generalized advantage estimation”. In:arXiv preprint arXiv:1506.02438(2015)

    2015 arXiv

  13. [21]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. “Deepseekmath: Pushing the limits of mathematical reasoning in open language models”. In:arXiv preprint arXiv:2402.03300(2024)

    2024 arXiv

  14. [22]

    Roformer: Enhanced trans- former with rotary position embedding

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. “Roformer: Enhanced trans- former with rotary position embedding”. In:Neurocomputing568 (2024), p. 127063

    2024

  15. [23]

    Dai, Anja Hauth, Katie Millican, et al.Gemini: A Family of Highly Capable Multimodal Models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalk- wyk, Andrew M. Dai, Anja Hauth, Katie Millican, et al.Gemini: A Family of Highly Capable Multimodal Models

  16. [24]

    Auxiliary-loss-free load balancing strategy for mixture-of-experts

    Lean Wang, Huazuo Gao, Chenggang Zhao, Xu Sun, and Damai Dai. “Auxiliary-loss-free load balancing strategy for mixture-of-experts”. In:arXiv preprint arXiv:2408.15664(2024)

    2024 arXiv

  17. [25]

    Mmlu-pro: A more robust and challenging multi-task language understand- ing benchmark, 2024

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. “Mmlu-pro: A more robust and challenging multi-task language understand- ing benchmark, 2024”. In:URL https://arxiv. org/abs/2406.01574(2024), p. 21

    2024 arXiv

  18. [26]

    Skywork-moe: A deep dive into training techniques for mixture-of-experts language models

    Tianwen Wei, Bo Zhu, Liang Zhao, Cheng Cheng, Biye Li, Weiwei Lü, Peng Cheng, Jianhao Zhang, Xiaoyu Zhang, Liang Zeng, et al. “Skywork-moe: A deep dive into training techniques for mixture-of-experts language models”. In:arXiv preprint arXiv:2406.06563(2024)

    2024 arXiv

  19. [27]

    Qwen3 Technical Report

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. “Qwen3 Technical Report”. In:ArXivabs/2505.09388 (2025)

    2025 arXiv

  20. [28]

    Qwen2.5 Technical Report

    Qwen An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxin Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, ...

    2024 arXiv

  21. [29]

    Instruction-following evaluation for large language models

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. “Instruction-following evaluation for large language models”. In:arXiv preprint arXiv:2311.07911(2023). 8

    2023 arXiv

  22. [2025]

    arXiv:2312.11805 [cs.CL].URL:https://arxiv.org/abs/2312.11805

    arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.