Pith. sign in

REVIEW 4 major objections 4 minor 51 references

Grove MoE: Towards Efficient and Superior MoE LLMs with Adjugate Experts

T0 review · 4 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Grove MoE's variable-size experts, activated by token complexity, let a 33B-parameter model run with 3.14–3.28B active parameters while matching larger open models.

desk verdict Heterogeneous-size experts with complexity-based activation is a plausible efficiency idea, but the paper as it stands can't be verified — the full text is corrupted and the routing mechanism is undisclosed. read the letter →

arxiv 2508.07785 v1 pith:QK5DIFKI submitted 2025-08-11 cs.CL

Haoyuan Wu , Haoxing Chen , Xiaodong Chen , Zhanchao Zhou , Tieyuan Chen , Yihong Zhuang , Guoshan Lu , Zenan Huang
show 5 more authors
Junbo Zhao Lin Liu Zhenzhong Lan Bei Yu Jianguo Li
This is my paper · ORCID
classification cs.CL
keywords mixtureofexpertsheterogeneousdynamicactivationtokencomplexityadjugateefficientLLMinferenceupcyclingsparse
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the standard Mixture-of-Experts recipe, in which every expert is the same size and every token activates a fixed number of experts, wastes computation on easy tokens. Grove MoE replaces uniform experts with adjugate experts of varying sizes, and adds a dynamic activation mechanism that chooses how much capacity a token receives based on its complexity. The authors build two 33B-parameter models by upcycling the Qwen3-30B-A3B-Base checkpoint, and report that only 3.14–3.28B parameters are active per token while performance stays comparable to open-source models of similar or larger size. If this holds, MoE inference can buy large-model quality with a much smaller compute budget.

What carries the argument

The load-bearing mechanism is the adjugate expert: a variable-size expert unit used inside the MoE layers, paired with a dynamic activation policy that reads per-token complexity and decides which expert size to invoke. This turns sparse activation from a fixed-count selection into a continuous capacity budget—the source of the 3.14–3.28B activated-parameter range. The big.LITTLE-inspired design is what allows the model to carry 33B parameters while spending only a tenth of them on an average token.

What would settle it

Take the trained GroveMoE-Base and run it with dynamic activation disabled, first forcing all tokens to the smallest experts and then forcing all to the largest, at matched total FLOPs. If either fixed policy achieves the same benchmark scores as the dynamic policy, then token-complexity routing is not the source of the reported efficiency; the claim collapses.

Watch

Extended reading notes

Core claim

Grove MoE's central proposal is to make sparsity complexity-adaptive rather than count-adaptive. In a conventional MoE, activation is sparse but homogeneous: each token uses the same number of same-sized experts, so a trivial token pays as much as a difficult one. Grove MoE instead maintains experts in different sizes and lets a router allocate more parameters to tokens that need them. The resulting models, GroveMoE-Base and GroveMoE-Inst, have 33B total parameters yet activate only 3.14–3.28B per token, and the paper reports that they match state-of-the-art open-source models of comparable or larger size. The design is presented as an upcycling of an existing strong MoE checkpoint, so the a

Load-bearing premise

The whole efficiency claim rests on the router being able to tell hard tokens from easy ones and pick the right expert size reliably, while the routing and load-balancing overhead stays small enough to be worth it.

Editorial extensions

If this is right

  • MoE quality no longer needs to be tied to a fixed activated-parameter count; capacity can be spread across expert sizes and spent selectively.
  • Inference costs become input-dependent, so serving systems can schedule harder tokens on larger experts and easy tokens on small ones, lowering average FLOPs per request.
  • Existing strong MoE checkpoints can be upcycled into heterogeneous-expert models, preserving quality while adding dormant capacity.
  • The reported 3.14–3.28B range implies fine-grained control over the compute-quality frontier, making per-token compute budgeting a usable design axis.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • I would expect the dynamic activation policy to create a measurable quality-per-FLOP frontier: if a fixed-size expert matches the dynamic model at the same average activated parameters, then the complexity signal is not buying anything; if it does not, complexity-routing is doing real work.
  • The 3.14–3.28B spread is narrow enough that a distribution analysis over tokens would clarify which tokens actually get the large experts; a concentrated distribution would mean most of the benefit comes from a single size, not from complexity adaptation.
  • The same adjugate idea could be applied to the number of layers or attention heads a token passes through, not just expert width, since the big.LITTLE principle is about allocating capacity where it matters.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. This paper proposes Grove MoE, an MoE architecture with heterogeneous ``adjugate'' experts of varying sizes, inspired by the big.LITTLE CPU design, and a dynamic activation mechanism that selects experts according to per-token complexity. The authors report GroveMoE-Base and GroveMoE-Inst, 33B-parameter models upcycled from Qwen3-30B-A3B-Base, claiming 3.14-3.28B dynamically activated parameters and performance comparable to SOTA open-source models of similar or larger size. The supplied full text is a mojibake rendering and no equations, tables, ablations, training details, or experimental protocols can be audited; the abstract is the only legible substantive component.

Significance. If the central efficiency claim were fully supported, Grove MoE would be a valuable step toward input-dependent MoE computation, and the big.LITTLE analogy is well motivated. The active-parameter range (3.14-3.28B) is specific and falsifiable in principle. However, because the full text cannot be read, none of the architectural or empirical support can be verified, and no code, proofs, or reproducibility artifacts are visible. The current manuscript therefore cannot yet contribute beyond its abstract; a complete, correctly submitted manuscript is required before its significance can be judged.

major comments (4)
  1. [Full text (passim)] The supplied full text is largely unreadable mojibake; no equation, table, or experimental description can be inspected. The abstract's claims about the architecture, dynamic activation, upcycling, and benchmark performance cannot be checked. This is the central load-bearing issue for the paper. Please resubmit a clean, correctly rendered PDF and source files.
  2. [Abstract and full text] The dynamic activation mechanism is not specified. The abstract states that 3.14-3.28B parameters are activated "based on token complexity" but does not define the complexity signal, the routing criterion, or the threshold policy. Without this, the reported active-parameter range is uninterpretable and the efficiency claim is not reproducible. A revision must state the routing rule and provide evidence that the router remains stable under load-balancing and entropy-regularization losses.
  3. [Abstract (upcycling comparison)] The comparison to Qwen3-30B-A3B-Base is a potential confound. GroveMoE has 33B total parameters versus the approximately 30B of the source model, so any performance improvement over the backbone could be due to the extra parameters rather than to the adjugate-expert/dynamic-activation design. Please provide controlled ablations with equal total parameter counts, equal active parameter counts, and a homogeneous-expert MoE baseline of the same scale.
  4. [Full-text header] The supplied full text contains the line "arXiv:2508.07795v3 [cs.CV] 14 Mar 2026", which does not match the claimed arXiv identifier (2508.07785) or field (cs.CL). This mismatch makes it impossible to confirm that the reviewed full text is the paper described in the abstract. Please submit the correct manuscript and verify the arXiv metadata.
minor comments (4)
  1. [Abstract] Define "adjugate experts" at first use; the term is not standard and the abstract does not explain what makes an expert "adjugate" beyond having a different size.
  2. [Abstract] Clarify whether the 3.14-3.28B active-parameter range is an empirical measurement across a benchmark or an analytic bound, and report the distribution of token routing across expert sizes.
  3. [Abstract] The phrase "similar or even larger size" is vague. Name the concrete SOTA baselines and report the win/loss/tie summary on each benchmark.
  4. [Full text (tables)] The readable fragments of the tables appear to contain blank cells and unexplained column headers. If these are model outputs, please report the full numerical values, standard errors, and the precise metric names.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity detected: benchmarks are external and the active-parameter claim is empirical, not definitional; the garbled full text prevents identifying any reduction.

full rationale

The central claims in the abstract are comparisons to external open-source models ("performance comparable to SOTA open-source models of similar or even larger size") and a reported active-parameter range ("dynamically activate 3.14-3.28B parameters"). These are empirical measurements against public benchmarks and fixed model configurations, not quantities defined in terms of one another. Nothing in the readable portion of the manuscript defines the reported active-parameter count as the output of the same fitted quantity later called a prediction, and no fitted parameter is renamed as a prediction. The upcycling from Qwen3-30B-A3B-Base introduces a potential confound: some of the gains could come from the added total parameters rather than the Grove architecture. That is a validity concern, not circularity. The full text is largely mojibake, so equations and ablations cannot be inspected; there is also an inserted arXiv identifier (2508.07795v3 [cs.CV]) inconsistent with the paper ID, which makes verification harder. However, under the required evidentiary standard, no specific equation or construction shows that the derivation reduces to its own inputs. Therefore the appropriate honest finding is no significant circularity.

Assumptions & free parameters 3 free parameters · 3 assumptions · 1 invented entities

Everything above is inferred from the abstract because the supplied full text is encoding-corrupted and unreadable. The ledger flags the routing policy, expert-size schedule, and data recipe as likely design decisions that are invisible here, plus the upcycling and complexity-signal assumptions that carry the architecture.

free parameters (3)
  • Token-complexity routing criteria and activation thresholds = undisclosed in abstract
    The 3.14-3.28B active-parameter range must come from some routing or threshold policy; if tuned on the evaluation benchmarks, the reported efficiency would be partly fitted.
  • Expert size configuration (count and width of small versus large experts) = undisclosed in abstract
    The heterogeneous expert schedule is a design choice typically set by experiment; not visible at abstract level.
  • Mid-training and post-training data mixing ratios = undisclosed in abstract
    Upcycled model quality depends on data recipe choices; absent from the abstract.
assumptions (3)
  • domain assumption Weights upcycled from Qwen3-30B-A3B-Base preserve and extend capability during mid-training and post-training
    The abstract stakes the models entirely on upcycling; if the backbone does not transfer, the central claim fails.
  • ad hoc to paper Per-token complexity is a learnable signal that predicts how much expert capacity a token needs
    Core premise of the dynamic activation mechanism; the abstract supplies no evidence that this signal is reliable at scale.
  • standard math Standard MoE routing and training math from prior literature holds for heterogeneous expert sizes
    The paper extends existing MoE machinery; correctness of that underlying machinery is assumed.
invented entities (1)
  • Adjugate experts (heterogeneous-size experts with dynamic activation) independent evidence
    purpose: Allow per-token variable compute by mixing small and large experts inside one MoE layer
    The design is falsifiable through the reported benchmarks and the 3.14-3.28B active-parameter claim, but that evidence is not shown in the abstract.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Grove MoE: Towards Efficient and Superior MoE LLMs with Adjugate Experts." pith.science (2026). https://pith.science/paper/QK5DIFKI

@misc{pith2026250807785,
  author       = {Pith},
  title        = {Pith review of: Grove MoE: Towards Efficient and Superior MoE LLMs with Adjugate Experts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QK5DIFKI}},
  note         = {Machine review of arXiv:2508.07785}
}
read the original abstract

The Mixture of Experts (MoE) architecture is a cornerstone of modern state-of-the-art (SOTA) large language models (LLMs). MoE models facilitate scalability by enabling sparse parameter activation. However, traditional MoE architecture uses homogeneous experts of a uniform size, activating a fixed number of parameters irrespective of input complexity and thus limiting computational efficiency. To overcome this limitation, we introduce Grove MoE, a novel architecture incorporating experts of varying sizes, inspired by the heterogeneous big.LITTLE CPU architecture. This architecture features novel adjugate experts with a dynamic activation mechanism, enabling model capacity expansion while maintaining manageable computational overhead. Building on this architecture, we present GroveMoE-Base and GroveMoE-Inst, 33B-parameter LLMs developed by applying an upcycling strategy to the Qwen3-30B-A3B-Base model during mid-training and post-training. GroveMoE models dynamically activate 3.14-3.28B parameters based on token complexity and achieve performance comparable to SOTA open-source models of similar or even larger size.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

51 extracted references · 18 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    AIME Problems and Solutions , 2025

    AIME . AIME Problems and Solutions , 2025. URL https://artofproblemsolving.com/wiki/index.php/AIME_Problems_and_Solutions

    work page 2025

  3. [3]

    Alternating Updates for Efficient Transformers

    Cenk Baykal, Dylan Cutler, Nishanth Dikkala, Nikhil Ghosh, Rina Panigrahy, and Xin Wang. Alternating Updates for Efficient Transformers . In Annual Conference on Neural Information Processing Systems (NIPS), 2023

    work page 2023

  4. [4]

    MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation

    Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, et al. MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation . IEEE Transactions on Software Engineering, 49 0 (7): 0 3675--3691, 2023

    work page 2023

  5. [5]

    Parallel Scaling Law for Language Models

    Mouxiang Chen, Binyuan Hui, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Jianling Sun, Junyang Lin, and Zhongxin Liu. Parallel Scaling Law for Language Models . arXiv preprint arXiv:2505.10475, 2025

    arXiv 2025

  6. [6]

    Training Verifiers to Solve Math Word Problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training Verifiers to Solve Math Word Problems . arXiv preprint arXiv:2110.14168, 2021

    arXiv 2021

  7. [7]

    SuperGPQA: Scaling llm evaluation across 285 graduate disciplines

    Xinrun Du, Yifan Yao, Kaijing Ma, Bingli Wang, Tianyu Zheng, King Zhu, Minghao Liu, Yiming Liang, Xiaolong Jin, Zhenlin Wei, et al. SuperGPQA: Scaling llm evaluation across 285 graduate disciplines . arXiv preprint arXiv:2502.14739, 2025

    arXiv 2025

  8. [8]

    DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning over Paragraphs

    Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning over Paragraphs . arXiv preprint arXiv:1903.00161, 2019

    arXiv 1903

Show all 51 references
  1. [9]

    Omni-MATH: A Universal Olympiad Level Mathematic Benchmark for Large Language Models

    Bofei Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, et al. Omni-MATH: A Universal Olympiad Level Mathematic Benchmark for Large Language Models . arXiv preprint arXiv:2410.07985, 2024

    2024 arXiv

  2. [10]

    Gemma 3 Technical Teport

    Gemma, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ram \'e , Morgane Rivi \`e re, et al. Gemma 3 Technical Teport . arXiv preprint arXiv:2503.19786, 2025

    2025 arXiv

  3. [11]

    Gemini2.5 Pro

    Google DeepMind . Gemini2.5 Pro . https://deepmind.google/technologies/gemini/pro/, 2025

    2025

  4. [12]

    The Llama 3 Herd of Models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Llama 3 Herd of Models . arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  5. [13]

    big.LITTLE Processing with ARL Cortex-A15 & Cortex-A7

    Peter Greenhalgh. big.LITTLE Processing with ARL Cortex-A15 & Cortex-A7 . ARM White paper, 17, 2011

    2011

  6. [14]

    CruxEval: A Benchmark for Code Reasoning, Understanding and Execution

    Alex Gu, Baptiste Rozi \`e re, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida I Wang. CruxEval: A Benchmark for Code Reasoning, Understanding and Execution . arXiv preprint arXiv:2401.03065, 2024

    2024 arXiv

  7. [15]

    Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient Transformer Models

    Yongxin Guo, Zhenglin Cheng, Xiaoying Tang, Zhaopeng Tu, and Tao Lin. Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient Transformer Models . arXiv preprint arXiv:2405.14297, 2024

    2024 arXiv

  8. [16]

    OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems

    Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems . arXiv preprint arXiv:2402...

    2024 arXiv

  9. [17]

    Measuring Massive Multitask Language Understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring Massive Multitask Language Understanding . arXiv preprint arXiv:2009.03300, 2020

    2009 arXiv

  10. [18]

    Measuring Mathematical Problem Solving with the Math Dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring Mathematical Problem Solving with the Math Dataset . arXiv preprint arXiv:2103.03874, 2021

    2021 arXiv

  11. [19]

    Harder Tasks Need More Experts: Dynamic Routing in MoE Models

    Quzhe Huang, Zhenwei An, Nan Zhuang, Mingxu Tao, Chen Zhang, Yang Jin, Kun Xu, Liwei Chen, Songfang Huang, and Yansong Feng. Harder Tasks Need More Experts: Dynamic Routing in MoE Models . arXiv preprint arXiv:2403.07652, 2024

    2024

  12. [20]

    C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models

    Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Yao Fu, et al. C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models . Advances in Neural Information Processing Systems, 36:...

    2023

  13. [21]

    dots.llm1 Technical Report

    Bi Huo, Bin Tu, Cheng Qin, Da Zheng, Debing Zhang, Dongjie Zhang, En Li, Fu Guo, Jian Yao, Jie Lou, et al. dots.llm1 Technical Report . arXiv preprint arXiv:2506.05767, 2025

    2025 arXiv

  14. [22]

    GPT-4o System Card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. GPT-4o System Card . arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  15. [23]

    LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code . arXiv preprint arXiv:2403.07974, 2024

    2024 arXiv

  16. [24]

    Mixtral of Experts

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of Experts . arXiv preprint arXiv:2401.04088, 2024

    2024 arXiv

  17. [25]

    MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Experts

    Peng Jin, Bo Zhu, Li Yuan, and Shuicheng Yan. MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Experts . arXiv preprint arXiv:2410.07348, 2024

    2024 arXiv

  18. [26]

    Sparse Upcycling: Training mixture-of-experts from dense checkpoints

    Aran Komatsuzaki, Joan Puigcerver, James Lee-Thorp, Carlos Riquelme Ruiz, Basil Mustafa, Joshua Ainslie, Yi Tay, Mostafa Dehghani, and Neil Houlsby. Sparse Upcycling: Training mixture-of-experts from dense checkpoints . In International Conference on Learning Representations (...

    2023

  19. [27]

    CMMLU: Measuring Massive Multitask Language Understanding in Chinese

    Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. CMMLU: Measuring Massive Multitask Language Understanding in Chinese . arXiv preprint arXiv:2306.09212, 2023

    2023 arXiv

  20. [28]

    From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and Benchbuilder Pipeline

    Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and Benchbuilder Pipeline . arXiv preprint arXiv:2406.11939, 2024

    2024 arXiv

  21. [29]

    Let's Verify Step by Step

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's Verify Step by Step . In The Twelfth International Conference on Learning Representations, 2023

    2023

  22. [30]

    DeepSeek-V3 Technical Report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. DeepSeek-V3 Technical Report . arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  23. [31]

    Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation . Advances in Neural Information Processing Systems, 36: 0 21558--21572, 2023

    2023

  24. [32]

    Decoupled Weight Decay Regularization

    Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization . arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  25. [33]

    The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation , 2025

    Meta-AI . The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation , 2025. URL https://ai.meta.com/blog/llama-4-multimodal-intelligence/

    2025

  26. [34]

    Mistral-Small-3.1

    Mistral AI . Mistral-Small-3.1 . https://mistral.ai/news/mistral-small-3-1, 2025

    2025

  27. [35]

    Drop-Upcycling: Training Sparse Mixture of Experts with Partial Re-Initialization

    Taishi Nakamura, Takuya Akiba, Kazuki Fujii, Yusuke Oda, Rio Yokota, and Jun Suzuki. Drop-Upcycling: Training Sparse Mixture of Experts with Partial Re-Initialization . arXiv preprint arXiv:2502.19261, 2025

    2025 arXiv

  28. [36]

    GPQA: A Graduate-Level Google-Proof Q&Q Benchmark

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. GPQA: A Graduate-Level Google-Proof Q&Q Benchmark . In First Conference on Language Modeling, 2024

    2024

  29. [37]

    sgl-project . SGLang . https://github.com/sgl-project/sglang, 2025

    2025

  30. [38]

    MoE Travels 3 , 2025

    Jianlin Su. MoE Travels 3 , 2025. URL https://kexue.fm/archives/10757

    2025

  31. [39]

    Hunyuan-Large: An Open-Source MoE Model with 52 Billion Activated Parameters by Tencent

    Xingwu Sun, Yanfeng Chen, Yiqing Huang, Ruobing Xie, Jiaqi Zhu, Kai Zhang, Shuaipeng Li, Zhen Yang, Jonny Han, Xiaobo Shu, et al. Hunyuan-Large: An Open-Source MoE Model with 52 Billion Activated Parameters by Tencent . arXiv preprint arXiv:2411.02265, 2024

    2024 arXiv

  32. [40]

    Challenging Big-Bench Tasks and Whether Chain-of-Thought Can Solve Them

    Mirac Suzgun, Nathan Scales, Nathanael Sch \"a rli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. Challenging Big-Bench Tasks and Whether Chain-of-Thought Can Solve Them . arXiv preprint arXiv:2210.09261, 2022

    2022 arXiv

  33. [41]

    MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark . In The Thirty-eight Conference on Neural Information P...

    2024

  34. [42]

    ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routing

    Ziteng Wang, Jun Zhu, and Jianfei Chen. ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routing . arXiv preprint arXiv:2412.14711, 2024 b

    2024 arXiv

  35. [43]

    Magicoder: Empowering Code Generation with OSS-Instruct

    Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. Magicoder: Empowering Code Generation with OSS-Instruct . arXiv preprint arXiv:2312.02120, 2023

    2023 arXiv

  36. [44]

    Parameter-Efficient Sparsity Crafting from Dense to Mixture-of-Experts for Instruction Tuning on General Tasks

    Haoyuan Wu, Haisheng Zheng, Zhuolun He, and Bei Yu. Parameter-Efficient Sparsity Crafting from Dense to Mixture-of-Experts for Instruction Tuning on General Tasks . In Empirical Methods in Natural Language Processing (EMNLP), 2024

    2024

  37. [45]

    Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing

    Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing . arXiv preprint arXiv:2406.08464, 2024

    2024 arXiv

  38. [46]

    Patil, Ion Stoica, and Joseph E

    Fanjia Yan, Huanzhi Mao, Charlie Cheng-Jie Ji, Tianjun Zhang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. Berkeley Function Calling Leaderboard . https://gorilla.cs.berkeley.edu/blogs/8_berkeley_function_calling_leaderboard.html, 2024

    2024

  39. [47]

    Qwen2.5 Technical Report

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2.5 Technical Report . arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  40. [48]

    Qwen3 Technical Report

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 Technical Report . arXiv preprint arXiv:2505.09388, 2025

    2025 arXiv

  41. [49]

    AdaMoE: Token-Adaptive Routing with Null Experts for Mixture-of-Experts Language Models

    Zihao Zeng, Yibo Miao, Hongcheng Gao, Hao Zhang, and Zhijie Deng. AdaMoE: Token-Adaptive Routing with Null Experts for Mixture-of-Experts Language Models . arXiv preprint arXiv:2406.13233, 2024

    2024 arXiv

  42. [50]

    AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models

    Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models . arXiv preprint arXiv:2304.06364, 2023

    2023 arXiv

  43. [51]

    Instruction-Following Evaluation for Large Language Models

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-Following Evaluation for Large Language Models . arXiv preprint arXiv:2311.07911, 2023

    2023 arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.