Pith. sign in

REVIEW 4 major objections 6 minor 3 cited by

dots.llm1 Technical Report

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that dots.llm1, a mixture-of-experts model that activates only 14B of its 142B parameters, matches a 72B dense model on benchmark performance after 11.2T tokens of pretraining and post-training, at roughly a quarter of…

desk verdict A useful industrial MoE report with open per-trillion checkpoints; the parity claim is plausible but hinges on an unreleased eval harness and a self-reported cost baseline. read the letter →

arxiv 2506.05767 v1 pith:DXRXOI5E submitted 2025-06-06 cs.CL cs.AI

Bi Huo , Bin Tu , Cheng Qin , Da Zheng , Debing Zhang , Dongjie Zhang , En Li , Fu Guo
show 19 more authors
Jian Yao Jie Lou Junfeng Tian Li Hu Ran Zhu Shengdong Chen Shuo Liu Su Guang Te Wo Weijun Zhang Xiaoming Shi Xinxin Peng Xing Wu Yawen Liu Yuqiu Ji Ze Wen Zhenhai Liu Zichao Li Zilong Liao
This is my paper · ORCID
classification cs.CLcs.AI
keywords mixtureofexpertsMoElargelanguagemodelpretrainingdataprocessingcost-efficienttrainingloadbalancingpost-trainingopen-sourcecheckpoints
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

dots.llm1 is a mixture-of-experts language model with 142B total parameters that activates only 14B per token. The paper's central claim is that this 14B-active model delivers performance comparable to the 72B dense Qwen2.5-72B after pretraining on 11.2T tokens and post-training, while using roughly a quarter of the GPU hours (130K versus 340K per trillion tokens, in the authors' accounting). The claim would matter because it suggests the cost-performance frontier of open models can be shifted by sparsity plus data quality rather than by raw dense scale. The report also documents the recipe behind the claim: a three-stage data pipeline, an auxiliary-loss-free load-balancing MoE, infrastructure optimizations for all-to-all communication and grouped GEMMs, and a pretraining regime that uses no synthetic data, with intermediate checkpoints released every trillion tokens.

What carries the argument

The load-bearing object is the sparse MoE layer: each of the 62 layers carries 128 routed experts and 2 shared experts, the router selects the top-6 routed experts per token (8 active experts total), each expert is a fine-grained two-layer SwiGLU FFN, and gating is computed in FP32. Load balance is maintained without an auxiliary routing loss by a per-expert bias term (the auxiliary-loss-free strategy) plus a sequence-wise balance loss, so no tokens are dropped during training. Around that layer sit the supporting mechanisms: the three-stage data pipeline (document preparation, rule-based processing, model-based processing with a line-level clutter-removal model and a 200-class category-balancing classifier) that produces the 11.2T-token corpus; the warmup-stable-decay learning-rate schedule with two annealing stages; the UtK long-context extension to 32K; and the infrastructure recipes, an interleaved 1F1B pipeline that overlaps all-to-all communication with computation, and an aligned-tile grouped GEMM that the authors measure 14% faster forward and 6.68% faster backward on average than Transformer Engine 2.1 on H800.

What would settle it

Re-running the released dots.llm1 base and instruction weights through the identical evaluation harness (same prompts, few-shot counts, decoding temperature, and benchmark versions) as the comparison models would settle the performance claim: if reproduced scores deviate from Table 2 or Table 5 by more than benchmark noise, with HellaSwag reported 88.2 for dots.llm1 against 94.2 for Qwen2.5-72B, the 'comparable' claim would lose support. The cost claim would be falsified by an independent measurement: reproducing a 14B-active MoE pretraining run on a comparable corpus should land near 130K GPU hours per trillion tokens, not near the 340K dense figure.

Watch

Extended reading notes

Core claim

The central claim, on the paper's own terms, is that a 142B-parameter MoE activating only 14B parameters per token can meet a 72B dense model on the standard benchmark suite. Trained on 11.2T tokens from a balanced 1:1 Chinese/English web corpus built by the three-stage pipeline, with no synthetic data, the base model averages 91.3 on Chinese benchmarks, 75.7 on English, 78.3 on math, and 59.6 on code, versus 90.3, 76.3, 77.3, and 59.0 for Qwen2.5-72B; after SFT on 400K instruction instances plus verifier-filtered rejection sampling, the instruction model posts C-Eval 92.2, AIME24 33.1, MATH500 84.8, and ArenaHard 87.1. The authors read these tables as showing that dots.llm1 is comparable to Qwen2.5-72B while costing about a quarter of the GPU hours, and they attribute the result to the data pipeline and to infrastructure work, interleaved 1F1B all-to-all overlap and an efficient grouped GEMM implementation, rather than to model size.

Load-bearing premise

The comparability claim assumes every model in the tables was scored with the same prompts, few-shot counts, decoding settings, and benchmark versions, and the cost claim assumes the 340K GPU-hours-per-trillion figure for the 72B dense baseline, measured on the authors' own optimized setup rather than a published training run, is the right comparison point.

Editorial extensions

If this is right

  • If the benchmark claim holds, a 14B-active MoE trained on 11.2T tokens can substitute for a 72B dense model on most evaluated tasks, making the reported ~4x pretraining compute reduction a practical path for other training efforts.
  • The efficiency argument extends to deployment: the model performs inference on a single node with eight GPUs (40GB or 80GB), and its long-context RULER scores are competitive at 8K and 16K and within a few points of Qwen2.5-72B at 32K.
  • On Chinese language understanding the base model averages 91.3, above both Qwen2.5-72B (90.3) and DeepSeek-V3 (89.5), indicating that the 1:1 Chinese/English balance and the category-balancing classifier carry real capability, not just architectural sparsity.
  • Open-sourcing checkpoints every trillion tokens means the claim of stable training (no irrecoverable loss spikes) is directly checkable and gives the community a resource for studying MoE learning dynamics.
  • The post-training results (MATH500 84.8, CNMO24 40.6, AlpacaEval2 64.4, ArenaHard 87.1) indicate that the two-phase SFT with verifier-filtered math and code data unlocks capability in a 14B-active model that base pretraining alone would not show.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • My inference: if the 130K-versus-340K GPU-hour comparison survives independent measurement, the efficient frontier for open pretraining shifts toward high-sparsity MoE at roughly a 10:1 total-to-active parameter ratio, and the fully documented data pipeline should transfer directly to other architectures.
  • My inference: the paper's own tables suggest data curation, not architecture, is the main differentiator; a testable extension would ablate the category-balanced corpus against an uncurated mix at the same token count using the released per-trillion checkpoints.
  • My inference: the released checkpoint sequence permits a test the paper does not run, namely whether MoE benchmark gains follow a smooth curve or jump after the annealing stage in which reasoning and knowledge data rise to 90% of the mix.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper reports dots.llm1, a Mixture-of-Experts language model with 142B total and 14B active parameters, pretrained on 11.2T Chinese/English tokens without synthetic data, extended from 8K to 32K context via the UtK strategy, and post-trained with SFT/RFT on 400K instruction instances. The central claim is that dots.llm1 performs comparably to Qwen2.5-72B on a broad suite of base and instruction-tuned benchmarks while reducing training cost by roughly 4x, and the authors state they will release intermediate checkpoints every 1T tokens. The paper also describes a three-stage web-data curation pipeline and a communication-computation overlap scheme for MoE training.

Significance. If the benchmark results are independently reproducible, this is a meaningful result: a model with 14B active parameters matching the aggregate performance of a 72B dense model would strengthen the case for MoE as a cost-effective scaling path, and the planned release of intermediate checkpoints is a concrete community asset. The data-curation pipeline and the interleaved-1F1B overlap scheme are potentially useful engineering contributions. The claims are evaluated on external public benchmarks rather than on quantities derived from the model's own training, so there is no circularity; the main risks are the unreleased evaluation harness and the self-reported efficiency baseline.

major comments (4)
  1. [§4.4.1, Tables 2 and 5] The comparability claim rests on the statement that all models were evaluated under identical conditions, but the manuscript does not release the evaluation harness, prompt templates, few-shot formatting, benchmark versions, or decoding settings. Since the empirical support for the headline 'comparable to Qwen2.5-72B' consists of table-level gaps such as MMLU 83.2 vs 85.4, HellaSwag 88.2 vs 94.2, and MMLU-Pro 61.9 vs 64.7, those numbers cannot be independently interpreted as real performance differences rather than evaluation artifacts. Please publish the evaluation harness or a benchmark-by-benchmark specification sufficient to reproduce the reported scores.
  2. [§1 (contributions) and §5.1] The contributions bullet states that the model's performance is achieved 'without reliance on synthetic data or model distillation,' but §5.1 says that a small subset of SFT responses was refined using powerful teacher models (e.g., DeepSeek-V3 0324). This is an internal contradiction in a stated contribution. Please either remove the distillation claim, restrict it explicitly to pretraining, or describe the teacher-model refinement in a way consistent with the contribution.
  3. [§4.5, Table 4] The 4x training-cost reduction compares dots.llm1 to 'Qwen2.5 72B model (in our optimized framework)' at 340K GPU hours per trillion tokens. This is an internal optimized baseline, not an officially published training cost for Qwen2.5-72B, so the cost-efficiency claim is not independently verifiable. In addition, the total-GPU-hours comparison mixes per-token efficiency with the different pretraining token budgets (18T vs 11.2T). Please report per-token cost relative to a public baseline and separate the effect of the smaller data budget.
  4. [§4.5, Figure 4] The claim that the authors' web data surpasses TxT360 is based on a single 1.5B-parameter probe model trained on a 350B-token sample, but the training configuration (optimizer hyperparameters, learning-rate schedule, batch size, evaluation protocol) and the number of independent runs are not reported. Without this information and without variance estimates, the superiority claim for the data pipeline is not established. Please report the full training setup and, if feasible, multiple seeds or error bars.
minor comments (6)
  1. [Throughout] The model name 'dots.llm1' is occasionally concatenated with following text (e.g., 'dots.llm1achieves' in the Figure 1 caption and 'dots.llm1and' in §4.4.2); please fix the typography.
  2. [Table 2] The AGIEval value for DeepSeek V2 Base is reported as '68.34' with two decimals while all other values in the table use one decimal; please align the formatting.
  3. [§4.4.2] The text states that dots.llm1 achieves better performance in the zero-shot setting than in the few-shot setting on mathematics, with an improvement of over 4 points, but no zero-shot numbers are shown in Table 2; please provide the supporting results.
  4. [§3.1] The proposed interleaved-1F1B overlap scheme is described qualitatively and compared with DualPipe as having a 'marginally higher bubble rate,' but no end-to-end throughput or memory measurements are reported; Table 1 reports kernel-level grouped-GEMM speedups only, which do not directly demonstrate the pipeline-overlap benefit.
  5. [§6] In Equation (1), the expert load is defined as N_{E_i,D}/N_D even though each token is routed to eight experts, so the load values do not sum to 1 across experts; the text should state explicitly that the measure is the fraction of domain tokens routed to each expert, not a probability distribution.
  6. [References] The Qwen blog reference reads 'Qwen2.5: A party of foundation models'; 'party' should presumably be 'family'.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: dots.llm1's parity and cost claims rest on external benchmarks and direct measurements, not on self-referential equations or fitted parameters.

full rationale

The report's central claims—(i) performance comparable to Qwen2.5-72B and (ii) roughly 4x lower training cost—are supported by benchmark measurements on external, publicly defined tasks (MMLU, HellaSwag, GSM8K, HumanEval, MMLU-Pro, RULER, and others) and by direct GPU-hour accounting, rather than by solving equations whose outputs are already contained in their inputs. Sections 4.4.1 and 4.4.2 state that evaluations used vLLM under claimed identical conditions; even if the harness is not fully released, that is a reproducibility and correctness risk, not a circularity. The data-quality claim in Section 4.5 is tested with a separate 1.5B dense probe against TxT360 on six external benchmarks, so the pipeline is not validated by its own output. The self-citations that appear—UtK for context extension (Tian et al., 2024b) and the QK-Norm usage (Tian et al., 2024a)—are ancillary design choices; the long-context claim is independently checked on RULER in Table 3, and no uniqueness theorem or forced-choice argument is imported from the authors' prior work. The internal inconsistency between the stated contribution 'without reliance on synthetic data or model distillation' and the Section 5.1 use of teacher models to refine a small subset of SFT responses concerns factual consistency, not circularity. No fitted parameter is renamed as a prediction, and no performance quantity is defined in terms of itself. The derivation chain is therefore self-contained with respect to circularity.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim is an empirical performance comparison, so the ledger contains training hyperparameters and data-curation thresholds rather than fitted scientific constants, plus assumptions about the evaluation harness and baseline cost figures. No new physical or mathematical entities are introduced.

free parameters (5)
  • Peak learning rate = 3e-4
    Hand-set following the WSD schedule; not fitted to benchmarks, but it shapes the final model being evaluated.
  • Batch size schedule = 64M to 96M at 6T, to 128M at 8.3T
    Manual schedule that affects training dynamics and final quality.
  • Data pipeline thresholds = multiple (e.g., language confidence 0.65, line frequency 200, semantic similarity 0.95)
    Hand-chosen cutoffs in Appendix C that define the corpus; not fitted to evaluation targets.
  • Annealing mix ratios = 90% reasoning/knowledge in stage 1; code/math/reasoning boost in stage 2
    Manual data-mix choices during the 1.2T-token annealing phases.
  • Routed expert count and top-k = 128 routed, top-6 plus 2 shared
    Architecture choice inherited from DeepSeekMoE that fixes the 14B active parameter count.
assumptions (4)
  • domain assumption Comparison models were evaluated under identical conditions using the same harness.
    Stated in Table 2 and Table 5 captions, but the harness is not shipped; the central parity claim depends on this assumption.
  • domain assumption The open-sourced checkpoints correspond to the architecture and training configuration described in the paper.
    Performance numbers are attributed to dots.llm1; no checksum or config diff is provided.
  • domain assumption The GPU-hour comparison uses a fair optimized internal baseline for Qwen2.5-72B.
    Table 4 reports 'in our optimized framework' for Qwen; official Qwen training costs are not used.
  • standard math Transformer, attention, SwiGLU, RMSNorm, and top-k MoE routing are valid building blocks.
    No derivations are provided; the report relies on prior published architectures.

how reviews work

0 comments
Cite this review

Pith. "Pith review of dots.llm1 Technical Report." pith.science (2026). https://pith.science/paper/DXRXOI5E

@misc{pith2026250605767,
  author       = {Pith},
  title        = {Pith review of: dots.llm1 Technical Report},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DXRXOI5E}},
  note         = {Machine review of arXiv:2506.05767}
}
read the original abstract

Mixture of Experts (MoE) models have emerged as a promising paradigm for scaling language models efficiently by activating only a subset of parameters for each input token. In this report, we present dots.llm1, a large-scale MoE model that activates 14B parameters out of a total of 142B parameters, delivering performance on par with state-of-the-art models while reducing training and inference costs. Leveraging our meticulously crafted and efficient data processing pipeline, dots.llm1 achieves performance comparable to Qwen2.5-72B after pretraining on 11.2T high-quality tokens and post-training to fully unlock its capabilities. Notably, no synthetic data is used during pretraining. To foster further research, we open-source intermediate training checkpoints at every one trillion tokens, providing valuable insights into the learning dynamics of large language models.

Figures

Figures reproduced from arXiv: 2506.05767 by the authors.

Figure 1
Figure 1. Performance and cost comparison of open MoE and dense language models. Circles ( [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Interleaved 1F1B based communication and computation overlap [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The loss curve highlights the consistent stability of the training process. At 6 trillion training [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Comparison of performance curves between the TxT360 dataset and our web data on the MMLU, [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Expert load heatmap across layers on the Pile test dataset. [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CuraWeb: Joint Optimization of Quality, Redundancy, and Diversity for Web-Scale Pretraining Data

    cs.AI 2026-06 conditional novelty 6.0 of 10

    CuraWeb jointly optimizes cleaning, deduplication, and sampling to build a 2T-token pretraining corpus that outperforms FineWeb-Edu, DCLM, Nemotron-CC, and Dolma3 by 1.82% average at 200B tokens.

  2. Grove MoE: Towards Efficient and Superior MoE LLMs with Adjugate Experts

    cs.CL 2025-08 unverdicted novelty 5.0 of 10

    Grove MoE uses unequal-size adjugate experts with complexity-based activation to run 33B-parameter models at roughly 3.1 to 3.3B active parameters while matching larger open models in benchmarks.

  3. UltraMemV2: Memory Networks Scaling to 120B Parameters with Superior Long-Context Learning

    cs.LG 2025-08 conditional novelty 4.0 of 10

    A redesigned memory-layer architecture with five engineering improvements reaches performance parity with 8-expert MoE at similar compute, with lower memory access and stronger long-context memorization.

Reference graph

Works this paper leans on

52 extracted references · 11 canonical work pages · cited by 3 Pith papers

  1. [1]

    Llama 3 model card, 2024a

    AI@Meta. Llama 3 model card, 2024a. URL https://github.com/meta-llama/llama3/blob/main/MODE L CARD.md. AI@Meta. The llama 4 herd: The beginning of a new era of natively multimodal ai innovation, 2024b. URLhttps://ai.meta.com/blog/llama-4-multimodal-intelligence/. Joshua Ainslie, James Lee-Thorp, Michiel De Jong, Yury Zemlyanskiy, Federico Lebr ´on, and Su...

    arXiv

  2. [6]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pond´e de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bava...

    arXiv

  3. [7]

    Think you have solved question answering? Try ARC, the AI2 reasoning challenge.CoRR, abs/1803.05457,

    12 Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? Try ARC, the AI2 reasoning challenge.CoRR, abs/1803.05457,

    arXiv

  4. [9]

    Damai Dai, Chengqi Deng, Chenggang Zhao, R. X. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y. Wu, Zhenda Xie, Y. K. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, and Wenfeng Liang. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models.CoRR, abs/2401.06066,

    arXiv

  5. [10]

    DeepSeek-AI

    URLhttps://doi.org/10.48550/arXiv.2401.06066. DeepSeek-AI. Deepseek LLM: scaling open-source language models with longtermism.CoRR, abs/2401.02954, 2024a. URLhttps://doi.org/10.48550/arXiv.2401.02954. DeepSeek-AI. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. CoRR, abs/2405.04434, 2024b. URLhttps://doi.org/10.48550/ar...

    arXiv doi:10.48550/arxiv.2401.06066

  6. [11]

    URL https://arxiv.org/abs/2412.19437. Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, An- dreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. InInternational Conference on Machine Learning, pp. 7480–7512. PMLR,

    arXiv

  7. [14]

    Length-controlled alpacaeval: A simple way to debias automatic evaluators.arXiv preprint arXiv:2404.04475,

    Yann Dubois, Bal´azs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators.arXiv preprint arXiv:2404.04475,

    arXiv

  8. [16]

    URLhttps://doi.org/10.48550/arXiv.2406.04127. Zhouhong Gu, Xiaoxuan Zhu, Haoning Ye, Lin Zhang, Jianchen Wang, Yixin Zhu, Sihang Jiang, Zhuozhi Xiong, Zihan Li, Weijie Wu, Qianyu He, Rui Xu, Wenhao Huang, Jingping Liu, Zili Wang, Shusen Wang, Weiguo Zheng, Hongwei Feng, and Yanghua Xiao. Xiezhi: An ever-updating benchmark for holistic domain knowledge evaluation,

    arXiv doi:10.48550/arxiv.2406.04127

Show all 52 references
  1. [17]

    Yancheng He, Shilong Li, Jiaheng Liu, Yingshui Tan, Weixun Wang, Hui Huang, Xingyuan Bu, Hangyu Guo, Chengwei Hu, Boren Zheng, et al

    URLhttps://arxiv.org/abs/2306.05783. Yancheng He, Shilong Li, Jiaheng Liu, Yingshui Tan, Weixun Wang, Hui Huang, Xingyuan Bu, Hangyu Guo, Chengwei Hu, Boren Zheng, et al. Chinese simpleqa: A chinese factuality evaluation for large language models.arXiv preprint arXiv:2411.07140,

    arXiv

  2. [18]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt

    URL https://developer.nvidia.com/blog/introducing-grouped-gemm-apis-in-cublas-and-more-perfo rmance-updates/. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. InICLR. Open...

    arXiv

  3. [19]

    MiniCPM: Unveiling the potential of small language models with scalable training strategies.CoRR, abs/2404.06395,

    Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, Xinrong Zhang, Zhen Leng Thai, Kai Zhang, Chongyi Wang, Yuan Yao, Chenyang Zhao, Jie Zhou, Jie Cai, Zhongwu Zhai, Ning Ding, Chao Jia, Guoyang Zeng, Dahai Li, Z...

    arXiv

  4. [20]

    Livecodebench: Holistic and contamination free evaluation of large language models for code.arXiv preprint arXiv:2403.07974,

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar- Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code.arXiv preprint arXiv:2403.07974,

    arXiv

  5. [21]

    Style mixture of experts for expressive text-to-speech synthesis.arXiv preprint arXiv:2406.03637,

    Ahad Jawaid, Shreeram Suresh Chandra, Junchen Lu, and Berrak Sisman. Style mixture of experts for expressive text-to-speech synthesis.arXiv preprint arXiv:2406.03637,

    arXiv

  6. [22]

    Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, L ´elio Renard Lavaud, Lucile Saulnier, Marie-An...

    arXiv

  7. [24]

    URLhttps://doi.org/10.1162/tacl a 00276

    doi: 10.1162/tacl\ a\ 00276. URLhttps://doi.org/10.1162/tacl a 00276. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention...

    doi

  8. [25]

    CMMLU: Measuring massive multitask language understanding in Chinese.CoRR, abs/2306.09212,

    Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timo- thy Baldwin. CMMLU: Measuring massive multitask language understanding in Chinese.CoRR, abs/2306.09212,

    arXiv

  9. [26]

    From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline.arXiv preprint arXiv:2406.11939,

    Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline.arXiv preprint arXiv:2406.11939,

    arXiv

  10. [27]

    A coordinated tiling and batching framework for efficient GEMM on gpus

    Xiuhong Li, Yun Liang, Shengen Yan, Liancheng Jia, and Yinghan Li. A coordinated tiling and batching framework for efficient GEMM on gpus. In Jeffrey K. Hollingsworth and Idit Keidar (eds.),Proceedings of the 24th ACM SIGPLAN Symposium on Principles and Practice of Parallel Pr...

    2019

  11. [28]

    URLhttps://doi.org/10.1145/3293883.3295734

    doi: 10.1145/3293883.3295734. URLhttps://doi.org/10.1145/3293883.3295734. Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. InThe Twelfth International Co...

  12. [29]

    Evaluating language models for efficient code generation.arXiv preprint arXiv:2408.06450, 2024a

    Jiawei Liu, Songrun Xie, Junhao Wang, Yuxiang Wei, Yifeng Ding, and Lingming Zhang. Evaluating language models for efficient code generation.arXiv preprint arXiv:2408.06450, 2024a. Junnan Liu, Hongwei Liu, Linchen Xiao, Ziyi Wang, Kuikun Liu, Songyang Gao, Wenwei Zhang, Songya...

    arXiv

  13. [30]

    American invitational mathematics examination - aime

    MAA. American invitational mathematics examination - aime. InAmerican Invitational Mathematics Examination - AIME 2024, February

    2024

  14. [31]

    URLhttps://mistral.ai/news/mistral-small-3. NVIDIA. pytorch - transformer engine 2.1.0 documentation, August 2024a. URL https://docs.nvidia. com/deeplearning/transformer-engine/user-guide/api/pytorch.html#transformer engine.pytorc h.GroupedLinear. NVIDIA. MoE A2A Interleaved 1...

    2025 arXiv

  15. [32]

    URLhttps://openai.com/index/introducing-simpleqa/. OpenAI. Introducing gpt-4.5, 2025a. URLhttps://openai.com/index/introducing-gpt-4-5/. OpenAI. Introducing openai o3 and o4-mini, 2025b. URL https://openai.com/index/introducing-o 3-and-o4-mini/. Denis Paperno, Germ´an Kruszews...

    2016

  16. [34]

    URLhttps://openreview.net/forum?id=n6SCkn2QaG. Qwen. Qwen2.5: A party of foundation models, 2024a. URLhttps://qwenlm.github.io/blog/qwen2.5. Qwen. Qwen1.5-moe: Matching 7b model performance with 1/3 activated parameters”, February 2024b. URLhttps://qwenlm.github.io/blog/qwen-m...

    arXiv

  17. [35]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level Google-proof Q&A benchmark. CoRR, abs/2311.12022,

    arXiv

  18. [36]

    Glu variants improve transformer.arXiv preprint arXiv:2002.05202,

    Noam Shazeer. Glu variants improve transformer.arXiv preprint arXiv:2002.05202,

    2002 arXiv

  19. [37]

    Time-moe: Billion-scale time series foundation models with mixture of experts.arXiv preprint arXiv:2409.16040,

    Xiaoming Shi, Shiyu Wang, Yuqi Nie, Dianqi Li, Zhou Ye, Qingsong Wen, and Ming Jin. Time-moe: Billion-scale time series foundation models with mixture of experts.arXiv preprint arXiv:2409.16040,

    arXiv

  20. [38]

    Junfeng Tian, Rui Wang, Cong Li, Yudong Zhou, Jun Liu, and Jun Wang

    URLhttps://arxiv.org/abs/2502.14739. Junfeng Tian, Rui Wang, Cong Li, Yudong Zhou, Jun Liu, and Jun Wang. Nyonic technical report.arXiv preprint arXiv:2404.15702, 2024a. Junfeng Tian, Da Zheng, Yang Cheng, Rui Wang, Colin Zhang, and Debing Zhang. Untie the knots: An efficient ...

    arXiv

  21. [39]

    Auxiliary-loss-free load balancing strategy for mixture-of-experts.CoRR, abs/2408.15664, 2024a

    Lean Wang, Huazuo Gao, Chenggang Zhao, Xu Sun, and Damai Dai. Auxiliary-loss-free load balancing strategy for mixture-of-experts.CoRR, abs/2408.15664, 2024a. URL https://doi.org/10.48550/arXiv .2408.15664. Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shigu...

    doi arXiv

  22. [40]

    Small-scale proxies for large-scale transformer training instabilities.arXiv preprint arXiv:2309.14322,

    Mitchell Wortsman, Peter J Liu, Lechao Xiao, Katie Everett, Alex Alemi, Ben Adlam, John D Co-Reyes, Izzeddin Gur, Abhishek Kumar, Roman Novak, et al. Small-scale proxies for large-scale transformer training instabilities.arXiv preprint arXiv:2309.14322,

    arXiv

  23. [41]

    URLhttps://x.ai/news/grok-3. Liang Xu, Hai Hu, Xuanwei Zhang, Lu Li, Chenjie Cao, Yudong Li, Yechen Xu, Kai Sun, Dian Yu, Cong Yu, Yin Tian, Qianqian Dong, Weitang Liu, Bo Shi, Yiming Cui, Junyi Li, Jun Zeng, Rongzhao Wang, Weijian Xie, Yanting Li, Yina Patterson, Zuoyu Tian, ...

    2020

  24. [42]

    URL https://doi.org/10.18653/v1/2020.coling-main.419

    doi: 10.18653/V1/2020.COLING-MAIN.419. URL https://doi.org/10.18653/v1/2020.coling-main.419. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report.arXiv preprint arXiv:2412.15115,

    2020 arXiv

  25. [43]

    Gated linear attention transformers with hardware-efficient training.arXiv preprint arXiv:2312.06635,

    Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware-efficient training.arXiv preprint arXiv:2312.06635,

    arXiv

  26. [44]

    URL https://doi.org/10.1 007/s11227-022-04336-3

    doi: 10.1007/S11227-022-04336-3. URL https://doi.org/10.1 007/s11227-022-04336-3. 17 Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? In Anna Korhonen, David R. Traum, and Llu´ıs M`arquez (eds.),Proce...

    doi

  27. [45]

    URLhttps://doi.org/10.18653/v1/p19-1472

    doi: 10.18653/v1/p19-1472. URLhttps://doi.org/10.18653/v1/p19-1472. Yaqing Zhang, Yaobin Wang, Zhangbin Mo, Yong Zhou, Tao Sun, Guang Xu, Chaojun Xing, and Liang Yang. Accelerating small matrix multiplications by adaptive batching strategy on GPU. In24th IEEE Int Conf on High ...

    doi

  28. [46]

    URL https://doi.org/10.110 9/HPCC-DSS-SmartCity-DependSys57074.2022.00143

    doi: 10.1109/HPCC-DSS-SMARTCITY-DEPENDSYS57074.2022.00143. URL https://doi.org/10.110 9/HPCC-DSS-SmartCity-DependSys57074.2022.00143. Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. AGIEval: A human-centric benc...

    2022

  29. [47]

    URL https://doi.org/10.48550/arXiv.2304

    doi: 10.48550/arXiv.2304.06364. URL https://doi.org/10.48550/arXiv.2304. 06364. Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models.arXiv preprint arXiv:2311.07911,

    doi arXiv

  30. [48]

    URLhttps://arxiv.org/abs/2406.15877. 18 Appendix A Authors Bi Huo Bin Tu Cheng Qin Da Zheng Debing Zhang Dongjie Zhang En Li Fu Guo Jian Yao Jie Lou Junfeng Tian Li Hu Ran Zhu Shengdong Chen Shuo Liu Su Guang Te Wo Weijun Zhang Xiaoming Shi Xinxin Peng Xing Wu Yawen Liu Yuqiu ...

    2024 arXiv

  31. [49]

    to extract the main text from webpages. To mitigate the presence of irrelevant content, we implement a series of custom optimizations fortrafilatura, including adjustments to HTML patterns, keyword filtering, and content length specifications. Language IdentificationLeveraging...

    2016

  32. [50]

    and Gopher (Rae et al., 2021), combining precise heuristic rules and statistical features to systematically remove low-quality content. Our pipeline includes empty content removal, advertisement and registration prompt filtering, domain/URL/title-based meta filtering, wiki/cod...

    2021

  33. [51]

    The process involves the following steps: First, we apply the same text standardization as in the Identity Removal step

    to perform approximate deduplication. The process involves the following steps: First, we apply the same text standardization as in the Identity Removal step. Next, we tokenize the text using the Jieba tokenizer 4 to handle Chinese text, followed by 5-gram processing. Using th...

  34. [52]

    Quality ModelThe quality model performs comprehensive multidimensional analysis to evaluate and score training samples (Qwen, 2024a; Penedo et al., 2024)

    We fine-tune a 1.5B model and find that it significantly improves the overall data quality. Quality ModelThe quality model performs comprehensive multidimensional analysis to evaluate and score training samples (Qwen, 2024a; Penedo et al., 2024). We design a comprehensive anno...

    2024

  35. [2016]

    URL https://doi.org/10.1 8653/v1/p16-1144

    doi: 10.18653/v1/p16-1144. URL https://doi.org/10.1 8653/v1/p16-1144. Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The RefinedWeb dataset for Falcon LLM: ou...

    arXiv

  36. [2017]

    doi: 10.18653/v1/P17-1147

    Association for Computational Linguistics. doi: 10.18653/v1/P17-1147. URLhttps://aclanthology.org/P17-1147. Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, H´erve J´egou, and Tomas Mikolov. Fasttext.zip: Compressing text classification models.arXiv preprint arX...

    arXiv

  37. [2018]

    Training verifiers to solve math word problems.CoRR, abs/2110.14168,

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems.CoRR, abs/2110.14168,

    arXiv

  38. [2019]

    URL https://doi.org/10.18653/v1/ n19-1246

    doi: 10.18653/V1/N19-1246. URL https://doi.org/10.18653/v1/ n19-1246. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, A...

    arXiv

  39. [2020]

    URL https://doi.org/10.1609/aaai.v34i05.6239

    doi: 10.1609/aaai.v34i05.6239. URL https://doi.org/10.1609/aaai.v34i05.6239. Andrei Z Broder. On the resemblance and containment of documents. InProceedings. Compression and Complexity of SEQUENCES 1997 (Cat. No. 97TB100171), pp. 21–29. IEEE,

    1997 doi

  40. [2021]

    Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi

    URL https: //aclanthology.org/2021.acl-demo.15. Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. PIQA: reasoning about physical commonsense in natural language. InThe Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Secon...

    2021

  41. [2022]

    Are we done with mmlu?CoRR, abs/2406.04127,

    Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong, Alessio Devoto, Alberto Carlo Maria Mancino, Rohit Saxena, Xuanli He, Yu Zhao, Xiaotang Du, Mohammad Reza Ghasemi Madani, Claire Barale, Robert McHardy, Joshua Harris, Jean Kaddour, Emile van Krieken, and Pasquale Minervini...

    arXiv

  42. [2023]

    DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs

    Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.),Proceedings of the 2019 Conference of t...

    2019

  43. [2024]

    Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu

    URL https://arxiv.org/abs/2406.07436. Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. Bge m3-embedding: Multi- lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation,

    arXiv

  44. [2025]

    Jacob Austin, Augustus Odena, Maxwell I

    URL https://assets.anthropic.com/m/785e231869ea8 b3b/original/claude-3-7-sonnet-system-card.pdf. Jacob Austin, Augustus Odena, Maxwell I. Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V . Le, and Charles Sutton. Program sy...

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.