Pith. sign in

REVIEW 5 major objections 6 minor 3 cited by

Yi-Lightning Technical Report

T0 review · 5 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Yi-Lightning, a Mixture-of-Experts model, debuted 6th on the Chatbot Arena leaderboard and 2nd in Chinese.

desk verdict A useful engineering report on a new MoE model: the architecture ideas are worth stealing, but the headline Arena rank is a single noisy snapshot and the efficiency numbers are unverifiable. read the letter →

arxiv 2412.01253 v5 pith:XZBT6SJF submitted 2024-12-02 cs.CL cs.AIcs.LG

Alan Wake , Bei Chen , C.X. Lv , Chao Li , Chengen Huang , Chenglin Cai , Chujie Zheng , Daniel Cooper
show 36 more authors
Fan Zhou Feng Hu Ge Zhang Guoyin Wang Heng Ji Howard Qiu Jiangcheng Zhu Jun Tian Katherine Su Lihuan Zhang Liying Li Ming Song Mou Li Peng Liu Qicheng Hu Shawn Wang Shijun Zhou Shiming Yang Shiyong Li Tianhang Zhu Wen Xie Wenhao Huang Xiang He Xiaobo Chen Xiaohui Hu Xiaoyi Ren Xinyao Niu Yanpeng Li Yongke Zhao Yongzhen Luo Yuchi Xu Yuxuan Sha Zhaodong Yan Zhiyuan Liu Zirui Zhang Zonghong Dai
This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords Mixture-of-ExpertslargelanguagemodelChatbotArenahumanpreferencealignmentDirectOptimizationKVcachereuseRLHFRAISEsafety
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Yi-Lightning is a Mixture-of-Experts large language model whose October 16, 2024 debut placed it 6th overall on the Chatbot Arena leaderboard, tied with Grok-2-08-13, with 2nd place in Chinese and top-four finishes in Math, Coding, and Hard Prompts. The paper attributes this result to a collection of engineering choices: finer expert segmentation, a partitioned expert-parallel load-balancing loss, hybrid sliding-window/full attention with cross-layer KV-cache reuse, multi-stage pre-training, and post-training built on two-stage SFT, reward modeling, and offline-plus-online DPO. The same system posts competitive numbers on GPQA, MATH, HumanEval, IFEval, WildBench, Arena-Hard, AlignBench, and MT-Bench, though those static scores are less dominant than the human-preference ranking. The authors read this gap as evidence that current academic benchmarks undercount the qualities that make users prefer a model in practice. A sympathetic reader takes away a concrete, cost-conscious recipe for reaching top human-preference territory with an MoE model.

What carries the argument

The load-bearing mechanism is the routing and load-balancing stack around the MoE backbone. Fine-grained expert segmentation shrinks each expert's hidden dimension while increasing the number of experts activated per token, which the paper says improves parameter utilization; the accompanying losses—Switch-Transformer's $L_{ST}$, the Expert-Parallel loss $L_{EP}$, and the partitioned EP loss $L_{PEP}$—balance token dispatch within groups and partitions so All-to-All communication does not stall training. On the attention side, hybrid blocks place three sliding-window attention layers ahead of one full-attention layer, and cross-layer KV-cache reuse shares key-value states between consecutive full-attention layers, halving that component's memory and yielding up to 82.8% overall long-sequence memory reduction. In post-training, the online DPO loop is the efficiency carrier: it samples 16 responses per prompt, forms preference pairs by reward-model score gap, caches reference-model log-probabilities instead of keeping the reference model resident, and reuses the shared context of positive and negative samples to cut redundant computation. RAISE wraps the lifecycle with four safety components spanning pre-training filtration, post-training reward engineering, input filtering, and output control.

What would settle it

Run the public Yi-Lightning API on held-out versions of Arena-Hard, WildBench, and MATH and compare with the reported values (91.8, 65.1, 76.4); scores well below these numbers, or an archived official leaderboard that does not show a 6th-place debut on October 16, 2024, would falsify the central performance claim.

Watch

Extended reading notes

Core claim

The central claim is that Yi-Lightning, judged by real human pairwise comparisons on Chatbot Arena at its debut, performs at the level of leading proprietary systems: an Arena score of 1287, 6th overall, 2nd in Chinese, 3rd in Multi-Turn and Math, and 4th in Coding, Hard Prompts, and Longer Query. The paper explains this through architecture and training: experts are split into fine-grained FFN units so each token activates more, smaller experts; routing balance is enforced at expert, expert-parallel-group, and partition levels with losses $L_{ST}$, $L_{EP}$, and $L_{PEP}$ tuned to $10^{-6}$, $10^{-4}$, and $10^{-3}$; KV state is reused across consecutive full-attention layers while sliding-window attention covers local context, cutting long-sequence memory by up to 82.8%; and post-training proceeds through 1.3M plus 300K SFT samples, two-stage reward modeling over cleaned public and human preference data, then offline DPO followed by two online DPO iterations. On academic benchmarks the model is strong but not uniformly first, with Arena-Hard 91.8, WildBench 65.1, MT-Bench 8.75, and MATH 76.4, and the paper argues the gap between these static numbers and its high human-preference rank reflects a deliberate focus on practical user experience.

Load-bearing premise

The reported Chatbot Arena snapshot and benchmark tables are accurate, representative, and reproducible from the deployed model, since the report gives no confidence intervals, run counts, or released weights to verify them.

Editorial extensions

If this is right

  • A MoE model with fine-grained expert segmentation and cross-layer KV-cache reuse can place among the best human-preference systems while cutting memory, so cost-sensitive labs have a concrete architectural recipe.
  • Two-stage SFT plus offline and two-iteration online DPO is presented as sufficient to push a strong base model into the top tier of human preference, without more elaborate RL.
  • The reported gap between static benchmarks and Chatbot Arena implies that high MATH, IFEval, or GPQA scores do not guarantee real-user preference, arguing for preference-based evaluation in model development.
  • The 20B-token, 64K-context extension stage with boosted RoPE base frequency maintains standard benchmark performance, providing a data-efficient path to long context.
  • The partitioned EP load-balancing loss attacks the All-to-All dispatch imbalance that slows large MoE training, a problem that otherwise worsens as expert segmentation increases dispatched tokens.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The leaderboard positions are a single October 2024 snapshot; as an inference, the durable contribution is the training and architecture recipe, not the rank, which newer models may surpass.
  • The benchmark-versus-human preference gap suggests a directly testable hypothesis: a model tuned to maximize Arena-Hard judge scores will lose on WildBench or in human voting, and ablating output-format constraints like math boxed answers could isolate the cause.
  • Since the report releases no weights or evaluation logs, independent API-based replication of the benchmark and preference numbers is the natural next step to confirm the architectural claims.
  • If the KV-cache and attention claims transfer, other MoE models adopting hybrid attention with cross-layer sharing should see memory reductions near the reported 82.8% without quality loss; that prediction can be checked on any open MoE checkpoint.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper presents Yi-Lightning, a Mixture-of-Experts (MoE) large language model developed by 01.AI, and describes its architecture (fine-grained expert segmentation, EP/PEP load balancing, hybrid attention with cross-layer KV cache sharing), pre-training and post-training strategies, infrastructure optimizations, and the RAISE safety framework. The headline claims are a 6th-place overall ranking on the Chatbot Arena leaderboard, top category rankings, up to 82.8% KV memory reduction, up to 70% training speedup, >99% goodput, and competitive performance on public academic benchmarks, alongside a claimed disparity between static benchmarks and human preferences.

Significance. If the performance and efficiency claims are substantiated, Yi-Lightning would be a competitive flagship system, and the proposed PEP load-balancing mechanism and cross-layer KV cache sharing are of technical interest to the MoE community. The paper also raises a legitimate question about the alignment of static benchmarks with human preferences. However, the manuscript currently provides no evaluation protocols, confidence intervals, or measurement details, and the model weights are not released, so the significance of the reported results cannot yet be independently assessed.

major comments (5)
  1. [§1, Fig. 1] The central claim of 6th place on Chatbot Arena rests on a single leaderboard snapshot (October 16, 2024) with no vote counts, confidence intervals, or time-series data. The reported margin over GPT-4o-0513 is 2 points, which is well within the typical bootstrapped error of Arena scores, so the rank may reflect a high-variance early-vote state rather than a stable property of the model. Please report number of votes, bootstrapped CIs, and scores over a longer window to substantiate the headline claim.
  2. [§2.3, §5.1, §5.2, §5.3] The quantitative efficiency claims—up to 82.8% KV memory reduction, up to 70% training speedup, 95% GPU utilization in high-concurrency scenarios, and goodput above 99%—are stated without measurement conditions, baselines, or ablations. For example, §2.3 asserts the memory reduction 'while maintaining model performance on long sequences' but provides no perplexity or downstream benchmark comparison. Please describe the experimental setups for each number and provide data or derivations.
  3. [§7, Tables 1 and 2] All academic benchmark scores are presented as point estimates without standard deviations, number of runs, or exact evaluation prompts. The LLM-as-a-judge evaluations on WildBench, Arena-Hard, AlignBench, and MT-Bench use GPT-4o-0513 as the judge, but no judge agreement or variance is reported. Since the 'competitive performance' claim in the abstract rests on these tables, the comparison is currently not reproducible. Please include the missing statistical detail and, ideally, release the evaluation scripts.
  4. [§7, Final Discussion] The paper interprets the gap between academic-benchmark results and Arena rankings as evidence that conventional benchmarks have limited utility. However, this conclusion is based on a single, statistically uncharacterized Arena snapshot and a small set of benchmarks. Moreover, within the paper's own tables, Yi-Lightning leads on Arena-Hard (91.8) and MT-Bench (8.75)—themselves LLM-as-a-judge benchmarks—so the claimed 'disparity' is not consistently observed even in the reported data. Please provide a more rigorous analysis, for example by controlling for benchmark variance and Arena vote counts, before drawing a broad conclusion about the utility of academic benchmarks.
  5. [Footnote 4, Table 2] The footnote states that Yi-Lightning's API serving trades off inference speed and accuracy and that API-based evaluation results may be lower than local-deployment results. This raises a critical ambiguity: if the Chatbot Arena rankings are obtained via the API, the headline result corresponds to a speed-degraded variant, while the academic benchmark tables may reflect the local model. Please clarify which results are from the API and which from local deployment, and quantify the performance trade-off.
minor comments (6)
  1. [Fig. 1 caption vs. §7] The Figure 1 caption says Yi-Lightning is 'tied with Grok-2-08-13,' while §7 says it performs 'on par with GPT-4o-0513' (ranked 7th, score 1285). Please reconcile these statements or specify the exact ties.
  2. [§3.1] The 30-gram decontamination criterion is described in one sentence; please specify which benchmark subsets (training and test) are used and how substring matches are detected.
  3. [Table 2] Footnote 4 appears after the table; move it before the table or into the main text to avoid confusion about which results are affected.
  4. [References] The reference for Shibata et al. contains a typo: 'Yusuxke' should be 'Yusuke Shibata'.
  5. [§2] The paper does not state the total number of parameters, active parameters, or training FLOPs, which are standard for MoE technical reports and would help contextualize the architecture and efficiency claims.
  6. [§5.3] The term 'goodput' is used without a formal definition; please define it and describe the measurement window and hardware environment.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the report is descriptive and its performance claims rest on external leaderboard and benchmark measurements, not on fitted parameters or self-citation chains.

full rationale

This technical report is primarily a description of model architecture, training data, and infrastructure, rather than a derivation of predictions from fitted inputs. The central claim (6th place overall on Chatbot Arena, with category ranks 2nd to 4th) is presented in Section 1 and Section 7 as an externally measured leaderboard snapshot from October 16, 2024, and is not derived from any parameter fitted in the paper. The load-balancing losses in Section 2.2 (L_ST, L_EP, L_PEP) are standard auxiliary losses with hand-tuned coefficients (alpha_PEP = 1e-3, alpha_EP = 1e-4, alpha_ST = 1e-6), and no evaluation result is obtained by plugging fitted values back into the same equations. Self-citations to 01.AI (2024) appear as background for the previous Yi models and the data processing pipeline (Sections 3 and 3.1), but they are not load-bearing for the headline performance claim or for any benchmark number. The academic benchmark tables report measurements against external benchmarks (GPQA, MATH, HumanEval, IFEval, WildBench, Arena-Hard, AlignBench, MT-Bench). The absence of confidence intervals, vote counts, or a detailed evaluation protocol for the Chatbot Arena snapshot is a legitimate reproducibility and robustness concern, but it is not circularity: the reported rank is not equivalent to the paper's inputs by construction, and no fitted parameter is renamed as a prediction. Therefore, no circular step can be identified under the required standard of quoting a specific reduction, and the appropriate score is 0.

Assumptions & free parameters 3 free parameters · 2 assumptions · 0 invented entities

The central claim (competitive performance) does not depend on new free parameters or invented entities. The three alpha coefficients are training hyperparameters tuned by hand. The only background assumptions are standard mathematical tools and empirical observations about activation sparsity. No new particles, forces, or dimensions are introduced.

free parameters (3)
  • alpha_ST (Switch Transformer load balancing coefficient) = 10^-6
    Chosen by hand-tuning in Section 2.2; affects training stability but not the main benchmark claim.
  • alpha_EP (Expert Parallel load balancing coefficient) = 10^-4
    Hand-tuned in Section 2.2 to balance load across EP groups.
  • alpha_PEP (Partitioned EP load balancing coefficient) = 10^-3
    Hand-tuned in Section 2.2; the main new routing mechanism.
assumptions (2)
  • standard math Lagrange multiplier method for minimizing the load balancing loss
    Used in Section 2.2 to show the loss is minimized when tokens are evenly distributed. Standard calculus of variations.
  • domain assumption Activation sparsity in large dense models
    Invoked in Section 2.1 to justify fine-grained expert segmentation, citing Zhang et al. 2024a,b. This is an empirical observation from prior work, not proven in this paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Yi-Lightning Technical Report." pith.science (2026). https://pith.science/paper/XZBT6SJF

@misc{pith2026241201253,
  author       = {Pith},
  title        = {Pith review of: Yi-Lightning Technical Report},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XZBT6SJF}},
  note         = {Machine review of arXiv:2412.01253}
}
read the original abstract

This technical report presents Yi-Lightning, our latest flagship large language model (LLM). It achieves exceptional performance, ranking 6th overall on Chatbot Arena, with particularly strong results (2nd to 4th place) in specialized categories including Chinese, Math, Coding, and Hard Prompts. Yi-Lightning leverages an enhanced Mixture-of-Experts (MoE) architecture, featuring advanced expert segmentation and routing mechanisms coupled with optimized KV-caching techniques. Our development process encompasses comprehensive pre-training, supervised fine-tuning (SFT), and reinforcement learning from human feedback (RLHF), where we devise deliberate strategies for multi-stage training, synthetic data construction, and reward modeling. Furthermore, we implement RAISE (Responsible AI Safety Engine), a four-component framework to address safety issues across pre-training, post-training, and serving phases. Empowered by our scalable super-computing infrastructure, all these innovations substantially reduce training, deployment and inference costs while maintaining high-performance standards. With further evaluations on public academic benchmarks, Yi-Lightning demonstrates competitive performance against top-tier LLMs, while we observe a notable disparity between traditional, static benchmark results and real-world, dynamic human preferences. This observation prompts a critical reassessment of conventional benchmarks' utility in guiding the development of more intelligent and powerful AI systems for practical applications. Yi-Lightning is now available through our developer platform at https://platform.lingyiwanwu.com.

Figures

Figures reproduced from arXiv: 2412.01253 by the authors.

Figure 1
Figure 1. Snapshot of the Chatbot Arena leaderboard on October 16, 2024, with Yi-Lightning’s initial [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Illustration of RAISE (Responsible AI Safety Engine) system. [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. OctoThinker: Mid-training Incentivizes Reinforcement Learning Scaling

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A two-stage mid-training recipe on math-heavy corpora turns Llama-3.2 base models into ones whose RL math performance matches Qwen2.5 at the same size.

  2. ProcessBench: Identifying Process Errors in Mathematical Reasoning

    cs.AI 2024-12 conditional novelty 6.0 of 10

    ProcessBench is a 3,400-case human-annotated benchmark measuring whether AI models can find the first erroneous step in math solutions, with evaluations showing process reward models lag behind prompted critic models.

  3. Overview of the NLPCC 2025 Shared Task: Gender Bias Mitigation Challenge

    cs.CL 2025-06 conditional novelty 4.0 of 10

    A Chinese gender-bias corpus and shared-task benchmark show detection and classification are feasible, while automatic mitigation remains weak.

Reference graph

Works this paper leans on

39 extracted references · 3 canonical work pages · cited by 3 Pith papers

  1. [1]

    Yi: Open foundation models by 01.ai

    01.AI. Yi: Open foundation models by 01.ai. arXiv preprint arXiv:2403.04652,

    arXiv

  2. [3]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774,

    arXiv

  3. [6]

    Training a helpful and harmless assistant with reinforcement learning from human feedback

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862,

    arXiv

  4. [8]

    Ralph Allan Bradley and Milton E Terry

    URL https: //engineering.fb.com/2022/10/18/open-source/ocp-summit-2022-grand-teton/ . Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345,

    work page 2022

  5. [10]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168,

    arXiv

  6. [11]

    Ultrafeedback: Boosting language models with high-quality feedback

    Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback. arXiv preprint arXiv:2310.01377,

    arXiv

  7. [12]

    Deepseekmoe: Towards ultimate expert specialization in mixture- of-experts language models

    Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y Wu, et al. Deepseekmoe: Towards ultimate expert specialization in mixture- of-experts language models. arXiv preprint arXiv:2401.06066,

    arXiv

  8. [13]

    Boosting large-scale parallel training efficiency with c4: A communication-driven approach

    Jianbo Dong, Bin Luo, Jun Zhang, Pengcheng Zhang, Fei Feng, Yikai Zhu, Ang Liu, Zian Chen, Yi Shi, Hairong Jiao, et al. Boosting large-scale parallel training efficiency with c4: A communication-driven approach. arXiv preprint arXiv:2406.04594,

    arXiv

Show all 39 references
  1. [14]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783,

    arXiv

  2. [15]

    Data engineering for scaling language models to 128k context

    Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, Hannaneh Hajishirzi, Yoon Kim, and Hao Peng. Data engineering for scaling language models to 128k context. arXiv preprint arXiv:2402.10171,

    arXiv

  3. [16]

    Gemini: A family of highly capable multimodal models

    Gemini Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: A family of highly capable multimodal models. arXiv preprint arXiv:2312.11805,

    arXiv

  4. [17]

    14 Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y . Wu, Y . K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. Deepseek-coder: When the large language model meets programming – the rise of code intelligence. arXiv preprint arXiv...

    arXiv

  5. [18]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874,

    arXiv

  6. [19]

    Minicpm: Unveiling the potential of small language models with scalable training strategies

    Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, et al. Minicpm: Unveiling the potential of small language models with scalable training strategies. arXiv preprint arXiv:2404.06395,

    arXiv

  7. [20]

    Simple and scalable strategies to continually pre-train large language models

    Adam Ibrahim, Benjamin Thérien, Kshitij Gupta, Mats L Richter, Quentin Anthony, Timothée Lesort, Eugene Belilovsky, and Irina Rish. Simple and scalable strategies to continually pre-train large language models. arXiv preprint arXiv:2403.08763,

    arXiv

  8. [21]

    Mistral 7b

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825,

    arXiv

  9. [22]

    Mixtral of experts

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024a. Ziheng Jiang, Haibin Lin, Yinmin...

    arXiv

  10. [24]

    Let’s verify step by step

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. arXiv preprint arXiv:2305.20050,

    arXiv

  11. [25]

    Wildbench: Benchmarking llms with challenging tasks from real users in the wild

    Bill Yuchen Lin, Yuntian Deng, Khyathi Chandu, Faeze Brahman, Abhilasha Ravichander, Valentina Pyatkin, Nouha Dziri, Ronan Le Bras, and Yejin Choi. Wildbench: Benchmarking llms with challenging tasks from real users in the wild. arXiv preprint arXiv:2406.04770,

    arXiv

  12. [26]

    Deepseek-v2: A strong, economical, and efficient mixture-of- experts language model

    Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. Deepseek-v2: A strong, economical, and efficient mixture-of- experts language model. arXiv preprint arXiv:2405.04434,

    arXiv

  13. [27]

    Alignbench: Benchmarking chinese alignment of large language models

    Xiao Liu, Xuanyu Lei, Shengyuan Wang, Yue Huang, Zhuoer Feng, Bosi Wen, Jiale Cheng, Pei Ke, Yifan Xu, Weng Lam Tam, et al. Alignbench: Benchmarking chinese alignment of large language models. arXiv preprint arXiv:2311.18743,

    arXiv

  14. [28]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2305.18290,

    arXiv

  15. [29]

    Gpqa: A graduate-level google-proof q&a benchmark

    15 David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311.12022,

    arXiv

  16. [31]

    Smith, Luke Zettlemoyer, Scott Yih, and Mike Lewis

    Weijia Shi, Sewon Min, Maria Lomeli, Chunting Zhou, Margaret Li, Gergely Szilvasy, Rich James, Xi Victoria Lin, Noah A. Smith, Luke Zettlemoyer, Scott Yih, and Mike Lewis. In-context pretraining: Language modeling beyond document boundaries. arXiv preprint arXiv:2310.10638,

    arXiv

  17. [33]

    github.io/blog/qwen2.5/

    URL https://qwenlm. github.io/blog/qwen2.5/. Shenzhi Wang, Chang Liu, Zilong Zheng, Siyuan Qi, Shuo Chen, Qisen Yang, Andrew Zhao, Chaofei Wang, Shiji Song, and Gao Huang. Boosting LLM agents with recursive contemplation for effective deception handling. In Findings of the Ass...

    arXiv

  18. [34]

    Iterative preference learning from human feedback: Bridging theory and practice for RLHF under KL-constraint

    Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. Iterative preference learning from human feedback: Bridging theory and practice for RLHF under KL-constraint. In ICLR 2024 Workshop on Mathematical and Empirical Understanding of F ound...

    2024

  19. [35]

    Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, et al

    URL https://openreview.net/forum?id=w3oJXMAQx2. Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, et al. Effective long-context scaling of foundation models. arXiv preprint arX...

    arXiv

  20. [36]

    Qwen2 technical report

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671,

    arXiv

  21. [37]

    Relu 2 wins: Discovering efficient activation functions for sparse llms

    Zhengyan Zhang, Yixin Song, Guanghui Yu, Xu Han, Yankai Lin, Chaojun Xiao, Chenyang Song, Zhiyuan Liu, Zeyu Mi, and Maosong Sun. Relu 2 wins: Discovering efficient activation functions for sparse llms. arXiv preprint arXiv:2402.03804, 2024a. Zhengyan Zhang, Chaojun Xiao, Qiuji...

    arXiv

  22. [38]

    Click: Controllable text generation with sequence likelihood contrastive learning

    16 Chujie Zheng, Pei Ke, Zheng Zhang, and Minlie Huang. Click: Controllable text generation with sequence likelihood contrastive learning. In Findings of the Association for Compu- tational Linguistics: ACL 2023 , pages 1022–1040, Toronto, Canada, July 2023a. Associa- tion for...

    2023 arXiv

  23. [39]

    Universal and transferable adversarial attacks on aligned language models

    Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043,

    arXiv

  24. [1952]

    Evaluating large language models trained on code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374,

    arXiv

  25. [1999]

    Roformer: Enhanced Transformer with Rotary Position Embedding

    Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced Transformer with Rotary Position Embedding. arXiv preprint arXiv:2104.09864,

    arXiv

  26. [2002]

    URL https://www.usenix.org/conference/fast-02/ gpfs-shared-disk-file-system-large-computing-clusters

    USENIX Association. URL https://www.usenix.org/conference/fast-02/ gpfs-shared-disk-file-system-large-computing-clusters . Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y . K. Li, Y . Wu, and Daya Guo. Deepseekmath: Pushi...

    arXiv

  27. [2018]

    From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline

    Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline. arXiv preprint arXiv:2406.11939,

    arXiv

  28. [2021]

    Qwen technical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609,

    arXiv

  29. [2022]

    Ocp summit 2022: Open hardware for ai infrastructure, Nov

    Alexis Bjorlin. Ocp summit 2022: Open hardware for ai infrastructure, Nov

    2022

  30. [2023]

    Program synthesis with large language models

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732,

    arXiv

  31. [2024]

    Phi-3 technical report: A highly capable language model locally on your phone

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219,

    arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.