Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

KAT-V1: Kwai-AutoThink Technical Report

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A 40-billion-parameter model claims to match much larger reasoning systems by learning when to think, while cutting token usage.

desk verdict KAT-V1 is a genuine industrial release with a plausible adaptive-reasoning pipeline, but the headline SOTA claims rest on a benchmark table that cannot be checked as written. read the letter →

arxiv 2507.08297 v3 pith:G7BYXZDY submitted 2025-07-11 cs.CL

Zizheng Zhan , Ken Deng , Huaixi Tang , Wen Xiang , Kun Wu , Weihao Li , Wenqiang Zhu , Jingxuan Xu
show 22 more authors
Lecheng Huang Zongxian Feng Shaojie Wang Shangpeng Yan Xuxing Chen Jiaheng Liu Zhongyuan Peng Zuchen Gao Haoyang Huang Xiaojiang Zhang Jinghui Wang Zheng Lin Mengtong Li Huiming Wang Ziqi Zhan Yanan Wu Yuanxing Zhang Jian Yang Guang Chen Haotian Zhang Bin Chen Bing Yu
This is my paper
classification cs.CL
keywords adaptivereasoningoverthinkingchain-of-thoughtmulti-tokenpredictionknowledgedistillationreinforcementlearningtokenefficiencythink-onthink-offgating
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

KAT-V1-40B is an open 40-billion-parameter language model built to test a simple proposition: a model that decides per query whether to reason step by step can match the accuracy of much larger models that always reason, and do it with fewer tokens. The paper tries to establish this by training the model in three phases—dual-regime distillation with multi-token prediction, a cold-start phase that teaches think-on/think-off mode selection from majority-vote labels, and a reinforcement learning stage whose rewards cover both choosing the right mode and producing the right answer. The reported outcome is that KAT matches or outperforms DeepSeek-R1-0528 and Qwen3-235B-A22B across math, code, and general reasoning benchmarks, and that it tops all open-source models and surpasses o3-mini on the leakage-controlled LiveCodeBench Pro. The claim matters because overthinking, or unnecessary chain-of-thought on easy queries, is what makes high-accuracy reasoning models slow and expensive in deployed assistants; if KAT is right, reasoning depth becomes a controllable efficiency dial rather than a fixed cost.

What carries the argument

The load-bearing mechanism is the AutoThink training pipeline, organized around a single gating decision the model must learn: think_on or think_off. The pipeline has three parts. First, a dual-regime dataset labels each query by complexity and assigns it to a non-reasoning (Think-off) or reasoning (Think-on) regime, with Think-off responses generated by DeepSeek-V3 and Think-on responses produced by a multi-agent solver-thinker-critic pipeline; distillation then transfers teacher logits into the student's Multi-Token Prediction (MTP) heads using a Universal Logit Distillation Loss, so the student learns to predict several future tokens at once rather than only the next token. Second, a cold-start phase uses majority voting over multiple model outputs, intent-aware prompting, and a <judge> segment that explains why a mode was chosen, giving the model an initial prior for autonomous mode selection. Third, Step-SRPO extends the SRPO/GRPO reinforcement learning framework with two rewards—a Judge Reward for choosing the correct reasoning mode and an Answer Reward for final correctness—so the model is pushed simultaneously toward accurate gating and accurate answers. The central identity is that token efficiency and accuracy are not traded off but co-optimized through one learned gating variable.

What would settle it

Re-running Table 4 in a single harness with identical sampling settings and the official evaluation protocols would settle the claim: if KAT-V1-40B's LiveCodeBench Pro score of 12.7 does not exceed the o3-mini and Seed scores under that benchmark's official protocol, or if the Qwen3-235B-A22B code rows revert to their published values and KAT no longer matches DeepSeek-R1-0528 on AIME2024 and AIME2025, the central comparison fails. A controlled token-count comparison with identical prompts and temperatures would likewise settle the efficiency claim.

Watch

Extended reading notes

Core claim

The paper's central discovery is that the overthinking problem can be attacked at the level of training objectives: instead of treating chain-of-thought as something a model always emits, KAT is trained to gate it with explicit mode tokens, <think_on> and <think_off>, so that reasoning depth becomes a learned per-query decision. The authors report that this gating is what lets a 40B dense model match or beat DeepSeek-R1-0528, a 671B-parameter system with 37B active parameters, and Qwen3-235B-A22B on tasks like AIME2024 and AIME2025, while using fewer tokens; on LiveCodeBench Pro, a benchmark built to resist data leakage, they report KAT scoring 12.7 and leading all open-source models, ahead of o3-mini. The same mechanism is reported working in production: the model switches to direct answers on trivial queries, sparing hundreds of tokens and seconds of latency, and switches to long structured reasoning on complex engineering queries. The paper also reports that the AutoThink paradigm is scaling: a 200B MoE variant with 40B active parameters shows early gains over the 40B dense model.

Load-bearing premise

The load-bearing premise is that every model in the comparison tables was evaluated under identical, fair conditions, meaning the same prompts, decoding parameters, pass counts, token accounting, and post-processing, and that the numbers quoted for competing models match their official protocols.

Editorial extensions

If this is right

  • A 40B open model can reach the accuracy tier of 671B-parameter reasoning systems on several math and code benchmarks, which undercuts the assumption that raw scale is the main driver of reasoning quality.
  • Making reasoning depth a learned decision yields measurable efficiency: the paper reports the average think-on activation rate falling from roughly 90% to 66% and average output tokens falling about 19% during RL training, with further savings compared with DeepSeek-R1-0528.
  • Leakage-controlled evaluation changes the leaderboard: on LiveCodeBench Pro, KAT's reported 12.7 places it first among open-source models and ahead of the proprietary o3-mini, suggesting that contamination-resistant benchmarks reward training methods rather than data memorization.
  • The AutoThink pipeline transfers to larger sparse models: the in-training 200B MoE variant with 40B active parameters already shows gains over the 40B dense KAT, indicating the paradigm is scale-compatible.
  • Deployable assistants can exploit controllable reasoning: in the reported production cases, the model skips chain-of-thought on simple queries, avoiding a roughly 400-token, 17-second verbose response, and engages deep reasoning on complex ones, giving users a dial between speed and depth.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An extension the paper leaves implicit is that the judge signal, the explanation of why a query needs reasoning, could be reused as a lightweight difficulty estimator, allowing mode selection to transfer to tasks without verifiable answers or unit tests.
  • The announced plan to release 1.5B, 7B, and 13B variants makes a direct test possible: if the same three-stage pipeline reproduces the gating behavior at smaller scales, then AutoThink is a training recipe rather than a property of 40B-scale models.
  • The paper's anecdotal user overrides, such as 'DO NOT THINK' and 'Think deeper', suggest a product-level control surface; a quantitative study of override reliability under distribution shift would tell whether the gating mechanism remains stable when users force it against the learned prior.
  • Because the headline comparison tables do not list o3-mini or Seed scores, an independent re-run of LiveCodeBench Pro under its official protocol is the cheapest way to verify the strongest claim.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. KAT-V1 reports on Kwaipilot-AutoThink, a 40B dense model obtained by upscaling Qwen2.5-32B to 80 layers and continuing training on a dual-regime corpus. Stage 1 combines heterogeneous knowledge distillation from DeepSeek-V3 and DeepSeek-R1-0528 with Multi-Token Prediction and a Universal Logit Distillation loss; Stage 2 adds a cold-start mode-selection initialization and a Step-SRPO reinforcement learning algorithm, and the model is evaluated on general, math, code, agentic, and scientific benchmarks. The abstract and introduction claim that KAT matches or outperforms DeepSeek-R1-0528 and Qwen3-235B-A22B, surpasses o3-mini and Seed on leakage-controlled LiveCodeBench Pro, and reduces token usage.

Significance. If the reported results are reproducible and the comparisons are fair, the paper demonstrates a useful direction: a 40B model with explicit control over reasoning depth that approaches much larger systems while reducing inference tokens. The release of the model weights, the in-production deployment discussion, and the analyses of think-on rate and token dynamics during training are concrete strengths. However, the headline claims rest almost entirely on Table 4, and the manuscript provides no evaluation protocol and contains at least one baseline number that appears inconsistent with public results. The significance is therefore conditional on fixing the evaluation reporting rather than on the architectural ideas alone.

major comments (4)
  1. [§5, Table 4] The central comparison lacks an evaluation protocol. The paper does not state the prompts, decoding parameters, pass@k, number of runs, answer-extraction rules, or token-counting method used for any model in Table 4. This is load-bearing because every headline claim is a cross-model comparison. The table also contains a direct internal inconsistency: Qwen3-235B-A22B is listed at HumanEval 30.5 and MBPP 65.4, while Qwen3-32B in the same table is listed at 90.2 and 74.6, and the Qwen3-235B-A22B figures are far below the scores reported in the Qwen3 technical report. Either the Qwen3-235B-A22B baseline was run in a non-standard configuration or the number is misreported; in neither case can the reader trust the column. Please provide the full evaluation protocol and verify every baseline number against its official source or reproduce it under a documented configuration.
  2. [Abstract, §1, §5, Table 4] The claim that KAT 'surpasses o3-mini' on leakage-controlled LiveCodeBench Pro is not checkable from the manuscript. The LCB-Pro-Med row of Table 4 contains entries only for KAT, DeepSeek-R1-0528, DeepSeek-V3, and LLaMA-4-Maverick; no o3-mini or Seed score appears, and no external source or evaluation condition is cited for these proprietary baselines. Since this is one of the two strongest named claims in the abstract, the paper must either add the measured baseline scores and their protocol or temper the claim to what Table 4 actually supports.
  3. [§4.2] Step-SRPO is presented as a core contribution, but the implementation is deferred: the text states that 'the detailed AutoThink RL implementation will be released in a forthcoming companion paper' and the Future Works section repeats that the RL code and data will be released later. The current description gives only a qualitative account of Judge Reward and Answer Reward, with no objective function, update rule, reward shaping, normalization, or details on how the intermediate supervision is inserted into the GRPO/SRPO framework. Because the paper's contribution includes the RL algorithm, this omission prevents independent verification. Please include a precise algorithm specification or release the code alongside this report.
  4. [§3.2, §5] No contamination analysis or data-cutoff information is reported for the training corpora. Given that Stage 1 uses roughly 10 million examples from public and proprietary sources and the paper emphasizes a 'leakage-controlled' benchmark (LiveCodeBench Pro) for one of its headline claims, the authors should state the training-data cutoff, describe any near-duplicate or overlap removal with the evaluation benchmarks, and report whether contamination checks were performed.
minor comments (4)
  1. [§3.1, §3.3] The Universal Logit Distillation Loss is named and depicted in Figure 3 but never defined; please provide the actual loss expression. In addition, the claim of 'approximately 5 percent improvement' from MTP-enhanced distillation would be easier to assess if the benchmark names and scores were shown in a small table rather than stated only in prose.
  2. [Table 1] Table 1 reports layers, tie-embedding, heads, and context length, but omits hidden size, intermediate size, vocabulary size, and the number of duplicated layers; adding these would let readers understand the upscaling step and compare with the Qwen2.5-32B base.
  3. [§4.2.3, Figures 2 and 4] The text around the Stage-3 difficulty distribution refers to 'Figure 2' and 'Figure 4' for data distributions; on first reading this is confusing because Figure 2 is the framework diagram and Figure 4 shows domain distributions. Please verify the cross-references and, if possible, add a dedicated figure for the difficulty distribution.
  4. [§5, Figures 7–9] The claimed reduction in token usage (from 9,887 to 8,037 tokens on average) is presented without the underlying numbers in a table or the token-counting convention (e.g., including or excluding the thinking segment, stop tokens, or tool outputs). A short table with per-benchmark token counts would make the efficiency claim independently checkable.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation found; the only self-referential element is the same-group SRPO base for Step-SRPO, which is a transparency gap rather than a reduction of the benchmark claims.

full rationale

This report is an empirical system description rather than a derivation, so the main circularity patterns do not apply. The training pipeline is self-contained: pre-training distills from external DeepSeek teachers, cold-start mode labels come from majority voting with DeepSeek-V3/R1 as generators and judges, and Step-SRPO rewards are defined by external verifiers (math equivalence, unit tests, option keys). No fitted parameter is later relabeled as a prediction, and no uniqueness theorem or ansatz is imported from same-author prior work to force the conclusions. The headline comparisons to DeepSeek-R1, Qwen3, Seed, and o3-mini rest on the unstated cross-model evaluation protocol; Table 4 even lists Qwen3-235B-A22B at 30.5 HumanEval and omits o3-mini/Seed from the LiveCodeBench Pro row. Those are verifiability and correctness concerns, not circularity. The one self-referential element is Section 4.2's Step-SRPO, which cites the same group's SRPO [30] and defers the detailed implementation to a forthcoming companion paper; since the benchmark numbers are not derived from SRPO itself, this is a transparency gap rather than load-bearing circularity.

Assumptions & free parameters 5 free parameters · 5 assumptions · 1 invented entities

The central claims depend on a long chain of unstated modeling choices: data mixture ratios chosen by hand, teacher reliability, majority-vote mode labels, coarse reward proxies, and comparability of self-reported benchmark scores. The paper provides no ablations showing which choices are load-bearing, so the ledger contains assumptions that a replication would need to test.

free parameters (5)
  • Stage 1 Think-on data proportion = 34.8% Think-on / 65.2% Think-off
    Hand-chosen mixture balancing reasoning and non-reasoning training regimes in Section 3.2; no sensitivity analysis is shown, yet it directly shapes the mode-switching behavior that is the paper's central contribution.
  • Stage 2 Think-on to Think-off ratio = approximately 2:1
    Hand-chosen ratio in Section 4.1.2; influences cold-start mode-selection prior, with no ablation.
  • Random mode assignment fraction = 1%
    Section 4.1.1 assigns 1% of queries a random mode to avoid overfitting; the value is chosen by hand and no sensitivity analysis is given.
  • RL training corpus difficulty distribution = biased toward high difficulty among 45K samples
    Section 4.2.3 says difficulty is broadly distributed with concentration on harder examples; no quantitative distribution is shown, only a figure without numeric labels.
  • MTP future-token prediction depth = two future tokens, Token B and Token C
    Section 3.1 uses MTP modules predicting two future tokens; no ablation over the number of future tokens is reported.
assumptions (5)
  • domain assumption Upscaling Qwen2.5-32B by duplicating near-identity layers preserves and improves capabilities.
    Section 2 relies on layer saturation analysis from reference [34] to select layers for duplication; no controlled validation is given for this specific model.
  • domain assumption DeepSeek-R1-0528 and DeepSeek-V3 outputs are trustworthy supervision for reasoning and non-reasoning responses.
    Sections 3.1 and 3.2 use these teachers for distillation and response generation; their correctness is assumed, not independently verified.
  • domain assumption Majority voting across model outputs identifies the correct thinking mode for each query.
    Section 4.1.1 uses majority votes to label cold-start data; no accuracy of the labeling is reported, and the 1% random labels acknowledge the signal is imperfect.
  • domain assumption Verifier rewards used in Step-SRPO measure answer correctness.
    Section 4.2.2 uses symbolic equivalence for math, unit tests for code, option-letter matching for science, and lightweight heuristics for general tasks; the heuristic reward in particular is acknowledged to be a coarse proxy.
  • domain assumption Benchmark numbers in Table 4 are comparable across models despite unspecified evaluation settings.
    Section 5 reports scores without decoding details, pass@k, or token-counting protocol; the implied comparability underpins every headline claim.
invented entities (1)
  • Control tokens <think_on>, <think_off>, <judge>, <think>, and <answer>
    purpose: Mark whether reasoning is activated, where reasoning text starts, where the answer begins, and where the mode-judgment rationale lives.
    Introduced in Section 3 and Table 3 as internal training and inference markers; they have no falsifiable handle outside the paper beyond example generations, and their effectiveness is not separately measured.

how reviews work

0 comments
Cite this review

Pith. "Pith review of KAT-V1: Kwai-AutoThink Technical Report." pith.science (2026). https://pith.science/paper/G7BYXZDY

@misc{pith2026250708297,
  author       = {Pith},
  title        = {Pith review of: KAT-V1: Kwai-AutoThink Technical Report},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/G7BYXZDY}},
  note         = {Machine review of arXiv:2507.08297}
}
read the original abstract

We present Kwaipilot-AutoThink (KAT), an open-source 40B large language model developed to address the overthinking problem in reasoning-intensive tasks, where an automatic thinking training paradigm is proposed to dynamically switch between reasoning and non-reasoning modes based on task complexity. Specifically, first, we construct the dual-regime dataset based on a novel tagging pipeline and a multi-agent synthesis strategy, and then we apply Multi-Token Prediction (MTP)-enhanced knowledge distillation, enabling efficient and fine-grained reasoning transfer with minimal pretraining cost. Besides, we implement a cold-start initialization strategy that introduces mode-selection priors using majority-vote signals and intent-aware prompting. Finally, we propose Step-SRPO, a reinforcement learning algorithm that incorporates intermediate supervision into the GRPO framework, offering structured guidance over both reasoning-mode selection and response accuracy. Extensive experiments across multiple benchmarks demonstrate that KAT consistently matches or even outperforms current state-of-the-art models, including DeepSeek-R1-0528 and Qwen3-235B-A22B, across a wide range of reasoning-intensive tasks while reducing token usage. Notably, KAT outperforms all open-source models and even surpasses o3-mini on the leakage-controlled LiveCodeBench Pro. Beyond academic evaluation, KAT has been successfully deployed in Kwaipilot (i.e., Kuaishou's internal coding assistant), where it improves real-world development workflows with high accuracy, efficiency, and controllable reasoning behaviors. Moreover, we are actively training a 200B Mixture-of-Experts (MoE) model with 40B active parameters, and early results already show significant gains, further demonstrating the scalability of the AutoThink paradigm.

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. R-4B: Incentivizing General-Purpose Auto-Thinking Capability in MLLMs via Bi-Mode Annealing and Reinforce Learning

    cs.CV 2025-08 conditional novelty 6.0 of 10

    R-4B uses bi-mode annealing plus a GRPO variant with forced thinking and non-thinking rollouts so a 4B multimodal model adaptively picks when to reason, reportedly matching 16B thinking models on several reasoning benchmarks.

Reference graph

Works this paper leans on

47 extracted references · 14 canonical work pages · cited by 1 Pith paper

  1. [1]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    arXiv 2025

  2. [2]

    Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model

    Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434, 2024

    arXiv 2024

  3. [3]

    Deepseek llm: Scaling open-source language models with longtermism

    Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954, 2024

    arXiv 2024

  4. [4]

    Deepseek-coder: When the large language model meets programming–the rise of code intelligence

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al. Deepseek-coder: When the large language model meets programming–the rise of code intelligence. arXiv preprint arXiv:2401.14196, 2024

    arXiv 2024

  5. [5]

    Qwen3 technical report

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025

    arXiv 2025

  6. [6]

    Qwen2.5 technical report, 2025

    Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li,...

    2025

  7. [7]

    Qwen2 technical report, 2024

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei...

    2024

  8. [8]

    Qwen technical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023

    arXiv 2023

Show all 47 references
  1. [9]

    The llama 4 herd: The beginning of a new era of natively multimodal ai inno- vation

    Meta-AI. The llama 4 herd: The beginning of a new era of natively multimodal ai inno- vation. https://ai.meta.com/blog/llama-4-multimodal-intelligence/ , 2025. Accessed: 2025-07-04

    2025

  2. [10]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. 21

    2024 arXiv

  3. [11]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  4. [12]

    Mtu-bench: A multi-granularity tool-use benchmark for large language models

    Pei Wang, Yanan Wu, Zekun Wang, Jiaheng Liu, Xiaoshuai Song, Zhongyuan Peng, Ken Deng, Chenchen Zhang, Jiakai Wang, Junran Peng, et al. Mtu-bench: A multi-granularity tool-use benchmark for large language models. arXiv preprint arXiv:2410.11710, 2024

    arXiv

  5. [13]

    A comprehensive survey on long context language modeling

    Jiaheng Liu, Dawei Zhu, Zhiqi Bai, Yancheng He, Huanxuan Liao, Haoran Que, Zekun Wang, Chenchen Zhang, Ge Zhang, Jiebin Zhang, et al. A comprehensive survey on long context language modeling. arXiv preprint arXiv:2503.17407, 2025

    2025

  6. [14]

    M2rc-eval: Massively multilingual repository-level code completion evaluation

    Jiaheng Liu, Ken Deng, Congnan Liu, Jian Yang, Shukai Liu, He Zhu, Peng Zhao, Linzheng Chai, Yanan Wu, Ke Jin, et al. M2rc-eval: Massively multilingual repository-level code completion evaluation. arXiv preprint arXiv:2410.21157, 2024

    arXiv

  7. [15]

    R2c2-coder: Enhancing and benchmarking real-world repository-level code completion abilities of code large language models

    Ken Deng, Jiaheng Liu, He Zhu, Congnan Liu, Jingxin Li, Jiakai Wang, Peng Zhao, Chenchen Zhang, Yanan Wu, Xueqiao Yin, et al. R2c2-coder: Enhancing and benchmarking real-world repository-level code completion abilities of code large language models. arXiv preprint arXiv:2406.0...

    2024 arXiv

  8. [16]

    Do not think that much for 2+ 3=? on the overthinking of o1-like llms

    Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, et al. Do not think that much for 2+ 3=? on the overthinking of o1-like llms. arXiv preprint arXiv:2412.21187, 2024

    2024 arXiv

  9. [17]

    Concise reason- ing via reinforcement learning

    Mehdi Fatemi, Banafsheh Rafiee, Mingjie Tang, and Kartik Talamadupula. Concise reason- ing via reinforcement learning. arXiv preprint arXiv:2504.05185, 2025

    2025

  10. [18]

    Stop overthinking: A survey on efficient reasoning for large language models

    Yang Sui, Yu-Neng Chuang, Guanchu Wang, Jiamu Zhang, Tianyi Zhang, Jiayi Yuan, Hongyi Liu, Andrew Wen, Shaochen Zhong, Hanjie Chen, et al. Stop overthinking: A survey on efficient reasoning for large language models. arXiv preprint arXiv:2503.16419, 2025

    2025 arXiv

  11. [19]

    Adacot: Pareto-optimal adaptive chain-of-thought triggering via reinforcement learning

    Chenwei Lou, Zewei Sun, Xinnian Liang, Meng Qu, Wei Shen, Wenqi Wang, Yuntao Li, Qingping Yang, and Shuangzhi Wu. Adacot: Pareto-optimal adaptive chain-of-thought triggering via reinforcement learning. arXiv preprint arXiv:2505.11896, 2025

    2025 arXiv

  12. [20]

    Think only when you need with large hybrid-reasoning models

    Lingjie Jiang, Xun Wu, Shaohan Huang, Qingxiu Dong, Zewen Chi, Li Dong, Xingxing Zhang, Tengchao Lv, Lei Cui, and Furu Wei. Think only when you need with large hybrid-reasoning models. arXiv preprint arXiv:2505.14631, 2025

    2025 arXiv

  13. [21]

    Thinkless: Llm learns when to think

    Gongfan Fang, Xinyin Ma, and Xinchao Wang. Thinkless: Llm learns when to think. arXiv preprint arXiv:2505.13379, 2025

    2025 arXiv

  14. [22]

    Learning when to think: Shaping adaptive reasoning in r1-style models via multi-stage rl

    Songjun Tu, Jiahao Lin, Qichao Zhang, Xiangyu Tian, Linjing Li, Xiangyuan Lan, and Dongbin Zhao. Learning when to think: Shaping adaptive reasoning in r1-style models via multi-stage rl. arXiv preprint arXiv:2505.10832, 2025

    2025

  15. [23]

    Adaptthink: Reasoning models can learn when to think

    Jiajie Zhang, Nianyi Lin, Lei Hou, Ling Feng, and Juanzi Li. Adaptthink: Reasoning models can learn when to think. arXiv preprint arXiv:2505.13417, 2025

    2025 arXiv

  16. [24]

    Z. Shao, P . Wang, Q. Zhu, R. Xu, J. Song, M. Zhang, Y. Li, Y. Wu, and D. Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. 22

    2024 arXiv

  17. [25]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  18. [26]

    Knowledge distillation: A survey

    Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey. International Journal of Computer Vision, 129(6):1789–1819, 2021

    2021

  19. [27]

    Distilling the knowledge in a neural network

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015

    2015 arXiv

  20. [28]

    Ddk: Distilling domain knowledge for efficient large language models

    Jiaheng Liu, Chenchen Zhang, Jinyang Guo, Yuanxing Zhang, Haoran Que, Ken Deng, Jie Liu, Ge Zhang, Yanan Wu, Congnan Liu, et al. Ddk: Distilling domain knowledge for efficient large language models. Advances in Neural Information Processing Systems, 37:98297–98319, 2024

    2024

  21. [29]

    Deepseek-v3 technical report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  22. [30]

    Srpo: A cross-domain imple- mentation of large-scale reinforcement learning on llm

    Xiaojiang Zhang, Jinghui Wang, Zifei Cheng, Wenhao Zhuang, Zheng Lin, Minglei Zhang, Shaojie Wang, Yinghan Cui, Chao Wang, Junyi Peng, et al. Srpo: A cross-domain imple- mentation of large-scale reinforcement learning on llm. arXiv preprint arXiv:2504.14286, 2025

    2025 arXiv

  23. [31]

    Livecodebench pro: How do olympiad medalists judge llms in competitive programming?, 2025

    Zihan Zheng, Zerui Cheng, Zeyu Shen, Shang Zhou, Kaiyuan Liu, Hansen He, Dongruix- uan Li, Stanley Wei, Hangyi Hao, Jianzhu Yao, Peiyao Sheng, Zixuan Wang, Wenhao Chai, Aleksandra Korolova, Peter Henderson, Sanjeev Arora, Pramod Viswanath, Jingbo Shang, and Saining Xie. Liveco...

    2025

  24. [32]

    Introduction to techniques used in seed1.6

    ByteDance Seed. Introduction to techniques used in seed1.6. https://seed.bytedance .com/en/seed1_6, 2025. Accessed: 2025-07-21

    2025

  25. [33]

    Introducing openai o3-mini

    OpenAI. Introducing openai o3-mini. https://openai.com/index/openai-o3-min i/, 2025. Published: January 31, 2025; Accessed: 2025-07-21

    2025

  26. [34]

    Yi: Open foundation models by 01

    Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Guoyin Wang, Heng Li, Jiangcheng Zhu, Jianqun Chen, et al. Yi: Open foundation models by 01. ai. arXiv preprint arXiv:2403.04652, 2024

    2024 arXiv

  27. [35]

    Llama-nemotron: Efficient reasoning models, 2025

    Akhiad Bercovich, Itay Levy, Izik Golan, Mohammad Dabbah, Ran El-Yaniv, Omri Puny, Ido Galil, Zach Moshe, Tomer Ronen, Najeeb Nabwani, Ido Shahaf, Oren Tropp, Ehud Karpas, Ran Zilberstein, Jiaqi Zeng, Soumye Singhal, Alexander Bukharin, Yian Zhang, Tugrul Konuk, Gerald Shen, A...

    2025

  28. [36]

    Mmlu-pro: A more robust and challenging multi-task language understanding benchmark

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. In The Thirty-eight Conference on Neural Information Pr...

    2024

  29. [37]

    DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs

    Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proc. of NAACL, 2019

    2019

  30. [38]

    Wildbench: Benchmarking llms with challenging tasks from real users in the wild, 2024

    Bill Yuchen Lin, Yuntian Deng, Khyathi Chandu, Faeze Brahman, Abhilasha Ravichander, Valentina Pyatkin, Nouha Dziri, Ronan Le Bras, and Yejin Choi. Wildbench: Benchmarking llms with challenging tasks from real users in the wild, 2024

    2024

  31. [39]

    Gpqa: A graduate-level google-proof q&a benchmark

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024

    2024

  32. [40]

    Math-500 dataset

    HuggingFaceH4. Math-500 dataset. https://huggingface.co/datasets/HuggingF aceH4/MATH-500, 2024. Accessed: 2025-07-10

    2024

  33. [41]

    Aime_2024 dataset

    Maxwell Jia. Aime_2024 dataset. https://huggingface.co/datasets/Maxwell-Jia /AIME_2024, 2024. Accessed: 2025-07-10

    2024

  34. [42]

    Aime2025 dataset

    OpenCompass. Aime2025 dataset. https://huggingface.co/datasets/opencomp ass/AIME2025, 2025. Accessed: 2025-07-10

    2025

  35. [43]

    Autologi: Automated generation of logic puzzles for evaluating reasoning abilities of large language models, 2025

    Qin Zhu, Fei Huang, Runyu Peng, Keming Lu, Bowen Yu, Qinyuan Cheng, Xipeng Qiu, Xuanjing Huang, and Junyang Lin. Autologi: Automated generation of logic puzzles for evaluating reasoning abilities of large language models, 2025

    2025

  36. [44]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott G...

    2021

  37. [45]

    Program synthesis with large language models

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021

    2021 arXiv

  38. [46]

    Livecodebench: Holistic and con- tamination free evaluation of large language models for code

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and con- tamination free evaluation of large language models for code. arXiv preprint, 2024

    2024

  39. [47]

    Patil, Huanzhi Mao, Charlie Cheng-Jie Ji, Fanjia Yan, Vishnu Suresh, Ion Stoica, and Joseph E

    Shishir G. Patil, Huanzhi Mao, Charlie Cheng-Jie Ji, Fanjia Yan, Vishnu Suresh, Ion Stoica, and Joseph E. Gonzalez. The berkeley function calling leaderboard (bfcl): From tool use to agentic evaluation of large language models. In Forty-second International Conference on Machi...

    2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.