Pith. sign in

REVIEW 3 major objections 5 minor 11 cited by

MiniCPM4: Ultra-Efficient LLMs on End Devices

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MiniCPM4 claims that an 8B model, trained on 22% of the tokens used by Qwen3-8B, can match that model's benchmark performance while decoding 128K-token documents about seven times faster on edge GPUs.

desk verdict Plausible data-efficiency story with real releases, but the headline 7x long-context speedup is unsupported by the evidence and contradicted by the paper's own conclusion. read the letter →

arxiv 2506.07900 v2 pith:3ZDXG3WW submitted 2025-06-09 cs.CL cs.AI

MiniCPM Team: Chaojun Xiao , Yuxuan Li , Xu Han , Yuzhuo Bai , Jie Cai , Haotian Chen , Wentong Chen , Xin Cong
show 74 more authors
Ganqu Cui Ning Ding Shengda Fan Yewei Fang Zixuan Fu Wenyu Guan Yitong Guan Junshao Guo Yufeng Han Bingxiang He Yuxiang Huang Baoxi Ji Cunliang Kong Qiuzuo Li Siyuan Li Wenhao Li Xin Li Yanghao Li Yishan Li Zhen Li Dan Liu Biyuan Lin Yankai Lin Xiang Long Quanyu Lu Yaxi Lu Peiyan Luo Hongya Lyu Litu Ou Yinxu Pan Lushi Pu Zekai Qu Qundong Shi Zijun Song Jiayuan Su Zhou Su Ao Sun Xianghui Sun Peijun Tang Fangzheng Wang Feng Wang Shuo Wang Yudong Wang Zheng Wang Yesai Wu Zhenyu Xiao Jie Xie Zihao Xie Xiaoyue Xu Yukun Yan Jiarui Yuan Jinqian Zhang Kaihuo Zhang Lei Zhang Linyue Zhang Xueren Zhang Yudi Zhang Hengyu Zhao Weilin Zhao Weilun Zhao Yuanqian Zhao Zhi Zheng Chuyue Zhou Ge Zhou Jie Zhou Wei Zhou Yanghao Zhou Zihan Zhou Zixuan Zhou Zhiyuan Liu Guoyang Zeng Chao Jia Dahai Li Maosong Sun
This is my paper
classification cs.CLcs.AI
keywords efficientlargelanguagemodelsend-sidedeploymentsparseattentiontrainingdatafilteringspeculativesamplingquantization-awarelong-contextprocessingreinforcementlearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MiniCPM4 is an 8B (and 0.5B) language model built for phones, robots, and other edge hardware, and the paper's central claim is that efficiency need not come at the cost of capability. Trained on 8.3T tokens, MiniCPM4-8B matches or exceeds the open Qwen3-8B model across standard benchmarks even though Qwen3 used 36T tokens, and it processes 128K-token documents about seven times faster on an edge GPU. The authors attribute this to four coordinated innovations: trainable sparse attention, filtered and synthesized training data, cheaper hyperparameter and reinforcement-learning training, and a purpose-built inference stack. If the head-to-head numbers hold under matched evaluation, the result implies that data quality and architecture can substitute for a large fraction of training compute.

What carries the argument

InfLLM v2, a trainable sparse-attention mechanism in which each query attends only the top-k key-value blocks selected by mean-pooled semantic kernels, with the first block and local-window blocks always included. It cuts attention computation and memory access during both prefilling and decoding without adding parameters, and the paper reports 81% attention sparsity in the trained model and as little as 5% of tokens attended on 128K inputs. On the data side, UltraClean's two-stage annealing verification with a lightweight linear classifier decides which web tokens are kept, and a ScalingBench-loss indicator replaces raw language-model loss for hyperparameter search; those two pieces carry the 22%-tokens claim. CPM.cu then couples the sparse-attention kernel with frequency-ranked speculative drafting and prefix-aware quantization to turn the architectural savings into measured wall-clock speed.

What would settle it

Rerun MiniCPM4-8B and Qwen3-8B through the same evaluation harness with identical prompts, few-shot exemplars, and decoding settings, repeating each benchmark several times to estimate noise; if the averaged gap over the eight tasks shrinks to within one point or flips on several tasks, the 22%-tokens claim loses support. A second probe would match the pre-training corpus against benchmark development and test sets to check whether the filtered web data already contains these questions or near-duplicates.

Watch

Extended reading notes

Core claim

The paper claims that an end-side LLM can reach the capability level of a much more expensively trained open model by concentrating compute on the right data and the right attention pattern. Concretely, MiniCPM4-8B is pre-trained on 8.3T tokens against Qwen3-8B's 36T, and on the paper's Table 8 it out-scores the 36T-token model on the average of eight knowledge and reasoning benchmarks while reducing the attention footprint on 128K inputs to about 5% of dense. The same design yields a sevenfold decoding speedup on an edge GPU, a hybrid reasoning variant that stays within about a point of its full-attention version, and a ternary 1B model that beats several 2B full-precision baselines after only 10% of the quantization-aware training tokens used by a from-scratch ternary model. The authors offer this as evidence that data quality and architecture can substitute for training scale when the goal is useful model quality on constrained hardware.

Load-bearing premise

The headline token-efficiency claim assumes the benchmark results in Table 8 were collected under identical evaluation conditions for every model, with the same harness version, prompts, scoring, and decoding settings, and no test-set contamination, since some reported differences are under one point and no variance or contamination checks are reported.

Editorial extensions

If this is right

  • If the head-to-head numbers survive matched re-evaluation, an 8B model trained on roughly 8T tokens can stand in for a 36T-token model on standard English and Chinese benchmarks, so token count alone is not the binding constraint at this scale.
  • Sparse attention with only a few percent of context tokens attended preserves 100% needle-in-a-haystack accuracy at 128K, implying long-context document processing can move to edge GPUs without the quadratic memory blow-up.
  • The hybrid reasoning model's sparse variant stays within about a point of full attention on reasoning benchmarks, so the speed-oriented sparse path can be offered as the default deployment mode without a major capability penalty.
  • Chunk-wise rollout cuts reinforcement-learning sampling time per step while holding AIME scores roughly steady, which makes long chain-of-thought RL practical on smaller GPU clusters than vanilla rollout.
  • BitCPM4's result that a ternary model can be obtained by continuing from a high-precision checkpoint with 10% of from-scratch quantization-aware-training tokens suggests extremely low-bit models can be treated as a deployment step rather than a separate training project.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, if the token-efficiency claim generalizes past these benchmarks, the main cost barrier for mid-size open models shifts from raw FLOPs to data curation and evaluation discipline, so reproducibility of the filtering and verification pipeline matters more than training corpus size.
  • The measured speedups are reported at 32K-128K inputs on specific edge GPUs; a natural extension is to profile at short context, where sparse-attention block overhead and draft-model fixed costs may erase part of the advantage, and the paper does not report that regime.
  • The 22%-tokens comparison would be strengthened by an end-to-end 8B ablation trained on unfiltered web data at the same budget; the paper reports classifier-level gains on 100B-token 1B-model probes but not that direct ablation, so the attribution to the data pipeline is partly indirect.
  • A testable extension is to run the same two-stage annealing verification on other candidate corpora and check whether 1B-model loss improvements predict 8B benchmark gains, which would make the data-efficiency recipe a general tool rather than a one-off pipeline.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper presents MiniCPM4, an 8B and 0.5B LLM family aimed at on-device deployment, along with a ternary variant (BitCPM4) and a hybrid reasoning model (MiniCPM4.1). The contributions span sparse attention (InfLLM v2), data filtering and synthesis (UltraClean, UltraChat v2), training-strategy search (ModelTunnel v2), RL stabilization (chunk-wise rollout), and inference systems (CPM.cu, ArkInfer). The headline empirical claims are that MiniCPM4-8B reaches performance comparable to Qwen3-8B using only 8.3T tokens (22% of Qwen3's 36T) and that it processes 128K-token documents roughly 7x faster on edge GPUs. Model weights and inference code are released.

Significance. If the central claims hold, the result is practically significant: it would demonstrate that high-quality data curation and trainable sparse attention can close much of the gap to models trained on 4x more data, and it would provide a working recipe for 128K-context inference on consumer and edge GPUs. The paper also contributes useful engineering artifacts: released weights, a CUDA inference framework, and a cross-platform deployment layer, and the ScalingBench loss-to-performance relation has potential value for hyperparameter search. The significance is tempered, however, by the fact that the headline speed and token-efficiency numbers are not yet supported by controlled measurement protocols.

major comments (3)
  1. [§5.4, Figure 1; §7] The claimed 7x decoding speedup on 128K documents is load-bearing for the paper's 'ultra-efficient on end devices' thesis, but the evidence in Figure 1 does not support it. There is no measurement methodology: no framework and version, batch size, decoding length, precision/quantization, number of runs, or variance, and no dense-attention control for MiniCPM4. The baselines are presumably run in their own stacks while MiniCPM4 uses InfLLM v2, FR-Spec, and P-GPTQ within CPM.cu, so the speedup could be partly an implementation artifact. The inconsistency with Section 7, which states a 5x speedup for long-sequence processing, underscores that the number is not stable. Please provide a controlled comparison, including a dense-attention MiniCPM4 baseline and identical decoding settings, and reconcile the 7x and 5x figures.
  2. [§5.2, Table 8] The token-efficiency claim ('comparable performance with 22% of the training data') rests on a 0.58-point average difference (81.13 vs 80.55 for the 8B models) computed from what appears to be a single run on OpenCompass. The paper does not report the evaluation configuration (OpenCompass version, prompt templates, decoding parameters, few-shot settings), per-task variance, or contamination checks for either MiniCPM4 or the baselines. Under these conditions the parity is not distinguishable from evaluation noise, so the claim that 8.3T tokens suffice because of the proposed data pipeline is not established. Please report condition-matched, repeated evaluations with error bars and standard configurations.
  3. [§2.1.3, §5.3] Related to the speed claim, the complexity analysis in §2.1.3 states that stage 1 (block retrieval) remains O(l^2) per query token, while §5.3 reports that 128K inputs require only 6K attended tokens. The prefilling speedup therefore depends on constants and kernel efficiency, not on asymptotic complexity; this makes it essential to report kernel-level profiling and controlled comparisons (batch size, sequence length, GPU clocks) rather than end-to-end throughput alone. Please include such profiling or qualify the speedup accordingly.
minor comments (5)
  1. [Abstract; §5.1; §2.3.1] There are several typos: 'tenary' should be 'ternary' in the abstract, 'bulid' in §5.1, and 'emergenet' in §2.3.1. A careful proofreading pass is needed.
  2. [Figure 7] The x-axis tick labels in Figure 7 are unreadable because the numeric values are concatenated without separators; please provide legible axis ticks and a clear numeric scale.
  3. [Table 3] The 'GPU Hour' column entries ('Vanilla 1M', 'µP 32') are visually ambiguous; they should be presented as separate columns for model type and GPU-hour cost so that the reader does not misread which method uses 1M GPU hours.
  4. [§5.3; §2] The sparsity figures are inconsistent in presentation: §2 says '81% attention sparsity' while §5.3 says 'the sparsity of MiniCPM4 is only 5%' at 128K. Please define both numbers explicitly (e.g., percentage of selected KV blocks versus percentage of attended tokens) and reconcile the statements.
  5. [Table 2] The delta notation in Table 2 is confusing: values such as '+2.96' appear in the FineWeb-edu column, but the text compares UltraFineWeb against both FineWeb and FineWeb-edu. Use separate delta columns or a clear convention to avoid misinterpretation.

Circularity Check

1 steps flagged · score 3.0 of 10

Main benchmark results are externally measured, but the ScalingBench 'performance indicator' is constructed from validation data of the very downstream tasks it is used to predict, creating a mild self-referential loop in the hyperparameter-selection chain.

  1. fitted input called prediction [Section 2.3.1 (Efficient Predictable Scaling with Improved Performance Indicator), page 14-15]
    "we construct ScalingBench from the validation datasets of downstream tasks. In the original downstream datasets, each instance consists of a user instruction and a human-annotated label that usually contains a few words. In ScalingBench, we use GPT-4o (OpenAI, 2023) to generate reasoning steps for all test instances. Then we directly calculate the conditional loss on the reasoning steps and labels, specifically the loss incurred when the model generates answers given task inputs. The loss can serve as a reasonable performance indicator."

    ScalingBench is built from the validation datasets of the same downstream benchmarks (MMLU, CMMLU, CEval, BBH, MATH, HumanEval, MBPP) whose test performances are later reported in Table 8. The fitted sigmoid 'relationship between ScalingBench loss and downstream performance' is therefore a relationship between a loss on a subset of each benchmark and the score on that same benchmark suite, not an independent out-of-task law. ModelTunnel v2 then uses this ScalingBench loss as its performance indicator for selecting pre-training hyperparameters, meaning the reported benchmark results are partially informed by the benchmarks' own validation data. The held-out 7B/80B points in Figure 4 validate the curve at new model sizes, but not at new tasks, so the construction loop is not broken.

full rationale

The headline claims—comparable performance to Qwen3-8B with 22% of training tokens and large end-side speedups—rest on external benchmarks and direct measurements rather than on a derivation that rewrites its inputs as outputs. Table 8 is an external OpenCompass comparison, and the ScalingBench sigmoid is checked against held-out 7B/80B models, so the central results are not forced by construction. The only identifiable circular element is ScalingBench itself: it is explicitly constructed from the validation datasets of the downstream tasks whose performance it is used to predict, and ModelTunnel v2 adopts it as the performance indicator for hyperparameter search. This is a fitted-proxy loop rather than a mathematical identity, and the final scores are measured rather than read off the curve, so the circularity is partial. The speed claim is internally inconsistent (7x in Section 1, 5x in Section 7) and Section 5.4/Figure 1 omit a measurement protocol and dense-attention control, but that is a verification gap, not a circularity, so it is not counted as a circular step. Self-citations to InfLLM, ModelTunnel, UltraChat, FR-Spec, SpecMQuant, and UltraFineWeb are used as component building blocks with the paper providing its own descriptions and external evaluations, so they are not load-bearing in a circular way. Overall, the paper's main benchmark result has independent empirical content, with a modest self-referential dependency in the ScalingBench-based search chain.

Assumptions & free parameters 7 free parameters · 6 assumptions · 0 invented entities

The paper's central efficiency claims depend on several hand-chosen hyperparameters (kernel size, top-k budget, chunk length, QAT ratio, WSD split), on empirical assumptions about attention sinks and classifier seed quality, and on external methods like YaRN for context extension. There are no new physical entities, but the algorithmic components rest on assumptions that are not fully validated in the text.

free parameters (7)
  • InfLLM v2 semantic kernel size and stride = p=32, s=16
    Manual trade-off between relevance precision and computational overhead; directly affects the sparsity and speed claims in Sections 2.1.2-2.1.3.
  • InfLLM v2 top-k context budget = 6K tokens attended at 128K context (5% sparsity)
    Controls the attention sparsity that drives the long-context speedup claims (Section 5.3).
  • P-GPTQ initial-token cutoff s = s=4
    Chosen by empirical layer analysis to exclude initial-token outliers from Hessian computation (Section 4.1.2).
  • FR-Spec vocabulary subset fraction = 25% of vocabulary
    Empirically identified as capturing 95% of token occurrences while reducing draft LM-head cost (Section 4.1.1).
  • Chunk-wise rollout chunk length = 8K tokens
    Selected as the best efficiency/performance trade-off in Table 5; directly affects the RL speedup claim.
  • QAT continual-training token ratio = 40% of total tokens
    Threshold from Figure 5 beyond which final loss approaches from-scratch QAT; used to set the BitCPM4 QAT budget (Section 3.3.1).
  • WSD pre-training token split = 7T stable + 1.3T annealing
    Determines the '8T tokens' headline and the long-context extension budget (Section 5.1).
assumptions (6)
  • domain assumption Initial tokens and local window blocks contain information essential for attention, so their relevance scores are set to infinity.
    Section 2.1.2 asserts that initial tokens and local blocks 'usually contribute a lot' to outputs; this is adopted from prior attention-sink work without proof in this paper.
  • ad hoc to paper High-quality seed data that benefits LLM training also benefits training a classifier that identifies high-quality data.
    Section 2.2.1 states this as a core hypothesis and uses it to select classifier training seeds; no independent validation is provided.
  • domain assumption Mean-pooled semantic kernel representations stay in the same semantic space as token key vectors and can be optimized indirectly through token-level key vectors.
    Section 2.1.3 justifies the training scheme for sparse retrieval with this claim; no formal or empirical analysis is given.
  • domain assumption A sigmoid relationship between ScalingBench loss and downstream task performance holds across model sizes.
    Section 2.3.1 fits this curve to team models and validates on two larger models; it is an empirical regularity, not a theorem, and is used to transfer hyperparameters.
  • domain assumption YaRN extends the 32K-trained context to 128K without further training.
    Section 5.3 relies on YaRN for long-context evaluation; correctness rests on an external method rather than evidence in this paper.
  • domain assumption FP8 mixed-precision training with block-wise quantization preserves model quality.
    Section 2.3.2 follows DeepSeek-V3's approach; the paper does not re-validate the effect of FP8 on MiniCPM4's final quality.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MiniCPM4: Ultra-Efficient LLMs on End Devices." pith.science (2026). https://pith.science/paper/3ZDXG3WW

@misc{pith2026250607900,
  author       = {Pith},
  title        = {Pith review of: MiniCPM4: Ultra-Efficient LLMs on End Devices},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3ZDXG3WW}},
  note         = {Machine review of arXiv:2506.07900}
}
read the original abstract

This paper introduces MiniCPM4, a highly efficient large language model (LLM) designed explicitly for end-side devices. We achieve this efficiency through systematic innovation in four key dimensions: model architecture, training data, training algorithms, and inference systems. Specifically, in terms of model architecture, we propose InfLLM v2, a trainable sparse attention mechanism that accelerates both prefilling and decoding phases for long-context processing. Regarding training data, we propose UltraClean, an efficient and accurate pre-training data filtering and generation strategy, and UltraChat v2, a comprehensive supervised fine-tuning dataset. These datasets enable satisfactory model performance to be achieved using just 8 trillion training tokens. Regarding training algorithms, we propose ModelTunnel v2 for efficient pre-training strategy search, and improve existing post-training methods by introducing chunk-wise rollout for load-balanced reinforcement learning and data-efficient tenary LLM, BitCPM. Regarding inference systems, we propose CPM.cu that integrates sparse attention, model quantization, and speculative sampling to achieve efficient prefilling and decoding. To meet diverse on-device requirements, MiniCPM4 is available in two versions, with 0.5B and 8B parameters, respectively. Furthermore, we construct a hybrid reasoning model, MiniCPM4.1, which can be used in both deep reasoning mode and non-reasoning mode. Evaluation results demonstrate that MiniCPM4 and MiniCPM4.1 outperform similar-sized open-source models across benchmarks, with the 8B variants showing significant speed improvements on long sequence understanding and generation.

Figures

Figures reproduced from arXiv: 2506.07900 by the authors.

Figure 1
Figure 1. Inference Speed Evaluation on end-side GPUs. – 1 – arXiv:2506.07900v2 [cs.CL] 4 Sep 2025 [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The illustration of InfLLM v2. Each query group selects parts of key-value blocks for attention computation, where the initial tokens and local tokens in the sliding window are always selected. 2.1 InfLLM v2: Trainable Sparse Attention for Prefilling and Decoding Following most open-source LLMs, we adopt Transformer (Vaswani et al., 2017) as our basic architecture. In consideration of the emerging needs to process l… view at source ↗
Figure 3
Figure 3. The illustration of high-quality data filtering pipelines. Traditional model-based data filtering methods (a) and (b) rely on human expertise for seed data selection and lack data quality verification. conditions. Regarding the choice of semantic kernel size, smaller kernel sizes enable more precise relevance score computation; however, smaller kernel sizes also incur greater computational overhead. Thus, to achieve… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: The sigmoid relationship between loss and downstream performance on ScalingBench. To verify the effectiveness of ScalingBench and establish the relationship between the loss and downstream performance, we evaluate the loss and performance of multiple models on ScalingB…
Figure 5
Figure 5. Figure 5: The relationship between language modeling loss and ratio of QAT post-training tokens (proportion of full stable-phase tokens). necessitating more effective QAT methods (Liu et al., 2024b). Some recent efforts, such as BitNet (Ma et al., 2025), even train extremely low…
Figure 6
Figure 6. Figure 6: The illustration of FR-Spec, which requires the draft model to use a reduced vocabulary subset. 4.1.1 Frequency-Ranked Vocabulary Construction and Draft Verification The effectiveness of speculative sampling relies heavily on the efficiency of both the drafting and ver…
Figure 7
Figure 7. Figure 7: The evaluation results for long sequence prefilling with sparse attention [PITH_FULL_IMAGE:figures/full_fig_p031_7.png]
Figure 8
Figure 8. Figure 8: The outline of MiniCPM4-Survey. parameter size, we can achieve consistent speedup in both prefilling and decoding scenarios. Specifically, compared to Qwen3-8B, we achieve approximately 7x decoding acceleration on Jetson AGX Orin. The results demonstrate the effectiven…

Discussion (0). Sign in to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CodePercept: Code-Grounded Visual STEM Perception for MLLMs

    cs.CV 2026-03 conditional novelty 6.5 of 10

    Perception, not reasoning, is the main bottleneck for MLLM STEM visual reasoning, and training on executable reconstruction code measurably fixes it.

  2. AgenticASR: Refining Speech Recognition in Real-World Scenarios via an Agentic Approach

    cs.AI 2026-07 conditional novelty 6.0 of 10

    An ASR–Refiner system emits and revises clean transcripts online over a bounded sliding context, outperforming offline spoken-to-written baselines on a new bilingual rubric benchmark.

  3. $\Sigma$-Mem: An Online Reliability Memory for LLM-based Multi-Agent Systems

    cs.MA 2026-07 conditional novelty 6.0 of 10

    Online symmetric reliability memory for LLM multi-agent systems accumulates bounded competence and peer-relationship evidence and supports steering, routing, and weighted voting without retraining.

  4. CoSA: Accelerating Long-Context Inference via Proxy-Kernel Co-Designed Sparse Attention

    cs.CL 2026-07 conditional novelty 6.0 of 10

    CoSA couples an order-aware sparse-mask proxy with a skip-capable attention kernel, achieving higher long-context accuracy at lower compute budgets than existing sparse-attention baselines.

  5. Athena-Brain Technical Report: An Efficient Robot Brain for General Intelligence and Embodied Interaction

    cs.AI 2026-07 conditional novelty 6.0 of 10

    An 8B LLM post-trained with SFT, RL, embodied-expert training, and model merging reaches high in-domain embodied-task success with very short responses.

  6. FreyaTTS: A Compact Tokenizer-Free Flow-Matching Transformer for Turkish-First Speech Synthesis

    cs.CL 2026-07 conditional novelty 6.0 of 10

    A compact tokenizer-free non-autoregressive flow-matching DiT synthesizes Turkish speech in frozen AudioVAE2 latents at WER 8.0% / CER 3.0%, beating larger open cloners while running at RTF 0.11 on consumer GPUs.

  7. TSHA: A Benchmark for Visual Language Models in Trustworthy Safety Hazard Assessment Scenarios

    cs.CV 2026-03 unverdicted novelty 6.0 of 10

    TSHA is a new 80,000-pair benchmark for indoor safety hazard assessment; current vision-language models score roughly 45-85, and fine-tuning on TSHA raised Qwen2.5-VL-3B by 18.3 points on TSHA's test set.

  8. Generative Semantic Multi-Object Tracking: A Large-Scale Benchmark and an MLLM-Driven Reasoning Framework

    cs.CV 2026-01 reject novelty 6.0 of 10

    A new LLM-generated dataset and an MLLM-based tracker claim state-of-the-art semantic multi-object tracking, but the evaluation protocol masks missed objects and ID switches.

  9. Psyche-R1: Towards Reliable Psychological LLMs through Unified Empathy, Expertise, and Reasoning

    cs.CL 2025-08 conditional novelty 6.0 of 10

    A 7B Chinese psychological LLM trained with curated reasoning questions and empathetic dialogues via SFT plus GRPO approaches DeepSeek-R1's exam accuracy at a fraction of the size.

  10. Stem: Rethinking Causal Information Flow in Sparse Attention

    cs.LG 2026-03 conditional novelty 5.0 of 10

    Stem prunes causal attention with a position-decaying budget and a value-magnitude-aware score, matching dense accuracy on LongBench/RULER at a 25-31% compute budget.

  11. iFairy: the First 2-bit Complex LLM with All Parameters in $\{\pm1, \pm i\}$

    cs.LG 2025-08 conditional novelty 5.0 of 10

    A complex-valued LLM quantized to {±1, ±i} claims to beat prior low-bit LLMs, but the gains may come from a parameter-rich architecture.

Reference graph

Works this paper leans on

44 extracted references · 1 canonical work pages · cited by 11 Pith papers

  1. [1]

    Phi-4 technical report

    Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, et al. Phi-4 technical report. arXiv preprint arXiv:2412.08905,

    arXiv

  2. [4]

    doi: 10.18653/v1/2024.findings-emnlp.74

    Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-emnlp.74. Johan Bjorck, Alon Benhaim, Vishrav Chaudhary, Furu Wei, and Xia Song. Scaling optimal lr across token horizons. arXiv preprint arXiv:2409.19913,

    arXiv 2024

  3. [5]

    Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S

    Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri S. Chatterji, Annie S. Chen, Kathleen Creel, Jared Quincy Davis, Dorottya Demszky, Chris Donahue, Moussa Doumbouya, Esin Du...

    arXiv

  4. [6]

    Accelerating large language model decoding with speculative sampling

    Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318,

    arXiv

  5. [8]

    Prefixquant: Static quantization beats dynamic through prefixed outliers in llms

    Mengzhao Chen, Yi Liu, Jiahao Wang, Yi Bin, Wenqi Shao, and Ping Luo. Prefixquant: Static quantization beats dynamic through prefixed outliers in llms. arXiv preprint arXiv:2410.05265,

    arXiv

  6. [9]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168,

    arXiv

  7. [11]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Team DeepSeek, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948,

    arXiv

  8. [12]

    Longrope: Extending LLM context window beyond 2 million tokens

    Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. Longrope: Extending LLM context window beyond 2 million tokens. CoRR, abs/2402.13753,

    arXiv

Show all 44 references
  1. [13]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al

    doi: 10.48550/ARXIV .2402.13753. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783,

    doi arXiv

  2. [14]

    Scaling synthetic data creation with 1,000,000,000 personas

    Tao Ge, Xin Chan, Xiaoyang Wang, Dian Yu, Haitao Mi, and Dong Yu. Scaling synthetic data creation with 1,000,000,000 personas. arXiv preprint arXiv:2406.20094,

    arXiv

  3. [15]

    Chatglm: A family of large language models from glm-130b to glm-4 all tools

    Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng, Hanlin Zhao, et al. Chatglm: A family of large language models from glm-130b to glm-4 all tools. arXiv preprint arXiv:2406.12793,

    arXiv

  4. [16]

    Apple intelligence foundation language models

    Tom Gunter, Zirui Wang, Chong Wang, Ruoming Pang, Andy Narayanan, Aonan Zhang, Bowen Zhang, Chen Chen, Chung-Cheng Chiu, David Qiu, et al. Apple intelligence foundation language models. CoRR, abs/2407.21075,

  5. [17]

    doi: 10.48550/ARXIV .2407.21075. Xu Han, Zhengyan Zhang, Ning Ding, Yuxian Gu, Xiao Liu, Yuqi Huo, Jiezhong Qiu, Yuan Yao, Ao Zhang, Liang Zhang, Wentao Han, Minlie Huang, Qin Jin, Yanyan Lan, Yang Liu, Zhiyuan Liu, Zhiwu Lu, Xipeng Qiu, Ruihua Song, Jie Tang, Ji-Rong Wen, Jin...

    doi

  6. [18]

    Rae, Oriol Vinyals, and Laurent Sifre

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osin...

    arXiv

  7. [19]

    – 40 – MiniCPM4 Xinyi Hou, Yanjie Zhao, Shenao Wang, and Haoyu Wang

    doi: 10.48550/ARXIV .2203.15556. – 40 – MiniCPM4 Xinyi Hou, Yanjie Zhao, Shenao Wang, and Haoyu Wang. Model Context Protocol (MCP): Landscape, Security Threats, and Future Research Directions, April

    doi arXiv

  8. [20]

    Minicpm: Unveiling the potential of small language models with scalable training strategies

    Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, et al. Minicpm: Unveiling the potential of small language models with scalable training strategies. CoRR, abs/2404.06395,

    arXiv

  9. [21]

    Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Yao Fu, et al

    doi: 10.48550/ARXIV .2404.06395. Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Yao Fu, et al. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. Proceedings of NeurIPS...

    doi arXiv

  10. [22]

    Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. CoRR, abs/2001.08361,

    2001 arXiv

  11. [23]

    Cmmlu: Measuring massive multitask language understanding in chinese

    Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. Cmmlu: Measuring massive multitask language understanding in chinese. In Findings of ACL: ACL 2024, pp. 11260–11285, 2024a. Houyi Li, Wenzheng Zheng, Jingcheng Hu, Qiufeng Wan...

    2024 arXiv

  12. [24]

    Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model

    – 41 – MiniCPM4 Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434, 2024a. Zechun Liu, ...

    2024 arXiv

  13. [26]

    American invitational mathematics examination-aime

    MAA. American invitational mathematics examination-aime. URL https://maa.org/ maa-invitational-competitions/. Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. Factscore: Fine-grained atomic ev...

    arXiv

  14. [27]

    GPT-4 technical report

    OpenAI. GPT-4 technical report. CoRR, abs/2303.08774,

    arXiv

  15. [28]

    Gpt-4o mini: advancing cost-efficient intelligence

    OpenAI. Gpt-4o mini: advancing cost-efficient intelligence. Technical Report, 2024a. URL https://openai. com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/ . OpenAI. Openai o1 system card. CoRR, abs/2412.16720, 2024b. OpenAI. Introducing deep research. https://openai...

    arXiv

  16. [29]

    Yarn: Efficient context window extension of large language models

    Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models. CoRR, abs/2309.00071,

    arXiv

  17. [30]

    Pre-trained models for natural language processing: A survey

    Xipeng Qiu, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai, and Xuanjing Huang. Pre-trained models for natural language processing: A survey. CoRR, abs/2003.08271,

    2003 arXiv

  18. [32]

    Gemma 3 technical report

    Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. arXiv preprint arXiv:2503.19786,

    arXiv

  19. [34]

    Liangdong Wang, Bo-Wen Zhang, Chengwei Wu, Hanyu Zhao, Xiaofeng Shi, Shuhao Gu, Jijie Li, Quanyue Ma, TengFei Pan, and Guang Liu. Cci3. 0-hq: a large-scale chinese dataset of high quality designed for pre-training large language models. arXiv preprint arXiv:2410.18505, 2024a. ...

    arXiv

  20. [35]

    Mimo: Unlocking the reasoning potential of language model–from pretraining to posttraining

    Bingquan Xia, Bowen Shen, Dawei Zhu, Di Zhang, Gang Wang, Hailin Zhang, Huaqiu Liu, Jiebao Xiao, Jinhao Dong, Liang Zhao, et al. Mimo: Unlocking the reasoning potential of language model–from pretraining to posttraining. arXiv preprint arXiv:2505.07608,

    arXiv

  21. [36]

    Densing law of llms

    Chaojun Xiao, Jie Cai, Weilin Zhao, Guoyang Zeng, Biyuan Lin, Jie Zhou, Zhi Zheng, Xu Han, Zhiyuan Liu, and Maosong Sun. Densing law of llms. arXiv preprint arXiv:2412.04315, 2024a. – 43 – MiniCPM4 Chaojun Xiao, Pengle Zhang, Xu Han, Guangxuan Xiao, Yankai Lin, Zhengyan Zhang,...

    arXiv

  22. [37]

    Xattention: Block sparse attention with antidiagonal scoring

    Ruyi Xu, Guangxuan Xiao, Haofeng Huang, Junxian Guo, and Song Han. Xattention: Block sparse attention with antidiagonal scoring. arXiv preprint arXiv:2503.16428,

    arXiv

  23. [38]

    Onebit: Towards extremely low-bit large language models

    Yuzhuang Xu, Xu Han, Zonghan Yang, Shuo Wang, Qingfu Zhu, Zhiyuan Liu, Weidong Liu, and Wanxiang Che. Onebit: Towards extremely low-bit large language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. An Yang, Anfeng Li, Baosong Yang, Bei...

    arXiv

  24. [39]

    Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer

    Greg Yang, Edward J Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer. arXiv preprint arXiv:2203.03466,

    arXiv

  25. [40]

    Minicpm-v: A gpt-4v level mllm on your phone

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800,

    arXiv

  26. [41]

    Dapo: An open-source llm reinforcement learning system at scale

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476, 2025a. Yijiong Yu, Ziyun Dai, Zekun Wang, Wei Wang, ...

    arXiv

  27. [42]

    Spargeattn: Accurate sparse attention accelerating any model inference

    Jintao Zhang, Chendong Xiang, Haofeng Huang, Jia Wei, Haocheng Xi, Jun Zhu, and Jianfei Chen. Spargeattn: Accurate sparse attention accelerating any model inference. arXiv preprint arXiv:2502.18137, 2025a. Yudi Zhang, Weilin Zhao, Xu Han, Tiejun Zhao, Wang Xu, Hailong Cao, and...

  28. [43]

    Fr-spec: Accelerating large-vocabulary language models via frequency-ranked speculative sampling

    Weilin Zhao, Tengyu Pan, Xu Han, Yudi Zhang, Ao Sun, Yuxiang Huang, Kaihuo Zhang, Weilun Zhao, Yuxuan Li, Jianyong Wang, et al. Fr-spec: Accelerating large-vocabulary language models via frequency-ranked speculative sampling. arXiv preprint arXiv:2502.14856,

    arXiv

  29. [44]

    Opencodeinterpreter: Integrating code generation with execution and refinement

    Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and Xiang Yue. Opencodeinterpreter: Integrating code generation with execution and refinement. arXiv preprint arXiv:2402.14658,

    arXiv

  30. [2017]

    Llm ×mapreduce-v2: Entropy-driven convolutional test-time scaling for generating long-form articles from extremely long resources, 2025a

    Haoyu Wang, Yujia Fu, Zhu Zhang, Shuo Wang, Zirui Ren, Xiaorong Wang, Zhili Li, Chaoqun He, Bo An, Zhiyuan Liu, and Maosong Sun. Llm ×mapreduce-v2: Entropy-driven convolutional test-time scaling for generating long-form articles from extremely long resources, 2025a. Hongyu Wan...

    arXiv

  31. [2020]

    Assisting in writing wikipedia-like articles from scratch with large language models

    – 42 – MiniCPM4 Yijia Shao, Yucheng Jiang, Theodore A Kanell, Peter Xu, Omar Khattab, and Monica S Lam. Assisting in writing wikipedia-like articles from scratch with large language models. arXiv preprint arXiv:2402.14207, 2024a. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu,...

    2023 arXiv

  32. [2021]

    LongAlign: A recipe for long context alignment of large language models

    Yushi Bai, Xin Lv, Jiajie Zhang, Yuze He, Ji Qi, Lei Hou, Jie Tang, Yuxiao Dong, and Juanzi Li. LongAlign: A recipe for long context alignment of large language models. In Findings of ACL: EMNLP 2024 , pp. 1376–1395, Miami, Florida, USA, November

    2024

  33. [2022]

    Deepseek-v3 technical report

    Team DeepSeek, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437,

    arXiv

  34. [2023]

    Evaluating large language models trained on code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374,

    arXiv

  35. [2024]

    Program synthesis with large language models

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732,

    arXiv

  36. [2025]

    The era of 1-bit llms: All large language models are in 1.58 bits

    Shuming Ma, Hongyu Wang, Lingxiao Ma, Lei Wang, Wenhui Wang, Shaohan Huang, Li Dong, Ruiping Wang, Jilong Xue, and Furu Wei. The era of 1-bit llms: All large language models are in 1.58 bits. arXiv preprint arXiv:2402.17764,

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.