The reviewed record of science sign in
Pith

arxiv: 2311.00502 · v2 · pith:MWRLQGOO · submitted 2023-11-01 · cs.LG · cs.AI· cs.CL

Efficient LLM Inference on CPUs

Haihao Shen , Hanwen Chang , Bo Dong , Yu Luo , Hengyu Meng This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel pith:MWRLQGOOrecord.jsonopen to challenge →

classification cs.LG cs.AIcs.CL
keywords cpusinferencellmsapproachlargememorymodelsaccelerate
0
0 comments X
read the original abstract

Large language models (LLMs) have demonstrated remarkable performance and tremendous potential across a wide range of tasks. However, deploying these models has been challenging due to the astronomical amount of model parameters, which requires a demand for large memory capacity and high memory bandwidth. In this paper, we propose an effective approach that can make the deployment of LLMs more efficiently. We support an automatic INT4 weight-only quantization flow and design a special LLM runtime with highly-optimized kernels to accelerate the LLM inference on CPUs. We demonstrate the general applicability of our approach on popular LLMs including Llama2, Llama, GPT-NeoX, and showcase the extreme inference efficiency on CPUs. The code is publicly available at: https://github.com/intel/intel-extension-for-transformers.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Sandwich: Joint Configuration Search and Hot-Switching for Efficient CPU LLM Serving

    cs.AR 2025-05 unverdicted novelty 6.0

    Sandwich delivers 2.01x average end-to-end speedup and up to 3.4x latency reduction for CPU LLM serving via phase-wise hot-switching, TopoTree hardware abstraction, and fast-start dynamic kernel generation.

  2. ODMA: On-Demand Memory Allocation Strategy for LLM Serving on LPDDR-Class Accelerators

    cs.AR 2025-12 unverdicted novelty 5.0

    ODMA raises KV-cache utilization by up to 19.25% and throughput by 23-27% on Cambricon MLU accelerators by dynamically adjusting prediction buckets and using a safety pool for LLM serving.

  3. Active Imitation Learning for Thermal- and Kernel-Aware LFM Inference on 3D S-NUCA Many-Cores

    cs.LG 2026-04 unverdicted novelty 4.0

    AILFM uses active imitation learning to learn thermal- and kernel-aware scheduling policies for LFM inference on 3D S-NUCA many-cores, outperforming baselines while maintaining thermal safety.

  4. Small Language Models (SLMs) Can Still Pack a Punch: A survey (updated 2026)

    cs.CL 2025-01 unverdicted novelty 2.0

    A literature survey of Small Language Models (1-8B parameters) that can perform comparably or better than larger models, covering general-purpose and task-specific approaches plus creation techniques.