Pith. sign in

REVIEW 5 cited by

Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2402.07033 v3 pith:4TQBR6TB submitted 2024-02-10 cs.LG cs.AIcs.DCcs.OS

Keisuke Kamahori , Tian Tang , Yile Gu , Kan Zhu , Baris Kasikci This is my paper · ORCID
classification cs.LGcs.AIcs.DCcs.OS
keywords fiddlerinferencemodelsresourcestimesbatchlongmixture-of-experts
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Large Language Models (LLMs) with the Mixture-of-Experts (MoE) architectures have shown promising performance on various tasks. However, due to the huge model sizes, running them in resource-constrained environments where the GPU memory is not abundant is challenging. Some existing systems propose to use CPU resources to solve that, but they either suffer from the significant overhead of frequently moving data between CPU and GPU, or fail to consider distinct characteristics of CPUs and GPUs. This paper proposes Fiddler, a resource-efficient inference system for MoE models with limited GPU resources. Fiddler strategically utilizes CPU and GPU resources by determining the optimal execution strategy. Our evaluation shows that, unlike state-of-the-art systems that optimize for specific scenarios such as single batch inference or long prefill, Fiddler performs better in all scenarios. Compared against different baselines, Fiddler achieves 1.26 times speed up in single batch inference, 1.30 times in long prefill processing, and 11.57 times in beam search inference. The code of Fiddler is publicly available at https://github.com/efeslab/fiddler.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Does Mixture-of-Experts Actually Help Inference on Consumer and Edge Hardware? An Empirical Study

    cs.PF 2026-06 accept novelty 6.5 of 10

    On bandwidth-bound edge hardware, MoE inference cost tracks total parameters rather than active ones, so sparse activation fails to deliver the expected throughput or energy gains.

  2. Automated Tensor Scheduling for Hybrid CPU-GPU LLM Inference on Consumer Devices

    cs.DC 2026-07 conditional novelty 6.0 of 10

    Tensor-granularity hybrid CPU-GPU offloading with static placement plus load-aware dynamic transfer yields up to 1.94× prefill and 3.29× decode throughput for local LLMs on consumer devices.

  3. Context-Aware CodeLLM Eviction for AI-assisted Coding

    cs.SE 2025-06 reject novelty 5.0 of 10

    CACE, a context-aware eviction policy, cuts code-model reloads and response latency in self-hosted AI-assistant serving compared with LRU, though its future-demand factor reads the actual test workload.

  4. Klotski: Efficient Mixture-of-Expert Inference via Expert-Aware Multi-Batch Pipeline

    cs.LG 2025-02 conditional novelty 5.0 of 10

    Klotski pipelines multiple batches of MoE inference, prefetches only likely-used 'hot' experts, and reorders expert computations to overlap I/O with compute, claiming up to 85.12x throughput gains.

  5. Exploring the Dynamic Scheduling Space of Real-Time Generative AI Applications on Emerging Heterogeneous Systems

    cs.LG 2025-07 conditional novelty 4.0 of 10

    A simulation-based study on one AMD Ryzen AI system finds that scheduling policies strongly affect deadline violation rates, time-to-first-token, and tokens-per-second for real-time generative AI workloads, with an LL...

Pith tools