BaseRT: Advancing Best-in-Class LLM Inference with Apple M5 Neural Accelerators

Fabian Waschkowski; Lukas Wesemann; Prabod Rathnayaka

REVIEW 3 major objections 5 minor 20 references

BaseRT claims that routing prompt-processing matrix multiplications through Apple M5's per-core Neural Accelerators via hand-written Metal 4 tensor kernels yields up to 6.4× higher prefill throughput than llama.cpp and 3.9× higher than MLX

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 14:22 UTC pith:VAON6VSD

load-bearing objection BaseRT's M5 prefill numbers are real and internally consistent, but the attribution to tensor cores is under-supported by the missing ablation and default-only baselines. the 3 major comments →

arxiv 2607.19438 v1 pith:VAON6VSD submitted 2026-07-21 cs.AR cs.AIcs.CLcs.DCcs.LGcs.PF

BaseRT: Advancing Best-in-Class LLM Inference with Apple M5 Neural Accelerators

Fabian Waschkowski , Prabod Rathnayaka , Lukas Wesemann This is my paper

classification cs.AR cs.AIcs.CLcs.DCcs.LGcs.PF

keywords LLM inferenceApple M5Neural AcceleratorsMetal 4 tensor APIcooperative-tensor kernelsprefill throughputmixture-of-expertson-device inference

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that Apple's M5 GPU, which places a dedicated matrix engine in every core, can be turned into a major speedup for on-device LLM inference if a runtime writes kernels directly against the new Metal 4 tensor API. The authors extend their BaseRT runtime with hand-written cooperative-tensor kernels for dense and mixture-of-experts matrix multiplications and for flash-attention prefill, applying them only to compute-bound phases. On an M5 Pro they report up to 6.4× faster prompt processing than llama.cpp and 3.9× faster than MLX, with decode gains of 1.75× and 1.33×. The claim matters because prefill dominates time-to-first-token, so these gains would directly lower interactive latency for long-prompt and agentic on-device workloads.

Core claim

BaseRT's central discovery is that the M5's per-core Neural Accelerators are the decisive lever for prompt-processing throughput on Apple Silicon, provided the runtime routes the right operations to them. The authors hand-write Metal 4 tensor-core kernels for dense GEMM, MoE expert GEMM, fused gate/up projections, and the QK^T and PV products of prefill attention, using the MPP matmul2d primitive to accumulate into register-resident cooperative tensors. Decode stays on the existing single-vector kernels because it is memory-bandwidth-bound. On an M5 Pro across fifteen model configurations, this split yields up to 6.4× prefill throughput over llama.cpp and 3.9× over MLX, with the largest marg

What carries the argument

The central object is the Metal 4 cooperative-tensor kernel family, built on the MPP matmul2d primitive that issues matrix products at simdgroup or threadgroup scope to the M5's dedicated Neural Accelerators. A kernel declares tensor views over its operands, accumulates tile products into a register-resident cooperative tensor, and can stream operands directly from device memory. The workload-aware dispatch selects this path for compute-bound prefill GEMM, MoE expert GEMM, and prefill attention, and falls back to SIMD kernels for memory-bound decode and on pre-M5 hardware.

Load-bearing premise

The reported speedups assume that default-settings llama.cpp and mlx-lm represent those baselines at their best on the M5 Pro; if those defaults are underconfigured, the margins would shrink.

What would settle it

Run the same fifteen model configurations on an M5 Pro with llama.cpp and MLX each given their recommended or best-known settings, and with BaseRT's tensor path disabled in an ablation; if the 6.4× prefill margin disappears or the tensor path contributes little, the central claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

If the reported margins hold, BaseRT sets a new prompt-processing throughput ceiling on Apple M5 hardware, with time-to-first-token gains that grow with prompt length.
Mixture-of-experts models, where matrix multiplication dominates per active parameter, benefit disproportionately; the four MoE models tested show the largest margins.
Decode throughput gains remain modest and bounded by memory bandwidth, so the tensor cores do not remove the memory ceiling for token generation.
Because the tensor path only activates on M5 hardware, pre-M5 users see no change, and the design cleanly preserves prior SIMD performance.
The gains are concentrated in prefill, so scheduling decisions that balance prefill and decode can leverage this asymmetry.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

A natural test the paper leaves implicit: measuring the same workload with baselines tuned beyond default settings would clarify how much of the margin comes from the tensor kernels versus baseline underconfiguration.
Speculative decoding, which converts part of decode into a batched, compute-bound verification step, is an obvious next target for the tensor path and could indirectly raise decode throughput.
The M5 base and Max tiers differ in core count and bandwidth; if the pattern generalizes, the prefill advantages should scale with the number of neural accelerators, while the decode ceiling scales with bandwidth.
Other Metal-based runtimes could adopt the same Metal 4 tensor path, but the paper suggests that doing so requires rewriting kernels at the Metal language level rather than relying on a higher-level framework.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

BaseRT's M5 prefill numbers are real and internally consistent, but the attribution to tensor cores is under-supported by the missing ablation and default-only baselines.

read the letter

Short take: this is a legitimate engineering result — a native Metal runtime with hand-written Metal 4 tensor-core kernels that shows large, internally consistent prefill gains on Apple's new M5 hardware. The claim that the tensor cores are "the decisive lever" is not actually proven by the data, but the headline numbers are worth paying attention to.

What's new: a family of cooperative-tensor kernels — dense and MoE GEMM, fused gate/up, and flash-attention prefill — plus a dispatch policy that keeps decode on SIMD kernels. That's a sensible design and the results are plausible: decode gains cluster near 1.0–1.75×, consistent with memory bandwidth; prefill gains grow with context length and are largest on MoE models, which is exactly where a matmul engine should help. Tables 2 and 3 are internally consistent with the text ratios. The limitations section (5.1) is honest: it acknowledges single-device, no batching, M5-only, and that pre-M5 falls back to SIMD.

Soft spots: the biggest one is attribution. There's no ablation that disables the tensor path on the same M5 and the same BaseRT kernel set, so we can't tell how much of the 2×–6× prefill margin comes from the M5 Neural Accelerators versus BaseRT's existing SIMD kernels, dispatch overhead, or data layout. The baselines are run with default settings. llama.cpp b9960 with its Metal backend may not be tuned for the M5's new tensor API — or even use it. If llama.cpp is on an older SIMD path, the comparison is partially "custom tensor-core runtime vs. general-purpose SIMD runtime" rather than "BaseRT vs. best possible baseline." No error bars, only five repetitions averaged. These aren't fatal flaws for an engineering report, but they should be acknowledged and ideally fixed before the "decisive lever" claim is made.

There's also a small observational point: the paper's own numbers show MLX winning at long-context prefill on two small Llama models, and decode ties on two large models. That's honest data, and it keeps the claims from being overreaching, though the abstract's "best-in-class" phrase is doing more work than the measurements support.

Who it's for: practitioners working on on-device LLM inference on Apple Silicon, and anyone comparing runtimes on M5. It deserves a serious referee; the engineering is real, the measurements are extensive, and the missing pieces (ablation, tuned baselines, raw logs) are all fixable in revision. I'd want to see those before accepting far-reaching conclusions, but I'd engage with it and would cite it if I were working on this stack.

Referee Report

3 major / 5 minor

Summary. This paper presents BaseRT, a native Metal LLM-inference runtime, extended with Metal 4 cooperative-tensor kernels for the M5's per-core Neural Accelerators. The system uses tensor-path kernels for prefill GEMM, MoE expert GEMM, fused gate/up projections, and prefill attention, while retaining SIMD kernels for decode. On an M5 Pro the authors report prefill throughput up to 6.4× higher than llama.cpp and 3.9× higher than mlx-lm across 15 model configurations, with smaller decode gains (up to 1.75× vs. llama.cpp, 1.33× vs. mlx-lm). They attribute the prefill gains to M5 tensor cores and note decode remains memory-bandwidth-bound.

Significance. If substantiated, the paper establishes a new practical performance point for on-device inference and provides a concrete recipe for exploiting a new hardware path. The measurement methodology is internally consistent: the ratios stated in the text match Tables 2–3, quantisation is matched, and the baseline engines are identified by version. The authors are explicit about limitations (single device, no batching/parallelism, Metal-only). However, the external-validity claim rests on three pieces of evidence that are currently missing: an optimized or at least well-understood baseline configuration, an ablation separating the tensor kernels from BaseRT's existing SIMD path, and variance/reproducibility data. Without these, the reported margins and the causal attribution to tensor cores are not fully supported.

major comments (3)

[§4.1, Tables 2–3] The comparison uses llama.cpp b9960 and mlx-lm 0.31.3 with 'default settings' only, while BaseRT is a custom-tuned runtime. Large prefill margins may partly reflect an unconfigured baseline. Please report exactly which llama.cpp flags were enabled/disabled (thread count, Metal flash-attention, mmap, batch size, etc.) and whether mlx-lm's default scheduler uses the same Metal backend. Provide a sensitivity run with recommended/optimized baseline flags, or compare BaseRT's tensor path against BaseRT's own SIMD path on the same device.
[Abstract, §3.2, §4.4] The paper's headline causal claim—that M5 tensor cores are the decisive lever—is inferred from whole-system comparisons. There is no ablation disabling the tensor path (e.g., forcing SIMD kernels for prefill GEMM/attention) on the same M5 Pro, and no pre-M5 run is reported. Since BaseRT's dispatch automatically selects kernels, the reader cannot separate the new tensor kernels from existing fusion and dispatch optimizations. Add a same-device tensor-path-off ablation and, if possible, a pre-M5 comparison; otherwise soften the causal claim.
[§4.1 and all tables] Only the average of five repetitions is reported. For differences of 0.98×–1.05× and claims such as 'all but three of seventy-five points', run-to-run variance, thermal state, and GPU clock throttling need to be quantified. Provide standard deviation or min/max, raw per-run logs, and exact benchmark scripts, prompt data, and model checkpoint identifiers. This is necessary for reproducibility of a claimed state-of-the-art result.

minor comments (5)

[Fig. 1] The x-axis label reads '20480' while the text and tables use 2048 tokens. Please correct the typo and ensure the axis matches the described prompt lengths.
[Table 2] The header 'll.cpp' is ambiguous; use 'llama.cpp' consistently and define all abbreviations in the caption.
[§4.1, Metrics] State whether the five repetitions include warmup, whether tokenization time is excluded, and how prefill/decode throughput are computed (prompt tokens vs. generated tokens).
[§4.1 and references] Provide a commit/SHA for BaseRT v0.1.6 and for the exact baseline binaries so the evaluated code is identifiable beyond version numbers.
[§5.1] The limitations are honestly listed, but the single-device and single-dataset caveat could be moved to a more prominent place in the evaluation, and a 'reproducibility' subsection would help readers locate the missing artifacts.

Circularity Check

0 steps flagged

No significant circularity: the paper's claims are measured benchmarks against independent external runtimes; the sole self-citation is not load-bearing.

full rationale

This is an empirical benchmarking paper rather than a derived mathematical result. The central claims—prefill and decode throughput margins up to 6.4× over llama.cpp and 3.9× over MLX—are comparisons of directly measured tokens/s on the same Apple M5 Pro device, using external baseline runtimes (llama.cpp build b9960 and mlx-lm 0.31.3) at matched quantisation. There is no fitted parameter that is later renamed as a prediction, and no quantity is defined in terms of the outcome it is said to predict. The only self-citation is to prior BaseRT work [13], used to describe the runtime's existing design; it does not carry the performance argument, which rests on the measured tables. The paper's interpretation that M5 tensor cores are 'the decisive lever' is not backed by an ablation that disables the tensor path, but that is a correctness/evidence limitation, not circularity. The limitation section itself acknowledges the single-device scope. Overall, the derivation chain is self-contained with respect to external baselines, so circularity burden is minimal.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

The paper's central claim rests on unverified hardware/toolchain claims and benchmark fairness assumptions. There are no fitted numerical parameters and no new invented entities; all evidence is empirical measurement.

axioms (4)

domain assumption M5 GPU cores contain dedicated Neural Accelerator matrix units reachable via the Metal 4 tensor API.
Section 2 is built entirely on Apple documentation [16-20]. If this hardware/toolchain claim is false, the kernels and the entire result cannot function as described.
domain assumption The M5 Pro device used has the core count, memory bandwidth, and thermal behavior implied by the measurements.
Section 4.1 specifies a single M5 Pro with 48 GB; Section 5.1 acknowledges base and Max tiers are unmeasured. The general 'new performance ceiling' claim depends on one device being representative.
domain assumption Averaging five repetitions without reporting variance is sufficient to establish the reported ratios.
Section 4.1 defines metrics as averages of five repetitions but provides no standard deviation, confidence interval, or run-to-run distribution. This affects the precision of claims like 6.4x.
domain assumption Default llama.cpp and mlx-lm settings provide fair, representative baselines.
Section 4.1 states llama.cpp uses its Metal backend with default settings and mlx-lm uses default generation settings. BaseRT is hand-tuned by the authors; if these defaults underperform, the comparison is not apples-to-apples.

pith-pipeline@v1.3.0-alltime-deepseek · 8334 in / 12926 out tokens · 103339 ms · 2026-08-01T14:22:14.880561+00:00 · methodology

0 comments

read the original abstract

Apple's M5 generation introduces a redesigned GPU architecture in which every core carries a dedicated Neural Accelerator: on-die matrix units exposed through the Metal~4 tensor API. We show that BaseRT, our native Metal inference runtime for large language models on Apple Silicon, exploits these units to push inference throughput on Apple hardware substantially beyond both llama.cpp and MLX. Building on BaseRT's framework-free design, we add a family of hand-written Metal~4 tensor-core kernels (including dense and mixture-of-experts GEMM and flash-attention prefill kernels) that route the compute-bound matrix multiplications of inference through the M5 Neural Accelerators while leaving the memory-bound decode path on our existing specialised kernels. On an Apple M5 Pro, across fifteen model configurations spanning the Qwen3, Qwen3.5/3.6, Llama~3.2, and Gemma~4 families from sub-1B to 35B parameters, BaseRT delivers up to $6.4\times$ higher prompt-processing throughput than llama.cpp and $3.9\times$ higher than MLX, with the largest margins on the mixture-of-experts models where matrix multiplication dominates, while maintaining its lead on decode of up to $1.75\times$ over llama.cpp and $1.33\times$ over MLX. These results establish a new performance ceiling for on-device LLM inference and show that the M5's tensor cores are the decisive lever for prompt processing on Apple Silicon. BaseRT is publicly available at https://github.com/basecompute/baseRT.

Figures

Figures reproduced from arXiv: 2607.19438 by Fabian Waschkowski, Lukas Wesemann, Prabod Rathnayaka.

discussion (0)

Reference graph

Works this paper leans on

20 extracted references · 4 linked inside Pith

[1]

The next big shifts in AI workloads and hyperscaler strategies

Chhavi Arora, Marc Sorel, Pankaj Sachdeva, et al. The next big shifts in AI workloads and hyperscaler strategies. McKinsey & Company, December 2025. URL https://www.mckinsey.com/industries/te chnology-media-and-telecommunications/our-insights/the-next-big-shifts-in-ai-workloa ds-and-hyperscaler-strategies. Accessed: April 2026

2025
[2]

Gartner predicts 40% of AI data breaches will arise from cross-border GenAI misuse by 2027

Joerg Fritsch and Gartner, Inc. Gartner predicts 40% of AI data breaches will arise from cross-border GenAI misuse by 2027. Gartner Newsroom, February 2025. URL https://www.gartner.com/en/new 7 sroom/press-releases/2025-02-17-gartner-predicts-forty-percent-of-ai-data-breaches-w ill-arise-from-cross-border-genai-misuse-by-2027. Accessed: April 2026

2027
[3]

Artificial intelligence risk management framework: Generative artificial intelligence profile

National Institute of Standards and Technology. Artificial intelligence risk management framework: Generative artificial intelligence profile. Technical Report NIST AI 600-1, U.S. Department of Commerce, July 2024. URLhttps://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf

2024
[4]

DaDu-Corki: Algorithm-Architecture Co-Design for Embodied AI-powered Robotic Manipulation

Xuan Huang et al. DaDu-Corki: Algorithm-Architecture Co-Design for Embodied AI-powered Robotic Manipulation. InProceedings of the 52nd Annual International Symposium on Computer Architecture (ISCA), 2025. URLhttps://arxiv.org/abs/2407.04292

Pith/arXiv arXiv 2025
[5]

Agentic Artificial Intelligence: Architectures, Taxonomies, and Evaluation of Large Language Model Agents.arXiv preprint arXiv:2601.12560, 2026

Tula Masterman et al. Agentic Artificial Intelligence: Architectures, Taxonomies, and Evaluation of Large Language Model Agents.arXiv preprint arXiv:2601.12560, 2026. URL https://arxiv.org/abs/ 2601.12560

arXiv 2026
[6]

Edge-First Language Model Inference: Models, Metrics, and Tradeoffs

SiYoung Jang and Roberto Morabito. Edge-First Language Model Inference: Models, Metrics, and Tradeoffs. In45th IEEE International Conference on Distributed Computing Systems (ICDCS), 2025. URLhttps://arxiv.org/abs/2505.16508

Pith/arXiv arXiv 2025
[7]

Anthropic Status Page: Incident History

Anthropic. Anthropic Status Page: Incident History. https://status.anthropic.com, 2026. Accessed: April 2026. Documented 167 incidents between October 2025 and April 2026

2026
[8]

OpenAI Status Page: Incident History

OpenAI. OpenAI Status Page: Incident History. https://status.openai.com, 2026. Accessed: April 2026

2026
[9]

The New Economics of Enterprise Technology in an AI World

McKinsey & Company. The New Economics of Enterprise Technology in an AI World. https: //www.mckinsey.com/capabilities/tech-and-ai/our-insights/, May 2025

2025
[10]

State of Open Source AI 2026

Mozilla Foundation. State of Open Source AI 2026. Technical report, Mozilla Foundation, 2026. URL https://stateofopensource.ai/state-of-open-source-ai-2026.pdf. Accessed: July 2026

2026
[11]

Profiling Large Language Model Inference on Apple Silicon: A Quantization Perspective.arXiv preprint arXiv:2508.08531, 2025

Afsara Benazir and Felix Xiaozhu Lin. Profiling Large Language Model Inference on Apple Silicon: A Quantization Perspective.arXiv preprint arXiv:2508.08531, 2025

Pith/arXiv arXiv 2025
[12]

Artificial Intelligence Index Report 2025

Nestor Maslej et al. Artificial Intelligence Index Report 2025. Technical report, Stanford Institute for Human-Centered Artificial Intelligence (HAI), 2025. arXiv:2504.07139

arXiv 2025
[13]

BaseRT: Best-in-Class LLM Inference on Apple Silicon via Native Metal.arXiv preprint arXiv:2607.00501, 2026

Prabod Rathnayaka, Fabian Waschkowski, and Lukas Wesemann. BaseRT: Best-in-Class LLM Inference on Apple Silicon via Native Metal.arXiv preprint arXiv:2607.00501, 2026. URL https://arxiv.org/ abs/2607.00501

Pith/arXiv arXiv 2026
[14]

llama.cpp: LLM inference in C/C++

Georgi Gerganov et al. llama.cpp: LLM inference in C/C++. https://github.com/ggml-org/llama .cpp, 2023

2023
[15]

MLX: Efficient and flexible machine learning on Apple silicon.https://github.com/ml-explore/mlx, 2023

Awni Hannun, Jagrit Digani, Angelos Katharopoulos, and Ronan Collobert. MLX: Efficient and flexible machine learning on Apple silicon.https://github.com/ml-explore/mlx, 2023

2023
[16]

Apple unleashes M5, the next big leap in AI performance for Apple silicon

Apple. Apple unleashes M5, the next big leap in AI performance for Apple silicon. Apple Newsroom, October 2025. URL https://www.apple.com/newsroom/2025/10/apple-unleashes-m5-the-next-b ig-leap-in-ai-performance-for-apple-silicon/. Accessed: July 2026

2025
[17]

Apple. Metal 4. Apple Developer Documentation, 2025. URL https://developer.apple.com/docume ntation/metal/metal-4. Accessed: July 2026

2025
[18]

Metal Performance Primitives (MPP) Programming Guide

Apple. Metal Performance Primitives (MPP) Programming Guide. Apple Inc., Version 1, March 2026. Accessed: July 2026

2026
[19]

Discover Metal 4 tensors and Metal performance primitives

Apple. Discover Metal 4 tensors and Metal performance primitives. Apple Worldwide Developers Conference (WWDC), 2025. URLhttps://developer.apple.com/videos/play/wwdc2025/

2025
[20]

Metal Shading Language Specification

Apple. Metal Shading Language Specification. Apple Inc., Version 4.1, 2026. Accessed: July 2026. 8

2026

[1] [1]

The next big shifts in AI workloads and hyperscaler strategies

Chhavi Arora, Marc Sorel, Pankaj Sachdeva, et al. The next big shifts in AI workloads and hyperscaler strategies. McKinsey & Company, December 2025. URL https://www.mckinsey.com/industries/te chnology-media-and-telecommunications/our-insights/the-next-big-shifts-in-ai-workloa ds-and-hyperscaler-strategies. Accessed: April 2026

2025

[2] [2]

Gartner predicts 40% of AI data breaches will arise from cross-border GenAI misuse by 2027

Joerg Fritsch and Gartner, Inc. Gartner predicts 40% of AI data breaches will arise from cross-border GenAI misuse by 2027. Gartner Newsroom, February 2025. URL https://www.gartner.com/en/new 7 sroom/press-releases/2025-02-17-gartner-predicts-forty-percent-of-ai-data-breaches-w ill-arise-from-cross-border-genai-misuse-by-2027. Accessed: April 2026

2027

[3] [3]

Artificial intelligence risk management framework: Generative artificial intelligence profile

National Institute of Standards and Technology. Artificial intelligence risk management framework: Generative artificial intelligence profile. Technical Report NIST AI 600-1, U.S. Department of Commerce, July 2024. URLhttps://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf

2024

[4] [4]

DaDu-Corki: Algorithm-Architecture Co-Design for Embodied AI-powered Robotic Manipulation

Xuan Huang et al. DaDu-Corki: Algorithm-Architecture Co-Design for Embodied AI-powered Robotic Manipulation. InProceedings of the 52nd Annual International Symposium on Computer Architecture (ISCA), 2025. URLhttps://arxiv.org/abs/2407.04292

Pith/arXiv arXiv 2025

[5] [5]

Agentic Artificial Intelligence: Architectures, Taxonomies, and Evaluation of Large Language Model Agents.arXiv preprint arXiv:2601.12560, 2026

Tula Masterman et al. Agentic Artificial Intelligence: Architectures, Taxonomies, and Evaluation of Large Language Model Agents.arXiv preprint arXiv:2601.12560, 2026. URL https://arxiv.org/abs/ 2601.12560

arXiv 2026

[6] [6]

Edge-First Language Model Inference: Models, Metrics, and Tradeoffs

SiYoung Jang and Roberto Morabito. Edge-First Language Model Inference: Models, Metrics, and Tradeoffs. In45th IEEE International Conference on Distributed Computing Systems (ICDCS), 2025. URLhttps://arxiv.org/abs/2505.16508

Pith/arXiv arXiv 2025

[7] [7]

Anthropic Status Page: Incident History

Anthropic. Anthropic Status Page: Incident History. https://status.anthropic.com, 2026. Accessed: April 2026. Documented 167 incidents between October 2025 and April 2026

2026

[8] [8]

OpenAI Status Page: Incident History

OpenAI. OpenAI Status Page: Incident History. https://status.openai.com, 2026. Accessed: April 2026

2026

[9] [9]

The New Economics of Enterprise Technology in an AI World

McKinsey & Company. The New Economics of Enterprise Technology in an AI World. https: //www.mckinsey.com/capabilities/tech-and-ai/our-insights/, May 2025

2025

[10] [10]

State of Open Source AI 2026

Mozilla Foundation. State of Open Source AI 2026. Technical report, Mozilla Foundation, 2026. URL https://stateofopensource.ai/state-of-open-source-ai-2026.pdf. Accessed: July 2026

2026

[11] [11]

Profiling Large Language Model Inference on Apple Silicon: A Quantization Perspective.arXiv preprint arXiv:2508.08531, 2025

Afsara Benazir and Felix Xiaozhu Lin. Profiling Large Language Model Inference on Apple Silicon: A Quantization Perspective.arXiv preprint arXiv:2508.08531, 2025

Pith/arXiv arXiv 2025

[12] [12]

Artificial Intelligence Index Report 2025

Nestor Maslej et al. Artificial Intelligence Index Report 2025. Technical report, Stanford Institute for Human-Centered Artificial Intelligence (HAI), 2025. arXiv:2504.07139

arXiv 2025

[13] [13]

BaseRT: Best-in-Class LLM Inference on Apple Silicon via Native Metal.arXiv preprint arXiv:2607.00501, 2026

Prabod Rathnayaka, Fabian Waschkowski, and Lukas Wesemann. BaseRT: Best-in-Class LLM Inference on Apple Silicon via Native Metal.arXiv preprint arXiv:2607.00501, 2026. URL https://arxiv.org/ abs/2607.00501

Pith/arXiv arXiv 2026

[14] [14]

llama.cpp: LLM inference in C/C++

Georgi Gerganov et al. llama.cpp: LLM inference in C/C++. https://github.com/ggml-org/llama .cpp, 2023

2023

[15] [15]

MLX: Efficient and flexible machine learning on Apple silicon.https://github.com/ml-explore/mlx, 2023

Awni Hannun, Jagrit Digani, Angelos Katharopoulos, and Ronan Collobert. MLX: Efficient and flexible machine learning on Apple silicon.https://github.com/ml-explore/mlx, 2023

2023

[16] [16]

Apple unleashes M5, the next big leap in AI performance for Apple silicon

Apple. Apple unleashes M5, the next big leap in AI performance for Apple silicon. Apple Newsroom, October 2025. URL https://www.apple.com/newsroom/2025/10/apple-unleashes-m5-the-next-b ig-leap-in-ai-performance-for-apple-silicon/. Accessed: July 2026

2025

[17] [17]

Apple. Metal 4. Apple Developer Documentation, 2025. URL https://developer.apple.com/docume ntation/metal/metal-4. Accessed: July 2026

2025

[18] [18]

Metal Performance Primitives (MPP) Programming Guide

Apple. Metal Performance Primitives (MPP) Programming Guide. Apple Inc., Version 1, March 2026. Accessed: July 2026

2026

[19] [19]

Discover Metal 4 tensors and Metal performance primitives

Apple. Discover Metal 4 tensors and Metal performance primitives. Apple Worldwide Developers Conference (WWDC), 2025. URLhttps://developer.apple.com/videos/play/wwdc2025/

2025

[20] [20]

Metal Shading Language Specification

Apple. Metal Shading Language Specification. Apple Inc., Version 4.1, 2026. Accessed: July 2026. 8

2026