Pith. sign in

REVIEW 4 major objections 6 minor 5 cited by

WaferLLM: Large Language Model Inference at Wafer Scale

T0 review · 4 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Wafer-scale LLM inference can beat GPU clusters by 10–20x, this paper claims.

desk verdict Solid real-hardware systems paper with credible 8B/13B results; treat the 34B/72B numbers as extrapolations until the authors run the full models. read the letter →

arxiv 2502.04563 v3 pith:LOE26RDX submitted 2025-02-06 cs.LG cs.AIcs.ARcs.DCcs.ET

Congjie He , Yeqi Huang , Pei Mu , Ziming Miao , Jilong Xue , Lingxiao Ma , Fan Yang , Luo Mai This is my paper · ORCID
classification cs.LGcs.AIcs.ARcs.DCcs.ET
keywords wafer-scalecomputingLLMinferencePLMRmodelMeshGEMMGEMVnetwork-on-chipKVcachemanagementCerebrasWSE-2
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

WaferLLM claims that LLM inference can be made an order of magnitude faster by running it on a wafer-scale accelerator, whose hundreds of thousands of mesh-connected cores act like one giant on-chip memory. The paper argues that existing GPU-style inference systems fail on this hardware because they assume shared memory and uniform latency, and it provides a hardware model, PLMR, to guide a redesign. On a Cerebras WSE-2, the resulting system reports 10–20x higher per-request throughput than SGLang on A100 GPU clusters and about 2.5x better energy efficiency, with GEMV operations 606x faster than on a single A100. If the claims hold, the memory-bandwidth bottleneck that dominates token generation can be broken by wafer-scale integration rather than by adding more GPUs.

What carries the argument

The load-bearing mechanism is the PLMR device model, which names four constraints of wafer-scale mesh accelerators: massive parallelism, non-uniform memory-access latency, constrained per-core local memory, and limited hardware routing paths. On top of it sit two algorithmic inventions: MeshGEMM, a distributed GEMM whose interleaved logical-to-physical mapping keeps every communication step on a two-hop path, giving $O(1)$ critical-path latency with $O(1/N^2)$ memory per core; and MeshGEMV, a distributed GEMV whose K-tree allreduce (with $K=2$ in the implementation) reduces the allreduce critical path from a chain of $N$ sequential routing stages to a balanced tree of $K$ parallel-reduction phases. The KV-cache shift method is the third piece: instead of concatenating new tokens at one end, each row passes its oldest cache upward, keeping all cores evenly loaded. Together these pieces replace the allgather and allreduce patterns that work on GPUs but break on a mesh NoC.

What would settle it

Run full CodeLLaMA-34B and QWen2-72B inference on a wafer-scale chip with enough on-chip memory, or on an emulated memory configuration, and compare measured end-to-end TPR against the layer-scaled estimates in the paper; if measured throughput falls materially below the linear projection, the scaling assumption is refuted.

Watch

Extended reading notes

Core claim

The paper's central claim is that wafer-scale mesh accelerators can serve full LLM inference at speeds GPU clusters cannot reach, provided the software respects four hardware facts: massive core parallelism, highly non-uniform memory-access latency, small per-core local memory, and severely limited routing resources. WaferLLM encodes these as the PLMR model and uses it to design wafer-scale LLM parallelism, a transpose-free prefill/decode split, two new operators called MeshGEMM and MeshGEMV, and a shift-based KV-cache layout. In evaluation, WaferLLM on WSE-2 outperforms T10 by 100–200x, Ladder by 200–400x, and SGLang on A100 clusters by 10–20x end to end, while cutting energy use by roughly 2.5x. The authors attribute the remaining gap to the theoretical peak to immature hardware and software, and expect the advantage to widen as wafer-scale systems mature.

Load-bearing premise

The load-bearing premise is that prefill and decode throughput scale linearly with the number of transformer layers, so timing a subset of layers of CodeLLaMA-34B and QWen2-72B and multiplying by layer count gives true full-model performance; if pipeline bubbles or memory effects break that linearity, the large-model and 10–20x claims are overstated.

Editorial extensions

If this is right

  • Per-request token-generation throughput becomes the headline metric: the same latency budget per token covers far longer outputs, which matters for reasoning-style workloads that generate many tokens.
  • Decode, the phase usually memory-bandwidth-bound, can run at near 1:1 memory-to-compute ratio on wafer-scale chips, so energy per generated token drops relative to GPU clusters.
  • Directly porting shared-memory serving stacks to mesh accelerators is not just suboptimal but can underperform a single GPU; system design must be redone around the NoC.
  • The PLMR model and its operators transfer to future mesh-based accelerators, such as larger-memory wafers, so the software investment is not tied to one chip.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the reported end-to-end speedups are per-request throughput, not necessarily total throughput under high batch concurrency; a head-to-head batch-serving comparison would determine when the wafer-scale advantage translates into serving cost.
  • Editorial extension: because the 34B and 72B results are linearly scaled from a subset of layers, the paper does not yet demonstrate full-model performance for large weights; running those full models on a chip with enough on-chip memory would settle it.
  • Editorial extension: the KV-cache shift method is demonstrated for maximum output length per request; adapting it to variable-length batched serving with different sequence lengths is a natural next experiment the paper does not report.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper presents WaferLLM, an LLM inference system for wafer-scale accelerators, centered on a PLMR device model (massive Parallelism, non-uniform memory-access Latency, constrained local Memory, limited Routing). The system contributes wafer-scale prefill and decode parallelism policies, MeshGEMM and MeshGEMV kernels, a shift-based KV-cache management scheme, and an implementation on Cerebras WSE-2. Evaluations report end-to-end inference throughput for LLaMA3-8B and LLaMA2-13B, prefill/decode throughput also for CodeLLaMA-34B and QWen2-72B (the latter two via subset-of-layers extrapolation), microbenchmark comparisons against SUMMA, Cannon, and Cerebras's GEMV, and comparisons with SGLang on A100 GPU clusters. The headline claim is a 10–20x end-to-end speedup over A100 clusters and 100–400x speedups over T10 and Ladder baselines.

Significance. If the measured results hold, this is a significant systems contribution: it is one of the first complete LLM inference stacks for wafer-scale hardware, it is open-sourced, and it provides concrete full-model measurements on real hardware for 8B/13B models. The PLMR model is a useful organizing framework for reasoning about mesh-based massive-core accelerators, and the proposed MeshGEMM/MeshGEMV ideas are plausible algorithmic advances. The paper is honest about several current limitations (pipeline-parallel bubbles, local SRAM constraints, edge-core underutilization). However, the headline generalization to 34B/72B models rests on an unvalidated linear-scaling assumption, and the T10/Ladder baselines are author reimplementations whose fidelity is not fully documented; these issues currently temper the strength of the claimed speedups.

major comments (4)
  1. [§7.1, Tables 3–4] The CodeLLaMA-34B and QWen2-72B results are not full-model measurements. The text states that these models exceed WSE-2 memory capacity and that the authors 'evaluate a subset of layers and scale the results proportionally due to their uniform layer structure.' This assumes per-layer prefill/decode time is independent of total layer count, which is precisely what pipeline-parallel execution on a mesh NoC does not guarantee: pipeline fill/drain bubbles, inter-layer communication, KV-cache memory pressure, and edge-core effects all depend on the full model mapping. Because the full 34B/72B configuration was never run while the SGLang baselines are full-model runs, the 10–20x speedup for large models is not established. Please provide full-model measurements (for example with weight offloading or on WSE-3), or a validated scaling model with error bars, or explicitly restrict the large-model claims to the extrapolated regime.
  2. [§7.1, experiment setup] T10 and Ladder are reimplemented by the authors on WSE-2 rather than run as original systems. T10 assumes an inter-core crossbar, which is structurally different from the mesh NoC, and Ladder assumes shared memory; the paper does not describe how the reimplementations preserve the algorithms' semantics, scheduling, and optimizations on the mesh, nor whether the reimplementation code is available. The 100–400x speedup claims over these baselines therefore depend on the fidelity of the reimplementations. Please provide a detailed mapping of T10's compute-shift and Ladder's tile-based scheduling to WSE-2 cores, and ideally release the baseline code as part of the artifact.
  3. [§5.2, Algorithm 1] The claim that the two-hop interleaved communication pattern is minimal and hence satisfies the L property is supported only by an informal argument ('mathematical impossibility') and a statement that the 1D analysis 'naturally extends to a 2D mesh.' This minimality is the central distinction of MeshGEMM over Cannon, and the proof of O(α) critical-path complexity is load-bearing for the PLMR-compliance claim. Please supply a formal proof (including edge cases of Algorithm 1 for even and odd N, and for non-square meshes) or explicitly label the minimality claim as a conjecture supported by the empirical results of Figure 9.
  4. [§6.1, K-tree allreduce] The latency expression for K-tree allreduce is garbled in the text ('reducing the critical path to only K√ NK 2 times routing and N hops') and no derivation is given for the critical path, the number of routing stages, or how K is chosen to respect the R constraint on WSE-2 (the root core requires K+1 paths, with the hardware limit stated as 25). The statement that K=2 is a good choice needs quantitative support. Please correct the formula, provide the derivation, and report the measured routing-path usage per core against the hardware limit.
minor comments (6)
  1. [Abstract] The phrase 'up to 200× higher accelerator utilization' is not defined; the evaluation reports throughput per request (TPR) rather than utilization. Please define the utilization metric or rephrase the claim to match the measured quantities.
  2. [§6.1] The expression 'K√ NK 2 times routing and N hops' appears to be a typesetting error; it should be written unambiguously (for example, with explicit superscripts and parentheses) so that the scaling law is clear.
  3. [§7, Table 6] The matrix notation '[1,16K]×[16K,16K]' is confusing; please specify the dimensions explicitly (e.g., a 1×16384 vector multiplied by a 16384×16384 matrix) in the caption or surrounding text.
  4. [§7.1, Table 2] The footnote marker on the LLaMA2-13B row is inconsistent with the footnote text; please place the footnote so that it clearly applies to the 16-GPU columns and state the exact architectural reason (e.g., attention-head count not divisible by 16).
  5. [§7.5] The energy-efficiency comparison should state whether the reported A100/WSE-2 ratios are for prefill, decode, or end-to-end inference, and how power was measured (chip-only vs. full server), since the ratios in Tables 7 and 8 differ by an order of magnitude between phases.
  6. [§5.4] The use of N_lcm for non-square meshes is mentioned but not evaluated; please add a remark on how this affects the critical-path and routing-path analysis, since a non-square logical grid may alter the two-hop distance argument.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: PLMR is a descriptive hardware model, and the speedups are measured against external baselines; layer extrapolation and overlapping-author baselines are validity concerns, not circular reductions.

full rationale

The paper's derivation chain is not circular. The PLMR model (Section 3.1) is a descriptive hardware model capturing massive parallelism, non-uniform mesh latency, constrained per-core memory, and limited routing resources; it is not fitted to the paper's performance results, and the subsequent MeshGEMM and MeshGEMV designs are evaluated on real WSE-2 hardware against SUMMA, Cannon, Cerebras' GEMV, T10, Ladder, and SGLang. The central speedup claims are therefore measured outcomes, not consequences of the model's assumptions. Two adjacent concerns, neither circular, deserve note. First, Tables 3-4 extrapolate CodeLLaMA-34B and QWen2-72B from 'a subset of layers' scaled 'proportionally due to their uniform layer structure' (Section 7.1); this is an untested linear-scaling assumption that weakens the large-model claims, but it is not a definitional reduction or a fitted parameter renamed as a prediction. Second, the T10 [25] and Ladder [45] baselines are implemented by the authors and come from partially overlapping research groups, so the magnitude of those speedups should be read with caution; however, the comparisons are empirical and externally checkable rather than load-bearing self-citation. The paper also openly states that CodeLLaMA-34B and QWen2-72B were 'not included due to the memory constraint of a single WSE-2 chip,' and full-model end-to-end TPR is reported only for LLaMA3-8B and LLaMA2-13B. These are validity limitations, not circularity. No self-definitional step, fitted-input-as-prediction, or self-citation chain forces the paper's conclusions.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

PLMR is a modeling abstraction, not a new physical entity; no new particles, forces, dimensions, or conserved quantities are introduced. The core claims rest on vendor hardware parameters, on the decision to keep all weights on-chip, on subset-layer extrapolation for 34B/72B, and on an analytic latency model. Core counts are autotuned per benchmark.

free parameters (2)
  • Prefill and decode core counts per model = LLaMA3-8B: 660x660 prefill, 360x360 decode; LLaMA2-13B: 750x750 prefill, 375x375 decode; other models use autotuned…
    Reported TPR numbers are obtained after offline autotuning over core counts; the chosen configurations are part of what produces the headline speedups.
  • K-tree arity K = K=2
    Authors choose K=2 for the implementation; the speedup over ring and pipeline allreduce depends on this choice, though it is a design constant rather than a fitted value.
assumptions (5)
  • domain assumption WSE-2 has 850,000 cores, each with 48KB SRAM, 40GB total on-chip memory, 22PB/s bandwidth, and 5-bit routing headers.
    The PLMR model and algorithm designs take these hardware parameters as given from vendor specifications; they are not independently measured in the paper.
  • domain assumption All weights and KV cache for evaluated full models fit in 40GB on-chip memory; no off-chip memory traffic during inference.
    The communication-free advantage relies on this. For 34B and 72B models the assumption does not hold, so the paper evaluates subsets of layers instead.
  • ad hoc to paper Throughput for CodeLLaMA-34B and QWen2-72B scales linearly with layer count, so subset-layer measurements can be extrapolated to the full model.
    This is stated in Section 7 setup and is required for the 8B to 72B claim; communication and pipelining effects may not scale linearly.
  • domain assumption The critical-path latency model with per-hop latency alpha and per-routing latency beta is an adequate proxy for actual NoC cost on WSE-2.
    Used to justify the complexity claims for MeshGEMM and K-tree allreduce; not validated against measured per-hop or per-route latencies in the paper.
  • standard math Cannon's algorithm correctness applies to the cyclic-shift structure of MeshGEMM.
    The paper says correctness follows by reasoning similar to Cannon; this is a standard distributed GEMM result.

how reviews work

0 comments
Cite this review

Pith. "Pith review of WaferLLM: Large Language Model Inference at Wafer Scale." pith.science (2026). https://pith.science/paper/LOE26RDX

@misc{pith2026250204563,
  author       = {Pith},
  title        = {Pith review of: WaferLLM: Large Language Model Inference at Wafer Scale},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LOE26RDX}},
  note         = {Machine review of arXiv:2502.04563}
}
abstract

Emerging AI accelerators increasingly adopt wafer-scale manufacturing technologies, integrating hundreds of thousands of AI cores in a mesh architecture with large distributed on-chip memory (tens of GB in total) and ultra-high on-chip memory bandwidth (tens of PB/s). However, current LLM inference systems, optimized for shared memory architectures like GPUs, fail to exploit these accelerators fully. We introduce WaferLLM, the first wafer-scale LLM inference system. WaferLLM is guided by a novel PLMR model (pronounced as "Plummer") that captures the unique hardware characteristics of wafer-scale architectures. Leveraging this model, WaferLLM pioneers wafer-scale LLM parallelism, optimizing the utilization of hundreds of thousands of on-chip cores. It also introduces MeshGEMM and MeshGEMV, the first GEMM and GEMV implementations designed to scale effectively on wafer-scale accelerators. Evaluations show that WaferLLM achieves up to 200$\times$ higher accelerator utilization than state-of-the-art methods. Leveraging a wafer-scale accelerator (Cerebras WSE2), WaferLLM delivers GEMV operations 606$\times$ faster and 16$\times$ more energy-efficient than on an NVIDIA A100 GPU. For full LLM inference, WaferLLM achieves 10-20$\times$ speedups over A100 GPU clusters running SGLang and vLLM. These advantages are expected to grow as wafer-scale AI models, software, and hardware continue to mature. WaferLLM is open-sourced at https://github.com/MeshInfra/WaferLLM.

Figures

Figures reproduced from arXiv: 2502.04563 by the authors.

Figure 1
Figure 1. Key components in LLM inference We implemented WaferLLM on the Cerebras WSE engine using approximately 7,000 lines of CSL (a C-like program￾ming language) for LLM parallelism, MeshGEMM, and MeshGEMV, and 2,000 lines of Python for loading LLM checkpoints, launching inference, and execution parallelism policy. We conducted end-to-end LLM inference experiments with various models, including full LLaMA3-8B and LLaMA2- 1… view at source ↗
Figure 2
Figure 2. Massive-scale mesh-based memory architecture [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 4
Figure 4. Decode parallelism plan. E yFx indicates the E di￾mension is replicated along the y-axis, and F is partitioned along the x-axis. routing constraints, ensuring compliance with the L, M, and R properties. This PLMR-compliant distributed GEMM is fully described in Section 5. Using transposed distributed GEMM to avoid matrix transpose. We propose a transpose-free parallelism plan for prefill to avoid matrix transpose, a… view at source ↗
Figures from the paper (6 more)
Figure 5
Figure 5. Figure 5: KV cache concatenation vs. KV cache shift [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: PLMR compliance in distributed GEMM These differ by performing dist-GEMM, dist-GEMV and dist￾GEMM-T locally after grouping by head dimensions. 5 Wafer-Scale GEMM In this section, we introduce MeshGEMM, a scalable dis￾tributed GEMM for massive-scale, mesh architectures.…
Figure 7
Figure 7. Figure 7: Design intuitions and scalability analysis. [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: PLMR compliance in distributed GEMV path: Maximal latency of the whole allreduce communication, mainly determined by the time from when the partial sum is sent from the farthest core to when the aggregated sum is received by the farthest core. We analyze current distri…
Figure 9
Figure 9. Figure 9: MeshGEMM vs. SUMMA & Cannon strict access order. While T10’s compute-shift paradigm does not consider the spatial locality of cores on a mesh NoC, it still accounts for distributed memory, which allows perfor￾mance gains when memory (communication) accesses are order-i…
Figure 10
Figure 10. Figure 10: MeshGEMV vs. GEMV-Cerebras Additionally, increasing the number of cores does not al￾ways bring performance gains, especially for small matrix GEMM tasks. For example, in GEMM 2K, when scaling from 360×360 to 720×720 cores, the end-to-end latency of SUMMA and Cannon in…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SHIFT: Dynamic Compute Relocation Framework for Communication-Aware Chiplet-Based Systems

    cs.AR 2026-06 unverdicted novelty 6.0 of 10

    SHIFT is a topology-agnostic dynamic compute relocation framework for chiplet-based systems that reports 75.2-97.9% successful relocations, up to 76.8% latency reduction, 12.5x throughput gain, and 4.9x/5.9x/1.8x impr...

  2. ClusterFusion: Expanding Operator Fusion Scope for LLM Inference via Cluster-Level Collective Primitive

    cs.DC 2025-08 conditional novelty 6.0 of 10

    ClusterFusion fuses QKV projection, attention, and output projection into one kernel using Hopper cluster collectives, cutting average decoding latency by about 1.6x on an H100.

  3. ELK: Exploring the Efficiency of Inter-core Connected AI Chips with Deep Learning Compiler Techniques

    cs.AR 2025-07 conditional novelty 6.0 of 10

    A compiler framework, Elk, jointly schedules compute, inter-core data exchange, and HBM preloading on inter-core connected AI chips, reaching 94.84 percent of an ideal roofline on LLM workloads.

  4. A Theory of Inference Compute Scaling: Reasoning through Directed Stochastic Skill Search

    cs.LG 2025-06 conditional novelty 6.0 of 10

    A skill-graph random-walk model gives closed-form accuracy-versus-compute formulas for four reasoning strategies and connects them to training scaling.

  5. TriADA: Massively Parallel Trilinear Matrix-by-Tensor Multiply-Add Algorithm and Device Architecture for the Acceleration of 3D Discrete Transformations

    cs.DC 2025-06 conditional novelty 4.0 of 10

    The paper proposes a triple-stage outer-product algorithm and an isomorphic 3D mesh architecture that computes separable 3D orthogonal transforms in N1+N2+N3 time steps.

Reference graph

Works this paper leans on

60 extracted references · 54 canonical work pages · cited by 5 Pith papers

  1. [1]

    Abadi, P

    M. Abadi, P. Barham, J. Chen, et al. TensorFlow: A system for large-scale machine learning. OSDI 2016, 2016

    work page 2016

  2. [2]

    AMD optimizes EPYC mem- ory with NUMA

    Advanced Micro Devices. AMD optimizes EPYC mem- ory with NUMA. White paper, Advanced Micro De- vices, Inc., 2023

    work page 2023

  3. [3]

    Gqa: Training generalized multi-query transformer models from multi-head checkpoints, 2023

    Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints, 2023

    2023

  4. [4]

    AMD XDNA adaptive architecture, 2023

    AMD. AMD XDNA adaptive architecture, 2023. Ac- cessed: 2024-11-29

    work page 2023

  5. [5]

    Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Nee- lakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Nee- lakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020

    arXiv 2005

  6. [6]

    A cellular computer to implement the kalman filter algorithm

    Lynn Elliot Cannon. A cellular computer to implement the kalman filter algorithm. PhD thesis, Montana State University, 1969

    work page 1969

  7. [7]

    GEMM with collective operations

    Cerebras Systems. GEMM with collective operations. Accessed: 2024-10-05

    work page 2024

  8. [8]

    100× defect tolerance: How cerebras solved the yield problem, 2022

    Cerebras Systems. 100× defect tolerance: How cerebras solved the yield problem, 2022. Accessed: 2025-04-29

    work page 2022

Show all 60 references
  1. [9]

    Benchmark GEMV collectives, 2023

    Cerebras Systems. Benchmark GEMV collectives, 2023. Accessed: 2024-11-29

    2023

  2. [10]

    Chen et al

    T. Chen et al. TVM: An automated end-to-end optimiza- tion stack for deep learning. SSP 2018, 2018

    2018

  3. [11]

    Dongarra, and David W

    Jaeyoung Choi, Jack J. Dongarra, and David W. Walker. Parallel matrix transpose algorithms on distributed memory concurrent computers. Parallel Computing, 21(9):1387–1405, 1995

    1995

  4. [12]

    FlashAttention-2: Faster attention with bet- ter parallelism and work partitioning

    Tri Dao. FlashAttention-2: Faster attention with bet- ter parallelism and work partitioning. In The Twelfth International Conference on Learning Representations, 2024

    2024

  5. [13]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  6. [14]

    SambaNova’s new AI chip and the quest for efficiency, 2023

    Mark Harris. SambaNova’s new AI chip and the quest for efficiency, 2023. Accessed: 2024-11-29

    2023

  7. [15]

    Flashdecoding++: Faster large language model infer- ence on GPUs

    Ke Hong, Guohao Dai, Jiaming Xu, Qiuli Mao, Xiuhong Li, Jun Liu, Kangdi Chen, Hanyu Dong, and Yu Wang. Flashdecoding++: Faster large language model infer- ence on GPUs. arXiv preprint arXiv:2311.01282, 2023

    2023 arXiv

  8. [16]

    Openai o1 system card

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richard- son, Ahmed El-Kishky, Aiden Low, Alec Helyar, Alek- sander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card. arXiv preprint arXiv:2412.16720, 2024

    2024 arXiv

  9. [17]

    Tensor processing units for machine learning: An introduction

    Norman Jouppi, Cliff Young, et al. Tensor processing units for machine learning: An introduction. Technical report, Google Inc., 2017

    2017

  10. [18]

    Tenstorrent Blackhole and Metalium for standalone AI processing, 2024

    Patrick Kennedy. Tenstorrent Blackhole and Metalium for standalone AI processing, 2024. ServeTheHome, Hot Chips 2024 Coverage

    2024

  11. [19]

    Chiplet/interposer co-design for power delivery network optimization in heterogeneous 2.5-d ICs

    Jinwoo Kim, Venkata Chaitanya Krishna Chekuri, Nael Mizanur Rahman, Majid Ahadi Dolatsara, Hakki Mert Torun, Madhavan Swaminathan, Saibal Mukhopadhyay, and Sung Kyu Lim. Chiplet/interposer co-design for power delivery network optimization in heterogeneous 2.5-d ICs. IEEE Trans...

    2021

  12. [20]

    Efficient memory man- agement for large language model serving with Page- dAttention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory man- agement for large language model serving with Page- dAttention. In Proceedings of the 29th Symposium on Operating Systems Principle...

    2023

  13. [21]

    TSMC bets big on advanced packaging,

    Mark LaPedus. TSMC bets big on advanced packaging,

  14. [22]

    ReSA: Reconfig- urable systolic array for multiple tiny DNN tensors

    Ching-Jui Lee and Tsung Tai Yeh. ReSA: Reconfig- urable systolic array for multiple tiny DNN tensors. ACM Transactions on Architecture and Code Optimiza- tion, 21(3):43:1–43:24, 2024

    2024

  15. [23]

    Gonzalez, and Ion Stoica

    Zhuohan Li, Lianmin Zheng, Yinmin Zhong, Vincent Liu, Ying Sheng, Xin Jin, Yanping Huang, Zhifeng Chen, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Al- paServe: Statistical multiplexing with model parallelism for deep learning serving. In 17th USENIX Sympo- sium on Operatin...

    2023

  16. [24]

    Cerebras architecture deep dive: First look in- side the hardware/software co-design for deep learning

    Sean Lie. Cerebras architecture deep dive: First look in- side the hardware/software co-design for deep learning. IEEE Micro, 43(3):18–30, 2023

    2023

  17. [25]

    Scaling deep learn- ing computation over the inter-core connected intelli- gence processor with T10

    Yiqi Liu, Yuqi Xue, Yu Cheng, Lingxiao Ma, Ziming Miao, Jilong Xue, and Jian Huang. Scaling deep learn- ing computation over the inter-core connected intelli- gence processor with T10. In Proceedings of the ACM 15 SIGOPS 30th Symposium on Operating Systems Princi- ples, pages ...

  18. [26]

    TENET: A framework for modeling tensor dataflow based on relation-centric notation

    Liqiang Lu, Naiqing Guan, Yuyue Wang, Liancheng Jia, Zizhang Luo, Jieming Yin, Jason Cong, and Yun Liang. TENET: A framework for modeling tensor dataflow based on relation-centric notation. In 2021 ACM/IEEE 48th Annual International Symposium on Computer Ar- chitecture (ISCA),...

    2021

  19. [27]

    Near- optimal wafer-scale reduce

    Piotr Luczynski, Lukas Gianinazzi, Patrick Iff, Leighton Wilson, Daniele De Sensi, and Torsten Hoefler. Near- optimal wafer-scale reduce. In Proceedings of the 33rd International Symposium on High-Performance Parallel and Distributed Computing, HPDC ’24, page 334–347. ACM, June 2024

    2024

  20. [28]

    Rammer: Enabling holistic deep learning compiler optimizations with rTasks

    Lingxiao Ma, Zhiqiang Xie, Zhi Yang, Jilong Xue, Youshan Miao, Wei Cui, Wenxiang Hu, Fan Yang, Lintao Zhang, and Lidong Zhou. Rammer: Enabling holistic deep learning compiler optimizations with rTasks. In 14th USENIX Symposium on Operating Systems Design and Implementation (OS...

    2020

  21. [29]

    An electrical-thermal co-simulation model of chiplet hetero- geneous integration systems

    Xiaoning Ma, Qinzhi Xu, Chenghan Wang, He Cao, Jianyun Liu, Daoqing Zhang, and Zhiqiang Li. An electrical-thermal co-simulation model of chiplet hetero- geneous integration systems. IEEE Transactions on Very Large Scale Integration (VLSI) Systems, 32(10):1769– 1781, 2024

    2024

  22. [30]

    Introducing MTIA: Meta’s next-generation training and inference accelerator for AI, 2024

    Meta AI. Introducing MTIA: Meta’s next-generation training and inference accelerator for AI, 2024. Ac- cessed: 2024-12-10

    2024

  23. [31]

    Azure Maia: For the era of AI from silicon to software to systems, 2023

    Microsoft Azure. Azure Maia: For the era of AI from silicon to software to systems, 2023. Accessed: 2024- 11-29

    2023

  24. [32]

    Efficient large-scale language model training on GPU clusters using Megatron-LM

    Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, Amar Phanishayee, and Matei Zaharia. Efficient large-scale language model training on GPU clusters u...

    2021

  25. [33]

    Openai o3 and o4-mini system card

    OpenAI. Openai o3 and o4-mini system card. https: //openai.com/index/o3-o4-mini-system-card/ ,

  26. [34]

    Paszke, S

    A. Paszke, S. Gross, S. Chintala, et al. Automatic differ- entiation in PyTorch. NIPS 2017, 2017

    2017

  27. [35]

    Efficiently scal- ing transformer inference

    Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently scal- ing transformer inference. Proceedings of Machine Learning and Systems, 5, 2023

    2023

  28. [36]

    Rock et al

    J. Rock et al. XLA: Optimizing TensorFlow for high performance. Google Research, 2017

    2017

  29. [37]

    Welder: Scheduling deep learning memory access via tile-graph

    Yining Shi, Zhi Yang, Jilong Xue, Lingxiao Ma, Yuqing Xia, Ziming Miao, Yuxiao Guo, Fan Yang, and Lidong Zhou. Welder: Scheduling deep learning memory access via tile-graph. In 17th USENIX Symposium on Oper- ating Systems Design and Implementation (OSDI 23), pages 701–718, 2023

    2023

  30. [38]

    Souri, Kaustav Banerjee, Amit Mehrotra, and Krishna C

    Shukri J. Souri, Kaustav Banerjee, Amit Mehrotra, and Krishna C. Saraswat. Multiple Si layer ICs: motiva- tion, performance analysis, and design implications. In Proceedings of the 37th Annual Design Automation Con- ference, pages 213–220, 2000

    2000

  31. [39]

    Cerebras and g42 break ground on condor galaxy 3, an 8 exaflops ai supercom- puter

    Cerebras Systems. Cerebras and g42 break ground on condor galaxy 3, an 8 exaflops ai supercom- puter. https://cerebras.ai/press-release/ cerebras-g42-announce-condor-galaxy-3 , 2024. Accessed: 2025-05-03

    2024

  32. [40]

    Cerebras powers perplex- ity sonar with industry’s fastest ai inference

    Cerebras Systems. Cerebras powers perplex- ity sonar with industry’s fastest ai inference. https://www.cerebras.ai/press-release/ cerebras-powers-perplexity-sonar-with\ -industrys-fastest-ai-inference , 2025. Ac- cessed: 2025-05-03

    2025

  33. [41]

    DOJO: The microarchitecture of Tesla’s exa-scale com- puter

    Emil Talpes, Douglas Williams, and Debjit Das Sarma. DOJO: The microarchitecture of Tesla’s exa-scale com- puter. In 2022 IEEE Hot Chips 34 Symposium (HCS), pages 1–28, 2022

    2022

  34. [42]

    R. A. Van De Geijn and J. Watts. SUMMA: scalable universal matrix multiplication algorithm. Concurrency: Practice and Experience, 9(4):255–274, 1997

    1997

  35. [43]

    Gomez, Łukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems 30 (NeurIPS 2017), pages 5998–6008. Curran Associates, Inc., 2017

    2017

  36. [44]

    Cerebras brings instant inference to mistral le chat

    James Wang. Cerebras brings instant inference to mistral le chat. https://cerebras.ai/blog/ mistral-le-chat, 2025. Accessed: 2025-05-03

    2025

  37. [45]

    Ladder: Enabling efficient low- precision deep learning computing through hardware- aware tensor transformation

    Lei Wang, Lingxiao Ma, Shijie Cao, Quanlu Zhang, Ji- long Xue, Yining Shi, Ningxin Zheng, Ziming Miao, Fan 16 Yang, Ting Cao, et al. Ladder: Enabling efficient low- precision deep learning computing through hardware- aware tensor transformation. In 18th USENIX Sympo- sium on O...

    2024

  38. [46]

    Application defined on-chip networks for heteroge- neous chiplets: An implementation perspective

    Tianqi Wang, Fan Feng, Shaolin Xiang, Qi Li, and Jing Xia. Application defined on-chip networks for heteroge- neous chiplets: An implementation perspective. InIEEE International Symposium on High-Performance Com- puter Architecture (HPCA), pages 1198–1210, 2022

    2022

  39. [47]

    Static random-access memory,

    Wikipedia contributors. Static random-access memory,

  40. [48]

    Wafer-scale integration, 2024

    Wikipedia contributors. Wafer-scale integration, 2024. Accessed: 2024-12-10

    2024

  41. [49]

    LoongServe: Efficiently serving long-context large language models with elas- tic sequence parallelism

    Bingyang Wu, Shengyu Liu, Yinmin Zhong, Peng Sun, Xuanzhe Liu, and Xin Jin. LoongServe: Efficiently serving long-context large language models with elas- tic sequence parallelism. In Proceedings of the ACM SIGOPS 30th Symposium on Operating Systems Princi- ples, pages 640–654....

    2024

  42. [50]

    DIS- TAL: the distributed tensor algebra compiler

    Rohan Yadav, Alex Aiken, and Fredrik Kjolstad. DIS- TAL: the distributed tensor algebra compiler. In Pro- ceedings of the 43rd ACM SIGPLAN International Con- ference on Programming Language Design and Imple- mentation, pages 286–300, 2022

    2022

  43. [51]

    Zhao et al

    Y . Zhao et al. Ansor: A compiler stack for auto-tuning tensor programs. IEEE Transactions on Software Engi- neering, 2020

    2020

  44. [52]

    Alpa: Automating inter-and intra-operator parallelism for dis- tributed deep learning

    Lianmin Zheng, Zhuohan Li, Hao Zhang, Yonghao Zhuang, Zhifeng Chen, Yanping Huang, Yida Wang, Yuanzhong Xu, Danyang Zhuo, Eric P Xing, et al. Alpa: Automating inter-and intra-operator parallelism for dis- tributed deep learning. In 16th USENIX Symposium on Operating Systems De...

    2022

  45. [53]

    Sglang: Efficient execution of structured language model programs

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Livia Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, et al. Sglang: Efficient execution of structured language model programs. Advances in Neural Information Pro- cessing Systems, ...

    2024

  46. [54]

    FlexTensor: An automatic schedule exploration and optimization framework for tensor com- putation on heterogeneous system

    Size Zheng, Yun Liang, Shuo Wang, Renze Chen, and Kaiwen Sheng. FlexTensor: An automatic schedule exploration and optimization framework for tensor com- putation on heterogeneous system. InProceedings of the Twenty-Fifth International Conference on Architectural Support for Pr...

    2020

  47. [55]

    Dist- Serve: Disaggregating prefill and decoding for goodput- optimized large language model serving

    Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. Dist- Serve: Disaggregating prefill and decoding for goodput- optimized large language model serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI ...

  48. [56]

    Exploring TensorRT to improve real-time inference for deep learning

    Yuxiao Zhou and Kecheng Yang. Exploring TensorRT to improve real-time inference for deep learning. In 2022 IEEE 24th International Conference on High Per- formance Computing & Communications, pages 2011–

    2022

  49. [57]

    ROLLER: Fast and efficient tensor compilation for deep learning

    Hongyu Zhu, Ruofan Wu, Yijia Diao, Shanbin Ke, Haoyu Li, Chen Zhang, Jilong Xue, Lingxiao Ma, Yuqing Xia, Wei Cui, Fan Yang, Mao Yang, Lidong Zhou, Asaf Cidon, and Gennady Pekhimenko. ROLLER: Fast and efficient tensor compilation for deep learning. In 16th USENIX Symposium on ...

    2022

  50. [2023]

    Accessed: 2024-11-29

    2024

  51. [2024]

    Accessed: 2024-12-10

    2024

  52. [2025]

    Accessed: 2025-05-03

    2025

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.