Pith. sign in

REVIEW 2 major objections 5 minor 72 references

FastTPS accelerates the sequential token phase of LLM inference by 6x on AI accelerators through reloading-free KV-cache management, fused RoPE attention, and pipelined MLP fusion while preserving accuracy.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 06:15 UTC pith:3JQIIWHG

load-bearing objection Solid NPU systems paper: three concrete fusions deliver measured 6× token-phase speedup and 93% bandwidth on AMD hardware, with the usual vendor-paper caveats on static KV and omitted CPU cost. the 2 major comments →

arxiv 2607.11211 v1 pith:3JQIIWHG submitted 2026-07-13 cs.LG

FastTPS: An Optimized Method for LLM Token Phase for AI accelerators

Wenzong Yang , Danyang Zhang , Kun Cao , Tejus Siddagangaiah , Rajeev Patwari , Zhanxing Pu , Siyin Kong , Zijiang Yang
show 12 more authors
Hao Zhu Varun Sharma Yue Gao Tianping Li Fan Yang Jicheng Chen Yushan Chen Fennian Zhao Aaron Ng Elliott Delaye Ashish Sirasao Sudip Nag
This is my paper
classification cs.LG
keywords LLM inferencetoken phaseKV cacheoperator fusionAI acceleratorsRoPE attentionMLP pipelineoperational intensity
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Decoder-only large language models generate tokens one by one, so the token phase has almost no parallelism and quickly becomes limited by memory bandwidth, especially once the key-value cache grows long. FastTPS attacks that bottleneck on general AI accelerators with three coordinated changes: a static, address-jumping KV cache that never reloads or concatenates on the host, a tiling scheme that fuses rotary embeddings with the attention score and value multiplies inside on-chip memory, and an interlaced gate-up weight layout that lets the MLP run as a single fine-grained pipeline. On an AMD Ryzen AI NPU the method reaches 6x higher tokens-per-second for Phi-3-mini (93 percent of peak bandwidth) and improves attention and MLP blocks by up to 13x and 3x respectively, with numerical error an order of magnitude smaller than FlashAttention. A sympathetic reader cares because the same memory-bound token phase appears on every edge and server accelerator; closing the utilization gap directly raises the practical speed of long-context generation without requiring new hardware or lower-precision approximations.

Core claim

The paper shows that three accelerator-native dataflow changes—global static KV-cache updates that eliminate host Concat, TPSFLAT fusion of RoPE-QK-Softmax-PV under a three-level tiling hierarchy, and interlaced gate-up MLP fusion with pipeline scheduling—raise operational intensity enough to convert the token phase from a bandwidth-starved sequential loop into a near-peak-bandwidth pipeline, delivering measured 6x end-to-end speedup and 93 percent bandwidth utilization on a commercial NPU while keeping absolute error below 0.22 percent.

What carries the argument

FastTPS, built from GKVC (static 3-D KV-cache writes by address jumping), TPSFLAT (FLAT-style Softmax tiling that also fuses RoPE inside L1), and Fusion MLP (interlaced gate-up weights enabling single-Matmul plus SiLU-Mul pipeline).

Load-bearing premise

A pre-allocated static KV-cache region sized for a fixed maximum length can be updated by pure address jumps on the accelerator without fragmentation or waste, and host-CPU overhead can be ignored when reporting end-to-end tokens-per-second.

What would settle it

Run the identical Phi-3 and Llama models on the same NPU while forcing sequences that exceed the pre-allocated KV-cache size or while measuring wall-clock time that includes every host-side launch and synchronization; if the 6x ratio or 93 percent bandwidth figure collapses, the claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Long-sequence token generation on bandwidth-limited NPUs and edge accelerators becomes practical without further model compression.
  • Attention and MLP blocks can be scheduled as continuous on-chip pipelines, raising average accelerator utilization above 80 percent for the models tested.
  • The same three techniques remain compatible with grouped-query attention and low-precision formats already used in production stacks.
  • Roofline analysis shows operational intensity of attention rising roughly 3x, moving the token phase out of the pure memory-bound region for typical head dimensions.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the optimizations are expressed as data layout and tiling rather than new operators, the same patterns should transfer to any accelerator whose memory hierarchy can be abstracted as three cache levels.
  • Static pre-allocation trades memory capacity for bandwidth; on devices with tight DRAM the method may force shorter context windows unless combined with paging or compression.
  • Ignoring host overhead is reasonable for pure NPU-bound kernels but may understate the real system gain once the full inference runtime (tokenization, sampling, multi-model serving) is included.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes FastTPS, a dataflow optimization for the autoregressive token phase of decoder-only LLMs on general AI accelerators with a three-level cache hierarchy. It comprises three components: (1) Global KV Cache management (GKVC), which pre-allocates a static 3-D KV region and updates it by address-jumping (Eq. 2) on the accelerator, eliminating the Concat operator and CPU involvement; (2) TPSFLAT, a RoPE-aware tiling of FLAT that fuses RoPE, QKT, Softmax and PV inside L1 while writing only the new K/V back to L3 (Alg. 3); and (3) Fusion MLP that interlaces gate/up weights, fuses the resulting Matmul with SiLU and element-wise Mul, and pipelines weight loads (Alg. 4). On an AMD Ryzen AI 300 NPU (BF16) the method reports up to 6 imes end-to-end TPS versus a non-fused baseline (Phi3-mini-4k-instruct, 93 % peak bandwidth), attention speed-ups of 70–93 % and MLP speed-ups of 20–68 % across five models, with maximum absolute error an order of magnitude lower than FlashAttention.

Significance. If the measurements hold, FastTPS supplies a concrete, accuracy-preserving recipe for raising operational intensity of the memory-bound token phase on edge NPUs. The combination of static address-jump KV management, RoPE-aware L1 fusion and interlaced-weight MLP fusion is a useful engineering contribution for the growing class of three-level-cache AI accelerators; the multi-model latency breakdowns, roofline points and explicit accuracy comparison against a float32 baseline strengthen the claim. The work is therefore of practical interest to systems and architecture venues that target LLM inference on resource-constrained hardware.

major comments (2)
  1. [§4, §3.1.1] §4 (and the explicit statement that “CPU overhead wasn’t taken into consideration”) and §3.1.1: the headline 6× / 93 % figures are obtained by comparing a fully accelerator-resident pipeline against a baseline that still performs Concat on the CPU. Because the paper never quantifies the residual host-side orchestration cost of the optimized path, it is unclear how much of the reported end-to-end gain would survive a realistic runtime that must still schedule kernels and manage the static KV region. A short measurement (or an explicit bound) of residual host overhead is needed to support the end-to-end claim.
  2. [§3.1.1, Eq. (2)] §3.1.1 and Eq. (2): GKVC relies on a static pre-allocation sized for a predetermined maximum sequence length Nr. The manuscript does not discuss fragmentation, capacity waste for shorter sequences, or the cost of re-allocation when the true length exceeds Nr. Because the speed-up curves in Fig. 6 grow with Nt, the practical utility of the method for variable-length or multi-tenant workloads remains incompletely characterized.
minor comments (5)
  1. [Abstract, §1] Abstract and §1: the phrase “since they emerge with numeric deviation” is ungrammatical and should be rewritten for clarity.
  2. [Table 3] Table 3 header “Theoretical TPS (ms)” appears to report a latency (ms) rather than a throughput (tokens/s); the units and the definition of the theoretical bound should be stated explicitly.
  3. [Fig. 2] Fig. 2 caption cites “2.84 GFLOPs” as peak performance; the unit is almost certainly GFLOP/s and should be corrected.
  4. [§3.1.3, App. A–C] Appendix A–C contain useful cache-occupancy formulas and tiling details that are only lightly referenced in the main text; a short pointer in §3.1.3 would help readers locate them.
  5. [Tables 1–3, Figs. 5–7] Several model names are inconsistently abbreviated (L3 / Llama3, P3 / Phi3, L3.2 / Llama3.2) across tables and figures; a single consistent notation would improve readability.

Circularity Check

0 steps flagged

No significant circularity: empirical systems paper whose speedups and bandwidth figures are direct wall-clock measurements of newly proposed dataflow algorithms, not quantities forced by fitted parameters or self-referential definitions.

full rationale

FastTPS presents three concrete engineering optimizations (GKVC address-jump static KV writes of Eq. 2 / Alg. 3, TPSFLAT RoPE+attention fusion with three-level tiling, and interlaced gate-up Fusion MLP of Alg. 4) and evaluates them by measuring latency, TPS, OI/roofline points, and absolute error on an AMD Ryzen AI 300 NPU against HuggingFace baselines and FlashAttention. The headline 6 imes / 93 % claims (Table 3, abstract) are instrumented wall-clock and bandwidth numbers for Phi3-mini-4k-instruct (BF16); they are not derived from free parameters fitted to a subset of the same data and then re-labeled as predictions. Citations to FLAT (Kao et al. 2023) and AMD hardware descriptions supply background or platform facts but are not load-bearing uniqueness theorems that force the present results. Cache-occupancy formulas (Table 4 / App. B) are straightforward buffer-size calculations, not circular redefinitions. No self-definitional loop, fitted-input-as-prediction, or ansatz-smuggling appears; the work is self-contained against external benchmarks and measurements.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 3 invented entities

The central performance claims rest on a three-level cache model of the target NPU, the feasibility of static pre-allocation, and a set of free tiling parameters chosen for that hardware. No new physical entities are postulated; the invented constructs are purely algorithmic.

free parameters (4)
  • Hb (outer-loop head-group size)
    Chosen to fit L2 capacity; directly controls tiling granularity and therefore measured latency.
  • tilen, tilek, tiled (MLP tile dimensions)
    Hand-tuned to L1 matrix-multiply unit size and L2 bandwidth; appear in Algorithm 4 and determine pipeline efficiency.
  • x, y, z (number of tiles moved L3↔L2)
    Adjusted dynamically for bandwidth and L2 capacity; free parameters of the scheduling policy.
  • maximum pre-allocated sequence length Nr
    Static KV-cache size is fixed at compile/load time; under- or over-estimation wastes memory or aborts long generations.
axioms (4)
  • domain assumption The target accelerator exposes a three-level cache hierarchy (L1 < L2 < L3 in capacity, reverse in speed) that can be programmed with explicit data movement.
    Stated in §2.1 and used throughout Algorithms 3–4; without it the tiling strategy collapses.
  • domain assumption A contiguous static region large enough for the maximum sequence length can be pre-allocated in L3 and written by address jumps without fragmentation.
    Core premise of GKVC (§3.1.1); if the memory manager cannot guarantee contiguity the Concat elimination fails.
  • ad hoc to paper CPU-side orchestration overhead can be ignored when reporting end-to-end token-phase speed-ups.
    Explicitly declared in §4; removes a real cost that would appear in production deployments.
  • standard math Softmax normalization dimension permits independent head-wise tiling without accuracy loss beyond floating-point truncation.
    Inherited from FLAT and used to justify TPSFLAT fusion (§3.1.2).
invented entities (3)
  • GKVC (Global KV Cache management) no independent evidence
    purpose: Eliminate Concat and keep the entire token pipeline on the accelerator by address-jump writes into a static buffer.
    New algorithmic construct introduced in §3.1.1; no independent existence outside the paper’s dataflow.
  • TPSFLAT no independent evidence
    purpose: Fuse RoPE + QKT + Softmax + PV inside L1 under the token-phase constraint B=N=1.
    Specialized extension of FLAT tailored to TPS; defined only in this work.
  • Fusion MLP with interlaced gate-up weights no independent evidence
    purpose: Reduce I/O from 7 to 2 transfers and enable fine-grain pipeline of gate/up/SiLU/Mul.
    New layout and scheduling scheme introduced in §3.2; not previously published as a unit.

pith-pipeline@v1.1.0-grok45 · 25740 in / 3226 out tokens · 38320 ms · 2026-07-14T06:15:05.318731+00:00 · methodology

0 comments
read the original abstract

The popularity of large language models (LLMs) escalates an ongoing demand for effective inference. However, due to the sequential processing of tokens during the token phase in decoder-only LLMs inference, the inherent low parallelism leads to reduced throughput and suboptimal utilization of the computing units on artificial intelligence (AI) accelerators, particularly when handling long-sequence inputs that impose significant memory overhead. Recently, many reported methods have been developed as potential solutions, since they emerge with numeric deviation. This paper presents FastTPS, a high performance and low-precision loss method for accelerating the token-phase in LLM inference on general AI accelerators which includes three key components: (1) AI accelerator-enabled reloading-free KV Cache concatenation which decreases memory access overhead as well as enables full fusion of Attention, (2) high-efficiency and high-accuracy 'RoPE' attention based on the tiling optimized FLAT, and (3) highly-fused MLP with fine-grain pipeline scheduling. Our results confirm that FastTPS significantly alleviates memory bottlenecks in the token phase, delivering a 6x speed improvement (compared to none-fusion) on an AMD Ryzen AI 300 series NPU with BF16 precision while sustaining 93% peak memory bandwidth utilization during Phi3-mini-4k-instruct inference.

Figures

Figures reproduced from arXiv: 2607.11211 by Aaron Ng, Ashish Sirasao, Danyang Zhang, Elliott Delaye, Fan Yang, Fennian Zhao, Hao Zhu, Jicheng Chen, Kun Cao, Rajeev Patwari, Siyin Kong, Sudip Nag, Tejus Siddagangaiah, Tianping Li, Varun Sharma, Wenzong Yang, Yue Gao, Yushan Chen, Zhanxing Pu, Zijiang Yang.

Figure 1
Figure 1. Figure 1: Structural overview of the standard attention and MLP blocks. Section ⃝1 corresponds to the attention block, and Section ⃝2 corresponds to the MLP block. The change of data dimension is illustrated. Transformer-based systems. Nt denotes the number of total tokens (KV Cache), d is the split hidden size per head and N is the current token’s sequence length (query). On the other hand, the MLP block is a feedf… view at source ↗
Figure 2
Figure 2. Figure 2: Roofline analysis of the attention and MLP blocks for some specific Nt, where the attention block incorporates the GQA (grouped query attention) optimization. Ori refers to the original implementation before FastTPS optimization, and Opt indicates the optimized version following the application of FastTPS. For the attention block, the analysis covers scenarios where the number of tokens gradually increases… view at source ↗
Figure 3
Figure 3. Figure 3: Structural overview of attention blocks after GKVC and TPSFLAT optimization. The red, orange, and green areas corre￾spond to the L3, L2, and L1 cache respectively. In GKVC, the KV vectors computed in the L1 cache are directly written to the pre-allocated memory locations in the L3 cache via the L2 cache, eliminating the need for a Concat operator. This strategy ensures memory without fragmentation. TPSFLAT… view at source ↗
Figure 4
Figure 4. Figure 4: The Fusion MLP workflow, each part represents the en￾tire execution process of data exchange between the L1 cache and the L3 cache. The part1 is the RMSNorm operation applied to normalize the input tensor. The part2 is the primary part of MLP which describes the methods of interlaced gate-up weights. The pipeline schedule in part2 demonstrates improvement by introduc￾ing interlaced weights which eliminates… view at source ↗
Figure 5
Figure 5. Figure 5: presents the latency performance of various trans￾former models, with numerical annotations above the green bars indicating the percentage reduction in TPSFLAT la￾tency relative to the attention baseline. The attention mech￾anism involves KV Cache concatenation (processed by the CPU) and subsequent computations on the AI accelera￾tor which leads to low OI. Our proposed TPSFLAT op￾timizations eliminate thes… view at source ↗
Figure 7
Figure 7. Figure 7 [PITH_FULL_IMAGE:figures/full_fig_p009_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Maximum error as sequence length of TPSFLAT. The error decreases as sequence length increases. The largest maximum error is less than 0.22%. upon FLAT (Kao et al., 2023) which has been validated on NVIDIA Tesla T4, and the KV Cache management and tiling strategies can be directly integrated without ad￾ditional computational overhead. Meanwhile, our MLP dataflow optimization primarily involves storage reorg… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

72 extracted references · 26 linked inside Pith

  1. [1]

    Natural Language Processing Journal , volume=

    A survey of GPT-3 family large language models including ChatGPT and GPT-4 , author=. Natural Language Processing Journal , volume=. 2024 , publisher=

    2024

  2. [2]

    Bert: Pre-training of deep bidirectional transformers for language understanding , author=. Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers) , pages=

    2019

  3. [3]

    Advances in neural information processing systems , volume=

    Attention is all you need , author=. Advances in neural information processing systems , volume=

  4. [4]

    OpenAI blog , volume=

    Chatgpt: Optimizing language models for dialogue , author=. OpenAI blog , volume=

  5. [5]

    arXiv preprint arXiv:2201.08239 , year=

    Lamda: Language models for dialog applications , author=. arXiv preprint arXiv:2201.08239 , year=

    Pith/arXiv arXiv

  6. [6]

    arXiv preprint arXiv:2307.10169 , year=

    Challenges and applications of large language models , author=. arXiv preprint arXiv:2307.10169 , year=

    Pith/arXiv arXiv

  7. [7]

    IEEE Communications Surveys & Tutorials , volume=

    End-edge-cloud collaborative computing for deep learning: A comprehensive survey , author=. IEEE Communications Surveys & Tutorials , volume=. 2024 , publisher=

    2024

  8. [8]

    13th USENIX symposium on operating systems design and implementation (OSDI 18) , pages=

    Ray: A distributed framework for emerging \ AI \ applications , author=. 13th USENIX symposium on operating systems design and implementation (OSDI 18) , pages=

  9. [9]

    arXiv preprint arXiv:1909.08053 , year=

    Megatron-lm: Training multi-billion parameter language models using model parallelism , author=. arXiv preprint arXiv:1909.08053 , year=

    Pith/arXiv arXiv 1909

  10. [10]

    Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining , pages=

    Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters , author=. Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining , pages=

  11. [11]

    18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24) , pages=

    \ ServerlessLLM \ : \ Low-Latency \ serverless inference for large language models , author=. 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24) , pages=

  12. [12]

    Managing Requirements Knowledge, International Workshop on , pages=

    Artificial inteIligence on personal computers , author=. Managing Requirements Knowledge, International Workshop on , pages=. 1985 , organization=

    1985

  13. [13]

    Proceedings of the European Conference on Computer Vision (ECCV) Workshops , pages=

    Ai benchmark: Running deep neural networks on android smartphones , author=. Proceedings of the European Conference on Computer Vision (ECCV) Workshops , pages=

  14. [14]

    arXiv preprint arXiv:2410.03613 , year=

    Understanding Large Language Models in Your Pockets: Performance Study on COTS Mobile Devices , author=. arXiv preprint arXiv:2410.03613 , year=

    arXiv

  15. [15]

    2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA) , pages=

    Splitwise: Efficient generative llm inference using phase splitting , author=. 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA) , pages=. 2024 , organization=

    2024

  16. [16]

    arXiv preprint arXiv:2207.09238 , year=

    Formal algorithms for transformers , author=. arXiv preprint arXiv:2207.09238 , year=

    Pith/arXiv arXiv

  17. [17]

    Advances in Neural Information Processing Systems , volume=

    Kvquant: Towards 10 million context length llm inference with kv cache quantization , author=. Advances in Neural Information Processing Systems , volume=

  18. [18]

    Advances in neural information processing systems , volume=

    Q-vit: Accurate and fully quantized low-bit vision transformer , author=. Advances in neural information processing systems , volume=

  19. [19]

    Advances in neural information processing systems , volume=

    Zeroquant: Efficient and affordable post-training quantization for large-scale transformers , author=. Advances in neural information processing systems , volume=

  20. [20]

    arXiv preprint arXiv:2104.08500 , year=

    Vision transformer pruning , author=. arXiv preprint arXiv:2104.08500 , year=

    Pith/arXiv arXiv

  21. [21]

    Advances in Neural Information Processing Systems , volume=

    A fast post-training pruning framework for transformers , author=. Advances in Neural Information Processing Systems , volume=

  22. [22]

    Advances in neural information processing systems , volume=

    Flashattention: Fast and memory-efficient exact attention with io-awareness , author=. Advances in neural information processing systems , volume=

  23. [23]

    arXiv preprint arXiv:2307.08691 , year=

    Flashattention-2: Faster attention with better parallelism and work partitioning , author=. arXiv preprint arXiv:2307.08691 , year=

    Pith/arXiv arXiv

  24. [24]

    Advances in Neural Information Processing Systems , volume=

    Flashattention-3: Fast and accurate attention with asynchrony and low-precision , author=. Advances in Neural Information Processing Systems , volume=

  25. [25]

    arXiv preprint arXiv:2405.02803 , year=

    Is flash attention stable? , author=. arXiv preprint arXiv:2405.02803 , year=

    Pith/arXiv arXiv

  26. [26]

    Advances in Neural Information Processing Systems , volume=

    Efficientformer: Vision transformers at mobilenet speed , author=. Advances in Neural Information Processing Systems , volume=

  27. [27]

    Advances in neural information processing systems , volume=

    Long-short transformer: Efficient transformers for language and vision , author=. Advances in neural information processing systems , volume=

  28. [28]

    Advances in neural information processing systems , volume=

    Improved precision and recall metric for assessing generative models , author=. Advances in neural information processing systems , volume=

  29. [29]

    arXiv preprint arXiv:2503.18908 , year=

    Ffn fusion: Rethinking sequential computation in large language models , author=. arXiv preprint arXiv:2503.18908 , year=

    Pith/arXiv arXiv

  30. [30]

    Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 , pages=

    Flat: An optimized dataflow for mitigating attention bottlenecks , author=. Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 , pages=

  31. [31]

    IEEE Micro , year=

    Amd xdna™ npu in ryzen™ ai processors , author=. IEEE Micro , year=

  32. [32]

    2023 IEEE Hot Chips 35 Symposium (HCS) , pages=

    AMD Ryzen™ 7040 series: Technology overview , author=. 2023 IEEE Hot Chips 35 Symposium (HCS) , pages=. 2023 , organization=

    2023

  33. [33]

    arXiv preprint arXiv:2508.00904 , year=

    Forecasting LLM Inference Performance via Hardware-Agnostic Analytical Modeling , author=. arXiv preprint arXiv:2508.00904 , year=

    Pith/arXiv arXiv

  34. [34]

    arXiv preprint arXiv:2406.12793 , year=

    Chatglm: A family of large language models from glm-130b to glm-4 all tools , author=. arXiv preprint arXiv:2406.12793 , year=

    Pith/arXiv arXiv

  35. [35]

    arXiv preprint arXiv:2307.09288 , year=

    Llama 2: Open foundation and fine-tuned chat models , author=. arXiv preprint arXiv:2307.09288 , year=

    Pith/arXiv arXiv

  36. [36]

    arXiv preprint arXiv:2407.21783 , year=

    The llama 3 herd of models , author=. arXiv preprint arXiv:2407.21783 , year=

    Pith/arXiv arXiv

  37. [37]

    arXiv preprint arXiv:2412.08905 , year=

    Phi-4 technical report , author=. arXiv preprint arXiv:2412.08905 , year=

    Pith/arXiv arXiv

  38. [38]

    Hugging Face , title =

  39. [39]

    arXiv preprint arXiv:2012.14913 , year=

    Transformer feed-forward layers are key-value memories , author=. arXiv preprint arXiv:2012.14913 , year=

    Pith/arXiv arXiv 2012

  40. [40]

    arXiv preprint arXiv:2407.18003 , year=

    Keep the cost down: A review on methods to optimize LLM's KV-cache consumption , author=. arXiv preprint arXiv:2407.18003 , year=

    Pith/arXiv arXiv

  41. [41]

    Neural networks , volume=

    Sigmoid-weighted linear units for neural network function approximation in reinforcement learning , author=. Neural networks , volume=. 2018 , publisher=

    2018

  42. [42]

    arXiv preprint arXiv:2404.10933 , year=

    Llmem: Estimating gpu memory usage for fine-tuning pre-trained llms , author=. arXiv preprint arXiv:2404.10933 , year=

    Pith/arXiv arXiv

  43. [43]

    2025 IEEE International Symposium on High Performance Computer Architecture (HPCA) , pages=

    Dynamollm: Designing llm inference clusters for performance and energy efficiency , author=. 2025 IEEE International Symposium on High Performance Computer Architecture (HPCA) , pages=. 2025 , organization=

    2025

  44. [44]

    arXiv preprint arXiv:2406.08413 , year=

    Memory is all you need: An overview of compute-in-memory architectures for accelerating large language model inference , author=. arXiv preprint arXiv:2406.08413 , year=

    Pith/arXiv arXiv

  45. [45]

    2020 , publisher=

    Data orchestration in deep learning accelerators , author=. 2020 , publisher=

    2020

  46. [46]

    2020 , publisher=

    Efficient processing of deep neural networks , author=. 2020 , publisher=

    2020

  47. [47]

    Proceedings of the 52nd Annual IEEE/ACM International Symposium on Microarchitecture , pages=

    Understanding reuse, performance, and hardware cost of dnn dataflow: A data-centric approach , author=. Proceedings of the 52nd Annual IEEE/ACM International Symposium on Microarchitecture , pages=

  48. [48]

    Proceedings of the 29th symposium on operating systems principles , pages=

    Efficient memory management for large language model serving with pagedattention , author=. Proceedings of the 29th symposium on operating systems principles , pages=

  49. [49]

    Applied Intelligence , volume=

    A comprehensive review of model compression techniques in machine learning , author=. Applied Intelligence , volume=. 2024 , publisher=

    2024

  50. [50]

    arXiv preprint arXiv:2501.12948 , year=

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning , author=. arXiv preprint arXiv:2501.12948 , year=

    Pith/arXiv arXiv

  51. [51]

    arXiv preprint arXiv:2505.24133 , year=

    R-KV: Redundancy-aware KV Cache Compression for Training-Free Reasoning Models Acceleration , author=. arXiv preprint arXiv:2505.24133 , year=

    arXiv

  52. [52]

    arXiv preprint arXiv:2502.01941 , year=

    Can LLMs Maintain Fundamental Abilities under KV Cache Compression? , author=. arXiv preprint arXiv:2502.01941 , year=

    Pith/arXiv arXiv

  53. [53]

    Proceedings of the 52nd Annual International Symposium on Computer Architecture , pages=

    Insights into deepseek-v3: Scaling challenges and reflections on hardware for ai architectures , author=. Proceedings of the 52nd Annual International Symposium on Computer Architecture , pages=

  54. [54]

    arXiv preprint arXiv:2504.16084 , year=

    Ttrl: Test-time reinforcement learning , author=. arXiv preprint arXiv:2504.16084 , year=

    Pith/arXiv arXiv

  55. [55]

    Journal of King Saud University Computer and Information Sciences , volume=

    HiEE: Hierarchical feature fusion for Chinese event extraction , author=. Journal of King Saud University Computer and Information Sciences , volume=. 2025 , publisher=

    2025

  56. [56]

    arXiv preprint arXiv:2105.02723 , year=

    Do you even need attention? a stack of feed-forward layers does surprisingly well on imagenet , author=. arXiv preprint arXiv:2105.02723 , year=

    Pith/arXiv arXiv

  57. [57]

    arXiv preprint arXiv:2302.13971 , year=

    Llama: Open and efficient foundation language models , author=. arXiv preprint arXiv:2302.13971 , year=

    Pith/arXiv arXiv

  58. [58]

    arXiv preprint arXiv:2305.13245 , year=

    Gqa: Training generalized multi-query transformer models from multi-head checkpoints , author=. arXiv preprint arXiv:2305.13245 , year=

    Pith/arXiv arXiv

  59. [59]

    arXiv preprint arXiv:2309.16609 , year=

    Qwen technical report , author=. arXiv preprint arXiv:2309.16609 , year=

    Pith/arXiv arXiv

  60. [60]

    Proceedings of the 44th annual international symposium on computer architecture , pages=

    In-datacenter performance analysis of a tensor processing unit , author=. Proceedings of the 44th annual international symposium on computer architecture , pages=

  61. [61]

    Proceedings of the 42nd annual international symposium on computer architecture , pages=

    ShiDianNao: Shifting vision processing closer to the sensor , author=. Proceedings of the 42nd annual international symposium on computer architecture , pages=

  62. [62]

    ACM SIGARCH Computer Architecture News , volume=

    Diannao: A small-footprint high-throughput accelerator for ubiquitous machine-learning , author=. ACM SIGARCH Computer Architecture News , volume=. 2014 , publisher=

    2014

  63. [63]

    2014 47th Annual IEEE/ACM International Symposium on Microarchitecture , pages=

    Dadiannao: A machine-learning supercomputer , author=. 2014 47th Annual IEEE/ACM International Symposium on Microarchitecture , pages=. 2014 , organization=

    2014

  64. [64]

    ACM SIGARCH Computer Architecture News , volume=

    Pudiannao: A polyvalent machine learning accelerator , author=. ACM SIGARCH Computer Architecture News , volume=. 2015 , publisher=

    2015

  65. [65]

    arXiv preprint arXiv:1701.06538 , year=

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer , author=. arXiv preprint arXiv:1701.06538 , year=

    Pith/arXiv arXiv

  66. [66]

    arXiv preprint arXiv:2401.04088 , year=

    Mixtral of experts , author=. arXiv preprint arXiv:2401.04088 , year=

    Pith/arXiv arXiv

  67. [67]

    2022 IEEE International Conference on Mechatronics and Automation (ICMA) , pages=

    Feature Fusion Transformer Network for Natural Language Inference , author=. 2022 IEEE International Conference on Mechatronics and Automation (ICMA) , pages=. 2022 , organization=

    2022

  68. [68]

    2024 IEEE 42nd International Conference on Computer Design (ICCD) , pages=

    Vision Transformer Inference on a CNN Accelerator , author=. 2024 IEEE 42nd International Conference on Computer Design (ICCD) , pages=. 2024 , organization=

    2024

  69. [69]

    MLCommons Training Benchmarks , author =

  70. [70]

    2024 IEEE Hot Chips 36 Symposium (HCS) , organization =

    Lunar lake architecture session , author =. 2024 IEEE Hot Chips 36 Symposium (HCS) , organization =

    2024

  71. [71]

    AI Benchmarks Archive , author =

  72. [72]

    LFM-2 Benchmark Results , author =