Pith. sign in

REVIEW 3 major objections 7 minor 28 references

Hybrid Systolic Array Accelerator with Optimized Dataflow for Edge Large Language Model Inference

T0 review · 3 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A hybrid systolic array with 4-bit MXINT4 weights sustains 100 percent utilization during LLM decode and cuts memory traffic, reaching 247 token/s/mm2 on a 1.3B model.

desk verdict Genuinely interesting hybrid systolic array and a sound RMSNorm fusion, but the headline efficiency ratios rest on an undisclosed baseline reconstruction that needs full disclosure. read the letter →

arxiv 2507.09010 v1 pith:7JIM3QZW submitted 2025-07-11 cs.AR cs.AI

Chun-Ting Chen , HanGyeol Mun , Jian Meng , Mohamed S. Abdelfattah , Jae-sun Seo This is my paper
classification cs.ARcs.AI
keywords edgeLLMinferencehybridsystolicarrayMXINT4quantizationmatrix-vectormultiplicationdataflowRetNetRMSNormrotarypositionembeddinghardwareaccelerator
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Edge large-language-model inference is dominated by two mismatched workloads: compute-heavy prefill and memory-bound decode, and this paper's central claim is that one accelerator can serve both if it changes how it routes data on the fly. The authors propose a hybrid systolic array that reuses activations and weights during prefill like a conventional systolic array, then splits into four independent vector-style clusters during decode so all 256 MAC units stay busy even with a single token. To shrink the dominant decode memory traffic they quantize weights to 4-bit MXINT4 with shift-based scaling and perform dequantization inside the array at negligible cost. They further fuse RMSNorm into the next layer's scaling and compute RoPE angles online, removing buffers and DRAM reads. Running RetNet 1.3B under edge DRAM bandwidth, the design reports 247/117 token/s/mm2 in LISO/SILO scenarios, which they estimate is over 2.45x/13.5x better area efficiency than existing accelerators while keeping W4A8 accuracy close to W8A8.

What carries the argument

The load-bearing object is the hybrid systolic array (HSA): 256 PEs in four clusters of $4\times 16$ with a shared activation SRAM, bucket selectors, and 4-bit shifters. In prefill it runs an output-stationary MMM dataflow; in decode each PE cluster works independently so the array behaves as four parallel vector units, eliminating the low utilization that stalls a conventional systolic array when the batch is one. The MXINT4 scaling trick is the second piece of machinery: a 4-bit weight is shifted by the two low bits of the group scale and accumulated with enable-gating controlled by the two high bits, so dequantization happens while data moves across the array and the extra hardware is 0.4% of chip area. The RMSNorm and RoPE units are also named machinery: RMSNorm is fused into the next layer's quantization scale and bias, removing the input buffer and pipelining the $\sigma^{-1}$ computation; RoPE reuses the embedding multipliers to advance $\sin(m\theta_i)$ and $\cos(m\theta_i)$ via trigonometric identities between tokens.

What would settle it

Run the same RetNet 1.3B LISO/SILO workloads on the actual prior chips compared in Table II, with DRAM bandwidth capped at 51.2 GB/s, and compare measured token/s/mm2 against the paper's reconstructed values; if the measured values are close to or above the paper's numbers, the claimed >2.45x/13.5x margins do not hold.

Watch

Extended reading notes

Core claim

This paper claims that the right design point for an edge LLM accelerator is not peak TOPS but end-to-end area efficiency under a narrow DRAM budget, and that a single hardware structure can serve both phases of autoregressive inference. The proposed hybrid systolic array (HSA) operates as a conventional 2D systolic array during prefill matrix-matrix multiplication, reusing activations and weights, then switches each of its four PE clusters to independent matrix-vector multiplication during decode so that all 256 PEs stay busy with batch size one. MXINT4 weight quantization, with a shift-based scaling factor computed as $S_g = \lfloor \log_2(\max|W_g|) \rfloor$ and restricted to [-9, +5], halves weight memory traffic; dequantization is folded onto otherwise-idle PEs with bucket selectors and shifters, costing only 0.4% area. On post-layout 28nm simulation running RetNet 1.3B, the design reports 247 and 117 token/s/mm2 area efficiency for LISO and SILO settings, and 24.06 mJ/token decode energy, with W4A8 perplexity on WikiText2 close to the W8A8 SmoothQuant baseline and GSM8K accuracy within about 3 points of FP16.

Load-bearing premise

The speedup comparison with prior accelerators depends on reconstructed RetNet throughput numbers for chips that were never measured with RetNet; if those reconstructed numbers are too optimistic, the 2.45x/13.5x ratios shrink.

Editorial extensions

If this is right

  • With MXINT4 weights, decode-stage DRAM traffic is roughly halved, so token generation speed improves directly on bandwidth-limited edge devices without bigger SRAM or more PEs.
  • A 256-PE array is enough for a 1.3B model at 500 MHz: because decode is memory-bound, adding MACs would raise area without raising token throughput.
  • The fused RMSNorm pipeline removes a 32kB buffer and 5-10% of latency, enabling on-chip end-to-end inference instead of offloading normalization to a host.
  • The online RoPE unit avoids loading precomputed sin/cos tables from DRAM by reusing embedding multipliers with a trigonometric recurrence, cutting memory access and area (about 4% of total).
  • Because modern LLMs share the same MMM-prefill/MVM-decode pattern, the utilization and area-efficiency gains should transfer to transformer-based models, subject to their added attention memory traffic.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the same shift-based MXINT4 trick could be applied to the KV cache of attention-based LLMs, not just weights; that would extend the memory-traffic reduction to models with growing context, an extension the paper does not evaluate.
  • The 100% decode utilization is demonstrated for batch size one; if an edge device ever serves multiple concurrent users, the MVM dataflow would need to batch activations, and the 100% utilization claim may not carry over unchanged.
  • The RMSNorm fusion into the next layer's scaling factor suggests that other elementwise operations (for example GELU or SwiGLU) could be folded into adjacent quantized layers in the same way, reducing on-chip buffers further.
  • Because the comparison depends on reconstructed baselines, a useful follow-up would be a public, cycle-accurate scaling script; the core architectural claim of 100% utilization and 0.4% dequantization area is separable from the speedup-versus-prior-work claim.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper presents a 28nm edge LLM inference accelerator built around a hybrid systolic array (HSA) that combines conventional 2D systolic-array reuse for prefill (MMM) with a vector-like mode for decode (MVM). It uses MXINT4 weight quantization with shift-based scaling to reduce external memory access, fuses RMSNorm into the next layer's scaling to remove a 32kB buffer and latency, and computes RoPE sin/cos online. Post-layout results on RetNet 1.3B are reported as 247/117 token/s/mm2 under LISO/SILO, >2.45x/13.5x area efficiency over prior accelerators, and 24.06 mJ/token decode energy with claimed 100% hardware utilization during decode.

Significance. The RMSNorm fusion in Eq. (4) is a clean parameter-free algebraic transformation, and the shift-based MXINT4 dequantization is hardware-friendly: Table V shows a 10-16x area and 7-10x power advantage over INT8/FP16 dequantization units. The quantization evaluation uses held-out perplexity and GSM8K rather than tuning scaling factors against the headline token rates, which is a strength. The 100%-utilization MVM dataflow is a construction, not a fitted result, so the core architectural claim is in principle checkable. If the baseline reconstruction in Table II is fully disclosed, the work would be a solid contribution to memory-bound edge LLM inference. The main weakness is exactly that the quantitative headline depends on an undocumented estimation methodology for prior accelerators.

major comments (3)
  1. [Section V-B, Table II, footnote f] The headline area-efficiency ratios (2.45x/13.5x) are computed against prior accelerators that were not published with RetNet results, and the reconstruction method is not disclosed. The manuscript only states that a DDR5 51.2GB/s bandwidth was used to simulate edge decoding; it does not state how each design's PE count, SRAM capacity, dataflow, frequency, quantization format, and process scaling are converted to token/s/mm2. For example, Keller's 5nm-to-28nm area scaling (0.153 to 4.8 mm2) is given without any frequency or energy scaling rule, and MECLA's 80% memory-traffic reduction is not reflected in a traffic model. Because the claimed improvements are direct ratios to these reconstructed numbers, the authors should provide the estimation formulas, per-baseline parameters, and a sensitivity analysis (e.g., varying DRAM bytes/token by +/-20%) in an appendix.
  2. [Section IV-A2, Fig. 4(c)] The '100% hardware utilization' during decode is asserted but not supported by a cycle-level schedule or a formal definition of utilization. From Fig. 4(c) it is not clear whether all 256 PEs perform useful MACs every cycle, how the 4-bit shift and the Sw[3:2] clock-gating enable interact with the 16 PE rows, or how the vertical drain and partial-sum accumulation overlap with the next weight tile. Please provide a schedule for a representative RetNet MVM (with stated layer dimensions) showing no idle PE cycles, and clarify whether utilization counts activation loading and DRAM weight streaming.
  3. [Section V-B, Table II] The text claims 'superior energy efficiency in token generation' and reports 24.06 mJ/token decode energy, which is the best in Table II. However, the same table shows prefill energy with EMA of 0.773 mJ/token, which is higher than all listed baselines (0.246, 0.449, 0.685, 0.187, 0.148 mJ/token). Since LISO includes 750 prompt tokens, the end-to-end energy in that scenario will be worse than several baselines. The energy comparison should be reported on an end-to-end basis or explicitly scoped to the decode-only token-generation phase.
minor comments (7)
  1. [Table I, footnote a] 'Estimated based on HSA by considering the dataflow differences' is too vague to assess the conventional systolic array baseline; provide the estimation model or label the row as an approximate internal comparison.
  2. [Section IV-A1] The MMM dataflow text says only the weight SRAM in the first PC is activated, but Fig. 4(b) shows bucket selectors in all PCs; clarify whether only the top PC's SRAM is read or whether all PCs participate in weight reuse.
  3. [Section IV-B1, Eq. (4)] The notation Y* is nonstandard and easily confused with conjugation; rename it (e.g., Y') and state explicitly that Bn+1 can be precomputed offline because Wn+1, beta, and Sn+1 are known at deployment.
  4. [Fig. 3 caption] The energy assumption 'MAC=0.5pJ/Byte' is dimensionally odd; if this is energy per MAC or per accessed byte, correct the unit.
  5. [Section V-C, Table III] Add the weight group size used for each reported model and state whether the shift range [-9,+5] is ever saturated; the text gives 16 for the PE-matched case, but the table does not specify this per model.
  6. [Section IV-B2] There is a typo in the text: 'preparing for the the next token' should read 'preparing for the next token.'
  7. [Abstract and Section V-B] The abstract says 'superior energy efficiency' while Section V-B says 'competitive energy efficiency'; align the wording and specify the metric (decode energy per token versus end-to-end energy).

Circularity Check

0 steps flagged · score 0.0 of 10

No circular dependency found; the core results are independently constructed or evaluated, though Table II baseline estimates are under-disclosed.

full rationale

The paper's main claims are self-contained rather than circular. The 100% utilization MVM dataflow is a cycle-level property of the 256-PE hybrid systolic array design, not a fitted or calibrated output; it is reported directly from post-layout simulation and the dataflow construction. The RMSNorm fusion in Eq. (4) is a parameter-free algebraic identity: the normalization scale and bias are absorbed into the next layer's scaling factor and bias without being tuned against the headline token rates. The RoPE online sin/cos update is a trigonometric identity reusing existing multipliers, and the MXINT4 quantization scaling factors are computed from pretrained weights and evaluated on held-out WikiText2 and GSM8K accuracy, independent of the token/s/mm2 numbers. No load-bearing argument reduces to self-citation: the cited MX format [21], bucket-based dequantization inspiration [13], and compute-utility observation [27] are external or standard results, and the Moon et al. comparison [15]—despite co-author overlap—is an external chip baseline, not a justification of this paper's architecture. The one genuinely concerning passage is Table II's footnote f and Section V-B, where the prior accelerators' token/s/mm2 values are reconstructed under an assumed DDR5 51.2 GB/s bandwidth with no disclosed per-chip scaling model. This is a reproducibility and correctness risk: the claimed 2.45x/13.5x improvements depend on those reconstructed baselines and cannot be independently checked from the paper. However, that is an estimation-disclosure problem, not circularity: those baseline numbers are inputs to the comparison, not outputs of a fitting loop, and nothing in the paper's derivation chain defines a target quantity in terms of itself.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The paper's central numbers rest on several design choices such as group size and shift range, and on assumptions about generalization to other LLMs and fair comparison to prior chips. The quantization scaling factors are computed from model weights, not fitted to the reported performance target, and no invented physical entities are introduced.

free parameters (3)
  • Weight group size g = 16
    Hand-chosen to match the capacity of each PE along the output channel, affecting quantization granularity and hardware mapping.
  • MXINT4 shift range = [-9, +5]
    Constraint on the shifting-based scaling to prevent overflow during post-training quantization; a design choice, not fitted to the reported performance targets.
  • Scaling factor bit width Sw = 4-bit
    Defined by the MXINT4 format from the cited MX standard; the width is fixed by the chosen data format rather than tuned to the accuracy results.
assumptions (3)
  • domain assumption The efficiency gains measured on RetNet 1.3B transfer to other modern LLMs such as Llama.
    The paper asserts all LLMs share MMM and MVM workloads, but transformer-based models also require attention and softmax, which are not implemented or demonstrated end-to-end. This enters in Section I and the Conclusion.
  • domain assumption The prior-work comparisons in Table II are fair estimates under the same DDR5 bandwidth and RetNet 1.3B workload.
    The token/s/mm2 values for Keller, MECLA, FACT, Kim, and Moon are reconstructed under the authors' workload assumptions, but the estimation model is not fully specified.
  • domain assumption The decode dataflow achieves 100% hardware utilization across all tokens under the stated tiling and scheduling.
    The utilization claim is asserted as a contribution but no detailed proof or memory-scheduling derivation is provided in Section IV-A2.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Hybrid Systolic Array Accelerator with Optimized Dataflow for Edge Large Language Model Inference." pith.science (2026). https://pith.science/paper/7JIM3QZW

@misc{pith2026250709010,
  author       = {Pith},
  title        = {Pith review of: Hybrid Systolic Array Accelerator with Optimized Dataflow for Edge Large Language Model Inference},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7JIM3QZW}},
  note         = {Machine review of arXiv:2507.09010}
}
read the original abstract

Edge inference for large language models (LLM) offers secure, low-latency, and cost-effective inference solutions. We emphasize that an edge accelerator should achieve high area efficiency and minimize external memory access (EMA) during the memory-bound decode stage, while maintaining high energy efficiency during the compute intensive prefill stage. This paper proposes an edge LLM inference accelerator featuring a hybrid systolic array (HSA) architecture that optimizes inference efficiency in both stages. To further reduce EMA, we adopt MXINT4 weight quantization and propose an optimized dataflow tailored for HSA, ensuring negligible dequantization overhead and achieving 100% hardware utilization with minimal accuracy loss under edge DRAM bandwidth constraints. For non-linear operations, we incorporate optimized root mean square normalization (RMSNorm) and rotary position embedding (RoPE) units, reducing their latency, area, and memory access overhead while enabling end-to-end inference on our accelerator. Our solution achieves 247/117 (token/s/mm2) while running a 1.3B LLM on long-input/long-output scenarios, providing >2.45x/13.5x improvement over existing approaches, while maintaining superior energy efficiency in token generation.

Figures

Figures reproduced from arXiv: 2507.09010 by the authors.

Figure 3
Figure 3. Normalized latency/energy comparison of Llama [PITH_FULL_IMAGE:figures/full_fig_p002_3.png] view at source ↗
Figure 4
Figure 4. (a) Left: Proposed hybrid SA architecture. There are in total of 4 PE clusters arranged vertically with a unified activation SRAM. (b) Middle: MMM dataflow illustration, both Act and Wgt use INT8 format. (c) Right: MVM dataflow illustration, where each PC works independently in MVM since there is no weight reuse. MXINT4 format is used for Wgt. and processing element, we follow the trend of recent state-of￾the-art (S… view at source ↗
Figure 5
Figure 5. Comparison of normalization pipeline between [PITH_FULL_IMAGE:figures/full_fig_p004_5.png] view at source ↗
Figures from the paper (2 more)
Figure 6
Figure 6. Figure 6: Proposed RoPE unit. Angle memory stores sin, cos [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]
Figure 7
Figure 7. Figure 7: 28nm accelerator layout and area breakdown. [PITH_FULL_IMAGE:figures/full_fig_p005_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

28 extracted references · 17 canonical work pages

  1. [1]

    Phi-3 technical report: A highly capable language model locally on your phone,

    M. Abdin, J. Aneja, H. Awadalla, A. Awadallah, A. A. Awan, N. Bach, A. Bahree, A. Bakhtiari, J. Bao, H. Behl et al. , “Phi-3 technical report: A highly capable language model locally on your phone,” arXiv preprint arXiv:2404.14219, 2024

    arXiv 2024

  2. [2]

    Architecting an energy-efficient dram system for gpus,

    N. Chatterjee, M. O’Connor, D. Lee, D. R. Johnson, S. W. Keckler, M. Rhu, and W. J. Dally, “Architecting an energy-efficient dram system for gpus,” in 2017 IEEE International Symposium on High Performance Computer Architecture (HPCA) . IEEE, 2017, pp. 73–84

    work page 2017

  3. [3]

    Sparc: Token similarity-aware sparse attention transformer accelerator via row- wise clustering,

    H. Cho, D. Kim, S.-E. Hwang, and J. Park, “Sparc: Token similarity-aware sparse attention transformer accelerator via row- wise clustering,” in Proceedings of the 61st ACM/IEEE Design Automation Conference , ser. DAC ’24. New York, NY , USA: Association for Computing Machinery, 2024. [Online]. Available: https://doi.org/10.1145/3649329.3655936

    work page arXiv 2024

  4. [4]

    Vs-quant: Per-vector scaled quantization for accurate low-precision neural network inference,

    S. Dai, R. Venkatesan, M. Ren, B. Zimmer, W. Dally, and B. Khailany, “Vs-quant: Per-vector scaled quantization for accurate low-precision neural network inference,” Proceedings of Machine Learning and Sys- tems, vol. 3, pp. 873–884, 2021

    work page 2021

  5. [5]

    Transformers are ssms: generalized models and efficient algorithms through structured state space duality,

    T. Dao and A. Gu, “Transformers are ssms: generalized models and efficient algorithms through structured state space duality,” in Proceed- ings of the 41st International Conference on Machine Learning , ser. ICML’24. JMLR.org, 2024

    work page 2024

  6. [6]

    The llama 3 herd of models,

    A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan et al. , “The llama 3 herd of models,” arXiv preprint arXiv:2407.21783 , 2024

    arXiv 2024

  7. [7]

    Gptq: Accurate post-training quantization for generative pre-trained transformers,

    E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh, “Gptq: Accurate post-training quantization for generative pre-trained transformers,” arXiv preprint arXiv:2210.17323, 2022

    arXiv 2022

  8. [8]

    Ita: An energy-efficient attention and softmax accelerator for quantized transformers,

    G. Islamoglu, M. Scherer, G. Paulin, T. Fischer, V . J. Jung, A. Garofalo, and L. Benini, “Ita: An energy-efficient attention and softmax accelerator for quantized transformers,” in 2023 IEEE/ACM International Sympo- sium on Low Power Electronics and Design (ISLPED) . IEEE, 2023, pp. 1–6

    work page 2023

Show all 28 references
  1. [9]

    A 95.6-tops/w deep learning inference accelerator with per-vector scaled 4-bit quantization in 5 nm,

    B. Keller, R. Venkatesan, S. Dai, S. G. Tell, B. Zimmer, C. Sakr, W. J. Dally, C. T. Gray, and B. Khailany, “A 95.6-tops/w deep learning inference accelerator with per-vector scaled 4-bit quantization in 5 nm,” IEEE Journal of Solid-State Circuits , vol. 58, no. 4, pp. 1129–1141, 2023

    2023

  2. [10]

    A modular digital vlsi flow for high-productivity soc design,

    B. Khailany, E. Khmer, R. Venkatesan, J. Clemons, J. S. Emer, M. Fojtik, A. Klinefelter, M. Pellauer, N. Pinckney, Y . S. Shao et al. , “A modular digital vlsi flow for high-productivity soc design,” in Proceedings of the 55th Annual Design Automation Conference , 2018, pp. 1–6

    2018

  3. [11]

    S. Kim, S. Kim, W. Jo, S. Kim, S. Hong, and H.-J. Yoo, “20.5 c- transformer: A 2.6-18.1 µj/token homogeneous dnn-transformer/spiking- transformer processor with big-little network and implicit weight gener- ation for large language models,” in 2024 IEEE International Solid-Sta...

    2024

  4. [12]

    Qserve: W4a8kv4 quantization and system co-design for efficient llm serving,

    Y . Lin, H. Tang, S. Yang, Z. Zhang, G. Xiao, C. Gan, and S. Han, “Qserve: W4a8kv4 quantization and system co-design for efficient llm serving,” arXiv preprint arXiv:2405.04532 , 2024

    2024 arXiv

  5. [13]

    Bucket getter: A bucket-based processing engine for low-bit block floating point (bfp) dnns,

    Y .-C. Lo and R.-S. Liu, “Bucket getter: A bucket-based processing engine for low-bit block floating point (bfp) dnns,” in Proceedings of the 56th Annual IEEE/ACM International Symposium on Microarchitecture , 2023, pp. 1002–1015

    2023

  6. [14]

    Pointer sentinel mixture models,

    S. Merity, C. Xiong, J. Bradbury, and R. Socher, “Pointer sentinel mixture models,” 2016

    2016

  7. [15]

    A 127.8 tops/w arbitrarily quantized 1-to-8b scalable-precision accelerator for general-purpose deep learning with reduction of storage, logic and latency waste,

    S. Moon, H.-G. Mun, H. Son, and J.-Y . Sim, “A 127.8 tops/w arbitrarily quantized 1-to-8b scalable-precision accelerator for general-purpose deep learning with reduction of storage, logic and latency waste,” in2023 IEEE International Solid-State Circuits Conference (ISSCC) . I...

    2023

  8. [16]

    Abstractive text summarization using sequence-to-sequence rnns and beyond,

    R. Nallapati, B. Zhou, C. Gulcehre, B. Xiang et al. , “Abstractive text summarization using sequence-to-sequence rnns and beyond,” arXiv preprint arXiv:1602.06023, 2016

    2016 arXiv

  9. [17]

    Nvidia jetson orin nano,

    Nvidia, “Nvidia jetson orin nano,” https://www.nvidia.com/en- us/autonomous-machines/embedded-systems/jetson-orin/, accessed: 19 November 2024

    2024

  10. [18]

    Fine-grained dram: Energy-efficient dram for extreme bandwidth systems,

    M. O’Connor, N. Chatterjee, D. Lee, J. Wilson, A. Agrawal, S. W. Keckler, and W. J. Dally, “Fine-grained dram: Energy-efficient dram for extreme bandwidth systems,” in Proceedings of the 50th Annual IEEE/ACM International Symposium on Microarchitecture , 2017, pp. 41–54

    2017

  11. [19]

    Fact: Ffn-attention co-optimized transformer architecture with eager correlation prediction,

    Y . Qin, Y . Wang, D. Deng, Z. Zhao, X. Yang, L. Liu, S. Wei, Y . Hu, and S. Yin, “Fact: Ffn-attention co-optimized transformer architecture with eager correlation prediction,” in Proceedings of the 50th Annual International Symposium on Computer Architecture , 2023, pp. 1–14

    2023

  12. [20]

    Mecla: Memory-compute-efficient llm accelerator with scaling sub-matrix partition,

    Y . Qin, Y . Wang, Z. Zhao, X. Yang, Y . Zhou, S. Wei, Y . Hu, and S. Yin, “Mecla: Memory-compute-efficient llm accelerator with scaling sub-matrix partition,” in 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA) . IEEE, 2024, pp. 1032– 1047

    2024

  13. [21]

    Microscaling data formats for deep learning,

    B. D. Rouhani, R. Zhao, A. More, M. Hall, A. Khodamoradi, S. Deng, D. Choudhary, M. Cornea, E. Dellinger, K. Denolf et al., “Microscaling data formats for deep learning,” arXiv preprint arXiv:2310.10537 , 2023

    2023 arXiv

  14. [22]

    Roformer: En- hanced transformer with rotary position embedding,

    J. Su, M. Ahmed, Y . Lu, S. Pan, W. Bo, and Y . Liu, “Roformer: En- hanced transformer with rotary position embedding,” Neurocomputing, vol. 568, p. 127063, 2024

    2024

  15. [23]

    Retentive network: A successor to transformer for large language models,

    Y . Sun, L. Dong, S. Huang, S. Ma, Y . Xia, J. Xue, J. Wang, and F. Wei, “Retentive network: A successor to transformer for large language models,” arXiv preprint arXiv:2307.08621 , 2023

    2023 arXiv

  16. [24]

    Llama 2: Open foundation and fine-tuned chat models,

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale et al. , “Llama 2: Open foundation and fine-tuned chat models,” arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  17. [25]

    Sole: Hardware-software co-design of softmax and layernorm for efficient transformer inference,

    W. Wang, S. Zhou, W. Sun, P. Sun, and Y . Liu, “Sole: Hardware-software co-design of softmax and layernorm for efficient transformer inference,” in 2023 IEEE/ACM International Conference on Computer Aided Design (ICCAD). IEEE, 2023, pp. 1–9

    2023

  18. [26]

    Smoothquant: Accurate and efficient post-training quantization for large language models,

    G. Xiao, J. Lin, M. Seznec, H. Wu, J. Demouth, and S. Han, “Smoothquant: Accurate and efficient post-training quantization for large language models,” in International Conference on Machine Learning . PMLR, 2023, pp. 38 087–38 099

    2023

  19. [27]

    Llmcompass: Enabling efficient hardware design for large language model inference,

    H. Zhang, A. Ning, R. B. Prabhakar, and D. Wentzlaff, “Llmcompass: Enabling efficient hardware design for large language model inference,” in 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA). IEEE, 2024, pp. 1080–1096

    2024

  20. [28]

    Atom: Low-bit quantization for efficient and accurate llm serving,

    Y . Zhao, C.-Y . Lin, K. Zhu, Z. Ye, L. Chen, S. Zheng, L. Ceze, A. Krishnamurthy, T. Chen, and B. Kasikci, “Atom: Low-bit quantization for efficient and accurate llm serving,” Proceedings of Machine Learning and Systems (MLSys) , 2024

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.