Pith. sign in

REVIEW 2 major objections 5 minor 68 references

A Transformer co-designed for FPGAs combines hybrid linear attention with ternary weights to cut model size 10× and KV cache 12.8× while beating GPU latency and energy at long context.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 00:54 UTC pith:AWYUJOZ3

load-bearing objection Solid first FPGA realization of hybrid-linear + ternary Transformers with a clean DSP-free PE; the headline 3.9 imes/3.2 imes numbers are real for the scaled board but do not yet prove production-scale multi-batch serving. the 2 major comments →

arxiv 2607.03652 v1 pith:AWYUJOZ3 submitted 2026-07-04 cs.AR cs.AI

ELiTeFormer: An Efficient Transformer for FPGAs

Victor Agostinelli , Nicolas Bohm Agostini , Antonino Tumeo This is my paper
classification cs.AR cs.AI
keywords FPGA LLM accelerationlinear attentionternary quantizationBitNet b1.58multiplier-free PEKV-cache compressionalgorithm-architecture co-design
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Standard Transformers waste silicon and energy because attention scales with sequence length and feed-forward layers still use full-precision multiplies. ELiTeFormer replaces both bottlenecks at once: hybrid linear-plus-sliding-window attention keeps the key-value state constant-size, and every linear projection is restricted to ternary weights so multiplies disappear. The authors then build a processing element that turns those ternary multiplies into simple bit-masking and accumulation, eliminating DSP blocks on an FPGA. On a Versal board the design delivers roughly 4× lower latency and 3× better energy than a comparable LLaMA-3 model on an A100 once context grows long, while MMLU stays within three points of the pure ternary baseline. The result shows that algorithm and FPGA datapath can be co-designed so that extreme compression becomes a hardware advantage rather than a software afterthought.

Core claim

Jointly applying hybrid linear attention and BitNet-style ternary linear projections yields a Transformer whose weights compress 10 imes and whose KV cache compresses 12.8 imes relative to LLaMA 3, yet still reaches 31.9 % MMLU; a multiplier-free PE that realizes the ternary MACs with bit-masks then lets an FPGA deliver up to 3.9 imes lower latency and 3.2 imes better energy than the same-scale model on an A100 at long contexts.

What carries the argument

The ELTF processing element: a micro-architecture that indexes ternary weight bits to select pass-through, negation or zero masks, then reduces them with an adder tree, thereby eliminating every multiplier and every DSP block from the linear projections.

Load-bearing premise

That the accuracy and speedups measured on a heavily scaled-down 1 B-parameter FPGA deployment and on 8 B simulations will hold for a full-scale production model under realistic multi-batch serving loads.

What would settle it

Deploy a full-size ELiTeFormer-8B (or larger) end-to-end on the same Versal board, measure wall-clock latency and energy per token at 16 k context against an identically sized LLaMA-3 baseline on A100, and check whether the claimed 3.9 imes / 3.2 imes advantages and the MMLU gap remain within three points.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. ELiTeFormer proposes a Transformer architecture that jointly combines hybrid linear attention (Hedgehog-style linear attention plus a sliding-window sparse component) with BitNet b1.58-style ternary linear projections, co-designed for FPGA generative inference. The design claims 10 imes model-weight and 12.8 imes KV-cache compression versus LLaMA 3 while remaining within 3 % MMLU of BitNet b1.58, and introduces a multiplier-free PE (ELTF PE) that replaces ternary multiplies by bitmasking so that linear projections consume no DSP blocks. The authors distill an 8 B checkpoint via a modified LoLCATs workflow, report block-level cycle counts (9.6 imes FFN, 4.4 imes attention) via Bambu HLS, and present both Vitis-HLS system estimates for an 8 B design and measured latency/power for a scaled-down ~1 B deployment (d_FFN=8096, L=6) on a Xilinx VCK5000, claiming up to 3.9 imes lower latency and 3.2 imes better energy efficiency than LLaMA 3 on an A100 at long contexts.

Significance. If the joint algorithmic–architectural claims hold at production scale, the work would be the first concrete demonstration that hybrid linear attention and ternary projections can be co-designed into a DSP-free FPGA datapath that simultaneously compresses weights and the KV cache while remaining competitive on standard LLM benchmarks. The explicit PE micro-architecture, the HLS-to-board flow, and the side-by-side comparison against vLLM and bitnet.cpp baselines are concrete engineering contributions that the community can build upon. The paper also surfaces a useful negative result—that post-distillation LoRA fine-tuning degrades rather than improves the ternary model—which is valuable for future distillation pipelines.

major comments (2)
  1. Abstract and §4.3 claim end-to-end 3.9× latency and 3.2× energy gains versus LLaMA 3 on an A100. The only board measurements are for a heavily scaled-down 1 B model (d_FFN=8096, L=6, 0.19 GB weights) on the VCK5000’s DDR fabric; the 8 B numbers are Vitis HLS system estimations that never leave simulation (Table 4). Because the design is memory-bound (Table 6: attn stage alone uses 60 % LUTs / 80 % BRAM; off-chip 512-bit AXI bursts dominate), latency and energy scale with DRAM bandwidth, bank conflicts and KV-cache pressure that grow with layer count and multi-batch serving. The paper therefore does not yet demonstrate the headline end-to-end gains for the 8 B model it claims to accelerate. A full-scale placement/routing result, or at least a bandwidth-normalized multi-batch study that isolates the PE contribution from the memory hierarchy, is required before those numbers can be treated
  2. §4.1 and Table 3 document a substantial quality gap relative to full-precision LLaMA 3 (MMLU 31.9 % vs 66.6 %) and a non-negligible drop versus the BitNet b1.58 teacher (34.9 %). The authors themselves note a “quality-of-data moat” and that post-distillation LoRA only hurt perplexity (Table 2). While the 3 % MMLU gap to BitNet is correctly stated, the abstract’s phrasing “maintaining competitive accuracy” is therefore overstated for a general LLM audience. Either additional distillation / continued pre-training results that close the gap, or a clearer framing that the accuracy claim is relative only to the ternary teacher, is needed for the central quality claim to be load-bearing.
minor comments (5)
  1. Figure 4 reports cycle counts but does not state the target clock frequency or the precise Bambu HLS configuration; adding these would make the 9.6× / 4.4× speed-ups reproducible.
  2. Table 7 normalizes throughput to U280 HBM bandwidth for a memory-bound design; the scaling factor and the assumption that the design remains memory-bound after the bandwidth increase should be stated explicitly.
  3. The packing scheme is described as “8× compression” (4 ternary weights per 8-bit frame) while the abstract claims “10×”; a short clarification of the theoretical versus implemented packing would remove the inconsistency.
  4. Equation (3) drops the √d scaling “for simplicity”; a parenthetical note that the same scaling is retained in the actual implementation would avoid confusion.
  5. Several citations appear only as arXiv numbers without venue or year (e.g., FlightLLM, LoLCATs); completing the bibliographic entries would improve archival quality.

Circularity Check

0 steps flagged

No circularity: compression ratios and speed-ups are direct architectural consequences and measured wall-clock/cycle results against external baselines, not self-referential derivations.

full rationale

ELiTeFormer is an empirical HW/SW co-design paper. The 10× weight compression follows immediately from BitNet-style ternary packing (explicitly 4 weights per 8-bit frame, §3.4.1); the 12.8× KV-cache compression follows from the fixed-size recurrent state of hybrid linear+sliding-window attention (Eq. 3 and Fig. 1). Latency (9.6× FFN, 4.4× attention in Bambu simulation; up to 3.9× end-to-end on VCK5000) and energy (3.2× tokens/J) numbers are obtained by synthesizing the design, running it, and comparing against independent LLaMA-3/vLLM and BitNet/bitnet.cpp baselines on an A100. No free parameter is fitted to a data subset and then re-presented as a prediction; no uniqueness theorem or ansatz is imported solely via self-citation to force the result; the novel PE (Fig. 3) simply replaces ternary multiplies by bit-masking and is validated by resource reports (zero DSPs). The derivation chain is therefore self-contained and externally falsifiable.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 1 invented entities

The central claims rest on standard Transformer and FPGA assumptions plus a handful of design choices (window size, mini-batch dimension, compression packing) that are free parameters of the implementation. No new physical entities are postulated; the ELTF PE is an engineered micro-architecture, not an invented scientific entity.

free parameters (4)
  • sliding_window_size = 256
    Chosen as 256 tokens after ablation (Table 1); directly controls both accuracy and the constant-size cache footprint.
  • mini_batch_dimension_mB = 2
    Set to 2 (or 4 in illustrations) to trade BRAM for weight reuse; free design knob that affects reported latency.
  • learning_rate_schedule = 2e-5 → 1e-7
    Starts at 2e-5, decays to 1e-7 via inverse-square; standard but fitted for the distillation run.
  • weight_packing_scheme = 4 weights / 8 bits
    8-bit frames holding 4 ternary weights (8× compression); more aggressive 5/3 packing mentioned but not used.
axioms (4)
  • domain assumption Linear attention with a fixed-size recurrent state plus a fixed sliding window yields a static, a-priori-known memory footprint suitable for HLS static compilation.
    Stated in §2.4 and §3.1 as the reason FPGAs prefer linear/hybrid attention over classical quadratic attention.
  • standard math Ternary weights (−1,0,+1) can be emulated exactly by bit-masking and sign flips, eliminating the need for DSP multipliers.
    Used to justify the ELTF PE micro-architecture in §3.4.2.
  • domain assumption Prefilling can be off-loaded to a heterogeneous host (GPU/CPU) so the FPGA only performs generation.
    Explicit design choice in §3 that removes unbounded-generation complications from the FPGA datapath.
  • domain assumption RMSNorm preceding each BitNet projection forces a full-token synchronization barrier between pipeline stages.
    Explains the non-overlapped stage boundaries in the top-level data-flow (§3.3).
invented entities (1)
  • ELTF PE (ELiTeFormer Processing Element) independent evidence
    purpose: Multiplier-free micro-architecture that realizes ternary MAC via indexed bit-masks and a reduction tree.
    Core hardware contribution; engineered construct rather than a new physical or mathematical object. Independent evidence is the synthesized resource counts and measured speed-ups on the VCK5000.

pith-pipeline@v1.1.0-grok45 · 26309 in / 3199 out tokens · 28371 ms · 2026-07-12T00:54:27.438874+00:00 · methodology

0 comments
read the original abstract

Transformer blocks are prevalent in large language model (LLM) but present deployment challenges due to their challenging computational and memory demands. While prior work has typically optimized attention mechanisms or feed-forward networks (FFNs) separately, few hardware (HW) architecture have jointly addressed both components with co-designed hardware acceleration. We present ELiTeFormer (Efficient Linear Ternary Transformer), the first Transformer model architecture that unifies hybrid linear attention with ultra-low-precision (ternary) linear projections, specifically co-designed for field-programmable gate array (FPGA) deployment. ELiTeFormer achieves 10x model weight compression and 12.8x key-value (KV) cache compression compared to LLaMA 3, while maintaining competitive accuracy (31.9% on the MMLU benchmark, within 3.0% of BitNet b1.58). Our key architectural contribution is a novel processing element (PE) micro-architecture that eliminates all multiplications in ternary linear projections through bitmasking operations, significantly reducing resource utilization by completely avoiding dedicated digital signal processing (DSP) blocks. We simulate, synthesize, and deploy ELiTeFormer targeting a Xilinx VCK5000 Versal board using high-level synthesis (HLS) flows. Block-level simulations show 9.6x speedup for FFN operations and 4.4x speedup for attention compared to standard implementations. End-to-end deployment achieves up to 3.9x lower latency and 3.2x better energy efficiency than LLaMA 3 on an NVIDIA A100 graphics processing unit (GPU) at long context lengths. This represents the first FPGA realization combining linear attention with ternary quantization, demonstrating the viability of algorithm-architecture co-design for next-generation LLM acceleration.

Figures

Figures reproduced from arXiv: 2607.03652 by Antonino Tumeo, Nicolas Bohm Agostini, Victor Agostinelli.

Figure 1
Figure 1. Figure 1: Depiction of ELiTeFormer’s attention block during generative time-steps. Hedgehog Linear Attention [59] + Sliding [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: An example illustration of the timing diagram for [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: An illustration of the proposed ELiTeFormer linear projection architecture (left) and PE-specific architecture (right) [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: A simulated comparison of latency in cycles (k) (lower is better) between a typical linear projection block and [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Observed latency (ms) (lower is better) on a single [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

68 extracted references · 7 canonical work pages · 1 internal anchor

  1. [1]

    Victor Agostinelli and Lizhong Chen. 2023. Improving Autoregressive NLP Tasks via Modular Linearized Attention. InMachine Learning and Knowledge Discovery in Databases: Research Track, Danai Koutra, Claudia Plant, Manuel Gomez Rodriguez, Elena Baralis, and Francesco Bonchi (Eds.). Springer Nature Switzerland, Cham, 90–106

    2023

  2. [2]

    Victor Agostinelli, Sanghyun Hong, and Lizhong Chen. 2024. LeaPformer: En- abling Linear Transformers for Autoregressive and Simultaneous Tasks via Learned Proportions. InForty-first International Conference on Machine Learning. https://openreview.net/forum?id=XhH1OKLANY

    2024

  3. [3]

    Simran Arora, Sabri Eyuboglu, Michael Zhang, Aman Timalsina, Silas Al- berti, Dylan Zinsley, James Zou, Atri Rudra, and Christopher Ré. 2024. Sim- ple linear attention language models balance the recall-throughput tradeoff. arXiv:2402.18668 [cs.CL] https://arxiv.org/abs/2402.18668

    Pith/arXiv arXiv 2024

  4. [4]

    Peters, and Arman Cohan

    Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The Long- Document Transformer. https://doi.org/10.48550/ARXIV.2004.05150

    Pith/arXiv arXiv doi:10.48550/arxiv.2004.05150 2020

  5. [5]

    Hongzheng Chen, Jiahao Zhang, Yixiao Du, Shaojie Xiang, Zichao Yue, Niansong Zhang, Yaohui Cai, and Zhiru Zhang. 2024. Understanding the Potential of FPGA- Based Spatial Acceleration for Large Language Model Inference.ACM Trans. Reconfigurable Technol. Syst.(April 2024). https://doi.org/10.1145/3656177 Just Accepted. 11 , , Victor Agostinelli, Nicolas Boh...

    doi:10.1145/3656177 2024

  6. [6]

    Zixiang Chen, Yihe Deng, Yue Wu, Quanquan Gu, and Yuanzhi Li. 2022. Towards Understanding the Mixture-of-Experts Layer in Deep Learning. In Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Vol. 35. Curran Associates, Inc., 23049–23062. https://proceedings.neurips.cc/paper_files/pa...

    2022

  7. [7]

    Zhaodong Chen, Zheng Qu, Yuying Quan, Liu Liu, Yufei Ding, and Yuan Xie

  8. [8]

    InProceedings of the 28th ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programming(Montreal, QC, Canada)(PPoPP ’23)

    Dynamic N:M Fine-Grained Structured Sparse Attention Mechanism. InProceedings of the 28th ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programming(Montreal, QC, Canada)(PPoPP ’23). Association for Computing Machinery, New York, NY, USA, 369–379. https://doi.org/10. 1145/3572848.3577500

    arXiv

  9. [9]

    Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019. Generating Long Sequences with Sparse Transformers. https://doi.org/10.48550/ARXIV.1904.10509

    Pith/arXiv arXiv doi:10.48550/arxiv.1904.10509 2019

  10. [10]

    Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, David Belanger, Lucy Colwell, and Adrian Weller. 2020. Rethinking Attention with Performers. https://doi.org/10.48550/ARXIV.2009.14794

    Pith/arXiv arXiv doi:10.48550/arxiv.2009.14794 2020

  11. [11]

    Jyotikrishna Dass, Shang Wu, Huihong Shi, Chaojian Li, Zhifan Ye, Zhongfeng Wang, and Yingyan Lin. 2023. ViTALiTy: Unifying Low-rank and Sparse Ap- proximation for Vision Transformer Acceleration with a Linear Taylor Attention. In2023 IEEE International Symposium on High-Performance Computer Architec- ture (HPCA). 415–428. https://doi.org/10.1109/HPCA5654...

    doi:10.1109/hpca56546.2023.10071081 2023

  12. [12]

    Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer

  13. [13]

    arXiv:2208.07339 [cs.LG] https://arxiv.org/abs/2208.07339

    LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale. arXiv:2208.07339 [cs.LG] https://arxiv.org/abs/2208.07339

    Pith/arXiv arXiv

  14. [14]

    Tim Dettmers and Luke Zettlemoyer. 2023. The case for 4-bit precision: k-bit inference scaling laws(ICML’23). JMLR.org, Article 307, 25 pages

    2023

  15. [15]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805 [cs.CL] https://arxiv.org/abs/1810.04805

    Pith/arXiv arXiv 2019

  16. [16]

    Harry Dong, Xinyu Yang, Zhenyu Zhang, Zhangyang Wang, Yuejie Chi, and Beidi Chen. 2024. Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference. InProceedings of the 41st Interna- tional Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 235), Ruslan Salakhutdinov, Zico Kolter, Katherin...

    2024

  17. [17]

    Hongxiang Fan, Thomas Chau, Stylianos I Venieris, Royson Lee, Alexandros Kouris, Wayne Luk, Nicholas D Lane, and Mohamed S Abdelfattah. 2022. Adapt- able Butterfly Accelerator for Attention-based NNs via Hardware and Algorithm Co-design. InMICRO-55: 55th Annual IEEE/ACM International Symposium on Microarchitecture

    2022

  18. [18]

    Chao Fang, Shouliang Guo, Wei Wu, Jun Lin, Zhongfeng Wang, Ming Kai Hsu, and Lingzhi Liu. 2022. An Efficient Hardware Accelerator for Sparse Transformer Neural Networks. In2022 IEEE International Symposium on Circuits and Systems (ISCAS). 2670–2674. https://doi.org/10.1109/ISCAS48785.2022.9937659

    doi:10.1109/iscas48785.2022.9937659 2022

  19. [19]

    Chao Fang, Aojun Zhou, and Zhongfeng Wang. 2022. An Algorithm–Hardware Co-Optimized Framework for Accelerating N:M Sparse Transformers.IEEE Transactions on Very Large Scale Integration (VLSI) Systems30, 11 (2022), 1573–

    2022

  20. [20]

    https://doi.org/10.1109/TVLSI.2022.3197282

    work page doi:10.1109/tvlsi.2022.3197282 2022

  21. [21]

    Fabrizio Ferrandi, Vito Giovanni Castellana, Serena Curzel, Pietro Fezzardi, Michele Fiorito, Marco Lattuada, Marco Minutoli, Christian Pilato, and Antonino Tumeo. 2021. Invited: Bambu: an Open-Source Research Framework for the High-Level Synthesis of Complex Applications. In2021 58th ACM/IEEE Design Au- tomation Conference (DAC). IEEE, 1327–1330. https:/...

    work page doi:10.1109/dac18074 2021

  22. [22]

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2024. A fra...

    doi:10.5281/zenodo.12608602 2024

  23. [23]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, and al et. 2024. The Llama 3 Herd of Models. arXiv:2407.21783 [cs.AI] https://arxiv.org/abs/2407.21783

    Pith/arXiv arXiv 2024

  24. [24]

    Oh, Yeonhong Park, Yoonho Song, Jung-Hun Park, Sanghee Lee, Kyoung Park, Jae W

    Tae Jun Ham, Sungjun Jung, Seonghak Kim, Young H. Oh, Yeonhong Park, Yoonho Song, Jung-Hun Park, Sanghee Lee, Kyoung Park, Jae W. Lee, and Deog-Kyoon Jeong. 2020. 𝐴3: Accelerating Attention Mechanisms in Neural Networks with Ap- proximation.2020 IEEE International Symposium on High Performance Computer Architecture (HPCA)(2020), 328–341. https://api.seman...

    2020

  25. [25]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-Rank Adaptation of Large Language Models. InInternational Conference on Learning Representations. https: //openreview.net/forum?id=nZeVKeeFYf9

    2022

  26. [26]

    AMD Inc. 2025. AMD Vitis Unified Software Platform. https://www.amd.com/ en/products/software/adaptive-socs-and-fpgas/vitis.html. Accessed: 2025-01-17

    2025

  27. [27]

    Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. 2024. A Survey on Large Language Models for Code Generation. arXiv:2406.00515 [cs.CL] https://arxiv.org/abs/2406.00515

    Pith/arXiv arXiv 2024

  28. [28]

    Praneeth Kacham, Vahab Mirrokni, and Peilin Zhong. 2024. PolySketchFormer: Fast Transformers via Sketching Polynomial Kernels. arXiv:2310.01655 [cs.LG] https://arxiv.org/abs/2310.01655

    Pith/arXiv arXiv 2024

  29. [29]

    Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret

  30. [30]

    https://doi.org/10.48550/ARXIV.2006.16236

    Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention. https://doi.org/10.48550/ARXIV.2006.16236

    Pith/arXiv arXiv doi:10.48550/arxiv.2006.16236 2006

  31. [31]

    Hamza Khan, Asma Khan, Zainab Khan, Lun Bin Huang, Kun Wang, and Lei He

  32. [32]

    InThe 2021 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays(Virtual Event, USA)(FPGA ’21)

    NPE: An FPGA-based Overlay Processor for Natural Language Processing. InThe 2021 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays(Virtual Event, USA)(FPGA ’21). Association for Computing Machinery, New York, NY, USA, 227. https://doi.org/10.1145/3431920.3439477

    work page doi:10.1145/3431920.3439477 2021

  33. [33]

    Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The Efficient Transformer. https://doi.org/10.48550/ARXIV.2001.04451

    Pith/arXiv arXiv doi:10.48550/arxiv.2001.04451 2020

  34. [34]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Mem- ory Management for Large Language Model Serving with PagedAttention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles

    2023

  35. [35]

    Bingbing Li, Santosh Pandey, Haowen Fang, Yanjun Lyv, Ji Li, Jieyang Chen, Mimi Xie, Lipeng Wan, Hang Liu, and Caiwen Ding. 2020. FTRANS: energy-efficient acceleration of transformers using FPGA. InProceedings of the ACM/IEEE Inter- national Symposium on Low Power Electronics and Design(Boston, Massachusetts) (ISLPED ’20). Association for Computing Machin...

    work page doi:10.1145/3370748.3406567 2020

  36. [36]

    Mushui Liu, Yuhang Ma, Yang Zhen, Jun Dan, Yunlong Yu, Zeng Zhao, Zhipeng Hu, Bai Liu, and Changjie Fan. 2024. LLM4GEN: Leveraging Semantic Repre- sentation of LLMs for Text-to-Image Generation. arXiv:2407.00737 [cs.CV] https://arxiv.org/abs/2407.00737

    Pith/arXiv arXiv 2024

  37. [37]

    Shuming Ma, Hongyu Wang, Lingxiao Ma, Lei Wang, Wenhui Wang, Shaohan Huang, Li Dong, Ruiping Wang, Jilong Xue, and Furu Wei. 2024. The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits. arXiv:2402.17764 [cs.CL] https://arxiv.org/abs/2402.17764

    Pith/arXiv arXiv 2024

  38. [38]

    OpenAI. 2024. GPT-4 Technical Report. arXiv:2303.08774 [cs.CL] https://arxiv. org/abs/2303.08774

    Pith/arXiv arXiv 2024

  39. [39]

    Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Mar- garet Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. 2024. The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale. arXiv:2406.17557 [cs.CL] https://arxiv.org/abs/2406.17557

    Pith/arXiv arXiv 2024

  40. [40]

    Hongwu Peng, Shaoyi Huang, Tong Geng, Ang Li, Weiwen Jiang, Hang Liu, Shusen Wang, and Caiwen Ding. 2021. Accelerating Transformer-based Deep Learning Models on FPGAs using Column Balanced Block Pruning. In2021 22nd International Symposium on Quality Electronic Design (ISQED). 142–148. https://doi.org/10.1109/ISQED51717.2021.9424344

    doi:10.1109/isqed51717.2021.9424344 2021

  41. [41]

    Hao Peng, Jungo Kasai, Nikolaos Pappas, Dani Yogatama, Zhaofeng Wu, Lingpeng Kong, Roy Schwartz, and Noah A. Smith. 2022. ABC: Attention with Bounded- memory Control. InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (Eds.). Associatio...

    doi:10.18653/v1/2022.acl-long.515 2022

  42. [42]

    Patrick Plagwitz, Frank Hannig, and Jürgen Teich. 2022. TRAC: Compilation- Based Design of Transformer Accelerators for FPGAs. In2022 32nd International Conference on Field-Programmable Logic and Applications (FPL). 17–23. https: //doi.org/10.1109/FPL57034.2022.00015

    doi:10.1109/fpl57034.2022.00015 2022

  43. [43]

    Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Im- proving language understanding with unsupervised learning. (2018)

    2018

  44. [44]

    Atul Rahman, Jongeun Lee, and Kiyoung Choi. 2016. Efficient FPGA acceleration of Convolutional Neural Networks using logical-3D compute array. In’2016 Design, Automation & Test in Europe Conference & Exhibition (DATE)’. 1393–1398

    2016

  45. [45]

    Rishov Sarkar, Hanxue Liang, Zhiwen Fan, Zhangyang Wang, and Cong Hao

  46. [46]

    In2023 IEEE/ACM International Conference on Computer Aided Design (ICCAD)

    Edge-MoE: Memory-Efficient Multi-Task Vision Transformer Architecture with Task-Level Sparsity via Mixture-of-Experts. In2023 IEEE/ACM International Conference on Computer Aided Design (ICCAD). 01–09. https://doi.org/10.1109/ ICCAD57390.2023.10323651

    arXiv 2023

  47. [47]

    Guan Shen, Jieru Zhao, Quan Chen, Jingwen Leng, Chao Li, and Minyi Guo

  48. [48]

    SALO: An Efficient Spatial Accelerator Enabling Hybrid Sparse Attention Mechanisms for Long Sequences

    SALO: An Efficient Spatial Accelerator Enabling Hybrid Sparse Attention Mechanisms for Long Sequences. https://doi.org/10.48550/arXiv.2206.14550 arXiv:2206.14550

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2206.14550

  49. [49]

    Kejia Shi, Manting Zhang, Keqing Zhao, Xiaoxing Wu, Yang Liu, Jun Yu, and Kun Wang. 2024. Fitop-Trans: Maximizing Transformer Pipeline Efficiency through Fixed-Length Token Pruning on FPGA. In2024 34th International Conference on Field-Programmable Logic and Applications (FPL). 243–249. https://doi.org/10. 1109/FPL64840.2024.00041 12 ELiTeFormer: An Effic...

    arXiv 2024

  50. [50]

    Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. 2024. A Simple and Effective Pruning Approach for Large Language Models. InThe Twelfth Interna- tional Conference on Learning Representations. https://openreview.net/forum?id= PxoFut3dWW

    2024

  51. [51]

    Hashimoto

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford Alpaca: An Instruction-following LLaMA model. https://github.com/tatsu-lab/stanford_ alpaca

    2023

  52. [52]

    Yi Tay, Dara Bahri, Liu Yang, Donald Metzler, and Da-Cheng Juan. 2020. Sparse Sinkhorn Attention. https://arxiv.org/abs/2002.11296

    Pith/arXiv arXiv 2020

  53. [53]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. InAdvances in Neural Information Processing Systems, I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.), Vol. 30. Curran Associates, Inc. https://proc...

    2017

  54. [54]

    Hongji Wang, Yueyin Bai, Jun Yu, and Kun Wang. 2024. TransFRU: Efficient Deployment of Transformers on FPGA with Full Resource Utilization. In2024 29th Asia and South Pacific Design Automation Conference (ASP-DAC). 521–526. https://doi.org/10.1109/ASP-DAC58780.2024.10473976 ISSN: 2153-697X

    doi:10.1109/asp-dac58780.2024.10473976 2024

  55. [55]

    Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Huaijie Wang, Lingxiao Ma, Fan Yang, Ruiping Wang, Yi Wu, and Furu Wei. 2023. BitNet: Scaling 1- bit Transformers for Large Language Models. arXiv:2310.11453 [cs.CL] https: //arxiv.org/abs/2310.11453

    Pith/arXiv arXiv 2023

  56. [56]

    Hanrui Wang, Zhekai Zhang, and Song Han. 2021. SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruning. In2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA). 97–110. https://doi.org/10.1109/HPCA51647.2021.00018

    doi:10.1109/hpca51647.2021.00018 2021

  57. [57]

    Li, Madian Khabsa, Han Fang, and Hao Ma

    Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. 2020. Linformer: Self-Attention with Linear Complexity. https://doi.org/10.48550/ ARXIV.2006.04768

    Pith/arXiv arXiv 2020

  58. [58]

    Saiqun Wang and Hao Zhang. 2024. Efficient FPGA-Based Transformer Ac- celerator Using In-Block Balanced Pruning. In2024 13th International Con- ference on Communications, Circuits and Systems (ICCCAS). 18–23. https: //doi.org/10.1109/ICCCAS62034.2024.10651591

    work page doi:10.1109/icccas62034.2024.10651591 2024

  59. [59]

    Teng Wang, Lei Gong, Chao Wang, Yang Yang, Yingxue Gao, Xuehai Zhou, and Huaping Chen. 2022. ViA: A Novel Vision-Transformer Accelerator Based on FPGA.IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems41, 11 (2022), 4088–4099. https://doi.org/10.1109/TCAD.2022.3197489

    work page doi:10.1109/tcad.2022.3197489 2022

  60. [60]

    Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, and Vikas Singh. 2021. Nyströmformer: A Nyström-based Algorithm for Approximating Self-Attention. (2021)

    2021

  61. [61]

    Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. 2024. Gated Linear Attention Transformers with Hardware-Efficient Training. InPro- ceedings of the 41st International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 235), Ruslan Salakhutdinov, Zico Kolter, Kather- ine Heller, Adrian Weller, Nuria Oliver, ...

    2024

  62. [62]

    Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim. 2024. Parallelizing Linear Transformers with the Delta Rule over Sequence Length. arXiv:2406.06484 [cs.LG] https://arxiv.org/abs/2406.06484

    Pith/arXiv arXiv 2024

  63. [63]

    Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. 2020. Big Bird: Transformers for Longer Sequences. (2020). https: //doi.org/10.48550/ARXIV.2007.14062

    Pith/arXiv arXiv doi:10.48550/arxiv.2007.14062 2020

  64. [64]

    Shulin Zeng, Jun Liu, Guohao Dai, Xinhao Yang, Tianyu Fu, Hongyi Wang, Wenheng Ma, Hanbo Sun, Shiyao Li, Zixiao Huang, Yadong Dai, Jintao Li, Zehao Wang, Ruoyu Zhang, Kairui Wen, Xuefei Ning, and Yu Wang. 2024. FlightLLM: Efficient Large Language Model Inference with a Complete Mapping Flow on FPGAs. http://arxiv.org/abs/2401.03868 arXiv:2401.03868

    Pith/arXiv arXiv 2024

  65. [65]

    Michael Zhang, Simran Arora, Rahul Chalamala, Alan Wu, Benjamin Spector, Aaryan Singhal, Krithik Ramesh, and Christopher Ré. 2024. LoLCATs: On Low- Rank Linearizing of Large Language Models. arXiv:2410.10254 [cs.LG] https: //arxiv.org/abs/2410.10254

    Pith/arXiv arXiv 2024

  66. [66]

    Michael Zhang, Kush Bhatia, Hermann Kumbong, and Christopher Re. 2024. The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicry. InThe Twelfth International Conference on Learning Representations. https://openreview.net/forum?id=4g02l2N2Nx

    2024

  67. [67]

    Wenxuan Zhang, Yue Deng, Bing Liu, Sinno Pan, and Lidong Bing. 2024. Senti- ment Analysis in the Era of Large Language Models: A Reality Check. InFindings of the Association for Computational Linguistics: NAACL 2024, Kevin Duh, Helena Gomez, and Steven Bethard (Eds.). Association for Computational Linguistics, Mexico City, Mexico, 3881–3906. https://doi.o...

    doi:10.18653/v1/2024.findings- 2024

  68. [68]

    Chen Zhu, Wei Ping, Chaowei Xiao, Mohammad Shoeybi, Tom Goldstein, Anima Anandkumar, and Bryan Catanzaro. 2021. Long-Short Transformer: Efficient Transformers for Language and Vision. InAdvances in Neural Information Processing Systems, M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (Eds.), Vol. 34. Curran Associates, Inc., 177...

    2021