Pith. sign in

REVIEW 5 major objections 6 minor 37 references

Hamming Attention Distillation: Binarizing Keys and Queries for Efficient Long-Context Transformers

T0 review · 5 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Hamming Attention Distillation shows that binarizing keys and queries to +1/-1 and keeping only the top-N links per query costs 1.78 points on GLUE and 2.5 on ImageNet while enabling a 79% smaller, 87% lower-power attention unit.

desk verdict A well-specified and honest engineering paper on selective Q/K binarization, but the long-context and hardware claims outrun the evidence; deserves review, not unconditional acceptance. read the letter →

arxiv 2502.01770 v1 pith:XTQKRVTP submitted 2025-02-03 cs.LG cs.AIeess.IV

Mark Horton , Tergel Molom-Ochir , Peter Liu , Bhavna Gopal , Chiyue Wei , Cong Guo , Brady Taylor , Deliang Fan
show 3 more authors
Shan X. Wang Hai Li Yiran Chen
This is my paper · ORCID
classification cs.LGcs.AIeess.IV
keywords transformerattentionbinarizationHammingdistancetop-Nsparsityknowledgedistillationlong-contextinferencecontent-addressablememoryefficienthardware
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Hamming Attention Distillation (HAD) tries to establish that a transformer's attention can be made much cheaper without much accuracy loss by compressing only the address-matching part of attention. Keys and queries are turned into $\{-1,+1\}$ vectors so that dot products become Hamming distances, and for each query only the top $N$ keys are kept before the softmax. The method is a four-stage distillation from a full-precision teacher, with a KL loss on attention logits and then on output logits. On GLUE, HAD loses 1.78 points versus 9.08 for a full attention-binarization baseline; on ImageNet it loses 2.5 points versus 12.14. The paper also simulates a content-addressable-memory hardware design that uses these algorithmic choices to achieve 79% area and 87% power reduction compared with a standard attention implementation.

What carries the argument

The carrying mechanism is a binarized attention lookup combined with top-$N$ sparsification. The query and key matrices are standardized with data-estimated coefficients $\sigma_Q, \sigma_K$, then passed through sign quantization so $QK^\top$ is computed by Hamming distance (an XNOR-plus-popcount operation) instead of floating-point multiplication. A top-$N$ mask keeps only the largest logits per row, so softmax and the value accumulation operate on a sparse matrix. Training uses a four-stage annealing schedule: a scaled tanh with a decaying scale $c$ first approaches the sign function smoothly, then a straight-through estimator finishes the binarization, with a KL divergence on attention logits used as an auxiliary distillation signal before being removed in the final refinement stage. The paper justifies the choice of $N$ with a Gaussian model of attention logits: for standard-normal inputs, the fraction of the largest softmax outputs needed to capture a fixed probability mass approaches a constant as the softmax grows, which is used to scale $N$ linearly with context length.

What would settle it

Run the HAD distillation at a 4K or 8K token context, keep $N$ scaled linearly from the paper's 128-1024 results, and compare against the same model with full attention; if accuracy falls by more than the roughly 3 points seen in the paper's shorter-context experiments, or if the pruned links are the ones with the largest gradient impact on the output, the Gaussian scaling premise is falsified.

Watch

Extended reading notes

Core claim

The paper's central claim is that selective binarization of keys and queries, rather than full attention binarization, is what lets a compressed transformer stay accurate. With keys and queries standardized and quantized to $\{-1,+1\}$, the attention logit $QK^\top$ becomes a Hamming-distance score, and distillation experiments show that this change alone leaves GLUE and ImageNet accuracy within about 2 points of the full-precision teacher. Sparsifying the attention matrix to the top $N$ entries per query, with $N$ scaling linearly with context length, then attacks the remaining $O(n^2)$ costs of softmax and value accumulation. On the QuALITY long-context benchmark, the method tracks the baseline's accuracy gains as context length grows from 128 to 1024 tokens. The same decisions allow the custom CAM-based hardware to replace floating-point matrix products with 1-bit XNOR associative matching, producing the reported 79% area and 87% power reductions.

Load-bearing premise

The load-bearing assumption, argued in Section 3.2 from a Gaussian model of attention logits, is that a fixed fraction of the largest attention links always carries the essential signal; real, data-dependent attention distributions could break this at longer contexts.

Editorial extensions

If this is right

  • If HAD is right, a pre-trained full-precision transformer can be adapted to long-context use by fine-tuning only the key/query projections and adding a top-$N$ mask, without redesigning the network.
  • The accuracy gap to full precision (1.78 on GLUE, 2.5 on ImageNet) is small enough that binary Q/K attention becomes a practical choice for deployment, whereas full attention binarization losses of 9 to 12 points were much harder to accept.
  • Because $N$ scales linearly with context length, the attention cost for an $n$-token sequence drops from $O(n^2)$ to $O(n \cdot N)$, and the hardware numbers show this translates into area and power savings rather than remaining a theoretical bound.
  • On long-context question answering, HAD's accuracy improves with context length in step with the full-precision baseline, supporting the claim that the compression does not selectively harm long-range reasoning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Under the binary-associative-memory view, HAD-style Q/K binarization may transfer to decoder-only LLMs, where binarizing the entire network has been far too lossy, but the paper leaves that as future work.
  • The large gap between HAD and the SAB ablation suggests a broader design principle: keep the value-readout stage of attention expressive and compress only the addressing stage (Q and K), which other compression efforts could adopt.
  • The hardware numbers are for a CAM-based accelerator; on conventional GPUs, top-N sparse accumulation and XNOR matching may not map onto dense tensor cores, so real speedups depend on matching hardware.
  • The reported 79% area and 87% power reductions cover only the attention unit, not the whole transformer, so system-level savings would be smaller; quantifying them would be a natural next step.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes Hamming Attention Distillation (HAD), a method that binarizes the query and key matrices of a pre-trained transformer to {-1,+1}, replacing dot products with Hamming-distance computations, and sparsifies the attention matrix by keeping only the top-N attention entries per query. A multi-stage distillation procedure (scaled-tanh annealing, straight-through-estimator training, then final refinement) is used to transfer the teacher model's behavior to the binary student. The method is evaluated on GLUE with BERT, on ImageNet with DeiT, and on the QuALITY long-context QA benchmark at context lengths up to 1024 tokens. The authors also report a CAM-based hardware simulation claiming a 79% area reduction and 87% power reduction compared with a conventional attention implementation. The central claim is that HAD preserves most of the accuracy of full-precision transformers at a fraction of the attention cost, outperforming prior full-binarization methods such as BiT and BiViT.

Significance. If the claims were fully supported, HAD would be a useful contribution to efficient attention: it targets exactly the O(n^2) attention operations, uses hardware-friendly XNOR/Hamming operations, and includes thoughtful ablations (w/ SAB, w/o AD, w/o Tanh) that give some insight into which components matter. The attention-logit distillation objective and the multi-stage annealing schedule are clearly specified, and the QuALITY experiments are a step toward long-context validation. However, the central long-context and hardware claims are currently not backed by sufficient evidence: the longest experiment is 1024 tokens, the N-scaling rule is justified by a synthetic Gaussian model rather than by real attention maps, and the comparison with BiT/BiViT is not apples-to-apples because HAD leaves weights, values, and the attention map in full precision. In addition, the loss labeled KL divergence in Eqs. (9)-(10) is not a KL divergence. These issues are load-bearing for the paper's main conclusions.

major comments (5)
  1. [Section 4.3, Figure 5] The long-context claim is not supported beyond 1024 tokens. QuALITY inputs are 2,000 to 6,000 words, yet the largest context evaluated is 1024 tokens, and the model is truncated to fit. The N-scaling rule of Section 3.2 is justified by Figure 4, which models attention logits as i.i.d. standard Gaussians, but the actual logits in Eq. (5) are integer Hamming distances with a data-dependent distribution; no evidence is given that the linear rule holds for real attention maps or for longer sequences. The performance drop at 512 tokens is attributed to 'noise' without error bars. I would need experiments at 2048/4096 tokens, or at minimum an analysis of the fraction of teacher attention mass retained by the top-N rule on real attention maps, to accept the scaling claim.
  2. [Section 3.3, Eqs. (9)-(10)] Equations (9) and (10) are not KL divergences as claimed. The KL divergence between two softmax distributions with logit vectors a and b is proportional to sum_i softmax(a)_i [(a_i - logsumexp(a)) - (b_i - logsumexp(b))]. Eq. (9) instead uses exp(A_t(i,j)) (A_t(i,j) - A_s(i,j)) with no softmax normalization and no log-sum-exp terms, and Eq. (10) has the same structure for output logits. This is not merely a naming issue: if the objective actually minimized is Eq. (11), the student is trained to match an unnormalized exponentially weighted logit difference rather than the teacher's probability distribution. The authors should either correct the equations to be true KL divergences or explicitly rename the loss and explain why the unnormalized form is preferable.
  3. [Tables 1 and 2, Abstract] The comparison with BiT and BiViT is apples-to-oranges. HAD binarizes only the key and query activations while keeping weights, values, and the attention map in full precision; BiT and BiViT binarize far more of the network. The abstract's claim of 'state-of-the-art performance among binarized Transformers' is therefore not established by these tables. The large gap between HAD and BiT/BiViT may reflect the fact that HAD does not binarize the attention map or weights, rather than an advantage of the distillation scheme. A matched comparison would include other Q/K-only binarization methods or baselines with the same compression budget, or should at least phrase the result as 'accuracy of a Q/K-binarized, top-N sparse transformer' rather than a head-to-head win over fully binarized models.
  4. [Section 4.4, Table 3] The hardware simulation is not sufficiently specified to support the 79% area and 87% power reduction claims. The text states only that Verilog for a smaller module was synthesized with Synopsys Design Compiler and scaled to the full design, with no technology node, supply voltage, clock frequency, or synthesis corner reported. The numbers also lack an analysis of data movement and memory traffic, which are often dominant for attention in long-context inference. Table 3 reports component-level area and power, but without the underlying synthesis assumptions and scaling method the reader cannot assess whether the gains are realistic or an artifact of the unstated configuration.
  5. [Section 3.2, Eq. (6)] The top-N sparsification with N scaled linearly in context length does not change the asymptotic complexity of attention: the QK^T operation is still O(n^2) and the AV accumulation is O(nN)=O(n^2). Thus the paper's framing as an 'efficient long-context' method, and the abstract's claim of 'drastically reducing the computational costs of long-context inference,' should be qualified as constant-factor hardware-level gains on a fixed context length, not subquadratic scaling. The title and introduction currently imply a stronger asymptotic benefit than the method actually provides.
minor comments (6)
  1. [Section 3.6, Eqs. (13)-(15)] Section 3.6 appears to contain a copy-paste error: it first displays the same c-scaled tanh transformation as Eq. (13), then states 'We approximate...' and introduces Eq. (15). The authors should clarify which transformation is actually used in stage 2 and remove the duplicated equation.
  2. [Figure 4] The caption does not state the probability threshold used to define 'the percentage of the largest softmax outputs required to sum to the threshold probability.' Without the threshold value and the number of softmax dimensions simulated, the 'approaches a constant' claim is not reproducible.
  3. [Tables 1 and 2, Section 4.3] No standard deviations, number of seeds, or error bars are reported for any of the accuracy numbers, and the 'noise' mentioned for the 512-token QuALITY result is never quantified. Adding repeated-run statistics and confidence intervals would materially strengthen the comparison and the long-context discussion.
  4. [General] No code, training configuration details (number of epochs per stage, exact c decay schedule, dataset splits), or checkpoints are provided, which makes the empirical results difficult to verify or reproduce.
  5. [General] There are several typographical errors: 'fine' should be 'find' in Section 3.2, 'trainind' should be 'training' in Section 4.3, 'degredation' in Section 2.2, and 'signficantly' in Section 2.3. References [27] and [28] both cite the same XNOR-Net paper and should be merged.
  6. [Eq. (12)] The standardization coefficients sigma_Q and sigma_K are computed per minibatch as the standard deviation over 'all elements within the corresponding matrix,' but it is not clear whether this yields one scalar per layer, per head, or per tensor, nor how these coefficients are applied in Eqs. (13)-(18). Clarifying the tensor shapes would remove ambiguity.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: HAD's claims are benchmark measurements and distillation against an independent teacher, not derivations from fitted parameters.

full rationale

The paper's derivation chain is empirical and self-contained rather than circular. The Q/K binarization (Eqs. 4-5), top-N sparsity (Eq. 6), and softmax over top-N logits (Eq. 7) define the student mechanism; the student is trained to match a fixed full-precision teacher's attention logits and output logits through the distillation losses in Eqs. 9-11. The teacher is an independent pretrained model, not a function of the student's fitted parameters, and the reported GLUE, ImageNet, and QuALITY numbers are held-out benchmark evaluations. The sparsity parameter N is selected by an empirical accuracy sweep (Fig. 3) and a Gaussian softmax heuristic (Fig. 4), but this is an experimental design choice rather than a parameter fitted to the evaluation outcome and then relabeled as a prediction; the long-context QuALITY results are directly measured. Hardware area and power figures come from synthesis of the proposed design, not from the accuracy claims. There are no load-bearing self-citations, no imported uniqueness theorem, and no step where the output is equivalent to an input by construction. The main caveats, such as the linear N-scaling rule being unvalidated beyond 1024 tokens, are assumptions about generalization, not circularity.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claim rests on two fitted quantities (N and the sigma standardization coefficients) and on three supporting assumptions from prior work: the Gaussian attention-logit heuristic, the scaling-law argument about attention capacity, and the Hopfield-network storage analogy. No new physical or architectural entities are introduced; the CAM hardware is an existing technology used in the simulation.

free parameters (2)
  • Top-N sparsity N = 30 for BERT and DeiT; scaled linearly from 15 to 120 for QuALITY contexts of 128 to 1024 tokens
    Selected empirically from a DeiT-T accuracy sweep (Figure 3) and a Gaussian probability-mass heuristic (Figure 4); directly controls the sparsification that enables efficiency gains and affects accuracy.
  • Standardization coefficients sigma_Q and sigma_K = Per layer, estimated from 100 minibatches of 16 training samples (Eq. 12)
    Data-dependent scaling of binarized Q/K; sets the effective temperature of the softmax and is necessary for models like T5; no independent theoretical basis is given.
assumptions (3)
  • ad hoc to paper Keys and queries follow approximately Gaussian distributions, so the fraction of top softmax outputs needed to reach a probability threshold is roughly constant with sequence length (Figure 4).
    Used in Section 3.2 to justify linearly scaling N with context length; no validation on real attention maps is provided.
  • domain assumption Transformer performance depends weakly on model shape and attention capacity, so binarizing keys and queries should not substantially hurt accuracy (citing Kaplan et al. scaling laws).
    Invoked in Section 2.3; scaling laws concern parameter count and data, not specifically attention precision, and the argument is used as motivation rather than proof.
  • domain assumption Modern Hopfield networks with binary patterns have exponential storage capacity, suggesting that binary keys and queries remain expressive (citing Ramsauer et al.).
    Section 2.3 presents this as an analogy; it is not a proof that transformer attention with sign-binarized Q/K retains the same representational capacity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Hamming Attention Distillation: Binarizing Keys and Queries for Efficient Long-Context Transformers." pith.science (2026). https://pith.science/paper/XTQKRVTP

@misc{pith2026250201770,
  author       = {Pith},
  title        = {Pith review of: Hamming Attention Distillation: Binarizing Keys and Queries for Efficient Long-Context Transformers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XTQKRVTP}},
  note         = {Machine review of arXiv:2502.01770}
}
abstract

Pre-trained transformer models with extended context windows are notoriously expensive to run at scale, often limiting real-world deployment due to their high computational and memory requirements. In this paper, we introduce Hamming Attention Distillation (HAD), a novel framework that binarizes keys and queries in the attention mechanism to achieve significant efficiency gains. By converting keys and queries into {-1, +1} vectors and replacing dot-product operations with efficient Hamming distance computations, our method drastically reduces computational overhead. Additionally, we incorporate attention matrix sparsification to prune low-impact activations, which further reduces the cost of processing long-context sequences. \par Despite these aggressive compression strategies, our distilled approach preserves a high degree of representational power, leading to substantially improved accuracy compared to prior transformer binarization methods. We evaluate HAD on a range of tasks and models, including the GLUE benchmark, ImageNet, and QuALITY, demonstrating state-of-the-art performance among binarized Transformers while drastically reducing the computational costs of long-context inference. \par We implement HAD in custom hardware simulations, demonstrating superior performance characteristics compared to a custom hardware implementation of standard attention. HAD achieves just $\mathbf{1.78}\%$ performance losses on GLUE compared to $9.08\%$ in state-of-the-art binarization work, and $\mathbf{2.5}\%$ performance losses on ImageNet compared to $12.14\%$, all while targeting custom hardware with a $\mathbf{79}\%$ area reduction and $\mathbf{87}\%$ power reduction compared to its standard attention counterpart.

Figures

Figures reproduced from arXiv: 2502.01770 by the authors.

Figure 1
Figure 1. Runtime analysis of BERT Base over increasing context [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Binarized attention mechanism in Hamming Attention [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Accuracies measured while progressively distilling a full [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Given standard gaussian inputs, the percentage of the [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Comparison of HAD and baseline accuracy across dif [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

37 extracted references · 22 canonical work pages

  1. [1]

    The groq software-defined scale-out tensor streaming multipro- cessor: From chips-to-systems architectural overview

    Dennis Abts, John Kim, Garrin Kimmell, Matthew Boyd, Kris Kang, Sahil Parmar, Andrew Ling, Andrew Bitar, Ibrahim Ahmed, and Jonathan Ross. The groq software-defined scale-out tensor streaming multipro- cessor: From chips-to-systems architectural overview. In 2022 IEEE Hot Chips 34 Symposium (HCS) , pages 1–69. IEEE Computer Society, 2022

    work page 2022

  2. [2]

    Vivit: A video vision transformer

    Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lu ˇci´c, and Cordelia Schmid. Vivit: A video vision transformer. In Proceedings of the IEEE/CVF international conference on computer vi- sion, pages 6836–6846, 2021

    work page 2021

  3. [3]

    Long- former: The long-document transformer

    Iz Beltagy, Matthew E Peters, and Arman Cohan. Long- former: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020

    arXiv 2004

  4. [4]

    Prime: A novel processing-in-memory architecture for neural net- work computation in reram-based main memory

    Ping Chi, Shuangchen Li, Cong Xu, Tao Zhang, Jishen Zhao, Yongpan Liu, Yu Wang, and Yuan Xie. Prime: A novel processing-in-memory architecture for neural net- work computation in reram-based main memory. ACM SIGARCH Computer Architecture News , 44(3):27–39, 2016

    work page 2016

  5. [5]

    On a model of associative memory with huge storage capacity

    Mete Demircigil, Judith Heusel, Matthias L ¨owe, Sven Upgang, and Franck Vermet. On a model of associative memory with huge storage capacity. Journal of Statisti- cal Physics, 168:288–299, 2017

    work page 2017

  6. [6]

    Imagenet: A large-scale hierarchical im- age database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical im- age database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248–255. Ieee, 2009

    work page 2009

  7. [7]

    Bert: Pre-training of deep bidirectional transformers for language understanding.arXiv preprint arXiv:1810.04805, 2018

    Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding.arXiv preprint arXiv:1810.04805, 2018

    arXiv 2018

  8. [8]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020

    arXiv 2010

Show all 37 references
  1. [9]

    Array multiplier using xnor

    Riya Garg and Navneet Kaur. Array multiplier using xnor. International Journal of Engineering Science and Technology (IJEST), 5(4):799–803, 2013

    2013

  2. [10]

    Mamba: Linear-time sequence modeling with selective state spaces

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023

    2023 arXiv

  3. [11]

    Sparse co-attention visual question answering networks based on thresholds

    Zihan Guo and Dezhi Han. Sparse co-attention visual question answering networks based on thresholds. Ap- plied Intelligence, 53(1):586–600, 2023

    2023

  4. [12]

    Bivit: Ex- tremely compressed binary vision transformers

    Yefei He, Zhenyu Lou, Luoming Zhang, Jing Liu, Wei- jia Wu, Hong Zhou, and Bohan Zhuang. Bivit: Ex- tremely compressed binary vision transformers. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pages 5651–5663, 2023

    2023

  5. [13]

    Binarized neural net- works

    Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Binarized neural net- works. Advances in neural information processing sys- tems, 29, 2016

    2016

  6. [14]

    In-datacenter performance analysis of a tensor process- ing unit

    Norman P Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, et al. In-datacenter performance analysis of a tensor process- ing unit. In Proceedings of the 44th annual international symposium ...

    2017

  7. [15]

    Scal- ing laws for neural language models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scal- ing laws for neural language models. arXiv preprint arXiv:2001.08361, 2020

    2001 arXiv

  8. [16]

    Reformer: The efficient transformer

    Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. arXiv preprint arXiv:2001.04451, 2020

    2001 arXiv

  9. [17]

    Self-binarizing networks

    Fayez Lahoud, Radhakrishna Achanta, Pablo M´arquez- Neila, and Sabine S ¨usstrunk. Self-binarizing networks. arXiv preprint arXiv:1902.00730, 2019

    1902 arXiv

  10. [18]

    Race: Large-scale reading com- prehension dataset from examinations

    Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. Race: Large-scale reading com- prehension dataset from examinations. arXiv preprint arXiv:1704.04683, 2017

    2017 arXiv

  11. [19]

    Power and area-efficient xnor-and hybrid binary neural networks using tft-type synaptic devices

    In-Seok Lee, Hyeongsu Kim, Min-Kyu Park, Joon Hwang, Ryun-Han Koo, Jae-Joon Kim, and Jong-Ho Lee. Power and area-efficient xnor-and hybrid binary neural networks using tft-type synaptic devices. IEEE Electron Device Letters, 43(12):1912–1915, 2022

    1912

  12. [20]

    Fp-bnn: Binarized neural network on fpga

    Shuang Liang, Shouyi Yin, Leibo Liu, Wayne Luk, and Shaojun Wei. Fp-bnn: Binarized neural network on fpga. Neurocomputing, 275:1072–1086, 2018

    2018

  13. [21]

    Cerebras architecture deep dive: First look inside the hw/sw co-design for deep learning: Cerebras systems

    Sean Lie. Cerebras architecture deep dive: First look inside the hw/sw co-design for deep learning: Cerebras systems. In 2022 IEEE Hot Chips 34 Symposium (HCS), pages 1–34. IEEE Computer Society, 2022

    2022

  14. [22]

    Transformer acceleration with dynamic sparse attention

    Liu Liu, Zheng Qu, Zhaodong Chen, Yufei Ding, and Yuan Xie. Transformer acceleration with dynamic sparse attention. arXiv preprint arXiv:2110.11299 , 2021

    2021 arXiv

  15. [23]

    Bit: Robustly binarized multi-distilled transformer

    Zechun Liu, Barlas Oguz, Aasish Pappu, Lin Xiao, Scott Yih, Meng Li, Raghuraman Krishnamoorthi, and Yashar Mehdad. Bit: Robustly binarized multi-distilled transformer. Advances in neural information processing systems, 35:14303–14316, 2022

    2022

  16. [24]

    Quality: Question answering with long input texts, yes! arXiv preprint arXiv:2112.08608, 2021

    Richard Yuanzhe Pang, Alicia Parrish, Nitish Joshi, Nikita Nangia, Jason Phang, Angelica Chen, Vishakh Padmakumar, Johnny Ma, Jana Thompson, He He, et al. Quality: Question answering with long input texts, yes! arXiv preprint arXiv:2112.08608, 2021

    2021 arXiv

  17. [25]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020

    2020

  18. [26]

    Hopfield networks is all you need

    Hubert Ramsauer, Bernhard Sch ¨afl, Johannes Lehner, Philipp Seidl, Michael Widrich, Thomas Adler, Lukas Gruber, Markus Holzleitner, Milena Pavlovi ´c, Geir Kjetil Sandve, et al. Hopfield networks is all you need. arXiv preprint arXiv:2008.02217, 2020

    2008 arXiv

  19. [27]

    Xnor-net: Imagenet classifica- tion using binary convolutional neural networks

    Mohammad Rastegari, Vicente Ordonez, Joseph Red- mon, and Ali Farhadi. Xnor-net: Imagenet classifica- tion using binary convolutional neural networks. In Eu- ropean conference on computer vision , pages 525–542. Springer, 2016

    2016

  20. [28]

    XNOR-Net: Imagenet classifica- tion using binary convolutional neural networks

    Mohammad Rastegari, Vicente Ordonez, Joseph Red- mon, and Ali Farhadi. XNOR-Net: Imagenet classifica- tion using binary convolutional neural networks. In Eu- ropean Conference on Computer Vision (ECCV) , pages 525–542. Springer, 2016

    2016

  21. [29]

    Efficient sparse-dense matrix-matrix multiplication on gpus us- ing the customized sparse storage format

    Shaohuai Shi, Qiang Wang, and Xiaowen Chu. Efficient sparse-dense matrix-matrix multiplication on gpus us- ing the customized sparse storage format. In 2020 IEEE 26th International Conference on Parallel and Dis- tributed Systems (ICPADS), pages 19–26. IEEE, 2020

    2020

  22. [30]

    Sextans: A streaming accelerator for general-purpose sparse-matrix dense-matrix multiplication

    Linghao Song, Yuze Chi, Atefeh Sohrabizadeh, Young- kyu Choi, Jason Lau, and Jason Cong. Sextans: A streaming accelerator for general-purpose sparse-matrix dense-matrix multiplication. In Proceedings of the 2022 ACM/SIGDA International Symposium on Field- Programmable Gate Arr...

    2022

  23. [31]

    Training data-efficient image transformers & distillation through attention

    Hugo Touvron, Matthieu Cord, Matthijs Douze, Fran- cisco Massa, Alexandre Sablayrolles, and Herv´e J´egou. Training data-efficient image transformers & distillation through attention. In International conference on ma- chine learning, pages 10347–10357. PMLR, 2021

    2021

  24. [32]

    Attention is all you need

    A Vaswani. Attention is all you need. Advances in Neu- ral Information Processing Systems, 2017

    2017

  25. [33]

    Audio trans- formers: Transformer architectures for large scale au- dio understanding

    Prateek Verma and Jonathan Berger. Audio trans- formers: Transformer architectures for large scale au- dio understanding. adieu convolutions. arXiv preprint arXiv:2105.00335, 2021

    2021 arXiv

  26. [34]

    Linformer: Self-attention with linear com- plexity

    Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear com- plexity. arXiv preprint arXiv:2006.04768, 2020

    2006 arXiv

  27. [35]

    Onebit: Towards extremely low-bit large language models

    Yuzhuang Xu, Xu Han, Zonghan Yang, Shuo Wang, Qingfu Zhu, Zhiyuan Liu, Weidong Liu, and Wanxiang Che. Onebit: Towards extremely low-bit large language models. arXiv preprint arXiv:2402.11295, 2024

    2024 arXiv

  28. [36]

    Pb- llm: Partially binarized large language models

    Zhihang Yuan, Yuzhang Shang, and Zhen Dong. Pb- llm: Partially binarized large language models. In The Twelfth International Conference on Learning Repre- sentations

  29. [37]

    Big bird: Transformers for longer se- quences

    Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago On- tanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer se- quences. Advances in neural information processing systems, 33:17283–17297, 2020

    2020

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.