Pith. sign in

REVIEW 4 major objections 5 minor 80 references

MAS-Attention: Memory-Aware Stream Processing for Attention Acceleration on Resource-Constrained Edge Devices

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Exact attention inference on edge NPUs can be up to 2.75x faster and 54% more energy-efficient by running softmax and matrix multiply as two parallel streams on separate compute units, without changing outputs.

desk verdict Genuine speedups on a shipping edge NPU, but the long-sequence story leans on a preemption primitive the paper never verifies. read the letter →

arxiv 2411.17720 v2 pith:YRWFK74Y submitted 2024-11-20 cs.DC cs.AIcs.PF

Mohammadali Shakerdargah , Shan Lu , Chao Gao , Di Niu This is my paper · ORCID
classification cs.DCcs.AIcs.PF
keywords attentionaccelerationedgeneuralprocessingunitstreammatrixmultiplysoftmaxoperatorfusiontilingmemory-awarescheduling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Attention is the part of a transformer that makes inference on phones and other small devices slow, because it needs quadratic memory and compute as context grows. This paper tries to establish that on edge neural accelerators with separate matrix and vector compute units, the attention block can be run as two parallel streams—matrix multiplies on the MAC unit and row-wise softmax on the vector unit—so neither unit sits idle while the other works. The method is exact, meaning the model's output is unchanged, and the paper reports up to 2.75x speedup and 54% lower energy in simulation, and up to 1.76x speedup on real edge hardware, against the best fused-attention baseline, FLAT. A careful multi-tiered tiling scheme and a proactive cache-overwrite rule are what make the overlap work within tight on-chip memory.

What carries the argument

The load-bearing mechanism is a two-stream, semi-synchronous pipeline. A MAC-unit stream computes the tiled products $Q_iK^T$ and $P_iV$ while a VEC-unit stream computes the row-wise softmax of the score tiles; the streams advance in rounds, with dependencies enforced by ordering so that a softmax tile only starts after its score tile exists and an output tile only after its probability tile exists. Two supporting mechanisms carry the practical feasibility: a multi-tiered tiling search that picks fine-grained sub-matrix tiles for $K$ and $V$ and row-granularity tiles for $C$ and $P$, and a proactive overwrite rule that, when on-chip memory fills, lets the softmax stream evict a $K$ or $V$ tile the MAC unit is still using, forcing the MAC to stop, reload from DRAM, and redo that multiply. The reload cost is the price paid to keep the pipeline from spilling softmax's on-chip partial results.

What would settle it

On an edge NPU that exposes no mid-MatMul preemption, run MAS-Attention and FLAT with a sequence length long enough to force the proactive overwrite path, and compare end-to-end cycles while counting the extra DRAM reloads and recomputed multiplies; if MAS-Attention is not faster than FLAT in that setting, the central speedup claim fails.

Watch

Extended reading notes

Core claim

The central claim is that exact attention inference on memory-constrained edge accelerators is not bound to run operators sequentially: the two matrix multiplications ($QK^T$ and $PV$) and the row-wise softmax can be scheduled as semi-synchronous streams on the MAC and VEC units, with each unit consuming its own tiled workload. The paper further claims that a multi-tiered tiling—fine sub-matrix tiles for $K$ and $V$, row tiles for the score and probability matrices—plus a proactive buffer-overwrite mechanism that sacrifices an in-flight $K$ or $V$ tile to make room for softmax data, lets this parallel execution survive cache pressure at long sequence lengths. On this basis the paper reports up to 2.75x cycle speedup and 54% energy reduction over FLAT on a simulated edge accelerator, and up to 1.76x speedup on a real edge NPU, with golden-data checks confirming identical output.

Load-bearing premise

The method assumes the accelerator can interrupt and discard an in-flight matrix multiply, overwrite its input buffers, and later resume by reloading those buffers from DRAM; if that preemption is unsupported or expensive, the cache-overwrite mechanism and its long-sequence benefits collapse.

Editorial extensions

If this is right

  • If the central claim holds, attention layers on edge NPUs with separate MAC and VEC units no longer need to serialize MatMul and softmax, so fused attention kernels can be re-scheduled as two streams instead of one.
  • The multi-tiered tiling search makes the scheme automatic across attention shapes, so BERT-, Llama-, T5-, ViT-, and XLM-style layers can each get near-optimal tiles without hand-tuning.
  • The proactive overwrite rule bounds on-chip memory pressure at long sequences, at the cost of extra DRAM reads of $K$ and $V$; the paper's measurements say this cost is outweighed by the pipeline gain.
  • End-to-end, the method cuts the largest attention unit's runtime by 29.4% and total inference latency by 6% in a Stable Diffusion UNet workload on a real edge device, so the gain survives outside isolated attention kernels.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same two-stream principle may generalize to any element-wise or row-wise op (layer normalization, GeLU, causal masking) that currently serializes with MatMul on edge NPUs, so the scheduling method could be packaged as a compiler pass rather than an attention-only kernel.
  • The speedup ceiling likely tracks the MAC-to-VEC throughput ratio on the target chip; on MAC-dominated parts the pipeline has little to overlap, so one testable prediction is that gains shrink as embedding size grows relative to sequence length.
  • Because the paper optimizes latency rather than energy (its FuseMax energy comparisons are mixed), retargeting the tiling search to an energy objective should widen the energy lead without changing the streaming structure.
  • The ~1M-token FP16 sequence ceiling on the simulated device, versus FLAT's ~2M, suggests a hybrid strategy would fall back to sequential FLAT beyond the point where the pipeline's extra on-chip rows no longer fit; whether that crossover exists on real hardware is an open, testable question.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes MAS-Attention, a scheme for exact attention inference on memory-constrained edge accelerators that overlaps tiled MatMul and softmax computations across heterogeneous MAC and VEC units. The method uses a multi-tiered tiling scheme, heuristic search for tiling factors, and a proactive cache overwrite strategy. The authors report simulations using a modified TileFlow/Timeloop/Accelergy stack showing up to 2.75x speedup and 54% energy savings over FLAT, and real-hardware experiments on a Huawei DaVinci NPU showing up to 1.76x speedup over FLAT with golden-data correctness checks.

Significance. If the results hold, the paper addresses a real and timely problem: edge NPUs with separate MAC and VEC units are often underutilized because attention kernels execute operators sequentially. The real-hardware speedups of 1.30–1.76x over FLAT are credible evidence that overlapping tiled MatMul and softmax can yield practical gains, and the exactness of the computation (with golden-data verification) is a strength. The paper is also honest about several limitations, including the reduced maximum sequence length relative to FLAT and the latency-only search objective. However, the simulation-based energy and long-sequence claims rest on an unvalidated hardware preemption primitive, and the headline numbers mask wide variance across workloads.

major comments (4)
  1. [§4.3] The proactive overwrite strategy requires preempting and discarding an in-flight MatMul on the MAC unit: when on-chip memory is full, Pi overwrites V or K, 'stop[ping] the MAC from continuing its operation', and the MAC later 'resume[s] its process by reloading either the V or K matrix from DRAM ... and redoing the MatMul calculation.' The paper does not state that the DaVinci NPU or the simulated TileFlow architecture exposes such preemption, nor does it model the cost of partial product loss, pipeline drain, or recomputation. Section 5.4.2 mentions extra DRAM reads for BERT and Llama workloads, but no cycle or energy cost for the recomputed MAC work is shown. The real-hardware experiments (Table 1) use sequence lengths up to 512, so they do not exercise the overwrite mechanism. This is load-bearing because the long-sequence and cache-limited performance story depends entirely on this primitive.
  2. [§5.2.1 and Table 3] The abstract highlights '54% reduction in energy consumption' as a headline result, but Table 3 shows this is the maximum over workloads, with a geometric mean of only 18.55% versus FLAT and near-zero savings for several workloads (0.02% for Llama3-8B, 1.98% for BERT-base, 3.34% for ViT-B/16). Moreover, MAS-Attention consumes more energy than FuseMax on several workloads (negative savings). The paper should present the distribution and clarify that the headline number is a best case, not typical.
  3. [§5.1 and §5.2] The simulated speedups (up to 2.75x over FLAT) are substantially larger than the real-hardware speedups (up to 1.76x), and the modified TileFlow simulator is not validated against the DaVinci NPU. The paper does not discuss this discrepancy or provide sensitivity analysis of the simulator's memory bandwidth, cache size, or MAC/VEC throughput assumptions. Since the energy and long-sequence claims are generated entirely by this simulator, the quantitative conclusions should be presented with appropriate caveats or additional validation.
  4. [Algorithm 1] Algorithm 1 presents the stream-processing schedule without incorporating the proactive overwrite logic described in §4.3. The pseudocode has no branch for halting a MatMul, overwriting K/V, or recomputing partial results. This makes it impossible to verify the correctness and cost of the claimed memory-management strategy from the formal algorithm, and it obscures how the overwrite mechanism interacts with the pipelined dependencies.
minor comments (5)
  1. [§1] There is a typo: 'optimizing the cachce management' should be 'cache management'.
  2. [§5.3] The text says 'Based on some literature studies, "pJ" (picojoule) is used as the unit for energy consumption reported by Accelergy.' The unit notation should be consistent; the table header reads 'Energy Consumption(10^9 pJ)', which is unconventional. Clarify whether the numbers are in 10^9 pJ or another unit.
  3. [References] Several references are malformed, e.g., '[app, b;c;a]' in §2 and 'app, b' in §1, and some URLs are missing titles or access dates. Please clean up the reference list.
  4. [§5.5] Figure 7 mentions 'proportionally reduces the number of plotted lines to approximately 2K' without explaining the selection criterion; also 'after around 10K iterations' is vague. Report the exact number of iterations and the convergence metric.
  5. [§5.2.2] The real-hardware results show no error bars or repeated measurements; since the speedups are modest (1.30–1.76x), run-to-run variability should be reported.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: MAS-Attention's speedups are empirical measurements against external baselines on a simulator and a real DaVinci NPU, and the tiling search is auto-tuning of implementation parameters rather than a prediction fitted to its own outputs.

full rationale

The paper's central claims are empirical performance results: cycle counts, energy estimates, and real-hardware runtimes measured against external baselines (Layer-Wise, Soft-Pipe, FLAT, TileFlow, FuseMax). The tiling-factor search is an offline auto-tuning procedure for implementation parameters, and reporting the best found configuration is an engineering optimization result, not a derivation whose output is defined by its input. There is no equation in which a predicted speedup is constructed from the same measured quantity it is claimed to predict, and no fitted parameter is renamed as a prediction. The paper cites prior work for tools and baselines but does not rely on any self-citation chain as load-bearing evidence. The unvalidated assumption that the MAC unit can be preempted mid-MatMul in the proactive overwrite strategy is a hardware-support and modeling risk, not a circularity: it affects whether the long-sequence results transfer to real hardware, but it does not make the reported speedups reduce to the paper's assumptions by construction. No significant circularity is present.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The method's performance depends on tuned tiling parameters and on hardware capabilities (separate MAC/VEC units, preemptible MatMul, sufficient cache for two rows) that are not universal. No new physical or mathematical entities are introduced.

free parameters (1)
  • Tiling factors (Bb, Hh, NQ, NK,V) per memory level and workload = Not reported; optimized per network by MCTS, GA, or grid search
    The cycle and energy numbers in Tables 2 and 3 are achieved at the search-found tiling. No sensitivity analysis is given, and the values are not shared, so the reported gains are tied to these tuned parameters.
assumptions (4)
  • domain assumption Edge accelerator provides at least one MAC unit and one VEC unit that can execute tiles concurrently without mutual interference.
    Sections 3 and 4.1 build the pipeline on this heterogeneous execution model; the method does not apply to accelerators without separate schedulable compute units.
  • ad hoc to paper The MAC unit can be preempted mid-MatMul: softmax may overwrite K or V on-chip, halting the MAC, which later reloads and recomputes the tile.
    Introduced in Section 4.3 (Figures 2 and 3). This preemption and recompute capability is assumed, not demonstrated as a standard hardware feature, and it is load-bearing for the long-sequence and energy/read tradeoffs.
  • domain assumption On-chip memory holds at least one row of Pi plus one row of Pi-1 or Ci+1 simultaneously.
    Section 5.6 states this requirement explicitly and uses it to explain why MAS-Attention caps at half of FLAT's maximum sequence length.
  • domain assumption The simulated edge architecture (3.75 GHz, 16 nm, 30 GB/s DRAM, 5 MB L1, 16x16 MAC mesh, 256 VEC mesh) is representative of resource-constrained edge NPUs.
    Section 5.1 defines the architecture 'after various stress tests'; claims are evaluated within this model plus one DaVinci NPU. If this architecture is unrepresentative, the simulated speedups may not generalize.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MAS-Attention: Memory-Aware Stream Processing for Attention Acceleration on Resource-Constrained Edge Devices." pith.science (2026). https://pith.science/paper/YRWFK74Y

@misc{pith2026241117720,
  author       = {Pith},
  title        = {Pith review of: MAS-Attention: Memory-Aware Stream Processing for Attention Acceleration on Resource-Constrained Edge Devices},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YRWFK74Y}},
  note         = {Machine review of arXiv:2411.17720}
}
read the original abstract

The advent of foundation models have revolutionized various fields, enabling unprecedented task accuracy and flexibility in computational linguistics, computer vision and other domains. Attention mechanism has become an essential component of foundation models, due to their superb capability of capturing correlations in a sequence. However, attention results in quadratic complexity in memory and compute as the context length grows. Although many fusion-based exact attention acceleration algorithms have been developed for datacenter-grade GPUs and accelerators leveraging multi-core parallelism and data locality, yet it remains a significant challenge to accelerate attention on resource-constrained edge neural accelerators with limited compute units and stringent on-chip caches. In this paper, we propose a scheme for exact attention inference acceleration on memory-constrained edge accelerators, by parallelizing the utilization of heterogeneous compute units, i.e., vector processing units and matrix processing units. Our method involves scheduling workloads onto these different compute units in a multi-tiered tiling scheme to process tiled vector workloads and matrix workloads in attention as two streams, respecting the workload dependencies. We search for tiling factors to maximize the parallelization of both compute units while considering I/O overhead, and propose a proactive cache overwrite strategy to avoid undesirable cache spills in reality. Extensive results based on open-sourced simulation frameworks show up to 2.75x speedup and 54% reduction in energy consumption as compared to the state-of-the-art attention fusion method (FLAT) in the edge computing scenario. Further experiments on a real-world edge neural processing unit demonstrate speedup of up to 1.76x for attention as compared to FLAT, without affecting model output accuracy.

Figures

Figures reproduced from arXiv: 2411.17720 by the authors.

Figure 1
Figure 1. Dataflow comparison between FLAT and MAS-Attention: FLAT executes tiled stages sequentially, while MAS-Attention performs MatMul and softmax operations semi-synchronously in parallel, maximizing compute utilization and significantly enhancing overall performance. 13-17 of Algorithm 1. For iterations i ≥ 3, the MAC unit computes the tile for the final MatMul operator as Oi−2 = Pi−2V . Meanwhile, the VEC unit computes… view at source ↗
Figure 4
Figure 4. Simulated Edge Architecture Design 5 EXPERIMENTS 5.1 Experimental Setup This section provides details on the simulation and modeling tools utilized, describes the hardware specifications, and outlines the experimental workloads and baseline algorithms used for analysis. We conduct a comprehensive evaluation of the proposed method, comparing it against state-of-the￾art attention fusion and acceleration techniques tai… view at source ↗
Figure 3
Figure 3. Selective Overwriting of K Matrix to Halt MatMul Op￾eration in MAS-Attention’s Memory Strategy. on chip, thereby interrupting the MAC unit’s process and preventing any further writes to the on-chip memory. Once the final result of Pi is fully calculated and stored on chip, the MAC unit can resume its process by reloading either the V or K matrix from DRAM to on-chip memory if it was overwritten and redoing the MatMu… view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: Normalized Execution Time Comparison Across Net￾works for Different Methods on Huawei MatePad Pro 13.2 with DaVinci DNN Accelerator et al., 2023), which limited us from deploying it on this edge device. Overall, the data validates MAS-Attention’s effectiveness in enhan…
Figure 6
Figure 6. Figure 6: Energy Consumption Breakdown for DDR, L1, L0 mem￾ories and PEs within MAC and VEC units Across Networks using Different Methods 5.4 DRAM Access Analysis Since the FLAT method is most comparable to MAS￾Attention in terms of both cycle and energy performance, we will foc…
Figure 7
Figure 7. Figure 7: Execution cycles vs. search time (both log scale) for different attention acceleration methods, demonstrating the impact of Genetic Algorithm (GA) and Monte Carlo Tree Search (MCTS) on each algorithm’s efficiency energy consumption metrics upon completion of the search…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

80 extracted references · 43 canonical work pages

  1. [1]

    https://docs.nvidia.com/deeplearning/tensorrt/archives/tensorrt-803/best-practices/index.html

    Nvidia, TensorRT . https://docs.nvidia.com/deeplearning/tensorrt/archives/tensorrt-803/best-practices/index.html. Accessed: 2024

    work page 2024

  2. [2]

    https://developer.apple.com/documentation/accelerate/bnns, a

    Apple, Accelerate Framework . https://developer.apple.com/documentation/accelerate/bnns, a . Accessed: 2023

    work page 2023

  3. [3]

    https://apple.github.io/coremltools/docs-guides/source/opt-palettization-overview.html, b

    Apple, Core ML Tools . https://apple.github.io/coremltools/docs-guides/source/opt-palettization-overview.html, b . Accessed: 2023

    work page 2023

  4. [4]

    https://developer.apple.com/documentation/metalperformanceshadersgraph, c

    Apple, Metal Performance Shaders Graph . https://developer.apple.com/documentation/metalperformanceshadersgraph, c . Accessed: 2023

    work page 2023

  5. [5]

    T. Dao, D. Haziza, F. Massa, G. Sizov, Flash-Decoding for long-context inference . https://crfm.stanford.edu/2023/10/12/flashdecoding.html. Accessed: 2023-10-12

    work page 2023

  6. [6]

    https://www.tensorflow

    Google, TensorFlow XLA . https://www.tensorflow. Accessed: 2021

    work page 2021

  7. [7]

    Y., Rajbhandari, S., Awan, A

    Aminabadi, R. Y., Rajbhandari, S., Awan, A. A., Li, C., Li, D., Zheng, E., Ruwase, O., Smith, S., Zhang, M., Rasley, J., et al. Deepspeed-inference: enabling efficient inference of transformer models at unprecedented scale. In SC22: International Conference for High Performance Computing, Networking, Storage and Analysis, pp.\ 1--15. IEEE, 2022

    2022

  8. [8]

    B., Del Sozzo, E., Akkas, A., Zhang, Y., Suriana, P., Kamil, S., and Amarasinghe, S

    Baghdadi, R., Ray, J., Romdhane, M. B., Del Sozzo, E., Akkas, A., Zhang, Y., Suriana, P., Kamil, S., and Amarasinghe, S. Tiramisu: A polyhedral compiler for expressing fast and portable code. In 2019 IEEE/ACM International Symposium on Code Generation and Optimization (CGO), pp.\ 193--205. IEEE, 2019

    work page 2019

Show all 80 references
  1. [9]

    \ TVM \ : An automated \ End-to-End \ optimizing compiler for deep learning

    Chen, T., Moreau, T., Jiang, Z., Zheng, L., Yan, E., Shen, H., Cowan, M., Wang, L., Hu, Y., Ceze, L., et al. \ TVM \ : An automated \ End-to-End \ optimizing compiler for deep learning. In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18), pp.\ 578...

    2018

  2. [10]

    Learning to optimize tensor programs

    Chen, T., Zheng, L., Yan, E., Jiang, Z., Moreau, T., Ceze, L., Guestrin, C., and Krishnamurthy, A. Learning to optimize tensor programs. Advances in Neural Information Processing Systems, 31, 2018 b

    2018

  3. [11]

    Eyeriss v2: A flexible accelerator for emerging deep neural networks on mobile devices

    Chen, Y.-H., Yang, T.-J., Emer, J., and Sze, V. Eyeriss v2: A flexible accelerator for emerging deep neural networks on mobile devices. IEEE Journal on Emerging and Selected Topics in Circuits and Systems, 9 0 (2): 0 292--308, 2019

    2019

  4. [12]

    A., Adya, S., and Rastegari, M

    Cho, M., Vahid, K. A., Adya, S., and Rastegari, M. Dkm: Differentiable k-means clustering layer for neural network compression. arXiv preprint arXiv:2108.12659, 2021

    2021 arXiv

  5. [13]

    Kv-runahead: Scalable causal llm inference by parallel key-value cache generation

    Cho, M., Rastegari, M., and Naik, D. Kv-runahead: Scalable causal llm inference by parallel key-value cache generation. arXiv preprint arXiv:2405.05329, 2024

    2024 arXiv

  6. [14]

    Flashattention-2: Faster attention with better parallelism and work partitioning

    Dao, T. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691, 2023

    2023 arXiv

  7. [15]

    Flashattention: Fast and memory-efficient exact attention with io-awareness

    Dao, T., Fu, D., Ermon, S., Rudra, A., and R \'e , C. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35: 0 16344--16359, 2022

    2022

  8. [16]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018

    2018 arXiv

  9. [17]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020

    2010 arXiv

  10. [18]

    Scaling rectified flow transformers for high-resolution image synthesis

    Esser, P., Kulal, S., Blattmann, A., Entezari, R., M \"u ller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F., et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning, 2024

    2024

  11. [19]

    Videoagent: A memory-augmented multimodal agent for video understanding

    Fan, Y., Ma, X., Wu, R., Du, Y., Li, J., Gao, Z., and Li, Q. Videoagent: A memory-augmented multimodal agent for video understanding. In European Conference on Computer Vision, pp.\ 75--92. Springer, 2025

    2025

  12. [20]

    A., Yang, Y., Sajjad, H., Nakov, P., Chen, D., and Winslett, M

    Ganesh, P., Chen, Y., Lou, X., Khan, M. A., Yang, Y., Sajjad, H., Nakov, P., Chen, D., and Winslett, M. Compressing large-scale transformer-based models: A case study on bert. Transactions of the Association for Computational Linguistics, 9: 0 1061--1080, 2021

    2021

  13. [21]

    Collective loop fusion for array contraction

    Gao, G., Olsen, R., Sarkar, V., and Thekkath, R. Collective loop fusion for array contraction. In Languages and Compilers for Parallel Computing: 5th International Workshop New Haven, Connecticut, USA, August 3--5, 1992 Proceedings 5, pp.\ 281--295. Springer, 1993

    1992

  14. [22]

    Improving alignment of dialogue agents via targeted human judgements

    Glaese, A., McAleese, N., Tr e bacz, M., Aslanides, J., Firoiu, V., Ewalds, T., Rauh, M., Weidinger, L., Chadwick, M., Thacker, P., et al. Improving alignment of dialogue agents via targeted human judgements. arXiv preprint arXiv:2209.14375, 2022

    2022 arXiv

  15. [23]

    V., Prabhala, H., Paul, S., and Von Platen, P

    Gupta, Y., Jaddipal, V. V., Prabhala, H., Paul, S., and Von Platen, P. Progressive knowledge distillation of stable diffusion xl using layer level loss. arXiv preprint arXiv:2401.02677, 2024

    2024 arXiv

  16. [24]

    Flashdecoding++: Faster large language model inference on gpus

    Hong, K., Dai, G., Xu, J., Mao, Q., Li, X., Liu, J., Chen, K., Dong, H., and Wang, Y. Flashdecoding++: Faster large language model inference on gpus. arXiv preprint arXiv:2311.01282, 2023

    2023 arXiv

  17. [25]

    Knowledge diffusion for distillation

    Huang, T., Zhang, Y., Zheng, M., You, S., Wang, F., Qian, C., and Xu, C. Knowledge diffusion for distillation. Advances in Neural Information Processing Systems, 36, 2024

    2024

  18. [26]

    Data movement is all you need: A case study on optimizing transformers

    Ivanov, A., Dryden, N., Ben-Nun, T., Li, S., and Hoefler, T. Data movement is all you need: A case study on optimizing transformers. Proceedings of Machine Learning and Systems, 3: 0 711--732, 2021

    2021

  19. [27]

    P., Young, C., Patil, N., Patterson, D., Agrawal, G., Bajwa, R., Bates, S., Bhatia, S., Boden, N., Borchers, A., et al

    Jouppi, N. P., Young, C., Patil, N., Patterson, D., Agrawal, G., Bajwa, R., Bates, S., Bhatia, S., Boden, N., Borchers, A., et al. In-datacenter performance analysis of a tensor processing unit. In Proceedings of the 44th annual international symposium on computer architecture...

    2017

  20. [28]

    P., Yoon, D

    Jouppi, N. P., Yoon, D. H., Kurian, G., Li, S., Patil, N., Laudon, J., Young, C., and Patterson, D. A domain-specific supercomputer for training deep neural networks. Communications of the ACM, 63 0 (7): 0 67--78, 2020

    2020

  21. [29]

    Flat: An optimized dataflow for mitigating attention bottlenecks

    Kao, S.-C., Subramanian, S., Agrawal, G., Yazdanbakhsh, A., and Krishna, T. Flat: An optimized dataflow for mitigating attention bottlenecks. In Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volum...

    2023

  22. [30]

    B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D

    Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020

    2001 arXiv

  23. [31]

    Kirk, D. et al. Nvidia cuda software and gpu parallel computing architecture. In ISMM, volume 7, pp.\ 103--104, 2007

    2007

  24. [32]

    Reformer: The efficient transformer

    Kitaev, N., Kaiser, ., and Levskaya, A. Reformer: The efficient transformer. arXiv preprint arXiv:2001.04451, 2020

    2001 arXiv

  25. [33]

    The tensor algebra compiler

    Kjolstad, F., Kamil, S., Chou, S., Lugato, D., and Amarasinghe, S. The tensor algebra compiler. Proceedings of the ACM on Programming Languages, 1 0 (OOPSLA): 0 1--29, 2017

    2017

  26. [34]

    Maeri: Enabling flexible dataflow mapping over dnn accelerators via reconfigurable interconnects

    Kwon, H., Samajdar, A., and Krishna, T. Maeri: Enabling flexible dataflow mapping over dnn accelerators via reconfigurable interconnects. ACM SIGPLAN Notices, 53 0 (2): 0 461--475, 2018

    2018

  27. [35]

    H., Gonzalez, J., Zhang, H., and Stoica, I

    Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pp.\ 611--626, 2023

    2023

  28. [36]

    and Conneau, A

    Lample, G. and Conneau, A. Cross-lingual language model pretraining. arXiv preprint arXiv:1901.07291, 2019

    1901 arXiv

  29. [37]

    Multimodal foundation models: From specialists to general-purpose assistants

    Li, C., Gan, Z., Yang, Z., Yang, J., Li, L., Wang, L., Gao, J., et al. Multimodal foundation models: From specialists to general-purpose assistants. Foundations and Trends in Computer Graphics and Vision , 16 0 (1-2): 0 1--214, 2024 a

    2024

  30. [38]

    onednn graph compiler: A hybrid approach for high-performance deep learning compilation

    Li, J., Qin, Z., Mei, Y., Cui, J., Song, Y., Chen, C., Zhang, Y., Du, L., Cheng, X., Jin, B., et al. onednn graph compiler: A hybrid approach for high-performance deep learning compilation. In 2024 IEEE/ACM International Symposium on Code Generation and Optimization (CGO), pp....

    2024

  31. [39]

    Q-vit: Accurate and fully quantized low-bit vision transformer

    Li, Y., Xu, S., Zhang, B., Cao, X., Gao, P., and Guo, G. Q-vit: Accurate and fully quantized low-bit vision transformer. Advances in neural information processing systems, 35: 0 34451--34463, 2022

    2022

  32. [40]

    and Gu, Q

    Li, Z. and Gu, Q. I-vit: Integer-only quantization for efficient vision transformer inference. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 17065--17075, 2023

    2023

  33. [41]

    Davinci: A scalable architecture for neural network computing

    Liao, H., Tu, J., Xia, J., and Zhou, X. Davinci: A scalable architecture for neural network computing. In 2019 IEEE Hot Chips 31 Symposium (HCS), pp.\ 1--44. IEEE Computer Society, 2019

    2019

  34. [42]

    Fq-vit: Post-training quantization for fully quantized vision transformer

    Lin, Y., Zhang, T., Sun, P., Li, Z., and Zhou, S. Fq-vit: Post-training quantization for fully quantized vision transformer. arXiv preprint arXiv:2111.13824, 2021

    2021 arXiv

  35. [43]

    Ring attention with blockwise transformers for near-infinite context

    Liu, H., Zaharia, M., and Abbeel, P. Ring attention with blockwise transformers for near-infinite context. arXiv preprint arXiv:2310.01889, 2023

    2023 arXiv

  36. [44]

    Post-training quantization for vision transformer

    Liu, Z., Wang, Y., Han, K., Zhang, W., Ma, S., and Gao, W. Post-training quantization for vision transformer. Advances in Neural Information Processing Systems, 34: 0 28092--28103, 2021

    2021

  37. [45]

    Tprune: Efficient transformer pruning for mobile devices

    Mao, J., Yang, H., Li, A., Li, H., and Chen, Y. Tprune: Efficient transformer pruning for mobile devices. ACM Transactions on Cyber-Physical Systems, 5 0 (3): 0 1--22, 2021

    2021

  38. [46]

    H., Cao, Q., Horton, M., Jin, Y., Sun, C., Mirzadeh, I., Najibi, M., Belenko, D., Zatloukal, P., et al

    Mehta, S., Sekhavat, M. H., Cao, Q., Horton, M., Jin, Y., Sun, C., Mirzadeh, I., Najibi, M., Belenko, D., Zatloukal, P., et al. Openelm: An efficient language model family with open-source training and inference framework. arXiv preprint arXiv:2404.14619, 2024

    2024 arXiv

  39. [47]

    Defines: Enabling fast exploration of the depth-first scheduling space for dnn accelerators through analytical modeling

    Mei, L., Goetschalckx, K., Symons, A., and Verhelst, M. Defines: Enabling fast exploration of the depth-first scheduling space for dnn accelerators through analytical modeling. In 2023 IEEE International Symposium on High-Performance Computer Architecture (HPCA), pp.\ 570--583...

    2023

  40. [48]

    Efficient large-scale language model training on gpu clusters using megatron-lm

    Narayanan, D., Shoeybi, M., Casper, J., LeGresley, P., Patwary, M., Korthikanti, V., Vainbrand, D., Kashinkunti, P., Bernauer, J., Catanzaro, B., et al. Efficient large-scale language model training on gpu clusters using megatron-lm. In Proceedings of the International Confere...

    2021

  41. [49]

    O., Pellauer, M., Emer, J

    Nayak, N., Wu, X., Odemuyiwa, T. O., Pellauer, M., Emer, J. S., and Fletcher, C. W. Fusemax: Leveraging extended einsums to optimize attention accelerator design. arXiv preprint arXiv:2406.10491, 2024

    2024

  42. [50]

    Dnnfusion: accelerating deep neural networks execution with advanced operator fusion

    Niu, W., Guan, J., Wang, Y., Agrawal, G., and Ren, B. Dnnfusion: accelerating deep neural networks execution with advanced operator fusion. In Proceedings of the 42nd ACM SIGPLAN International Conference on Programming Language Design and Implementation, pp.\ 883--898, 2021

    2021

  43. [51]

    Training language models to follow instructions with human feedback

    Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35: 0 27730--27744, 2022

    2022

  44. [52]

    S., Chen, Y.-H., Ying, V

    Parashar, A., Raina, P., Shao, Y. S., Chen, Y.-H., Ying, V. A., Mukkara, A., Venkatesan, R., Khailany, B., Keckler, S. W., and Emer, J. Timeloop: A systematic approach to dnn accelerator evaluation. In 2019 IEEE international symposium on performance analysis of systems and so...

    2019

  45. [53]

    Splitwise: Efficient generative llm inference using phase splitting

    Patel, P., Choukse, E., Zhang, C., Shah, A., Goiri, \'I ., Maleki, S., and Bianchini, R. Splitwise: Efficient generative llm inference using phase splitting. Power, 400 0 (700W): 0 1--75, 2023

    2023

  46. [54]

    and Xie, S

    Peebles, W. and Xie, S. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 4195--4205, 2023

    2023

  47. [55]

    Accelerating transformer-based deep learning models on fpgas using column balanced block pruning

    Peng, H., Huang, S., Geng, T., Li, A., Jiang, W., Liu, H., Wang, S., and Ding, C. Accelerating transformer-based deep learning models on fpgas using column balanced block pruning. In 2021 22nd International Symposium on Quality Electronic Design (ISQED), pp.\ 142--148. IEEE, 2021

    2021

  48. [56]

    Sensimix: Sensitivity-aware 8-bit index & 1-bit value mixed precision quantization for bert compression

    Piao, T., Cho, I., and Kang, U. Sensimix: Sensitivity-aware 8-bit index & 1-bit value mixed precision quantization for bert compression. PloS one, 17 0 (4): 0 e0265621, 2022

    2022

  49. [57]

    T., and Mildenhall, B

    Poole, B., Jain, A., Barron, J. T., and Mildenhall, B. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209.14988, 2022

    2022 arXiv

  50. [58]

    Improving language understanding by generative pre-training

    Radford, A., Narasimhan, K., Salimans, T., Sutskever, I., et al. Improving language understanding by generative pre-training. 2018

    2018

  51. [59]

    Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 0 (140): 0 1--67, 2020

    2020

  52. [60]

    Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters

    Rasley, J., Rajbhandari, S., Ruwase, O., and He, Y. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp.\ 3505--3506, 2020

    2020

  53. [61]

    Flashattention-3: Fast and accurate attention with asynchrony and low-precision

    Shah, J., Bikshandi, G., Zhang, Y., Thakkar, V., Ramani, P., and Dao, T. Flashattention-3: Fast and accurate attention with asynchrony and low-precision. arXiv preprint arXiv:2407.08608, 2024

    2024 arXiv

  54. [62]

    Megatron-lm: Training multi-billion parameter language models using model parallelism

    Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., and Catanzaro, B. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019

    1909 arXiv

  55. [63]

    Patient knowledge distillation for bert model compression

    Sun, S., Cheng, Y., Gan, Z., and Liu, J. Patient knowledge distillation for bert model compression. arXiv preprint arXiv:1908.09355, 2019

    1908 arXiv

  56. [64]

    Improving the efficiency of transformers for resource-constrained devices

    Tabani, H., Balasubramaniam, A., Marzban, S., Arani, E., and Zonooz, B. Improving the efficiency of transformers for resource-constrained devices. In 2021 24th Euromicro Conference on Digital System Design (DSD), pp.\ 449--456. IEEE, 2021

    2021

  57. [65]

    Llama: Open and efficient foundation language models

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozi \`e re, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  58. [66]

    N., Kaiser, ., and Polosukhin, I

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, ., and Polosukhin, I. Attention is all you need. Advances in neural information processing systems, 30, 2017

    2017

  59. [67]

    Mobile-agent: Autonomous multi-modal mobile device agent with visual perception

    Wang, J., Xu, H., Ye, J., Yan, M., Shen, W., Zhang, J., Huang, F., and Sang, J. Mobile-agent: Autonomous multi-modal mobile device agent with visual perception. arXiv preprint arXiv:2401.16158, 2024

    2024 arXiv

  60. [68]

    C., Venkataramani, S., Sen, S., Chen, C.-Y., El Maghraoui, K., Srinivasan, V

    Wang, N., Liu, C.-C. C., Venkataramani, S., Sen, S., Chen, C.-Y., El Maghraoui, K., Srinivasan, V. V., and Chang, L. Deep compression of pre-trained transformer models. Advances in Neural Information Processing Systems, 35: 0 14140--14154, 2022

    2022

  61. [69]

    Cluster-former: Clustering-based sparse transformer for long-range dependency encoding

    Wang, S., Zhou, L., Gan, Z., Chen, Y.-C., Fang, Y., Sun, S., Cheng, Y., and Liu, J. Cluster-former: Clustering-based sparse transformer for long-range dependency encoding. arXiv preprint arXiv:2009.06097, 2020 a

    2009 arXiv

  62. [70]

    Minilmv2: Multi-head self-attention relation distillation for compressing pretrained transformers

    Wang, W., Bao, H., Huang, S., Dong, L., and Wei, F. Minilmv2: Multi-head self-attention relation distillation for compressing pretrained transformers. arXiv preprint arXiv:2012.15828, 2020 b

    2012 arXiv

  63. [71]

    Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers

    Wang, W., Wei, F., Dong, L., Bao, H., Yang, N., and Zhou, M. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. Advances in Neural Information Processing Systems, 33: 0 5776--5788, 2020 c

    2020

  64. [72]

    N., Emer, J

    Wu, Y. N., Emer, J. S., and Sze, V. Accelergy: An architecture-level energy estimation methodology for accelerator designs. In 2019 IEEE/ACM International Conference on Computer-Aided Design (ICCAD), pp.\ 1--8. IEEE, 2019

    2019

  65. [73]

    Zeroquant: Efficient and affordable post-training quantization for large-scale transformers

    Yao, Z., Yazdani Aminabadi, R., Zhang, M., Wu, X., Li, C., and He, Y. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. Advances in Neural Information Processing Systems, 35: 0 27168--27183, 2022

    2022

  66. [74]

    Boost vision transformer with gpu-friendly sparsity and quantization

    Yu, C., Chen, T., Gan, Z., and Fan, J. Boost vision transformer with gpu-friendly sparsity and quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 22658--22668, 2023

    2023

  67. [75]

    Width & depth pruning for vision transformers

    Yu, F., Huang, K., Wang, M., Cheng, Y., Chu, W., and Cui, L. Width & depth pruning for vision transformers. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp.\ 3143--3151, 2022 a

    2022

  68. [76]

    Unified visual transformer compression

    Yu, S., Chen, T., Shen, J., Yuan, H., Tan, J., Yang, S., Liu, J., and Wang, Z. Unified visual transformer compression. arXiv preprint arXiv:2203.08243, 2022 b

    2022 arXiv

  69. [77]

    Appagent: Multimodal agents as smartphone users

    Zhang, C., Yang, Z., Liu, J., Han, Y., Chen, X., Huang, Z., Fu, B., and Yu, G. Appagent: Multimodal agents as smartphone users. arXiv preprint arXiv:2312.13771, 2023

    2023 arXiv

  70. [78]

    Tileflow: A framework for modeling fusion dataflow via tree-based analysis

    Zheng, S., Chen, S., Gao, S., Jia, L., Sun, G., Wang, R., and Liang, Y. Tileflow: A framework for modeling fusion dataflow via tree-based analysis. In Proceedings of the 56th Annual IEEE/ACM International Symposium on Microarchitecture, pp.\ 1271--1288, 2023

    2023

  71. [79]

    and Yang, K

    Zhou, Y. and Yang, K. Exploring tensorrt to improve real-time inference for deep learning. In 2022 IEEE 24th Int Conf on High Performance Computing & Communications; 8th Int Conf on Data Science & Systems; 20th Int Conf on Smart City; 8th Int Conf on Dependability in Sensor, C...

    2022

  72. [80]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.