Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

Towards 3D Acceleration for low-power Mixture-of-Experts and Multi-Head Attention Spiking Transformers

T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Pith's one-line read This paper claims that face-to-face-bonded 3D integration can accelerate spiking Mixture-of-Experts and Multi-Head Attention transformers, beating 2D CMOS on area, frequency, memory-access latency, and power.

desk verdict Genuine first 3D architecture for spiking MoE transformers, but the F2F-specific PPA claims rest on a monolithic-3D flow and a misreported table. read the letter →

arxiv 2412.05540 v1 pith:F7AAES3W submitted 2024-12-07 cs.NE cs.AIcs.AR

classification cs.NEcs.AIcs.AR
keywords spikingneuralnetworkstransformersmixture-of-expertsmulti-headattention3Dintegrationface-to-facebondinghardwareacceleratorlowpower
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that face-to-face-bonded 3D integration is a practical way to build accelerators for spiking Mixture-of-Experts (MoE) and Multi-Head Attention (MHA) transformers, and it reports concrete gains over a 2D CMOS baseline: 3-5.1% higher effective frequency, 39-41% area reduction, 26.9-30% lower memory-access latency, and up to 14.4% lower total power. The motivation is that spiking transformers, which combine event-driven binary activations with attention and conditional expert routing, currently lack dedicated hardware that exploits their parallel distributed structure. If the claim holds, 3D integration offers a path toward low-power, scalable on-chip hardware for brain-inspired transformer workloads.

What carries the argument

The load-bearing object is the modularized two-tier spiking expert core: a bottom-tier dense systolic PE array performs synaptic integration of 1-bit spiking activities with multi-bit expert weights, and a top-tier spiking generator accumulates membrane potentials and conditionally emits spikes, with synaptic-integration results extracted vertically through dedicated readout ports. A spiking token router with an expert-routing score array handles conditional top-K routing, while reconfigurable attention PEs compute both $QK^T$ attention maps and $AV$ integrations without moving multi-bit attention data out of the array. This machinery turns expert parallelism in MoE and head parallelism in MHA into spatially distributed 3D compute blocks with short inter-tier wires.

What would settle it

Re-run the 2D baseline with a tighter floorplan or a published spiking-transformer accelerator as reference and measure PPA; if the 3D design no longer shows the claimed 39-41% area reduction and up to 14.4% power savings, the central claim fails. Alternatively, a thermal or yield simulation of the 0.5-1 µm F2F stack that shows performance-degrading hotspots would falsify the practical deployment claim.

Watch

Extended reading notes

Core claim

The paper presents, to its knowledge, the first dedicated 3D accelerator architecture for spiking MoE transformers, assembling modularized two-tier Spiking Expert cores for MoE layers and modularized Spiking Attention cores for MHA, connected by face-to-face (F2F) bonding. In a 28nm PDK, the 3D implementation of the four-expert system runs at 2.24 GHz (MHA) and 1.74 GHz (MoE) versus 2.13 GHz and 1.69 GHz for the 2D versions, occupies 3.36 mm² versus 5.53 mm² (MHA) and 1.75 mm² versus 2.97 mm² (MoE), cuts memory-access latency by 30% (MHA) and 15% (MoE), and reduces total power by up to 14.4% (MoE). The gains are attributed to shortened vertical wires and the memory-on-logic and logic-on-logic partitioning enabled by 3D stacking.

Load-bearing premise

The 3D gains are all measured against the authors' own 2D implementations of the same RTL, and the design assumes 0.5-1 µm F2F bond stacks are manufacturable with negligible yield and thermal penalties; if either premise fails, the reported improvements could shrink or disappear.

Editorial extensions

If this is right

  • If the reported PPA gains hold, F2F-bonded 3D integration can narrow the gap between spiking algorithms and the energy efficiency they promise on hardware, without requiring novel memory devices.
  • Lower memory-access latency and power at the global-buffer and local-buffer levels make scaling to more experts more plausible, since shared expert weights and activation traffic become cheaper in the vertical stack.
  • Kernel-fused attention that keeps multi-bit attention maps in registers removes a class of data-movement costs, so adding attention heads should cost less memory traffic than in a 2D implementation.
  • The modular core-based methodology can be reused for other spiking transformer variants, not just the CIFAR-10/CIFAR-100 configurations evaluated here.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The accuracy figures in Table I are model-level results and do not by themselves validate the hardware; the hardware claim rests entirely on the 2D-versus-3D PPA comparison, so the choice of 2D baseline is the decisive test.
  • If the per-expert modularity scales to larger expert counts, tiling more two-tier cores could support hundreds of experts, but inter-expert routing bandwidth and thermal dissipation in the stack, which the paper does not model, would bound that scaling.
  • A testable extension would apply the same 3D partition to a dense (non-MoE) spiking transformer; if the memory-access gains persist without expert diversity, the benefit is general wirelength reduction rather than expert parallelism.
  • The 0.5-1 µm F2F bond pitch is assumed manufacturable with negligible yield or thermal penalty; a foundry-validated stack could change the power and latency numbers, so the reported improvements should be read as design-space estimates.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes a 3D-integrated accelerator for spiking Mixture-of-Experts (MoE) and Multi-Head Attention (MHA) transformers. The architecture maps spiking experts and attention heads onto modularized two-tier cores, with memory-on-logic and logic-on-logic stacking, and employs kernel-fused dataflows across spiking routing, synaptic integration, and spike generation. The evaluation compares 2D and 3D physical implementations using a 28nm PDK and reports accuracy scaling on CIFAR-10/CIFAR-100 with increasing numbers of experts, plus PPA gains for the 3D designs: 3%-5.1% higher effective frequency, 39%-41% area reduction, 14.9%-30% memory access latency reduction, and up to 14.4% total power reduction. The paper claims to be the first dedicated 3D accelerator architecture for spiking MoE transformers.

Significance. If the F2F-specific PPA results are substantiated, the work would provide a useful demonstration that face-to-face 3D integration can benefit spiking transformer workloads, and the proposed modularized expert/attention cores are a sensible architectural contribution. The accuracy scaling in Table I is a reasonable algorithmic sanity check. However, the central physical-design claim is currently not supported by the evidence: the 3D implementations are run through the pin-3D flow, which targets monolithic 3D integration, and the paper does not model F2F bond parasitics or thermal effects. The reported gains therefore cannot yet be attributed to F2F bonding. The paper also contains a direct numerical misreporting in the abstract. These issues are load-bearing for the main claim, so the manuscript requires major revision.

major comments (3)
  1. [Section V-A, reference [27]] The 3D designs are implemented using the pin-3D flow, which is a physical synthesis and post-layout optimization flow for heterogeneous monolithic 3D ICs. The manuscript states only the F2F bond pitch (0.5 to 1 um) and does not report extraction of bond resistance/capacitance, bond-aware timing or power analysis, or thermal signoff. Since the abstract and conclusion attribute the PPA improvements to F2F bonding, the experimental flow does not distinguish an F2F advantage from a monolithic-3D or idealized-bond advantage. Please either use an F2F-aware physical design methodology with bond parasitic and thermal modeling, or reframe the claims to generic 3D integration.
  2. [Abstract and Introduction, Section V-B Table II] The abstract and introduction state that the 3D accelerators provide a '26.9%-29% memory access latency reduction,' but Table II shows latency reductions of 30% for MHA (160 ps to 112 ps) and only 14.9% for MoE (202 ps to 172 ps). The values 29% and 26.9% are the memory access power reductions, not latency reductions. This misreporting should be corrected so that the stated claims match the data.
  3. [Section V-B, Tables II and III] The 'Memory Access Latency' metric in Table II is inconsistent with the hierarchical latencies in Table III. For example, the MHA 2D activation GLB latency is 220 ps in Table III, yet the overall memory access latency in Table II is 160 ps, which is smaller than this single component and does not correspond to any obvious sum, maximum, or weighted combination of the listed hierarchical latencies. The metric should be defined precisely and the numbers reconciled, since the claimed latency reductions depend on this quantity.
minor comments (6)
  1. [Section VI] The conclusion contains a typo: 'acclerators' should be 'accelerators.'
  2. [Section V-B heading] The heading 'Layout Comparision between 2D and 3D' should read 'Comparison.'
  3. [Abstract] The phrase 'the number of computational operation' should be 'the number of computational operations.'
  4. [Section V-A] The text says 'we adapt two 8K ×128b SRAM units' where 'adopt' is the intended word; similar usage appears elsewhere.
  5. [References] References [11] and [17] are the same paper ('Parallel time batching: Systolic-array acceleration of sparse spiking neural computation') and should be consolidated.
  6. [Figures 4 and 5] The layout figures are difficult to read in the current resolution; please provide zoomed views or higher-resolution images so that the placement details discussed in the text are visible.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the 2D-vs-3D PPA comparison is an in-paper physical-design evaluation, and self-cited tools are used as implementation aids rather than load-bearing premises.

full rationale

The paper is an implementation and measurement study rather than a derivation from first principles. Its central claim, that a face-to-face-bonded 3D implementation of spiking MoE and MHA accelerators improves frequency, area, memory-access latency, and power over a 2D implementation, is supported by in-paper physical synthesis results (Table II) comparing 2D and 3D implementations of the same RTL. No parameter is fitted to the reported outcomes, and no quantity presented as a prediction is constructed from the data it claims to predict. The cited pin-3D flow [27], which shares an author with the present paper, is invoked as an EDA tool for performing the 3D physical implementation; the paper's conclusion rests on the resulting measured metrics, not on any previously asserted theorem or uniqueness claim from that citation. Other self-citations ([11], [16], [17], [24], [25]) appear in background discussion of prior neuromorphic and 3D LSM work and are not load-bearing in the evaluation. Therefore no circular step can be exhibited from the paper's text. Concerns about whether pin-3D adequately models true F2F bond parasitics and thermal effects are correctness or validation issues, not circularity.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central 2D vs 3D comparison does not fit any parameters, so the free-parameter list is empty. The load-bearing assumptions are the manufacturability of the F2F stack, the fairness of the in-house 2D baseline, and the LIF workload model. No new physical entities are introduced.

assumptions (3)
  • domain assumption F2F bonding with 0.5-1 µm bond pitch yields manufacturable two-tier stacks at 28nm with negligible yield and thermal impact.
    The entire 3D PPA comparison assumes the stacked dies can be bonded reliably and that thermal and yield effects do not change the electrical results; none of these are modeled in the paper.
  • domain assumption The 2D baseline is a fairly optimized design using the same cell library, memory macros, and synthesis effort as the 3D version.
    All reported improvements are computed against the authors' own 2D floorplan; an under-optimized baseline would inflate the 3D gains, and no external 2D accelerator is compared.
  • standard math The LIF neuron model and binary spike matrix multiplication in Eq. 1-2 correctly capture spiking transformer computation.
    The accelerator PEs implement these equations; this is standard background, but the hardware validation assumes the equations are the right workload model.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards 3D Acceleration for low-power Mixture-of-Experts and Multi-Head Attention Spiking Transformers." pith.science (2026). https://pith.science/paper/F7AAES3W

@misc{pith2026241205540,
  author       = {Pith},
  title        = {Pith review of: Towards 3D Acceleration for low-power Mixture-of-Experts and Multi-Head Attention Spiking Transformers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/F7AAES3W}},
  note         = {Machine review of arXiv:2412.05540}
}
read the original abstract

Spiking Neural Networks(SNNs) provide a brain-inspired and event-driven mechanism that is believed to be critical to unlock energy-efficient deep learning. The mixture-of-experts approach mirrors the parallel distributed processing of nervous systems, introducing conditional computation policies and expanding model capacity without scaling up the number of computational operations. Additionally, spiking mixture-of-experts self-attention mechanisms enhance representation capacity, effectively capturing diverse patterns of entities and dependencies between visual or linguistic tokens. However, there is currently a lack of hardware support for highly parallel distributed processing needed by spiking transformers, which embody a brain-inspired computation. This paper introduces the first 3D hardware architecture and design methodology for Mixture-of-Experts and Multi-Head Attention spiking transformers. By leveraging 3D integration with memory-on-logic and logic-on-logic stacking, we explore such brain-inspired accelerators with spatially stackable circuitry, demonstrating significant optimization of energy efficiency and latency compared to conventional 2D CMOS integration.

Figures

Figures reproduced from arXiv: 2412.05540 by the authors.

Figure 1
Figure 1. (a) Model Architecture of the Spiking Mixture-of-Experts (MoE) Transformer, comprising a core backbone with spiking MoE and spiking MHA [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. (a) Proposed 3D Architecture for processing Spiking Mixture-of-Experts layers: Two-tier 3D Partitioning and Dataflow (b) 3D Extractable PE Design [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. (a) Proposed 3D Architecture for processing Spiking Multi-head Attention layers (b) Reconfigurable PE Design [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: The placement and layout comparison of 2D and 3D spiking MoE [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: The placement and layout of 2D and 3D spiking MHA accelerators. [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SpikeX: Exploring Accelerator Architecture and Network-Hardware Co-Optimization for Sparse Spiking Neural Networks

    cs.NE 2025-05 conditional novelty 6.0 of 10

    A simulated spike-based accelerator with spatiotemporal dispatch and sparsity-aware training claims 15.1x to 150.87x better energy-delay product than a prior SNN systolic baseline.

Reference graph

Works this paper leans on

26 extracted references · 18 canonical work pages · cited by 1 Pith paper

  1. [1]

    An image is worth 16x16 words: Transformers for image recognition at scale,

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” in International Conference on Learning Representations , 2021. [Online]. Available: https://openreview.net/forum?id=YicbFdNTTy

  2. [2]

    Zero-shot text-to-image generation,

    A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. V oss, A. Radford, M. Chen, and I. Sutskever, “Zero-shot text-to-image generation,” in Proceedings of the 38th International Conference on Machine Learning , ser. Proceedings of Machine Learning Research, M. Meila and T. Zhang, Eds., vol. 139. PMLR, 18–24 Jul 2021, pp. 8821–8831. [Online]. Available: https://proce...

  3. [3]

    Mistral 7b,

    A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. l. Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier et al. , “Mistral 7b,” arXiv preprint arXiv:2310.06825 , 2023

  4. [4]

    Unified scaling laws for routed language models,

    A. Clark, D. de Las Casas, A. Guy, A. Mensch, M. Paganini, J. Hoff- mann, B. Damoc, B. Hechtman, T. Cai, S. Borgeaud et al. , “Unified scaling laws for routed language models,” in International conference on machine learning . PMLR, 2022, pp. 4057–4086

  5. [5]

    Scaling vision with sparse mixture of experts,

    C. Riquelme, J. Puigcerver, B. Mustafa, M. Neumann, R. Jenatton, A. Susano Pinto, D. Keysers, and N. Houlsby, “Scaling vision with sparse mixture of experts,” Advances in Neural Information Processing Systems, vol. 34, pp. 8583–8595, 2021

  6. [6]

    M3vit: Mixture-of-experts vision transformer for efficient multi- task learning with model-accelerator co-design,

    Z. Fan, R. Sarkar, Z. Jiang, T. Chen, K. Zou, Y . Cheng, C. Hao, Z. Wang et al., “M3vit: Mixture-of-experts vision transformer for efficient multi- task learning with model-accelerator co-design,” Advances in Neural Information Processing Systems , vol. 35, pp. 28 441–28 457, 2022

  7. [7]

    Networks of spiking neurons: The third generation of neural network models,

    W. Maass, “Networks of spiking neurons: The third generation of neural network models,” Neural Networks , vol. 10, no. 9, pp. 1659–1671, 1997. [Online]. Available: https://www.sciencedirect.com/ science/article/pii/S0893608097000117

  8. [8]

    Towards spike-based machine intel- ligence with neuromorphic computing,

    K. Roy, A. Jaiswal, and P. Panda, “Towards spike-based machine intel- ligence with neuromorphic computing,” Nature, vol. 575, p. 607–617, 2019

Show all 26 references
  1. [9]

    Truenorth: Design and tool flow of a 65 mw 1 million neuron programmable neurosynaptic chip,

    F. Akopyan, J. Sawada, A. Cassidy, R. Alvarez-Icaza, J. Arthur, P. Merolla, N. Imam, Y . Nakamura, P. Datta, G.-J. Nam et al. , “Truenorth: Design and tool flow of a 65 mw 1 million neuron programmable neurosynaptic chip,” IEEE transactions on computer- aided design of integra...

  2. [10]

    Loihi: A neuromorphic manycore processor with on-chip learning,

    M. Davies, N. Srinivasa, T.-H. Lin, G. Chinya, Y . Cao, S. H. Choday, G. Dimou, P. Joshi, N. Imam, S. Jain et al. , “Loihi: A neuromorphic manycore processor with on-chip learning,” Ieee Micro, vol. 38, no. 1, pp. 82–99, 2018

  3. [12]

    Spikformer: When spiking neural network meets transformer,

    Z. Zhou, Y . Zhu, C. He, Y . Wang, S. Y AN, Y . Tian, and L. Yuan, “Spikformer: When spiking neural network meets transformer,” in The Eleventh International Conference on Learning Representations , 2023

  4. [13]

    Spiking transformers for event-based single object tracking,

    J. Zhang, B. Dong, H. Zhang, J. Ding, F. Heide, B. Yin, and X. Yang, “Spiking transformers for event-based single object tracking,” in 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 8791–8800

  5. [14]

    Spikegpt: Generative pre-trained language model with spiking neural networks,

    R.-J. Zhu, Q. Zhao, G. Li, and J. K. Eshraghian, “Spikegpt: Generative pre-trained language model with spiking neural networks,” 2023

  6. [15]

    Spike- driven transformer,

    M. Yao, J. Hu, Z. Zhou, L. Yuan, Y . Tian, B. Xu, and G. Li, “Spike- driven transformer,” arXiv preprint arXiv:2307.01694 , 2023

  7. [16]

    Reconfigurable dataflow optimization for spatiotem- poral spiking neural computation on systolic array accelerators,

    J.-J. Lee and P. Li, “Reconfigurable dataflow optimization for spatiotem- poral spiking neural computation on systolic array accelerators,” in 2020 IEEE 38th International Conference on Computer Design (ICCD) , 2020, pp. 57–64

  8. [17]

    Parallel time batching: Systolic- array acceleration of sparse spiking neural computation,

    J.-J. Lee, W. Zhang, and P. Li, “Parallel time batching: Systolic- array acceleration of sparse spiking neural computation,” in 2022 IEEE International Symposium on High-Performance Computer Architecture (HPCA), 2022, pp. 317–330

  9. [18]

    Loas: Fully temporal- parallel dataflow for dual-sparse spiking neural networks,

    R. Yin, Y . Kim, D. Wu, and P. Panda, “Loas: Fully temporal- parallel dataflow for dual-sparse spiking neural networks,”arXiv preprint arXiv:2407.14073, 2024

  10. [19]

    Spinalflow: An architecture and dataflow tailored for spiking neural networks,

    S. Narayanan, K. Taht, R. Balasubramonian, E. Giacomin, and P.-E. Gaillardon, “Spinalflow: An architecture and dataflow tailored for spiking neural networks,” in Proceedings of the ACM/IEEE 47th Annual International Symposium on Computer Architecture , ser. ISCA ’20. IEEE Pres...

  11. [20]

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity,

    W. Fedus, B. Zoph, and N. Shazeer, “Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity,” Journal of Machine Learning Research , vol. 23, no. 120, pp. 1–39, 2022

  12. [21]

    3d-carbon: An analytical carbon modeling tool for 3d and 2.5 d integrated circuits,

    Y . Zhao, Y . Zhao, C. Wan, and Y . Lin, “3d-carbon: An analytical carbon modeling tool for 3d and 2.5 d integrated circuits,” in Proceedings of the 61st ACM/IEEE Design Automation Conference , 2024, pp. 1–6

  13. [22]

    Nebula: A neuromorphic spin-based ultra-low power architecture for snns and anns,

    S. Singh, A. Sarma, N. Jao, A. Pattnaik, S. Lu, K. Yang, A. Sengupta, V . Narayanan, and C. R. Das, “Nebula: A neuromorphic spin-based ultra-low power architecture for snns and anns,” in 2020 ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA) . IEEE, ...

  14. [23]

    30.2 a 22nm 0.26 nw/synapse spike-driven spiking neural network processing unit using time-step-first dataflow and sparsity-adaptive in-memory computing,

    Y . Liu, Y . Ma, N. Shang, T. Zhao, P. Chen, M. Wu, J. Ru, T. Jia, L. Ye, Z. Wang et al. , “30.2 a 22nm 0.26 nw/synapse spike-driven spiking neural network processing unit using time-step-first dataflow and sparsity-adaptive in-memory computing,” in 2024 IEEE International Sol...

  15. [24]

    Design and architectural co-optimization of monolithic 3d liquid state machine-based neuromorphic processor,

    B. W. Ku, Y . Liu, Y . Jin, S. Samal, P. Li, and S. K. Lim, “Design and architectural co-optimization of monolithic 3d liquid state machine-based neuromorphic processor,” in Proceedings of the 55th Annual Design Automation Conference , ser. DAC ’18. New York, NY , USA: Associa...

  16. [25]

    Area-efficient and low-power face-to-face-bonded 3d liquid state machine design,

    B. W. Ku, Y . Liu, Y . Jin, P. Li, and S. K. Lim, “Area-efficient and low-power face-to-face-bonded 3d liquid state machine design,” in 2018 IEEE/ACM International Conference on Computer-Aided Design (ICCAD). IEEE, 2018, pp. 1–6

  17. [26]

    The cifar-10 dataset,

    A. Krizhevsky, V . Nair, and G. Hinton, “The cifar-10 dataset,” online: http://www. cs. toronto. edu/ kriz/cifar .html , 2014

  18. [27]

    Pin-3d: a physical synthesis and post-layout optimization flow for heterogeneous monolithic 3d ics,

    S. S. K. Pentapati, K. Chang, V . Gerousis, R. Sengupta, and S. K. Lim, “Pin-3d: a physical synthesis and post-layout optimization flow for heterogeneous monolithic 3d ics,” in Proceedings of the 39th International Conference on Computer-Aided Design , ser. ICCAD ’20. New York...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.