Pith. sign in

REVIEW 3 major objections 4 minor 21 references

XAMBA: Enabling Efficient State Space Models on Resource-Constrained Neural Processing Units

T0 review · 3 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash

Pith's one-line read XAMBA turns Mamba-2's serial sums and activations into parallel matrix operations, slashing NPU latency by up to 4.8x while keeping quality within 1.5 points.

desk verdict Genuinely useful first result for SSMs on commercial NPUs; CumBA/ReduBA hold up, but ActiBA's speedup and accuracy are measured on two different functions. read the letter →

arxiv 2502.06924 v4 pith:CO57YG7U submitted 2025-02-10 cs.LG cs.AI

classification cs.LGcs.AI
keywords statespacemodelsMambaMamba-2NPUoptimizationcumulativesumasmatrixmultiplypiecewiselinearactivationapproximationedgeAIinferencelatencyreduction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

XAMBA argues that state-space models like Mamba and Mamba-2, despite their linear-scaling appeal, run inefficiently on commercial laptop NPUs because their sequential operations—cumulative sum, reduce sum, and nonlinear activations—execute serially on the DSP while the NPU's parallel MAC array sits idle. The paper replaces the sequential CumSum with a matrix multiplication against a precomputed triangular mask (CumBA) and ReduceSum with a matrix-vector product (ReduBA), offloading both to the parallel MPUs. For activations, ActiBA maps Swish and Softplus onto the NPU's piecewise linear unit using a configurable lookup table. On an Intel Core Ultra Series 2 NPU, these changes cut Mamba-2 130M latency by up to 4.8x and Mamba 130M activation-bound latency by up to 2.6x, with accuracy changes under 1.5 percentage points. The point is that models can be optimized to existing NPU hardware rather than requiring new accelerators.

What carries the argument

The central object is the lower-triangular CumBA mask $M_{\mathrm{CumBA}}$ with $M_{\mathrm{CumBA}}(i,j)=1$ iff $j \le i$, turning the row-wise cumulative sum of $X \in \mathbb{R}^{m\times n}$ into the matrix product $C = M_{\mathrm{CumBA}}X$; ReduBA uses an all-ones vector $M_{\mathrm{ReduBA}}$ so that row-summing becomes $R = M_{\mathrm{ReduBA}}X$. This identity swaps a serial $m$-cycle DSP loop for a parallel MatMul/MVM on the MAC array. The second mechanism is the NPU's Piecewise Linear Unit with its configurable lookup table, which stores slopes and intercepts so that $f(x) \approx m_k x + c_k$ on each interval $[x_k, x_{k+1}]$, letting Swish and Softplus run during the drain phase without a sequential DSP kernel. ZVC compression exploits the mask's ~50% sparsity to reduce memory traffic.

What would settle it

Run the actual PLU-configured Swish/Softplus kernels (not ReLU-substituted) on the same Intel Core Ultra Series 2 NPU with the same 4-token prefill/decoding harness and compare wall-clock latency per token against the reported 260 tokens/s; if the real kernels run measurably slower than the ReLU emulation, the ActiBA speedup is not representative.

Watch

Extended reading notes

Core claim

The paper claims that the dominant bottlenecks of Mamba-2 and Mamba on an NPU are not their matrix multiplications but their sequential operators: the CumSum b inside the SSD step-1 of Mamba-2 accounts for over 99.9% of CumSum time on a 256x256 matrix, and Swish/Softplus dominate Mamba's runtime when executed as DSP loops. XAMBA's central discovery is that these serial operators can be reformulated as data-parallel matrix operations that exactly match the NPU's MAC-array strengths. CumBA computes the cumulative sum as C = M_CumBA * X with a lower-triangular binary mask, ReduBA computes ReduceSum as a matrix-vector product with an all-ones mask, and both exploit the resulting sparsity and reuse to cut latency and memory traffic. ActiBA stores piecewise-linear slopes and intercepts in the PLU's configurable lookup table so Swish and Softplus evaluate during the drain phase, effectively for free. Measured on a commercial Intel Core Ultra Series 2 NPU, CumBA alone reduces Mamba-2 130M latency by 2.7x, ReduBA by 1.2x, and the combination by 4.8x, while ActiBA reduces Mamba 130M latency by up to 2.6x; the accuracy table shows a maximum average-accuracy drop of 1.36 percentage points on the smallest Mamba model and essentially no change on Mamba-2 variants.

Load-bearing premise

ActiBA's speedup is measured with ReLU standing in for Swish/Softplus in the latency tests, while the accuracy table uses the real piecewise-linear PLU mapping, so the 2.6x transfer depends on those two implementations costing the same on the NPU.

Editorial extensions

If this is right

  • Mamba-2's CumSum bottleneck drops from over 50% of baseline latency to a minor component after CumBA, directly improving tokens per second.
  • Because CumBA's mask is half zeros, ZVC and sparsity-based compute skipping turn the triangular mask into a memory and compute saving, not just a data layout change.
  • ActiBA's PLU approximations keep average accuracy within 1.5 percentage points for the smallest 130M models and within 0.1 points for most larger variants.
  • The Mamba-130M decoding throughput rises from 100 to 260 tokens/s, clearing the 50 tokens/s KPI defined for client-side responsiveness.
  • The techniques are described as architecture-generic: any NPU with parallel MAC units plus a DSP can adopt the matrix-mask and PLU-LUT mappings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The prefix-sum-as-matmul identity is classical; XAMBA's contribution is showing that on an MPU+DSP NPU the matrix form wins over the serial DSP loop. A natural next test is whether the same mask trick helps other sequential primitives such as cumprod or selective-scan state updates, where the mask would be non-triangular.
  • The 4.8x figure comes from a single-block, 4-token Mamba-2 measurement. Longer sequences increase the matmul cost (m rows in the mask), so the speedup could shrink or grow depending on tile sizes; that tradeoff is not explored in the paper.
  • ActiBA trades accuracy for speed, and the paper reports the knob: adding PLU segments near the origin recovers accuracy. An automatic per-layer segment-count search could push throughput further while meeting a quality floor.
  • The reported gains are specific to Intel's NPU software stack; whether AMD or Qualcomm NPUs with different PLU and sparsity support receive the same factors is an open empirical question.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. XAMBA presents a three-step methodology for running Mamba and Mamba-2 state-space models on commercial Intel Core Ultra Series 2 NPUs: first enabling the models on the NPU via ONNX/OpenVINO conversion, then replacing sequential CumSum and ReduceSum with masked matrix operations (CumBA, ReduBA), and finally approximating SiLU and Softplus activations with piecewise-linear C-LUT mappings on the NPU's PLU (ActiBA). The paper reports up to 4.8x latency reduction for Mamba-2 130M from CumBA plus ReduBA, and up to 2.6x latency reduction for Mamba 130M from ActiBA, with Table 1 reporting 'negligible' quality loss for PLU-based approximations across several Mamba and Mamba-2 model sizes.

Significance. If the claims hold, XAMBA is a useful practical contribution: it is, to my knowledge, the first reported framework for deploying SSMs on a commercial COTS NPU, and it demonstrates a convincing 4.8x speedup for Mamba-2 through two mathematically exact reformulations. The CumBA and ReduBA ideas are simple, elegant, and appear exact, and the use of measured performance on a real Intel NPU against an OpenVINO baseline is a strength. The public code availability is also a plus. The ActiBA contribution, however, has a load-bearing experimental gap between the latency measurements and the quality measurements, and the quality numbers themselves are not uniformly 'negligible.' These issues are fixable with additional experiments and careful reporting, so the paper does not warrant rejection, but it needs a major revision.

major comments (3)
  1. [Section 3 and Section 4 (Fig. 4(c), Table 1)] The ActiBA latency result and the ActiBA quality result are measured on two different approximations. Section 3 states 'ActiBA is emulated by replacing activation functions with ReLU,' and Figure 4(c) labels the measured variants 'Si-Sp-Rl,' yet Table 1 reports accuracy for 'PLU SiLU & Softplus.' ReLU is a one-segment clamp; the PLU/C-LUT datapath in Figure 2(e) includes segment-range comparison, slope/intercept lookup, and saturation, so there is no evidence that the ReLU-emulated execution time equals the PLU execution time. Consequently, the paper's joint claim of 'up to 2.6x with negligible quality loss' for ActiBA is not established: the 2.6x speedup was measured on a substitute operator, while the quality numbers were obtained with the actual PLU approximation. Please run the latency experiment with the exact PLU configuration used for Table 1, or provide a rigorous argument that a ReLU datapath has identical cost, and report both metrics for the same variant.
  2. [Section 2.2 and Table 1] The accuracy of the PLU approximation and its hardware cost both depend on the number and placement of linear segments, but the paper never reports the C-LUT configuration used to produce Table 1. Section 2.2 states that 'increasing the number of linear segments... can further reduce this loss,' which makes the segment count and breakpoints a first-order parameter of the ActiBA tradeoff. Without this information, the Table 1 results are not reproducible, and one cannot assess whether the 'negligible' accuracy loss is an artifact of using very fine segmentation that would invalidate the 2.6x latency claim. Please specify the segment counts, boundaries, slopes/intercepts (or a reference to the exact C-LUT configuration), and state how the PLU was emulated for the quality evaluation.
  3. [Table 1 and Section 4] The claim of 'negligible quality loss' is weakened by component-level accuracy drops that the paper does not discuss. For Mamba-130M, the PLU variant drops from 35.25 to 31.83 on PIQA and from 64.47 to 59.19 on Winogrande, even though the average accuracy drops only from 42.03 to 40.67. The paper's statement that the 'largest accuracy drop' is 1.36% refers to the average and obscures these benchmark-specific degradations. If the intended claim is that the average degradation is small, it should be stated as such; otherwise, 'negligible' is too strong. Please report and discuss the per-benchmark variance, and explain why these component-wise drops are acceptable for the intended deployment scenarios.
minor comments (4)
  1. [Section 3 and Figure 4] The experimental methodology should state whether reported latencies are means or medians, how many benchmark iterations were run, whether warm-up was performed, and the exact input shape and batch size used beyond 'fixed input tokens of 4.' Figure 4(a) says 'average inference latency' while Figure 4(c) says 'first inference latency'; these metrics should be defined and used consistently.
  2. [Figure 4(a)] The combined 4.8x speedup from CumBA (2.7x) and ReduBA (1.2x) does not follow trivially from the individual speedups, since 2.7x and 1.2x would combine to about 3.2x if applied independently. Please clarify how the optimizations interact and why the combined speedup is 4.8x.
  3. [Table 1] The LAMBADA PPL entry for Mamba-130M-PLU is reported as '—'; either provide the missing value or explain why it is unavailable, since omitting one of the main benchmarks for the model most affected by ActiBA weakens the quality claim.
  4. [Throughout] There are typos in Table 1 and Figure 5: 'Mmaba2-1.3B' and 'Mmaba2-2.7B' should be 'Mamba2-1.3B' and 'Mamba2-2.7B,' and 'Swish' is sometimes written inconsistently. These should be corrected before publication.

Circularity Check

0 steps flagged · score 1.0 of 10

No circularity in XAMBA's derivation: speedups are measured against an external OpenVINO baseline; ActiBA's ReLU/PLU emulation is a validity gap, not a circular reduction.

full rationale

The central claims are empirical, not derived by construction. CumBA ('CumSum computed as C = MCumBA · X' with a lower-triangular mask) and ReduBA ('R = MReduBA · X' with an all-ones vector) are exact algebraic identities for CumSum and ReduceSum; the reported 2.7X, 1.2X, and 4.8X speedups are measured on an Intel Core Ultra Series 2 NPU using the OpenVINO benchmark app against an external baseline, so no prediction reduces to its input. ActiBA's accuracy results (Table 1) are for PLU piecewise-linear SiLU/Softplus, while Section 3 states 'ActiBA is emulated by replacing activation functions with ReLU.' This mismatch means the joint '2.6X with negligible quality loss' claim is not fully verified, but it is a correctness/validity gap rather than circularity: the ReLU latency measurement is not derived from the PLU accuracy table, and no parameter is fitted to force the speedup. The only self-citation is the NPU architecture description 'inspired by Raha et al. (2024)' (FlexNN), whose authors overlap with the present paper; however, this architectural model is motivational, and the performance conclusions are validated against real hardware, so the self-citation is not load-bearing. No equation is equivalent to another by construction in a way that assumes the claimed result, and no fitted quantity is renamed as a prediction. Score 1 reflects the minor, non-load-bearing self-citation; the ActiBA ReLU-emulation issue is a limitation to be weighed under correctness, not circularity.

Assumptions & free parameters 1 free parameters · 2 assumptions · 0 invented entities

The central claims rest on two hardware assumptions about the commercial NPU and on the unstated piecewise linear configuration for ActiBA. There are no new physical entities introduced.

free parameters (1)
  • PLU C-LUT segment slopes/intercepts for SiLU and Softplus = Not disclosed
    The accuracy results in Table 1 depend on this piecewise linear approximation, but the paper does not report the number of segments, breakpoints, slopes, or intercepts. The choice of these values determines both the accuracy loss and the hardware behavior.
assumptions (2)
  • domain assumption The Intel Core Ultra Series 2 NPU provides a Piecewise Linear Unit (PLU) with a Configurable Lookup Table as described in Figure 2(e).
    ActiBA relies on this hardware feature, but the paper provides no direct measurement or datasheet detail on the C-LUT's capacity or segment granularity.
  • domain assumption The output-stationary MPU architecture illustrated in Figure 2(a), including Zero Value Compression and sparsity-aware compute skipping, matches the behavior of the commercial NPU used in the experiments.
    The architecture is inspired by FlexNN (Raha et al.), and the paper assumes the commercial NPU exposes the same mechanisms that make CumBA's ZVC and compute-skip effective.

how reviews work

0 comments
Cite this review

Pith. "Pith review of XAMBA: Enabling Efficient State Space Models on Resource-Constrained Neural Processing Units." pith.science (2026). https://pith.science/paper/CO57YG7U

@misc{pith2026250206924,
  author       = {Pith},
  title        = {Pith review of: XAMBA: Enabling Efficient State Space Models on Resource-Constrained Neural Processing Units},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CO57YG7U}},
  note         = {Machine review of arXiv:2502.06924}
}
read the original abstract

State-Space Models (SSMs) have emerged as efficient alternatives to transformers for sequential data tasks, offering linear or near-linear scalability with sequence length, making them ideal for long-sequence applications in NLP, vision, and edge AI, including real-time transcription, translation, and contextual search. These applications require lightweight, high-performance models for deployment on resource-constrained devices like laptops and PCs. Designing specialized accelerators for every emerging neural network is costly and impractical; instead, optimizing models for existing NPUs in AI PCs provides a scalable solution. To this end, we propose XAMBA, the first framework to enable and optimize SSMs on commercial off-the-shelf (COTS) state-of-the-art (SOTA) NPUs. XAMBA follows a three-step methodology: (1) enabling SSMs on NPUs, (2) optimizing performance to meet KPI requirements, and (3) trading accuracy for additional performance gains. After enabling SSMs on NPUs, XAMBA mitigates key bottlenecks using CumBA and ReduBA, replacing sequential CumSum and ReduceSum operations with matrix-based computations, significantly improving execution speed and memory efficiency. Additionally, ActiBA enhances performance by approximating expensive activation functions (e.g., Swish, Softplus) using piecewise linear mappings, reducing latency with minimal accuracy loss. Evaluations on an Intel Core Ultra Series 2 AI PC show that XAMBA achieves up to 4.8X speed-up over the baseline. Our implementation is available at https://github.com/arghadippurdue/XAMBA.

Figures

Figures reproduced from arXiv: 2502.06924 by the authors.

Figure 1
Figure 1. Execution bottlenecks for Mamba and Mamba-2 on Intel® Core™ Ultra Series 2 NPU. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. XAMBA: (a) NPU architecture (b) Sequential CumSum and ReduceSum computation on [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. CumBA: Enhancing memory, bandwidth, and compute efficiency by exploiting CumBA [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Latency reduction for Mamba and Mamba-2 130M models on Intel® Core™ Ultra Series [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Mamba Gu & Dao (2024) vs. Mamba-2 Dao & Gu (2024) showcasing structural dif [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 14 canonical work pages

  1. [1]

    Transformers are SSMs: generalized models and efficient algorithms through structured state space duality

    Tri Dao and Albert Gu. Transformers are SSMs: generalized models and efficient algorithms through structured state space duality . In ICML, 2024

  2. [2]

    NITRO: LLM Inference on Intel Laptop NPUs

    Anthony Fei and Mohamed S. Abdelfattah. NITRO: LLM Inference on Intel Laptop NPUs , 2024. URL https://arxiv.org/abs/2412.11053

  3. [3]

    Mamba: Linear-Time Sequence Modeling with Selective State Spaces

    Albert Gu and Tri Dao. Mamba: Linear-Time Sequence Modeling with Selective State Spaces . In COLM, 2024

  4. [4]

    HiPPO: Recurrent Memory with Optimal Polynomial Projections

    Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher R \'e . HiPPO: Recurrent Memory with Optimal Polynomial Projections . In NeurIPS, 2020

  5. [5]

    Efficiently Modeling Long Sequences with Structured State Spaces

    Albert Gu, Karan Goel, and Christopher Re. Efficiently Modeling Long Sequences with Structured State Spaces . In ICLR, 2022

  6. [6]

    Intel \ Distribution of OpenVINO \ Toolkit

    Intel. Intel \ Distribution of OpenVINO \ Toolkit . URL https://www.intel.com/content/www/us/en/developer/tools/openvino-toolkit/overview.html. Accessed: Feb. 07, 2025

  7. [7]

    Intel® Core™ Ultra series mobile processors product brief , 2024 a

    Intel Corporation . Intel® Core™ Ultra series mobile processors product brief , 2024 a . URL https://www.intel.com/content/www/us/en/products/docs/processors/core-ultra/core-ultra-series-2-mobile-product-brief.html. Accessed: Feb. 07, 2025

  8. [8]

    Intel® Core™ Ultra series 1 product brief , 2024 b

    Intel Corporation . Intel® Core™ Ultra series 1 product brief , 2024 b . URL https://www.intel.com/content/www/us/en/products/docs/processors/core-ultra/core-ultra-series-1-product-brief.html. Accessed: Feb 07, 2025

Show all 21 references
  1. [9]

    J. Li, S. Huang, J. Xu, J. Liu, L. Ding, N. Xu, and G. Dai. MARCA: Mamba Accelerator with ReConfigurable Architecture . In arXiv, 2024. URL https://arxiv.org/abs/2409.11440

  2. [10]

    MobileLLM: optimizing sub-billion parameter language models for on-device use cases

    Zechun Liu, Changsheng Zhao, Forrest Iandola, Chen Lai, Yuandong Tian, Igor Fedorov, Yunyang Xiong, Ernie Chang, Yangyang Shi, Raghuraman Krishnamoorthi, Liangzhen Lai, and Vikas Chandra. MobileLLM: optimizing sub-billion parameter language models for on-device use cases . In ...

  3. [11]

    OpenVINO IR Format: Operation Sets and Specifications

    OpenVINO Documentation . OpenVINO IR Format: Operation Sets and Specifications . https://docs.openvino.ai/2024/documentation/openvino-ir-format/operation-sets/operation-specs.html, 2024. Accessed: Feb. 07, 2025

  4. [12]

    Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challenges

    Badri Narayana Patro and Vijay Srinivas Agneeswaran. Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challenges . In arXiv, 2024. URL https://arxiv.org/abs/2404.16112

  5. [13]

    Mathaikutty, Soumendu K

    Arnab Raha, Deepak A. Mathaikutty, Soumendu K. Ghosh, and Shamik Kundu. FlexNN: A Dataflow-aware Flexible Deep Learning Accelerator for Energy-Efficient Edge Devices . In arXiv, 2024. URL https://arxiv.org/abs/2403.09026

  6. [14]

    Flex-SFU: Accelerating DNN Activation Functions by Non-Uniform Piecewise Approximation

    Enrico Reggiani, Renzo Andri, and Lukas Cavigelli. Flex-SFU: Accelerating DNN Activation Functions by Non-Uniform Piecewise Approximation . In DAC, 2023

  7. [15]

    Minsoo Rhu, Mike O'Connor, Niladrish Chatterjee, Jeff Pool, Youngeun Kwon, and Stephen W. Keckler. Compressing DMA Engine: Leveraging Activation Sparsity for Training Deep Neural Networks . In HPCA, 2018

  8. [16]

    Wang et al

    X. Wang et al. Efficient Inference of Recurrent Neural Networks on Neural Processors . IEEE Transactions on Neural Networks and Learning Systems, 2018

  9. [17]

    Kinsy, Nanning Zheng, and Pengju Ren

    Tao Yang, Yadong Wei, Zhijun Tu, Haolun Zeng, Michel A. Kinsy, Nanning Zheng, and Pengju Ren. Design Space Exploration of Neural Network Activation Function Circuits . IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 2019

  10. [18]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  11. [19]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  12. [20]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  13. [21]

    jU HA ZUu^ޚz HcYqYIcv/IO[x r@z V = <V Lj +2Ǥ=Fiͬ1 d>v qC ѨM@ ȡ6q 6( [Aeiw w'8 H : Z׿ w |o [ zj <>uwj^\ ҄;?հ#FO2&=̹ڈ

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.