{"id":"bd683957-3d5e-43f0-ab99-22688863ff35","arxiv_id":"2505.18975","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"FastMamba is a co-designed FPGA accelerator for Mamba2 that combines Hadamard-based 8-bit quantization, power-of-two quantization, and first-order linear approximations of nonlinear functions, reporting large CPU/GPU speedups and improved energy efficiency.","lead":"FastMamba is an FPGA accelerator for Mamba2, a state space language model, that uses Hadamard-based 8-bit quantization and power-of-two arithmetic to cut computation. It reports up to 68.8x CPU and 8.9x GPU speedups for prefill, and 1.65x better energy efficiency than an RTX 3090 for decode.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Hardware quantization scales are static in Fig. 6, while Algorithm 1 uses per-input dynamic max; the reported 0.4% accuracy drop may not hold on the actual FPGA.","rationale":"The reader's weakest assumption focuses on extrapolation of quantization accuracy from Mamba2-130M to Mamba2-2.7B. That is a valid gap: the decode energy-efficiency result has no associated accuracy check. However, I find an even more immediate concern at the 130M scale itself. The pseudocode in Algorithm 1 suggests dynamic, input-dependent scale factors, while the hardware description indicates fixed scale coefficients and shifts with no max-reduction circuitry. If the hardware actually uses static scales derived from calibration, the accuracy numbers in Table II may not be reproducible on the FPGA, because a static scale may not capture the per-input maximum of the Hadamard-transformed activations. This would directly affect the validity of the 'accurate quantization' contribution and, by extension, the speedup and efficiency comparisons that use the quantized hardware. I do not claim the authors are misleading; the paper may simply omit the calibration step. But the ambiguity is load-bearing, and the proposed test would settle it. My verdict remains CONDITIONAL, matching the reader's, because the paper needs to clarify or verify this point before full acceptance. I partially agree with the reader because the missing 2.7B accuracy is also a real gap, but my primary concern is the scale-consistency issue at the implementation level.","tokens_in":9399,"tokens_out":8273,"duration_ms":51591,"concrete_test":"Re-run the Mamba2-130M evaluation in Table II using static scale factors, one per linear layer, estimated from a calibration set (e.g., 512 samples from Wikitext) and then applied to all test inputs, matching the hardware quantization path (multiply+shift). Compare average accuracy to the FP16 baseline. If the degradation exceeds the reported 1% (or the Table II FastMamba average of 42.2 vs 42.6), the hardware engine's quantization is not validated. Also check the RTL/bitstream for any runtime max-reduction logic; its absence confirms static scaling.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Algorithm 1 (Sec. III-A) computes quantization scales sX and sW from the max of all Hadamard-transformed values (lines 7-8), which is an input-dependent, two-pass operation. The hardware architecture in Sec. IV-B (Fig. 6) quantizes intermediate values with a multiplier ('×scoe') and shifter ('>>sshift') and contains no max-reduction unit, implying precomputed static scales. The paper never reconciles this mismatch. If the FPGA uses static scales, outlier activations in XH may exceed the 8-bit range for inputs different from the calibration set, degrading accuracy more than the 0.4% average drop reported in Table II. Since the central claim is 'accurate 8-bit quantization' enabling the speedup and energy results, an unvalidated static-scale implementation would undermine those results. The manuscript's Section V.A accuracy evaluation does not state whether scales are dynamic or static.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"FastMamba presents a hardware-algorithm co-design for deploying Mamba2 on FPGA. The algorithm contributions are Hadamard-transform-based 8-bit quantization for linear layers, power-of-two quantization for convolution and SSM blocks, and first-order linear approximation of exponential and SoftPlus. The hardware contributions are vector processing units, a Hadamard-based linear module, an SSM module with a nonlinear approximation unit, and a pipelined dataflow. Experiments on Xilinx VC709 report that W8A8 quantization of Mamba2-130M stays within about 1% average accuracy of FP16, prefill speedups up to 68.80x over a Xeon CPU and 8.90x over an RTX 3090, and 1.65x energy efficiency over the RTX 3090 for decode on Mamba2-2.7B.","tokens_in":9525,"tokens_out":5595,"duration_ms":49100,"significance":"The problem is timely and the paper contains useful engineering: it gives a full resource-utilization breakdown, quantifies DSP/FF savings of the nonlinear unit, and evaluates against two standard platforms. If the accuracy results are reproducible on the actual hardware, the Hadamard-based quantization plus custom datapath is a credible approach to SSM inference at the edge. However, the central claims currently rest on an unresolved mismatch between the dynamic quantization algorithm and the static-looking hardware, and on accuracy evidence only at the smaller model scale.","major_comments":[{"comment":"Algorithm 1 (lines 7-8) computes the activation scale sX from the maximum of the Hadamard-transformed activations XH, which is an input-dependent, two-pass procedure, and the hardware in Fig. 6 and Section IV-B uses fixed coefficients (x scoe, >> sshift) with no max-reduction or scale-computation unit. The manuscript never states whether the implemented module uses dynamic or static scales. If static, the Table II accuracy results (which appear to use dynamic scales) do not validate the FPGA implementation, because activations outside the calibration range would be clipped in 8 bits. Please either describe the hardware dynamic-scale path, or report accuracy and quantization-error statistics using the same static-scale procedure as the hardware, including the calibration set and clipping behavior.","section":"III-A, IV-B"},{"comment":"The accuracy evaluation (Table II) is performed only on Mamba2-130M, while the decode-stage throughput and energy-efficiency results (Table III, Section V.C.2) use Mamba2-2.7B. The paper assumes that the Hadamard 8-bit linear quantization and the PoT quantization of the SSM block preserve accuracy at 2.7B scale, but no perplexity/zero-shot accuracy or outlier statistics are reported for that model. Since the energy-efficiency claim is for the 2.7B configuration, provide accuracy results (or at least an outlier-range analysis) for Mamba2-2.7B under the exact quantization scheme used by the hardware.","section":"V.A, V.C.2"},{"comment":"The speedup comparison in Section V.C.1 reports maximum 68.80x vs CPU and 8.90x vs GPU but does not describe the CPU/GPU software configuration: framework, batch size, sequence length, thread count, FP16/int8 kernels, or whether the comparison uses optimized prefill/decode kernels. The paper also does not compare against any existing Mamba accelerator, despite citing Marca [9] in the introduction. Without these details the speedup numbers are difficult to interpret. Provide the exact evaluation protocol and, ideally, a comparison with prior FPGA/ASIC Mamba accelerators on the same workload.","section":"V.C.1, Table III"},{"comment":"Energy efficiency is reported as token/(s·W) without any description of how power is measured (e.g., board-level power meter, vendor tools, or model estimates) or which components are included. The GPU and FPGA run at different frequencies and use different memory systems, so the 1.65x energy-efficiency ratio needs a stated measurement methodology to be credible.","section":"Table III, V.C.2"}],"minor_comments":[{"comment":"The abstract states a 6x energy-efficiency improvement in the decode experiment, while Section V.C.2, Table III, and the conclusion report 1.65x. Please make these numbers consistent.","section":"Abstract, V.C.2"},{"comment":"Line 4, 'FindHadamard(X[i])', suggests the Hadamard matrix is derived from data; since H depends only on the dimension, rename to something like 'FindHadamard(dim(X[i]))' to avoid confusion.","section":"Algorithm 1"},{"comment":"The Throughput row shows GPU 111 token/s and FastMamba 5.68 token/s, yet the text emphasizes higher energy efficiency for FastMamba. State explicitly that the throughput is for decode of Mamba2-2.7B and clarify the workload and batch size so readers can understand the throughput/energy trade-off.","section":"Table III"},{"comment":"The 'Half Float Nonlinear Unit' in Fig. 10 is not defined in the text; specify whether it is an FP16 reference implementation and how its resource numbers were obtained.","section":"Fig. 10, IV-D"},{"comment":"Equation (5) states SoftPlus(x) ≈ e^x without specifying the domain; the subsequent symmetry argument implies x ≤ 0. Add the domain and quantify the approximation error of the first-order linear method.","section":"III-B.2"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the scope of CS.AR and the co-design idea is relevant. The main concerns -- static/dynamic scale mismatch, missing accuracy evidence at 2.7B, and insufficient baseline methodology -- are fixable with additional experiments and clarifications. I would not reject, but the current version overclaims relative to what is demonstrated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick verdict: FastMamba is a real FPGA implementation of Mamba2 with a sensible quantization stack, and the accuracy tables suggest the quantization actually works. But the version you showed me has a few loose ends that need attention before I'd trust the headline numbers: an abstract/body inconsistency on energy efficiency, and a mismatch between the algorithm's input-dependent quantization scales and the apparent static scales in the hardware.\n\nWhat's good: the paper adapts QuaRot-style Hadamard rotation to Mamba2's linear layers, adds power-of-two quantization for the SSM and convolution, and shares a single approximation unit for exp and SoftPlus. That's a coherent co-design, not just a repackaging of known tricks. The accuracy results on Mamba2-130M are genuinely decent: FastMamba-LQ matches FP16 average accuracy (42.6 vs 42.6), and the full quantized model drops only to 42.2 with PPL 17.9 vs 16.9. For an 8-bit linear + PoT SSM that's a good outcome. The nonlinear approximation unit saves 56% DSP and 49% FF versus an FP16 unit, which is a concrete hardware win. The implementation uses real resources (77% LUT, 92% DSP) and the throughput/energy numbers are reported on actual hardware.\n\nNow the soft spots. First, the abstract in the arXiv listing says 6x energy efficiency over RTX 3090, but the body consistently says 1.65x. That's a serious discrepancy; it needs to be fixed and explained. Second, Algorithm 1 computes quantization scales from the max over the input (a two-pass, input-dependent operation), while the Hadamard linear module in Fig. 6 quantizes with a multiplier and shifter and has no max-reduction unit. The paper never says whether the hardware uses static scales. If it does, the reported accuracy may not hold for inputs outside the calibration set. The accuracy evaluation doesn't state which version was measured. This isn't necessarily fatal, but it's a gap that needs to be closed. Third, the speedup comparisons to CPU and GPU don't tell us what software optimizations were used (e.g., thread count, batch size, library), and there's no comparison to prior Mamba accelerators like MARCA, which makes it hard to place the 8.9x speedup. Fourth, the 2.7B decode measurements have no accuracy validation at that scale, so we're asked to trust that the quantization transfers from 130M.\n\nOverall: the core idea is sound, and the accuracy evidence from the 130M model is the strongest part. The hardware implementation is real and resource reporting is transparent. The issues are about completeness and verification, not about a fundamentally broken approach. I'd send this to peer review, but I'd want the authors to reconcile abstract and body, explain the scale computation in hardware, and add at least one comparison with an existing Mamba accelerator.","headline":"A solid but under-verified FPGA accelerator for Mamba2—good quantization results on the 130M model, but an energy-efficiency discrepancy and an algorithm/hardware scale mismatch need fixing before the headline numbers are trusted.","tokens_in":10076,"tokens_out":4053,"would_cite":true,"duration_ms":46933,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Hadamard outlier suppression lets an FPGA run Mamba2 prefill 8.9x faster than an RTX 3090.","keywords":["Mamba","Mamba2","State Space Models","FPGA acceleration","8-bit quantization","Hadamard transformation","power-of-two quantization","algorithm-hardware co-design"],"falsifier":"Run the same Hadamard-based W8A8 quantization on Mamba2-2.7B and compare Lambada perplexity and zero-shot accuracy to FP16; a degradation well above the roughly one percentage point seen at 130M, or post-Hadamard activations that still fall outside the 8-bit clipping range, would break the central claim.","tokens_in":1738,"feed_emoji":"⚡","tokens_out":2103,"duration_ms":101997,"temperature":0.7,"pith_summary":"FastMamba aims to show that Mamba2, a state-space language model, can be deployed efficiently on FPGAs without losing accuracy. It combines an 8-bit quantization scheme that uses a Hadamard transform to spread out outlier activation values, a power-of-two fixed-point format for the SSM block and convolution layer, and first-order linear approximations for the exponential and SoftPlus functions. On the Xilinx VC709 FPGA, the design reports a maximum 68.80x speedup over an Intel Xeon 4210R CPU and 8.90x over an NVIDIA RTX 3090 during prompt prefill, the context-ingestion pass, on Mamba2-130M, and 1.65x better energy efficiency than the RTX 3090 when decoding Mamba2-2.7B. If these results hold, resource-constrained edge devices could run state-space models locally instead of relying on cloud GPUs.","feed_headline":"FPGA accelerator runs Mamba2 prefill 8.9x faster than an RTX 3090","feed_subtitle":"Hadamard-based 8-bit quantization holds accuracy within 1% while power-of-two arithmetic and linearized nonlinearities cut hardware cost.","key_machinery":"The load-bearing identity is the Hadamard factorization $Y=(XH)(H^T W^T)$, where the Hadamard matrix, normalized by $\\sqrt{n}$, is orthonormal, so the rotation concentrates outlier energy and gives 8-bit quantization a narrower dynamic range. The SSM block and convolution use fine-grained power-of-two scaling factors $2^p$, turning multiplications into shifts, and the nonlinearities are reduced to an 8-segment first-order linear approximation of $2^v$ built from $e^x = 2^{x\\log_2 e}$ and the symmetry $\\mathrm{SoftPlus}(x)=x+\\mathrm{SoftPlus}(-x)$. On the hardware side, five vector processing units (parallel adder, parallel multiplier, parallel multiply-add, Hadamard adder tree, and multiplier adder tree) execute these fixed-point operations in a pipelined dataflow.","core_discovery":"The paper claims that Mamba2's three deployment obstacles—outlier-heavy linear-layer activations, irregular element-wise SSM operations, and hardware-unfriendly nonlinearities—can all be removed algorithmically. Hadamard rotations spread activation and weight outliers before 8-bit quantization, power-of-two scaling makes SSM and convolution arithmetic shift-dominated, and exp and SoftPlus collapse into one linear-approximation unit. The implemented FPGA reaches a maximum 68.80x prefill speedup over the CPU and 8.90x over the RTX 3090 on Mamba2-130M, and a 1.65x energy-efficiency gain over the RTX 3090 when decoding Mamba2-2.7B, with reported accuracy degradation within about one percentage point on the 130M model.","pith_inferences":["If the Hadamard-based quantization holds up at larger scale, the same outlier-rotation trick should transfer to other state-space and attention architectures whose activation tensors have extreme channels; the paper only demonstrates it on the 130M model.","The energy-efficiency result covers decode only; a prefill comparison at 2.7B would require either more on-chip memory or a sequence-chunked schedule, so the measured 1.65x advantage may not carry over to that stage.","A direct testable extension is to record the post-Hadamard activation ranges at every linear layer of Mamba2-2.7B; if any remain outside the 8-bit clipping range, the accuracy-degradation assumption breaks before any hardware measurement."],"forward_implications":["The Hadamard-based W8A8 linear quantizer beats both normal quantization and SmoothQuant on perplexity and seven zero-shot tasks for Mamba2-130M, so outlier rotation rather than per-channel smoothing carries the accuracy.","Power-of-two quantization plus linearized nonlinearities reduces the SSM block to fixed-point shifts, adds, and matrix products that map directly onto vector processing units.","The pipelined FPGA reaches a maximum 68.80x prefill speedup over the Intel Xeon 4210R CPU and 8.90x over the NVIDIA RTX 3090, with average speedups of 55.70x and 6.06x at typical evaluation lengths.","The shared Nonlinear Approximation Unit saves 56% of DSP resources and 49% of flip-flop resources compared with an FP16 nonlinear unit.","The decode-stage comparison on Mamba2-2.7B reports 1.65x energy efficiency over the RTX 3090, but the accuracy of the quantized 2.7B model is not measured in the paper."],"supporting_citations":[{"why":"Supplies the Hadamard rotation method that the linear-layer quantizer uses to suppress activation and weight outliers.","marker":"[8]"},{"why":"Defines the Mamba2 architecture and state-space-duality formulation whose computations the accelerator targets.","marker":"[2]"},{"why":"Provides the SmoothQuant W8A8 baseline against which the Hadamard quantizer is compared on perplexity and zero-shot accuracy.","marker":"[7]"},{"why":"Introduces the selective state-space block that Mamba2 generalizes and whose nonlinearities the paper approximates.","marker":"[1]"},{"why":"Provides the shift-based exponential approximation used for $e^\\Delta$ in the SSM block.","marker":"[15]"},{"why":"Gives the symmetry identity that lets SoftPlus reuse the exponential approximation unit.","marker":"[16]"},{"why":"Supplies the evaluation harness used to produce the Mamba2-130M perplexity and zero-shot accuracy numbers.","marker":"[17]"}],"fun_headline_variants":["Mamba2 prefill: 8.9x GPU speedup, 68.8x CPU on FPGA","Hadamard tamed outliers for 8-bit Mamba2 FPGA, 8.9x GPU","FPGA gives Mamba2 6x energy efficiency vs RTX 3090","FastMamba: 8-bit quantization on FPGA beats GPU 8.9x"],"cache_read_input_tokens":12288,"weakest_assumption_plain":"The design assumes that the method that keeps accuracy loss under about one percent on the small 130M model will also keep accuracy acceptable on the 2.7B model used in the decode tests, but no accuracy measurements are reported at that scale.","fun_headline_variants_meta":{"raw":{"variants":["Mamba2 prefill: 8.9x GPU speedup, 68.8x CPU on FPGA","Hadamard tamed outliers for 8-bit Mamba2 FPGA, 8.9x GPU","FPGA gives Mamba2 6x energy efficiency vs RTX 3090","FastMamba: 8-bit quantization on FPGA beats GPU 8.9x"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001613,"raw_usage":{"total_tokens":6451,"prompt_tokens":1006,"completion_tokens":5445,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":622,"completion_tokens_details":{"reasoning_tokens":5344}},"tokens_in":622,"tokens_out":5445,"duration_ms":32418,"temperature":1.0,"reasoning_tokens":5344,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:21:46.424505+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same Hadamard-based W8A8 quantization on Mamba2-2.7B and compare Lambada perplexity and zero-shot accuracy to FP16; a degradation well above the roughly one percentage point seen at 130M, or post-Hadamard activations that still fall outside the 8-bit clipping range, would break the central claim.","supporting_citations":[{"cited_title":"QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs,","cited_arxiv_id":null,"evidence_quote":"Supplies the Hadamard rotation method that the linear-layer quantizer uses to suppress activation and weight outliers."},{"cited_title":"Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality,","cited_arxiv_id":null,"evidence_quote":"Defines the Mamba2 architecture and state-space-duality formulation whose computations the accelerator targets."},{"cited_title":"SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models,","cited_arxiv_id":null,"evidence_quote":"Provides the SmoothQuant W8A8 baseline against which the Hadamard quantizer is compared on perplexity and zero-shot accuracy."},{"cited_title":"Efficient Precision-Adjustable Architecture for Softmax Function in Deep Learning,","cited_arxiv_id":null,"evidence_quote":"Provides the shift-based exponential approximation used for $e^\\Delta$ in the SSM block."},{"cited_title":"A High-Precision Flexible Symmetry-Aware Architec- ture for Element-Wise Activation Functions,","cited_arxiv_id":null,"evidence_quote":"Gives the symmetry identity that lets SoftPlus reuse the exponential approximation unit."}],"review_version":1}