{"id":"b9afa308-7890-44a7-863e-12c3b47aae59","arxiv_id":"2411.17847","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"An integer-only Softmax approximation from I-BERT, mapped onto associative processors, can cut Softmax energy by up to 1300x and latency by up to 12.58x versus GPUs, but with small perplexity loss at the advertised precision.","lead":"SoftmAP is a software-hardware co-design that runs a low-precision, integer-only approximation of Softmax on in-memory associative processors, and reports large energy and latency savings versus GPUs for Llama2 models. It also includes a precision sensitivity study showing which bit widths keep perplexity close to the floating-point baseline on WikiText-2.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline 1300x energy and 12.58x latency gains rest on an unvalidated AP simulator and a 16nm-vs-7nm/8nm comparison, so the quantitative central claim is not yet supported.","rationale":"The reader's weakest_assumption identifies the most load-bearing threat to the central claim: the hardware evaluation is produced by an unvalidated simulator and a process-imbalanced comparison. I agree with that assessment because the headline numbers—1300x energy, 12.58x latency, and EDP ratios up to 8851x—are direct outputs of this model, and no independent evidence is provided that the Table II cycle counts match real associative-processor behavior. The process-node mismatch (16nm AP versus 7nm A100 and 8nm RTX3090) further undermines the quantitative comparison, since no normalization is attempted. The perplexity degradation at the chosen M=6 precision is a secondary but real issue: the abstract claims 'without compromising performance,' yet the paper's own Tables III and IV show an 8-9% perplexity gap on WikiText-2 at the advertised setting, and the closer M=8 configuration is not used in the hardware analysis. I do not see these issues as proof that the method is wrong; they are addressable with the proposed validation and normalization steps. Therefore the existing CONDITIONAL verdict remains appropriate, and I do not recommend changing it. The concrete test would settle whether the efficiency claim survives validation and fair process comparison.","tokens_in":12470,"tokens_out":5412,"duration_ms":48316,"concrete_test":"Construct or obtain a cycle-accurate simulator (or FPGA prototype) of the 2D SRAM-based associative processor at 16nm, implement the Fig. 5 dataflow for the M=6, N=16 configuration, and compare measured cycle counts and energy for add, multiply, and reduction against Table II. If any operation exceeds the Table II formula by more than 20%, recompute the energy and latency ratios in Figs. 6 and 7. Separately, rescale the AP energy and latency to 7nm using published technology-scaling factors and recompute the 1300x energy and 12.58x latency claims. If the normalized ratios no longer reach 'three orders of magnitude' in EDP, the headline claim should be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim—the reported reductions in energy, latency, and EDP in Section V-B—rests entirely on a Python AP simulator whose per-operation runtime and energy formulas (Table II) are taken from prior work [34] without validation against a fabricated or cycle-accurate associative processor. No measured hardware, RTL model, or independent simulation cross-check is provided for the specific Softmax dataflow in Fig. 5. If the Table II cycle formulas are optimistic—for example, an M-bit multiplication is listed as 2M+8M^2+2M cycles, which for M=6 gives 312 cycles—the claimed AP advantage would shrink substantially. In addition, the comparison is not technology-neutral: the AP is modeled at 16nm while the A100 is 7nm and the RTX3090 is 8nm, and no process-node scaling is applied. Since energy and delay depend strongly on the technology node, part of the advertised advantage may be an artifact of comparing an older-node AP against modern GPUs. A secondary but related concern is that the 'without compromising performance' claim is strained by the paper's own data: at the advertised M=6, N=16 setting, Llama2-7b perplexity is 5.92 versus the floating-point 5.47 (8.2% worse), and Llama2-13b is 5.35 versus 4.88 (9.6% worse), while the better M=8 setting is not used for the hardware evaluation. Thus both halves of the central claim—efficiency and preserved model quality—are not fully substantiated as presented.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SoftmAP, a software-hardware co-design for integer-only low-precision Softmax on associative processors (APs). The method uses a second-order polynomial approximation of the exponential with Barrett reduction, maps the computation to a 2D-AP dataflow, and evaluates perplexity on Llama2-7b/13b/70b over WikiText-2 across precision combinations (vcorr, M, N). The paper reports up to 1300x energy and 12.58x latency reductions versus A100 and RTX3090 GPUs, and up to ~3 orders of magnitude EDP improvement, based on a Python AP simulator.","tokens_in":12803,"tokens_out":4259,"duration_ms":37358,"significance":"If substantiated, the work would be significant: it presents the first integer-only low-precision Softmax tailored to associative processors for LLMs, with a detailed hardware mapping and a precision sensitivity analysis. The idea of accelerating non-linear operators on APs is timely and the approximation is grounded in prior I-BERT work. However, the quantitative headline claims rest entirely on an unvalidated simulator and a non-technology-neutral comparison, and the accuracy claim is weakened by post-hoc precision selection and a non-trivial perplexity gap at the chosen configuration. The contribution is promising but not yet convincing as presented.","major_comments":[{"comment":"The advertised energy and latency reductions (up to 1300x energy, 12.58x latency, and EDP ratios in Table V) are computed with a Python AP simulator whose per-operation cycle and energy formulas are taken from prior work [34] without validation against fabricated hardware, RTL simulation, or an independent cycle-accurate model, and no sensitivity analysis is reported. Since these numbers constitute the central quantitative claim of the paper, the claim is not yet supported; please provide at least an independent cross-check or a documented sensitivity study of the Table II formulas.","section":"Section V-B, Table II, Figs. 6-8"},{"comment":"The 'best precision combination' (vcorr=M, M=6, N=16) and the clipping threshold TC are selected using the same WikiText-2 perplexity numbers that are later used to support the 'without compromising performance' claim. This is a post-hoc selection on the evaluation benchmark. Moreover, the selected configuration shows perplexity 5.92 vs 5.47 for Llama2-7b and 5.35 vs 4.88 for Llama2-13b (8-10% degradation), while M=8 achieves 5.51 and 4.93, respectively, but is not used in the hardware evaluation. The abstract's 'without compromising performance' is therefore overstated.","section":"Section V-A, Tables III and IV"},{"comment":"The GPU comparison is not process-node-neutral: the AP is modeled at 16nm while the A100 is 7nm and the RTX3090 is 8nm, and no voltage/technology scaling is applied. Because energy and delay depend strongly on the technology node, part of the claimed AP advantage may be an artifact of comparing an older-node model against modern GPUs. Please normalize to a common node or report a technology-scaled comparison.","section":"Section V-B, Figs. 6-7"}],"minor_comments":[{"comment":"The sentence 'The ratios are 4× higher with A100 compared to RTX3090' contradicts Table V, which shows the RTX3090 EDP ratios are higher; it should read 'with RTX3090 compared to A100'.","section":"Section V-B, after Table V"},{"comment":"Many annotations in Figures 4 and 5 are garbled (e.g., 'BKLLM($$%&'!(∗F/2#))', '()++)'), making the dataflow difficult to follow, especially for the shift and Barrett-reduction steps.","section":"Figures 4 and 5"},{"comment":"The AP simulator description is a single sentence; please provide the energy model equations (or identify the specific equations in [34]) so that the evaluation is reproducible.","section":"Section IV, AP simulator"},{"comment":"In line 11, the shift amount '⌊−vstable/vln2⌋' is non-negative because vstable is non-positive and vln2 is positive; stating this explicitly would help readers verify the integer-only property.","section":"Section III, Algorithm 1"},{"comment":"The text consistently spells 'Barret reduction' and 'Barret' (Algorithm 1, Section III); the standard spelling is 'Barrett reduction' (as in reference [27]).","section":"Section III and references"}],"recommendation":"major_revision","confidential_remarks":"The paper is a plausible engineering contribution, but the evaluation methodology needs to be strengthened before it can support the advertised numbers. The main risks are the unvalidated simulator, the process-node mismatch, and the post-hoc precision/threshold selection on the same benchmark used for the accuracy claim. These are fixable: the authors could add validation or sensitivity analysis, apply process-node scaling, and report the M=8 configuration in the hardware evaluation. I would not reject, but the quantitative claims are not yet substantiated as written."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. The genuinely new thing is the precision sensitivity analysis of integer-only Softmax on Llama2-7b/13b/70b and the mapping of that dataflow onto the 2D associative processor. The approximation itself is I-BERT's polynomial, and the AP is prior work from the same group, but the study of which mixed precisions preserve perplexity on modern LLMs is a real contribution. The paper is also honest about some limits: it reports the Amdahl's law cap of 10.7% end-to-end speedup for Llama2-70b, it openly states M=4 fails, and it notes offloading costs are out of scope.\n\nThe soft spots are real but not fatal. The 'without compromising performance' claim doesn't survive contact with the paper's own tables: at the advertised M=6, N=16, perplexity is 5.92 vs 5.47 for 7b and 5.35 vs 4.88 for 13b. That's an 8-9% degradation, which is a trade-off, not a free lunch. The clipping threshold and the precision combination are both chosen using WikiText-2, and the same benchmark is used to report the final perplexity, so part of the accuracy result is fitted. The authors disclose the manual selection, but the headline claim would need a separate calibration/validation split.\n\nThe bigger issue is the hardware evaluation. All energy/latency numbers come from a Python simulator using formulas from prior work [34], with no fabricated chip or cycle-accurate model. The process comparison is 16nm AP vs 7nm A100 and 8nm RTX3090, with no node scaling. If the Table II multiplication formula (2M+8M^2+2M cycles) is optimistic, the 1300x energy advantage shrinks substantially. That doesn't invalidate the architectural idea, but it means the quantitative claim is not yet supported.\n\nThe paper deserves a serious referee. The right fixes are clear: validate the simulator against an RTL model or fabricated test chip, apply process-node normalization, split calibration from evaluation, and present the M=6 perplexity gap honestly as a trade-off. I'd send it to review with those expectations.","headline":"Genuine precision study and a plausible AP mapping, but the headline efficiency claim rests on an unvalidated simulator and the 'no compromise' claim is undercut by the paper's own perplexity numbers.","tokens_in":13359,"tokens_out":2346,"would_cite":false,"duration_ms":19342,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SoftmAP implements integer-only Softmax on associative processors, reporting up to 1300x energy and 12.58x latency savings over A100 and RTX3090 GPUs for Llama2 models.","keywords":["large language models","Softmax","quantization","in-memory computing","associative processors","integer-only approximation","energy-delay product","Llama2"],"falsifier":"Run the same integer-only Softmax dataflow on a cycle-accurate or fabricated 16nm associative processor and measure energy and latency; if the measured energy-delay product advantage over an A100 falls below an order of magnitude, the paper's central hardware claim collapses.","tokens_in":1748,"feed_emoji":"⚡","tokens_out":3735,"duration_ms":79484,"temperature":0.7,"pith_summary":"The paper tries to show that Softmax, a non-linear operator that increasingly bottlenecks large language models at long sequence lengths, can be approximated with integer-only arithmetic and run on in-memory associative processors without a meaningful perplexity penalty. If correct, this would remove a quantization-resistant obstacle to deploying LLMs on resource-constrained devices, because the approximation uses only multiplications, additions, and shifts. The authors evaluate the approximation on Llama2-7b, 13b, and 70b over WikiText-2, and report that the associative processor beats A100 and RTX3090 GPUs by up to 1300x in energy and 12.58x in latency for the Softmax operation, reducing the energy-delay product by up to three orders of magnitude.","feed_headline":"Softmax goes integer-only: 1300x energy savings over GPUs","feed_subtitle":"An associative in-memory processor runs Llama2 softmax in low-precision integers with near-identical perplexity.","key_machinery":"The load-bearing object is the integer-only Softmax approximation adapted from I-BERT: after subtracting the maximum for numerical stability, the negative input is split into a quotient and remainder relative to $\\ln(2)$, the remainder is passed through the polynomial $(v_{\\text{corr}} + v_b)^2 + v_c$, and the result is shifted by the quotient to approximate the exponential. Barrett reduction replaces the modulo division with a precomputed constant multiplication and shifts, keeping everything in integer arithmetic. On the hardware side, a 2D associative processor stores the vocabulary entries as rows and executes the step sequence in Fig. 5 bit-serially, word-parallel, so each Softmax over sequence-length/2 words is done in parallel; the paper identifies $v_{\\text{corr}} = M$, $M = 6$, $N = 16$ as the best precision combination across the three Llama2 models.","core_discovery":"The central claim is that the exponential inside Softmax can be faithfully replaced by a second-order polynomial plus a shift, computed entirely in low-precision integers, and that the resulting approximation can be mapped onto a two-dimensional associative processor operating bit-serially across words in parallel. With a clipping threshold of -7, a Softmax input precision of $M=6$, and a sum precision of $N=16$, the approximation keeps WikiText-2 perplexity at 5.92 for Llama2-7b and 5.35 for Llama2-13b, against floating-point baselines of 5.47 and 4.88, respectively. The paper further claims that this mixed-precision integer-only implementation on a 16nm associative processor consumes up to 1300x less energy and runs up to 12.58x faster than the Softmax operation on A100 and RTX3090 GPUs, with the largest advantages at long sequence lengths and small batch sizes.","pith_inferences":["If the Table II energy/latency formulas hold, the same integer polynomial approach could be extended to LayerNorm, the other non-linear quantization bottleneck, potentially removing all floating-point ops from a transformer block; the paper does not address this.","The 16nm AP vs 7nm/8nm GPU comparison is not process-node normalized; on a common node the 1300x energy claim would likely shrink, though the in-memory computing advantage may persist.","A testable extension is to run the same approximation on Llama3/3.1-70b and on long-context benchmarks like C4 or PG-19; the paper predicts no change for structurally similar models, but this has not been measured.","The AP is only latency-competitive at sequence lengths of 1024 or more, so a practical deployment would need a dispatcher that offloads Softmax to the AP only for long-context inference."],"forward_implications":["Softmax can be computed in integer-only arithmetic with a second-order polynomial and Barrett reduction, enabling quantization-friendly LLM inference without floating-point units.","For sequence lengths above 1024, where Softmax can reach 38% of Llama2-7b's runtime on an A100, the AP's latency advantage translates into overall execution time reductions (e.g., 10.71% for Llama2-70b at length 4096).","The best mixed-precision setting ($M=6$, $N=16$, $v_{\\text{corr}}=M$) keeps perplexity within 0.45-0.47 of floating-point for Llama2-7b/13b on WikiText-2, so the approximation is usable for generation tasks.","The AP's edge over GPUs grows with sequence length and shrinks at short lengths, meaning the technique pairs naturally with long-context LLM inference.","SoftmAP is orthogonal to parallelization methods like FlashAttention-2, so it can be applied per-partition to further accelerate Softmax."],"supporting_citations":[{"why":"Supplies the integer-only second-order polynomial exponential approximation and the shift-based evaluation that the paper adapts.","marker":"[28]"},{"why":"Provides Barrett reduction, used to compute the modulo $v_{\\text{corr}}$ with multiplications and shifts instead of division in integer arithmetic.","marker":"[27]"},{"why":"Source of the 2D AP energy and latency formulas in Table II that produce the paper's headline savings numbers.","marker":"[34]"},{"why":"Defines the two-dimensional associative processor architecture on which the Softmax dataflow is mapped.","marker":"[26]"},{"why":"Supplies the Llama2 model family (7B, 13B, 70B) used for perplexity and energy/latency evaluation.","marker":"[20]"},{"why":"Provides the WikiText-2 dataset on which perplexity is measured and the clipping threshold is calibrated.","marker":"[30]"}],"fun_headline_variants":["Integer-only softmax on associative cores: 1300x less energy","1300x energy cut for LLM softmax via integer-only design","SoftmAP: integer softmax in associative memory, 1300x savings","Associative processors run softmax in integers, 1300x cheaper"],"cache_read_input_tokens":15360,"weakest_assumption_plain":"The 1300x energy and 12.58x latency claims rest on unvalidated simulator formulas and a cross-process-node comparison against GPUs.","fun_headline_variants_meta":{"raw":{"variants":["Integer-only softmax on associative cores: 1300x less energy","1300x energy cut for LLM softmax via integer-only design","SoftmAP: integer softmax in associative memory, 1300x savings","Associative processors run softmax in integers, 1300x cheaper"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0005,"raw_usage":{"total_tokens":2400,"prompt_tokens":853,"completion_tokens":1547,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":469,"completion_tokens_details":{"reasoning_tokens":1478}},"tokens_in":469,"tokens_out":1547,"duration_ms":9025,"temperature":1.0,"reasoning_tokens":1478,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:46:22.301575+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same integer-only Softmax dataflow on a cycle-accurate or fabricated 16nm associative processor and measure energy and latency; if the measured energy-delay product advantage over an A100 falls below an order of magnitude, the paper's central hardware claim collapses.","supporting_citations":[{"cited_title":"Implementing the rivest shamir and adleman public key en- cryption algorithm on a standard digital signal processor,","cited_arxiv_id":null,"evidence_quote":"Provides Barrett reduction, used to compute the modulo $v_{\\text{corr}}$ with multiplications and shifts instead of division in integer arithmetic."},{"cited_title":"A two-dimensional as- sociative processor,","cited_arxiv_id":null,"evidence_quote":"Defines the two-dimensional associative processor architecture on which the Softmax dataflow is mapped."}],"review_version":1}