Pith. sign in

REVIEW 4 major objections 5 minor 77 references

Efficient Nonlinear Function Approximation in Analog Resistive Crossbars for Recurrent Neural Networks

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read By programming a memristor column with the inverse of a nonlinear activation, a ramp ADC can compute that activation during digitization, eliminating the digital nonlinearity step that bottlenecks recurrent neural network inference…

desk verdict The nonlinear-ramp ADC concept is sound and the memristor ramp generation is genuinely measured, but the paper oversells the hardware demonstration: the ADC's comparator and integrator are software models, not on-chip circuits. read the letter →

arxiv 2411.18271 v1 pith:QHEINMV7 submitted 2024-11-27 cs.AR

Junyi Yang , Ruibin Mao , Mingrui Jiang , Yichuan Cheng , Pao-Sheng Vincent Sun , Shuai Dong , Giacomo Pedretti , Xia Sheng
show 4 more authors
Jim Ignowski Haoliang Li Can Li Arindam Basu
This is my paper · ORCID
classification cs.AR
keywords analogin-memorycomputingmemristorcrossbarnonlinearactivationfunctionrampADCLSTMinferencekeywordspottinghardware-awaretraining
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Recurrent networks such as LSTM apply sigmoid and tanh after every matrix-vector product, and in analog in-memory computing those nonlinearities are a latency and energy bottleneck. The paper's claim is that these activations can be computed during analog-to-digital conversion—inside a nonlinear ADC (NL-ADC)—rather than in a separate digital processor. The NL-ADC programs an extra memristor column with the inverse of the desired activation, so the ramp voltage is predistorted and the comparator crossing time directly encodes the activated value. On-chip experiments on a 12-class keyword-spotting LSTM reach 88.5% measured accuracy at 5-bit, and system-level estimates show the LSTM layer at roughly 6.2X area efficiency and 1.46X energy efficiency versus a same-precision conventional ramp-ADC baseline. A scaled simulation on a 6.1-million-weight character-prediction LSTM stays within about 0.015 bits per character of software accuracy, indicating the approach carries over to larger recurrent models.

What carries the argument

The central object is the inverse-function ramp identity $t_{\mathrm{in}} = g^{-1}(V_{\mathrm{in}})$: when the ADC ramp $f(t)$ is chosen so that $f = g^{-1}$, the comparator crossing time becomes the activation value $g(V_{\mathrm{in}})$, so the activation is computed during digitization. The hardware that realizes this is a dedicated memristor column in the same crossbar: each of the $2^b$ ramp steps is a programmable conductance $G_{\mathrm{adc},k} \propto \Delta V_k$, integrated on a feedback capacitor to build the predistorted ramp, with the comparator being the existing sense amplifier. A few bias memristors set the ramp's start voltage and perform a one-point calibration that corrects programming errors and stuck devices, and using the same memristor circuits for MAC and reference makes read-voltage noise common-mode.

What would settle it

Fabricate a complete macro with real integrators, comparators, and ripple counters; measure the 5-bit sigmoid and tanh transfer functions' INL and the keyword-spotting LSTM accuracy, and compare measured LSTM-layer area and energy against a same-precision conventional ramp-ADC baseline. If the realized INL exceeds roughly 1 LSB, or the measured accuracy falls below the reported 88.5% by more than a few points, or the area/energy ratios do not approach the projected 6.2X and 1.46X, the central hardware claim is refuted.

Watch

Extended reading notes

Core claim

The central claim is that the nonlinear activation of a recurrent network can be made physically part of the analog-to-digital conversion step in a memristive crossbar. The method replaces the linear ramp of a conventional ramp ADC with a predistorted ramp whose shape is the inverse of the desired activation $g^{-1}$; the comparator then trips at a time proportional to $g(V_{\mathrm{in}})$ instead of $V_{\mathrm{in}}$. In the crossbar, the ramp is generated by one extra memristor column whose conductances $G_{\mathrm{adc},k}$ encode the successive step sizes $\Delta V_k = g^{-1}(t_k) - g^{-1}(t_{k-1})$, with a small set of bias memristors providing a one-point calibration. Because the MAC result and the ADC reference come from the same memristor circuits, read-voltage fluctuations cancel, giving maximum integral nonlinearity (INL) between 0.02 and 0.44 LSB when read voltage varies from 0.15 V to 0.25 V. The paper supports the claim with on-chip inference of a 12-class keyword-spotting LSTM (32 hidden neurons, 9216 memristors) reaching 88.5% accuracy at 5-bit, and with system-level estimates showing roughly 6.2X area efficiency and 1.46X energy efficiency over a same-precision conventional ramp-ADC baseline, plus a scaled 6.1-million-weight character-prediction LSTM that stays within 0.015 bits per character of software.

Load-bearing premise

The load-bearing premise is that the integrator and comparator, which are not fabricated on the chip and are instead emulated in software after reading the crossbar output, can be physically built with the area, speed, noise, and power assumed in the circuit simulations and system-level estimates; if those real circuits are larger, slower, noisier, or more power-hungry than modeled, the projected efficiency gains and the on-chip inference accuracy would not carry over to a complete system.

Editorial extensions

If this is right

  • LSTM inference can be executed with the nonlinear activations produced directly by the ADC, so the digital processor that normally computes sigmoid and tanh is removed from the critical path.
  • On the fabricated 9216-memristor crossbar, the 5-bit NL-ADC achieved 88.5% measured accuracy on a 12-class keyword-spotting task; the 4-bit and 3-bit versions achieved 86.6% and 85.2%.
  • At the system level the LSTM layer is projected to be about 6.2X more area-efficient and 1.46X more energy-efficient than a same-precision conventional ramp-ADC baseline, and roughly 9.9X and 4.5X better than earlier LSTM circuits.
  • For a 6.1-million-weight character-prediction LSTM, simulated 5-bit NL-ADC reaches 1.349 bits per character under measured write and read noise, close to the 1.334 software baseline.
  • Because the ramp and the MAC result use the same memristor circuits, read-voltage variation cancels: maximum INL stays between 0.02 and 0.44 LSB across a 0.15-0.25 V read sweep, versus 4.12-5.5 LSB for a conventional ADC.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The inverse-ramp construction is not limited to activations; any bijective function of the MAC output, such as a quantizer, a normalizer, or a log-softmax term, could in principle be fused into the same conversion by reprogramming the ramp column.
  • The decisive next experiment is a full macro that includes the real integrators, comparators, and ripple counters the current chip omits; measuring the LSTM-layer energy, area, and accuracy directly would verify the projected gains rather than relying on estimates.
  • Because the supplement shows non-monotonic GELU and Swish can be approximated by splitting the inverse into monotonic pieces, the same mechanism is a plausible route to transformer-style workloads, where those activations are common.
  • Passing the pulse-width-modulated output directly to the next layer, which the paper mentions but does not quantify, would compound the savings in deep RNNs by also removing input DAC or PWM-generation circuits, so the efficiency advantage should grow with depth.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The manuscript proposes an in-memory nonlinear ADC (NL-ADC) for analog resistive crossbars, in which an extra memristor column is programmed with conductances proportional to the steps of the inverse of a desired activation function. Integration of that column's current produces a nonlinear ramp, and comparison with the MAC result yields a thermometer code that approximates the activation. The authors derive the mapping, experimentally program six activation functions on a memristor array, demonstrate a one-point calibration scheme, show robustness to read-voltage variations, and estimate accuracy and efficiency for keyword spotting (88.5% claimed) and Penn Treebank character prediction through simulations. The fabricated chip includes the memristor array but not the integrator and comparator, which are implemented in software after the crossbar output is obtained.

Significance. If the central hardware claim were fully validated, the NL-ADC would attack a genuine bottleneck in IMC-based RNNs by merging nonlinear activation with digitization and removing the digital activation processor. The mathematical derivation is sound and parameter-free in the sense that the ramp shape is not fitted to data, and the measured conductance-mapped ramps for six activations, the one-point calibration, and the read-voltage-tracking property are valuable experimental results. The public code and the detailed efficiency-estimation methodology are also strengths. However, because the ADC periphery is not fabricated, the paper does not currently demonstrate an integrated NL-ADC; the headline accuracy and efficiency claims are estimates contingent on simulated peripherals.

major comments (4)
  1. [Results, 'In-memory Implementation of Nonlinear ADC and Vector Matrix Multiply in a Crossbar Array'] The paper states, 'The fabricated chip does not include the integrators and the comparator which are implemented in software after obtaining the crossbar output.' This means the transfer functions in Fig. 3 and the KWS accuracy of 88.5% in Fig. 4d combine measured memristor ramp generation with a software model of the analog periphery; they are not end-to-end hardware measurements. The Discussion's claim that the NL-ADC 'removes the need for any digital processor to implement nonlinear activations' is therefore not yet experimentally established. Please either fabricate and characterize the integrator and comparator or clearly re-label all such results as memristor measurements with simulated peripherals and temper the corresponding claims.
  2. [Supplementary Note S3, Tab. S3] The macro-level energy budget assigns 324.42 pJ to the integrator and 33.10 pJ to the comparator out of a total of 557.79 pJ (Tab. S3), sourced from references [51] and [45] and scaled to 16 nm; these are not measured values for the present design. The SPICE validation in Supplementary Note S2 varies only integrator DC gain and gain-bandwidth product and does not model comparator offset, noise, mismatch, or layout parasitics. Since the integrator and comparator together dominate the macro energy, the claimed 1.46X system-level energy-efficiency advantage over a conventional 5-bit ADC baseline is not robust unless these peripheral assumptions are bounded by measurement or by a sensitivity analysis.
  3. [Table 1 and Fig. 4] Table 1 reports 'KWS task on GSCD (Accuracy %): 88.5' without qualification, and the abstract claims 'experimentally demonstrate the implementation of a non-linear activation function integrated with a ramp ADC.' Because the comparator and integrator are software models, this overstates the level of integration actually demonstrated. The comparisons to prior integrated LSTM chips (e.g., 9.9X area efficiency, 4.5X energy efficiency) should be relabeled as estimates or projections rather than measured system performance.
  4. [Supplementary Note S4.b and Fig. 5] The NLP scalability results in Fig. 5 and Tabs. S14–S17 are based on NeuroSim system-level simulations in which the ADC is replaced by the proposed NL-ADC model. These results are therefore simulation-based, not experimentally demonstrated, and the main text should state this explicitly; the current phrasing in the 'Scaling to large RNNs' section, relying on 'experimentally validated nonideality models,' is potentially misleading.
minor comments (5)
  1. [Eq. (2)] The notation 'f () =g−1()' obscures the argument; write f(t) = g^{-1}(t) for clarity.
  2. [Abstract and Introduction] The phrase 'tasted limited success' in the abstract is awkward; 'had limited success' would be clearer. There is also a typo in the Introduction: 'network ouptut' should be 'network output.'
  3. [Fig. 3b] Please state explicitly how the 'Normal ADC' INL results are generated (e.g., a simulated linear ramp with the same Vread sweep), since the comparison is central to the robustness claim.
  4. [Table 1] In the header of Table 1, 'Nature'2332' contains a stray apostrophe, and reference formatting is inconsistent across the paper; please unify the citation style.
  5. [Discussion] The statement that the NL-ADC 'removes the need for any digital processor' should be qualified: non-monotonic activations such as GELU and Swish still require additional logic as described in Supplementary Note S12.

Circularity Check

0 steps flagged · score 1.0 of 10

No material circularity: the inverse-ramp construction is a mathematical implementation identity, not a fitted prediction; the reported accuracy and efficiency claims rest on measured chip data and externally sourced component data.

full rationale

The central derivation is the identity in Eq. (2): the paper sets t_in = f^{-1}(V_in) = g(V_in), which forces the ramp to be f() = g^{-1}(). This is the design equation of a predistorted ramp ADC, not a claim that g is empirically predicted from something independent of g. The paper's actual hardware claim is that a memristor column can be programmed to realize this inverse ramp accurately and that the resulting comparator output implements g during digitization; that claim is tested by measured INL values, by the measured transfer functions in Fig. 3, and by the measured 88.5% KWS accuracy in Fig. 4d. The one-point calibration is not a fit of the nonlinear shape: it only shifts Vinit so that the zero crossing of the programmed ramp matches the theoretical ramp, as stated in the Calibration section, leaving the 32 programmed step conductances fixed. The self-citation that is most load-adjacent is ref. 41 for defect-aware training, by co-authors of the present paper, but the method is fully restated as Algorithm 1 in Methods and is used as a standard robustness technique; it is not the evidence for the inverse-ramp construction or for the measured accuracy. Other self-references supply device characterization or component-level energy/area data and are not used to forbid alternatives. The paper also explicitly discloses a real limitation: 'The fabricated chip does not include the integrators and the comparator which are implemented in software after obtaining the crossbar output.' This means the full end-to-end ADC periphery is unbuilt, which is a correctness and external-validity risk for the projected system-level efficiency numbers, but it is not a circularity: the missing circuits are independent of the mathematical identity and are modeled using external component data. The NLP scalability results are simulations using noise distributions measured from the same chip; this is a model-based projection rather than a circular prediction, because the noise model is an external measurement and the simulated network behavior is not assumed to equal the target result. Overall, no derivation step reduces to its own input in a way that would make the central claim forced by definition or by self-citation.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The derivation of the nonlinear ramp is parameter-free (ramp = inverse activation), but the hardware implementation depends on several device and circuit assumptions, most notably the un-fabricated integrator/comparator performance and the programmable multi-level memristor conductances.

free parameters (5)
  • Maximum conductance gmax = 150 µS
    Chosen based on the TaOx memristor device characteristics from prior work (ref 48). Used to normalize both weight conductances (gamma = gmax/|w|max) and NL-ADC step conductances. A design constant, not fitted to the target results, but it sets the absolute scale of the ramp.
  • Weight clipping bound |w|max = 2
    Weights are clipped to [-2, 2] before mapping (Eq. 6), determining the gamma scaling factor. This is a hand-chosen hyperparameter.
  • One-point calibration offset Vinit,new = Per-chip value via Ncali bias memristors
    The zero-crossing of the ramp is matched by adjusting the bias conductance. This is a calibration parameter fitted to each chip/programming run; it corrects offset but does not change the function shape.
  • Training noise sigma = 5 µS
    Defect-aware training injects Gaussian noise with sigma = 5 µS, chosen to exceed the measured programming error of 2.67 µS. A training hyperparameter.
  • Read voltage Vread = 0.2 V
    Operating point for MAC and NL-ADC reads. The NL-ADC is designed to be insensitive to Vread variations, so this value is not critical to the method.
assumptions (5)
  • standard math Comparator acts as an ideal Heaviside threshold function (Eq. 1).
    The ramp ADC model treats the comparator transition as instantaneous when Vramp(t) equals Vin. Real comparators have offset, noise, and finite gain, which are not modeled in the mathematical derivation.
  • domain assumption Memristor conductances are programmable to arbitrary values within [0, 150 µS].
    The multi-level capability of the TaOx devices (ref 48) is relied on to encode each ramp step delta_Vk into a single memristor. Experimental programming achieves 2.67 µS standard deviation, supporting this approximately.
  • domain assumption The charge integrator and comparator operate with the performance assumed in SPICE (gain ~1000, GBW ~200 MHz) and with the area/energy figures from refs 15, 45, 51.
    These circuits are not fabricated on the chip; their behavior and cost are taken from simulation and literature. If the real circuits deviate, the system-level efficiency estimates change.
  • standard math The activation functions are bijective (or split into monotonic segments).
    The inverse-ramp identity requires an invertible function. For GELU and Swish the paper splits at extrema (Supp. Note S12). This is a standard mathematical condition.
  • domain assumption Conductance drift is slow enough that inference accuracy remains acceptable over the relevant time window.
    The drift measurements (Supp. Note S13) show up to 6% accuracy drop over 500,000 s without drift-aware training, and <2% with it. The method relies on this drift being manageable.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Efficient Nonlinear Function Approximation in Analog Resistive Crossbars for Recurrent Neural Networks." pith.science (2026). https://pith.science/paper/QHEINMV7

@misc{pith2026241118271,
  author       = {Pith},
  title        = {Pith review of: Efficient Nonlinear Function Approximation in Analog Resistive Crossbars for Recurrent Neural Networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QHEINMV7}},
  note         = {Machine review of arXiv:2411.18271}
}
read the original abstract

Analog In-memory Computing (IMC) has demonstrated energy-efficient and low latency implementation of convolution and fully-connected layers in deep neural networks (DNN) by using physics for computing in parallel resistive memory arrays. However, recurrent neural networks (RNN) that are widely used for speech-recognition and natural language processing have tasted limited success with this approach. This can be attributed to the significant time and energy penalties incurred in implementing nonlinear activation functions that are abundant in such models. In this work, we experimentally demonstrate the implementation of a non-linear activation function integrated with a ramp analog-to-digital conversion (ADC) at the periphery of the memory to improve in-memory implementation of RNNs. Our approach uses an extra column of memristors to produce an appropriately pre-distorted ramp voltage such that the comparator output directly approximates the desired nonlinear function. We experimentally demonstrate programming different nonlinear functions using a memristive array and simulate its incorporation in RNNs to solve keyword spotting and language modelling tasks. Compared to other approaches, we demonstrate manifold increase in area-efficiency, energy-efficiency and throughput due to the in-memory, programmable ramp generator that removes digital processing overhead.

Figures

Figures reproduced from arXiv: 2411.18271 by the authors.

Figure 1
Figure 1. Limitation of current In-memory computing (IMC) for Recurrent Neural Networks and our proposed solution. a A survey of DNN accelerators show the improvement in energy efficiency offered by IMC over digital architectures. However, the improvement does not extend to recurrent neural networks (RNN) such as LSTM and there exists a gap in energy efficiency between RNNs and feedforward architectures. Details of the survey… view at source ↗
Figure 2
Figure 2. Overview of in-memory nonlinear ADC. a The concept of traditional ramp-based ADC. b The schematic and timing of in-memory computing circuits with embedded nonlinear activation function generation. c The Inverse of the sigmoid function illustrates the shape of the required ramp voltage. d The value of each step of the ramp voltage Vramp denoted by ∆Vk is proportional to memristor conductances Gadc,k used to program t… view at source ↗
Figure 3
Figure 3. Experimentally demonstrated NL-ADC on crossbar arrays a Calibration process for accurate NL-ADC programming. The left panel shows the ramp function of the ideal case, programming without bias calibration and with bias calibration. The case with bias calibration shows better INL performance. The right panel shows the actual conductance mapping on the crossbar arrays on two blocks of 8 arbitrary selected columns. The … view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: | LSTM for KWS task. a Architecture of LSTM network on-chip inference. b Mapping of LSTM network onto the chip. Weights and nonlinearities (Sigmoid and Tanh) of LSTM layer are programmed crossbar arrays as conductance. Input and output (I/O) data of LSTM layer are sent…
Figure 5
Figure 5. Figure 5: | LSTM for NLP task. a Architecture of LSTM network for on-chip inference in character prediction task. b Comparison in the LSTM layer between the number of neurons and operations per timestep in the NLP model for character prediction and the KWS model. c Simulation re…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

77 extracted references · 66 canonical work pages

  1. [51]

    Cai, F. & et. al. Power-efficient combinatorial optimization using intrinsic noise in memristor hopfield neural networks. Nat. Electron. 3, 409–418 (2020)

    work page 2020

  2. [45]

    Yu, C., Yoo, T., Chai, K. T. C., Kim, T. T.-H. & Kim, B. A 65-nm 8T SRAM compute-in-memory macro with column ADCs for processing neural networks. IEEE J. Solid-State Circuits 57, 3466–3476 (2022)

    work page 2022

  3. [1]

    Kolesnikov, A. et al. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In The International Conference on Learning Representations (ICLR) (2021)

    work page 2021

  4. [2]

    & Hinton, G

    Graves, A., Mohamed, A.-R. & Hinton, G. Speech recognition with deep recurrent neural networks. In 2013 IEEE International Conference on Acoustics, Speech and Signal Processing , 6645–6649 (Ieee, 2013)

    work page 2013

  5. [3]

    Silver, D. et al. Mastering the game of go with deep neural networks and tree search. Nature 529, 484–489 (2016)

    2016

  6. [4]

    Senior, A. W. et al. Improved protein structure prediction using potentials from deep learning. Nature 577, 706–710 (2020)

    2020

  7. [5]

    Vaswani, A. et al. Attention is all you need. Adv. Neural Inf. Process. Syst. 30 (2017)

    2017

  8. [6]

    Computing’s energy problem (and what we can do about it)

    Horowitz, M. Computing’s energy problem (and what we can do about it). In 2014 IEEE International Solid-State Circuits Conference digest of technical papers (ISSCC) , 10–14 (IEEE, 2014)

    work page 2014

Show all 77 references
  1. [7]

    & Williams, S

    Yang, J., Strukov, D. & Williams, S. Memristive devices for computing. Nat. Nanotechnol. 8, 13–24 (2013)

    2013

  2. [8]

    Prezioso, M. et al. Training and operation of an integrated neuromorphic network based on metal-oxide memristors.Nature 521, 61–64 (2015)

    2015

  3. [9]

    & Strukov, D

    Alibart, F., Zamanidoost, E. & Strukov, D. Pattern classification by memristive crossbar circuits using ex situ and in situ training. Nat. Commun. 4 (2013)

    2013

  4. [10]

    Can, L. & et. al. Efficient and self-adaptive in-situ learning in multilayer memristor neural networks. Nat. Commun. 9 (2018)

    2018

  5. [11]

    & Eleftheriou, E

    Sebastian, A., Le Gallo, M., Khaddam-Aljameh, R. & Eleftheriou, E. Memory devices and applications for in-memory computing. Nat. Nanotechnol. 15, 529–544 (2020)

    2020

  6. [12]

    Peng, Y . & et. al. Fully hardware-implemented memristor convolutional neural network.Nature 577, 641–646 (2020)

    2020

  7. [13]

    Kiani, F. & et. al. A fully hardware-based memristive multilayer neural network. Sci. Adv. 7 (2021)

    2021

  8. [14]

    Yang, K. et al. Transiently chaotic simulated annealing based on intrinsic nonlinearity of memristors for efficient solution of optimization problems. Sci. advances 6, eaba9901 (2020)

    2020

  9. [15]

    Jiang, M., Shan, K., He, C. & Li, C. Efficient combinatorial optimization by quantum-inspired parallel annealing in analogue memristor crossbar. Nat. Commun. 14, 5927 (2023)

    2023

  10. [16]

    John, R. A. et al. Halide perovskite memristors as flexible and reconfigurable physical unclonable functions. Nat. Commun. 12, 3681 (2021)

    2021

  11. [17]

    Mao, R. et al. Experimentally validated memristive memory augmented neural network with efficient hashing and similarity search. Nat. Commun. 13, 6284 (2022)

    2022

  12. [18]

    Sheridan, P. M. et al. Sparse coding with memristor networks. Nat. nanotechnology 12, 784–789 (2017). 17/52

    2017

  13. [19]

    Zidan, M. A. et al. A general memristor-based partial differential equation solver. Nat. Electron. 1, 411–420 (2018)

    2018

  14. [20]

    Le Gallo, M. & et. al. Mixed-precision in-memory computing. Nat. Electron. 1, 246–253 (2018)

    2018

  15. [21]

    Khwa, W. S. et al. A 40-nm, 2M-cell, 8b-precision, hybrid SLC-MLC PCM computing-in-memory macro with 20.5–65.0TOPS/W for tiny-Al edge devices. In 2022 IEEE International Solid-State Circuits Conference-(ISSCC) , 1–3 (IEEE, 2022)

    2022

  16. [22]

    J-.M, H. & et. al. A four-megabit compute-in-memory macro with eight-bit precision based on CMOS and resistive random-access memory for AI edge devices. Nat. Electron. 4, 921–930 (2021)

    2021

  17. [23]

    Le Gallo, M. et al. A 64-core mixed-signal in-memory compute chip based on phase-change memory for deep neural network inference. Nat. Electron. 6, 680–693 (2023)

    2023

  18. [24]

    Giraldo, J. S. P. & Verhelst, M. Laika: A 5 µW programmable LSTM accelerator for always-on keyword spotting in 65nm CMOS. In ESSCIRC 2018-IEEE 44th European Solid State Circuits Conference (ESSCIRC) , 166–169 (IEEE, 2018)

    2018

  19. [25]

    & Seo, J.-S

    Kadetotad, D., Yin, S., Berisha, V ., Chakrabarti, C. & Seo, J.-S. An 8.93 TOPS/W LSTM recurrent neural network accelerator featuring hierarchical coarse-grain sparsity for on-device speech recognition. IEEE J. Solid-State Circuits 55, 1877–1887 (2020)

    2020

  20. [26]

    & Yoo, H.-J

    Shin, D., Lee, J., Lee, J. & Yoo, H.-J. DNPU: An 8.1 TOPS/W reconfigurable CNN-RNN processor for general-purpose deep neural networks. In 2017 IEEE International Solid-State Circuits Conference (ISSCC) , 240–241 (IEEE, 2017)

    2017

  21. [27]

    & Benini, L

    Conti, F., Cavigelli, L., Paulin, G., Susmelj, I. & Benini, L. Chipmunk: A systolically scalable 0.9 mm 2, 3.08 Gop/s/mW@ 1.2 mW accelerator for near-sensor recurrent neural network inference. In 2018 IEEE Custom Integrated Circuits Conference (CICC), 1–4 (IEEE, 2018)

    2018

  22. [28]

    Yin, S. et al. A 1.06-to-5.09 TOPS/W reconfigurable hybrid-neural-network processor for deep learning applications. In 2017 Symposium on VLSI Circuits , C26–C27 (IEEE, 2017)

    2017

  23. [29]

    Li, C. et al. Long short-term memory networks in memristor crossbar arrays. Nat. Mach. Intell. 1, 49–57 (2019)

    2019

  24. [30]

    Tsai, H. et al. Inference of long-short term memory networks at software-equivalent accuracy using 2.5 M analog phase change memory devices. In 2019 Symposium on VLSI Technology, T82–T83 (IEEE, 2019)

    2019

  25. [31]

    Survey of neuromorphic and machine learning accelerators in SOVC, ISSCC and Nature/Science series of journals from 2017 onwards,

    “Survey of neuromorphic and machine learning accelerators in SOVC, ISSCC and Nature/Science series of journals from 2017 onwards,”. https://docs.google.com/spreadsheets/d/1_j-R-QigJTuK6W5Jg8w2Yl85Tn2J_S-x/edit?usp=drive_link& ouid=117536134117165308204&rtpof=true&sd=true. Acce...

    2017

  26. [32]

    Ambrogio, S. et al. An analog-AI chip for energy-efficient speech recognition and transcription. Nature 620, 768–775 (2023)

    2023

  27. [33]

    A twofold lookup table architecture for efficient approximation of activation functions

    Xie, Y .et al. A twofold lookup table architecture for efficient approximation of activation functions. IEEE Transactions on V ery Large Scale Integration (VLSI) Syst.28, 2540–2550 (2020)

    2020

  28. [34]

    Arvind, T. K. et al. Hardware implementation of hyperbolic tangent activation function for floating point formats. In 2020 24th International Symposium on VLSI Design and Test (VDAT) , 1–6 (IEEE, 2020)

    2020

  29. [35]

    Kwon, D. et al. A 1ynm 1.25 v 8gb 16gb/s/pin gddr6-based accelerator-in-memory supporting 1tflops mac operation and various activation functions for deep learning application. IEEE J. Solid-State Circuits 58, 291–302 (2022)

    2022

  30. [36]

    Raut, G. & et. al. A CORDIC based Configurable Activation Function for ANN Applications. In International Symposium on VLSI (ISVLSI) (IEEE, 2020)

    2020

  31. [37]

    Chong, Y . & et. al. Efficient implementation of activation functions for LSTM accelerators. In VLSI System on Chip (VLSI-SOC) (IEEE, 2021)

    2021

  32. [38]

    Pasupuleti, S. K. et al. Low complex & high accuracy computation approximations to enable on-device rnn applications. In 2019 IEEE International Symposium on Circuits and Systems (ISCAS) , 1–5 (IEEE, 2019)

    2019

  33. [39]

    Feng, X. et al. A high-precision flexible symmetry-aware architecture for element-wise activation functions. In 2021 International Conference on Field-Programmable Technology (ICFPT), 1–4 (IEEE, 2021)

    2021

  34. [40]

    & Wang, L

    Li, Y ., Cao, W., Zhou, X. & Wang, L. A low-cost reconfigurable nonlinear core for embedded dnn applications. In2020 International Conference on Field-Programmable Technology (ICFPT), 35–38 (IEEE, 2020)

    2020

  35. [41]

    Mao, R., Wen, B., Jiang, M., Chen, J. & Li, C. Experimentally-Validated Crossbar Model for Defect-Aware Training of Neural Networks. IEEE Transactions on Circuits Syst. II: Express Briefs 69, 2468–2472, DOI: 10.1109/TCSII.2022. 3160591 (2022). 18/52

    2022 doi

  36. [42]

    Speech commands: A dataset for limited-vocabulary speech recognition

    Warden, P. Speech commands: A dataset for limited-vocabulary speech recognition. arXiv preprint arXiv:1804.03209 (2018)

    2018 arXiv

  37. [43]

    & Marcinkiewicz, M

    Marcus, M., Santorini, B. & Marcinkiewicz, M. A. Building a large annotated corpus of English: The Penn Treebank. Comput. Linguist. 19 (1993)

    1993

  38. [44]

    E and Holberg, D

    Allen, P. E and Holberg, D. CMOS Analog Circuit Design (Oxford University Press, 2011)

    2011

  39. [46]

    Rao, M. & et al. Thousands of conductance levels in memristors integrated on CMOS. Nature (2023)

    2023

  40. [47]

    Song, W. & et al. Programming memristor arrays with arbitrarily high precision for analog computing. Science (2024)

    2024

  41. [48]

    Sheng, X. & et al. Low-Conductance and Multilevel CMOS-Integrated Nanoscale Oxide Memristors. Adv. Electron. Mater. 5 (2019)

    2019

  42. [49]

    Kim, K. et al. A 23µW solar-powered keyword-spotting ASIC with ring-oscillator-based time-domain feature extraction. In 2022 IEEE International Solid-State Circuits Conference (ISSCC) , vol. 65, 1–3 (IEEE, 2022)

    2022

  43. [50]

    Kim, K. et al. A 23-µW keyword spotting IC with ring-oscillator-based time-domain feature extraction.IEEE J. Solid-State Circuits 57, 3298–3311 (2022)

    2022

  44. [52]

    Liu, Q. et al. A fully integrated analog ReRAM based 78.4 TOPS/W compute-in-memory chip with fully parallel MAC computing. In 2020 IEEE International Solid-State Circuits Conference-(ISSCC) , 500–502 (IEEE, 2020)

    2020

  45. [53]

    Zhang, W. et al. Edge learning using a fully integrated neuro-inspired memristor chip. Science 381, 1205–1211 (2023)

    2023

  46. [54]

    & Basu, A

    Yi, C., Wang, Z., Patil, A. & Basu, A. A 2.86-TOPS/W Current Mirror Cross-Bar Based Machine-Learning and Physical Unclonable Function Engine for Internet-of-Things Applications. IEEE Trans. on CAS-I 66, 2240–52 (2018)

    2018

  47. [55]

    Yan, B., Q., Y . & et. al. RRAM-based spiking nonvolatile computing-in-memory processing engine with precision- configurable in situ nonlinear activation. In 2019 Symposium on VLSI Technology (SOVC), T86–T87 (IEEE, 2019)

    2019

  48. [56]

    W, L. & et al. A 40-nm MLC-RRAM Compute-in-Memory Macro With Sparsity Control, On-Chip Write-Verify, and Temperature-Independent ADC References. IEEE J. Solid-State Circuits (JSSC) 2868–77 (2022)

    2022

  49. [57]

    Jiang, H. & et al. A 40nm Analog-Input ADC-Free Compute-in-Memory RRAM Macrowith Pulse-Width Modulation between Sub-arrays. In 2022 Symposium on VLSI Circuits (IEEE, 2022)

    2022

  50. [58]

    Yue, J. et al. A 65nm 0.39-to-140.3 TOPS/W 1-to-12b unified neural network processor using block-circulant-enabled transpose-domain acceleration with 8.1× higher TOPS/mm 2 and 6T HBST-TRAM-based 2D data-reuse architecture. In 2019 IEEE International Solid-State Circuits Confer...

    2019

  51. [59]

    Jouppi, N. P. et al. In-datacenter performance analysis of a tensor processing unit. In Proceedings of the 44th annual international symposium on computer architecture, 1–12 (2017)

    2017

  52. [60]

    & Seo, J.-s

    Yin, S., Sun, X., Yu, S. & Seo, J.-s. High-throughput in-memory computing for binary deep neural networks with monolithically integrated rram and 90-nm cmos. IEEE Transactions on Electron Devices 67, 4185–4192 (2020)

    2020

  53. [61]

    He, W. et al. 2-bit-per-cell rram-based in-memory computing for area-/energy-efficient deep learning. IEEE Solid-State Circuits Lett. 3, 194–197 (2020)

    2020

  54. [62]

    Cai, F. et al. A fully integrated reprogrammable memristor–cmos system for efficient multiply–accumulate operations. Nat. electronics 2, 290–299 (2019)

    2019

  55. [63]

    Huo, Q. et al. A computing-in-memory macro based on three-dimensional resistive random-access memory. Nat. Electron. 5, 469–477 (2022)

    2022

  56. [64]

    A 510-nW wake-up keyword-spotting chip using serial-FFT-based MFCC and binarized depthwise separable CNN in 28-nm CMOS

    Shan, W.et al. A 510-nW wake-up keyword-spotting chip using serial-FFT-based MFCC and binarized depthwise separable CNN in 28-nm CMOS. IEEE J. Solid-State Circuits 56, 151–164 (2020)

    2020

  57. [65]

    DNN+NeuroSim framework

    “DNN+NeuroSim framework”. https://github.com/neurosim/DNN_NeuroSim_V2.1

  58. [66]

    Kingma, D. & J, B. Adam: A Method for Stochastic Optimization. In The International Conference on Learning Representations (ICLR) (2014)

    2014

  59. [67]

    Evaluation Metrics for Language Modeling,

    “Evaluation Metrics for Language Modeling,”. https://thegradient.pub/understanding-evaluation-metrics-for-language-models/. Accessed: 2024-1-31. 19/52

    2024

  60. [68]

    Efficient Nonlinear Function Approximation in Analog Resistive Crossbars for Recurrent Neural Networks

    "Efficient Nonlinear Function Approximation in Analog Resistive Crossbars for Recurrent Neural Networks". https: //github.com/CityU-BRAINSys-Lab/NLADC_code

  61. [69]

    H, L. & et al. Resistive RAM-centric computing: Design and modeling methodology. IEEE Trans. on Circuits Syst. - I (TCAS-I) 2263–2273 (2017)

    2017

  62. [70]

    Chen, P.-Y ., Peng, X. & Yu, S. Neurosim: A circuit-level macro model for benchmarking neuro-inspired architectures in online learning. IEEE Transactions on Comput. Des. Integr. Circuits Syst. 37, 3067–3080 (2018)

    2018

  63. [71]

    Esmanhotto, E. & et al. Experimental Demonstration of Multilevel Resistive Random Access Memory Programming for up to Two Months Stable Neural Networks Inference Accuracy. Adv. Intell. Syst. (2022)

    2022

  64. [72]

    Park, J. S. & et al. A 2.2mW 12-bit 200MS/s 28nm CMOS Pipelined SAR ADC with Dynamic Register-Based High-Speed SAR Logic. In 2020 IEEE Asian Solid-State Circuits Conference digest of technical papers (A-SSCC) (IEEE, 2020)

    2020

  65. [73]

    Charan, G. & et al. Accurate Inference with Inaccurate RRAM Devices: Statistical Data, Model Transfer, and On-line Adaptation. In 2020 Design Automation Conference (DAC) (IEEE, 2020)

    2020

  66. [74]

    R., P. & et. al. SWISH: A SELF-GATED ACTIV ATION FUNCTION.arXiv preprint arXiv:1710.05941 (2017)

    2017 arXiv

  67. [75]

    Hendrycks, D. & et. al. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415 (2016)

    2016 arXiv

  68. [76]

    Dosovitskiy, A. & et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)

    2020 arXiv

  69. [77]

    k" represents the number of processors in the system, which signifies the degree of parallelism in processing nonlinear functions. A higher value of

    Tan, M. & Le, Q. V . Mixed depthwise convolutional kernels.arXiv preprint arXiv:1907.09595 (2019). 20/52 Supplementary Supplementary Note S1. Nonlinear function approximation by ramp ADC Six commonly used nonlinear functions in neural networks and their inverse functions are p...

    2019 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.