HadamRNN: Binary and Sparse Ternary Orthogonal RNNs

ANITI); Armand Foucault (IMT; Fran\c{c}ois Malgouyres (IMT); Franck Mamalet (ANITI)

REVIEW 3 major objections 5 minor 29 references

HadamRNN: Binary and Sparse Ternary Orthogonal RNNs

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that binary and sparse ternary recurrent weights are trainable when parameterized as sign-flipped Hadamard matrices, producing the first binary-weight RNN that solves the 1,000-step copy task.

desk verdict Sound Hadamard-based construction with a real first on binary copy task, but the attribution to the learned sign vector is under-tested; still deserves serious peer review. read the letter →

arxiv 2502.00047 v4 pith:JSHHBK6W submitted 2025-01-28 cs.LG cs.AI

Armand Foucault (IMT , ANITI) , Franck Mamalet (ANITI) , François Malgouyres (IMT) This is my paper · ORCID

classification cs.LGcs.AI MSC 15B3468T0715B10

keywords HadamardmatricesbinaryrecurrentweightsorthogonalRNNsparseternaryquantizationcopytaskstraight-throughestimatorlong-termdependencies

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to break a long-standing barrier: attempts to binarize the recurrent weights of vanilla RNNs had previously failed, with earlier work reporting that BinaryConnect on recurrent weights never worked. The authors construct recurrent weight matrices as sign-flipped Sylvester (Hadamard) matrices, which are exactly orthogonal while remaining binary or sparse ternary. They introduce HadamRNN and Block-HadamRNN, and report performance comparable to full-precision ORNNs and LSTMs on copy, permuted and sequential MNIST, IMDB, GLUE, and IoT benchmarks while cutting model size by up to 290-fold. Their central claim is that HadamRNN is the first solution with binary recurrent weights capable of learning the copy task with more than 1000 timesteps. If correct, this shows that binarized recurrent weights are not inherently incapable of long-term memorization.

What carries the argument

The central object is the Sylvester matrix $S_{2^k}$, the power-of-two Hadamard matrix built recursively from the $2\times2$ block with entries $1,1,1,-1$ via $S_{2^k} = S_2 \otimes S_{2^{k-1}}$. The trainable part is a $d_h$-bit sign vector $u$, and the map $u \mapsto \operatorname{diag}(u) S_{2^k}$ generates $2^{d_h}$ distinct exactly orthogonal matrices because switching row signs preserves orthogonality. This parameterization reduces the recurrent computation to additions and bit-shifts, and the same construction with $I_q \otimes S_{2^k}$ yields a sparse ternary orthogonal matrix whose recurrent cost is $q$ times lower. The straight-through estimator is what makes the discrete sign vector optimizable.

What would settle it

Train HadamRNN with $d_h = 128$ on the 1020-step copy task using the paper's hyperparameters. The central claim stands if the cross-entropy drops from the 0.021 baseline toward the reported $1.6\times10^{-7}$; a failure to learn would break the claim that binary recurrent weights can handle 1000-step dependencies.

Watch

Extended reading notes

Core claim

The central claim is that a restricted family of binary orthogonal matrices is both trainable and expressive enough for recurrent networks. The paper constructs the recurrent weight as $W(u) = \mathrm{diag}(u) S_{2^k} / \sqrt{d_h}$, where $S_{2^k}$ is the Sylvester matrix and $u$ is a trainable sign vector; because flipping row signs of a Hadamard matrix preserves the Hadamard property, $W(u)$ is orthogonal by construction yet stored in only $d_h$ bits. The paper reports that the straight-through estimator successfully optimizes $u$, and that the resulting HadamRNN solves the 1020-step copy task with cross-entropy $1.6\times10^{-7}$ while full-precision and quantized LSTMs fail. It further reports that Block-HadamRNN, built from $I_q \otimes S_{2^k}$, keeps most of the performance while cutting recurrent additions by a factor $q$. The headline result is that this is the first solution with binary recurrent weights able to learn the copy task over 1000 timesteps.

Load-bearing premise

The load-bearing premise is that the sign vector $u$ can be trained reliably with the straight-through estimator and that the restricted family of sign-flipped Sylvester matrices is expressive enough for the tasks; the paper gives empirical support but no convergence or expressivity guarantee.

Editorial extensions

If this is right

Binary recurrent weights alone do not prevent long-term memorization; the 1-bit orthogonal parameterization solves the 1,020-step copy task, where full-precision and quantized LSTMs fail.
Fully quantized HadamRNN retains its accuracy after activation quantization, reaching 94.90 percent on pMNIST and 96.34 percent on sMNIST with a 3.58 kB model.
Block-HadamRNN trades a small accuracy drop for a factor-$q$ reduction in recurrent additions, giving a tunable balance between performance and compute.
HadamRNN is over 36 times smaller than the previous 5-bit quantized ORNN on the copy task, and up to 290 times smaller than full-precision ORNNs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because the sign-flip construction works for any Hadamard matrix, not only Sylvester matrices, hidden sizes other than powers of two could be covered using Hadamard matrices of order $4n$ up to 2000, giving finer control over size and compute.
The exact orthogonality of $W(u)$ is the property that separates this construction from approximately orthogonal quantized RNNs, so the same parameterization may help other architectures that rely on orthogonal recurrence, such as gated or state-space models.
The linear recurrent unit carries much of the memorization: the paper's own ablation shows a ReLU version fails the copy task, so extending the method to nonlinear long-context models needs a different mechanism.
It remains an open check whether the straight-through estimator keeps optimizing $u$ at hidden sizes well beyond 512 or at delays beyond 1000 steps; the paper gives no guarantee.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The thing to know: this paper has a genuinely nice construction and a real first for binary recurrent weights, but the copy-task headline is less cleanly attributed than the paper suggests. The Hadamard sign-flip trick is simple, correct, and new for binarizing orthogonal RNN weights; the copy-task result is indeed the first binary recurrent weight solution over 1000 timesteps, and the model sizes are impressively small. The orthogonality proofs in Appendix B are self-contained and correct, and the empirical coverage is broad—copy task, pMNIST/sMNIST, IMDB, GLUE, IoT. The ablations (linear vs. ReLU units, row vs. column sign flips) are the right ones to run. Credit where due: this is a competent and mostly honest paper.

Where it gets soft. The copy task is solved with a linear recurrent unit, and the paper's own ablation (Table 8) shows that ReLU kills the copy task entirely. So the memorization story is largely about linearity plus orthogonality, not about the binary parameterization or about learning the sign vector u. There is no control experiment with u fixed to a random sign vector and only U, V trained. If that baseline also solves the copy task—and it very well might, given the norm-preserving orthogonal map and learned read/write matrices—then the claim that the restricted 2^dh family is expressive and that STE can optimize u is not actually demonstrated. The result would still be a first for binary recurrent weights, but the mechanism would be less impressive. Second, the main tables show single-seed numbers, and Appendix E.2 documents sizable seed variance for p=2 on IMDB (std 5.09 at lr 1e-3). The copy-task row is a single seed, so I wouldn't hang too much on the exact 1.6e-7 cross-entropy. Third, I could not verify code availability from the text—the 'code available at hadamRNN' line has no URL in the version I have. For a methods paper, that is a real gap.

None of this changes my bottom line. The construction is mathematically sound, the copy-task result is a genuine first if it stands, and the paper deserves serious engagement. I would send it to peer review. The reading group will get a good discussion out of the random-u baseline—that is exactly the right next experiment.

Referee Report

3 major / 5 minor

Summary. The paper proposes a parameterization of a subset of binary and sparse ternary orthogonal matrices using Sylvester Hadamard matrices, and uses them as recurrent weight matrices in orthogonal RNNs. Specifically, HadamRNN uses W(u) = (1/sqrt(d_h)) diag(u) S_{2^k} and Block-HadamRNN uses W(u) = (1/sqrt(2^k)) diag(u)(I_q \otimes S_{2^k}), where u is a trainable sign vector optimized with the straight-through estimator. The orthogonality of these matrices is proved in Appendix B. The models are evaluated on the copy task with a 1000-step delay, permuted and sequential MNIST, IMDB, two GLUE tasks, and two IoT benchmarks, achieving kilobyte-scale model sizes. The headline claim is that HadamRNN is the first binary recurrent-weight model capable of solving the copy task over 1000 timesteps, and the results show competitive performance relative to full-precision ORNNs and to the QORNN baseline.

Significance. If the claims are substantiated, this is a solid contribution to quantized recurrent networks: the Hadamard construction gives exactly orthogonal recurrent matrices with 1-bit or sparse ternary weights, eliminates multiplications in the recurrent layer, and yields strong empirical performance on long-term dependency benchmarks at kilobyte-scale model sizes. The orthogonality proofs in Appendix B are correct and self-contained, the paper ships a code link (though the URL is incomplete), and the ablations on linear versus ReLU units, row versus column sign switches, and activation quantization are useful. The main significance hinges on whether the experiments actually demonstrate that the binary sign vector u is learnable and that the restricted Hadamard family is sufficiently expressive. The copy-task result, while impressive, may be largely attributable to the linear ORNN dynamics and the learned input/output matrices rather than to the training of u; this must be clarified before the central claims can be fully accepted.

major comments (3)

[Section 4.2.1 / Table 2 / Eq. (1)] The copy-task result does not isolate the contribution of the learned binary sign vector u. Because the hidden dynamics are linear (Eq. (1)) and U and V are learned at 4-bit or full precision, a fixed orthogonal W from the same Hadamard family (e.g., a random sign-flipped Sylvester matrix) may suffice for the 1000-step memorization task once U and V are optimized; the paper's own ablation in Appendix G.1 shows that replacing the linear unit with ReLU makes the copy task fail, indicating that the linear dynamics are the main enabler. The authors should add an ablation comparing learned u with a fixed u (e.g., all-ones or a random draw from {+-1}^{dh}) on the copy task and on at least one classification task (pMNIST or IMDB). Without such a baseline, the claims that 'u can be optimized using standard methods like the STE' (Section 3.2) and that the 2^{dh}-member matrix family is 'sufficiently expressive' (Section 3.2) are not supported by the reported experiments.
[Appendix E.2 / Tables 2 and 3] The main results are reported as single-run point estimates, while Appendix E.2 documents substantial seed-to-seed variability for the lowest bitwidth (e.g., a standard deviation of 5.09 accuracy points for p=2 on IMDB at learning rate 1e-3, and up to 8.05 at 5e-4). The headline copy-task value (1.6e-7) and the small differences between configurations (e.g., 94.88 vs 94.90 on pMNIST in Table 2) cannot be assessed without error bars. Please report the mean and standard deviation over at least five seeds for all main rows in Tables 2 and 3, or at minimum for the copy-task rows and all p=2 rows, and state how seeds are used in hyperparameter selection.
[Section 5 / Section 3.2] The conclusion states that 'we demonstrate that the parameters of such matrices can be learned using standard methods like the STE,' but the experiments provide no evidence that optimizing u changes the outcome relative to drawing u at random. Since the linear ORNN with any full-rank orthogonal W can represent the copy task in a sufficiently large hidden state (the desired outputs are linear functions of the inputs), the copy-task success alone does not constitute such a demonstration. The authors should either supply the random-u baseline, provide a theoretical expressivity or convergence argument for the u-optimization, or explicitly soften the claim to say that the binary orthogonal family contains a solution and that U and V are learned.

minor comments (5)

[Section 3.3] There is a typo: 'reffered to as Hadamard RNN' should be 'referred to as Hadamard RNN.'
[Section 3.2] There is a typo: 'Hadamarad matrix' should be 'Hadamard matrix.'
[Tables 2 and 3] The copy-task tables label the sequence length as T=1000, but Section 4.1 defines T=L+2K=1020 with L=1000; please correct the headers or clarify that L=1000.
[Appendix E.2 / Table 5] The text states that p=2 results in high variability, but the largest standard deviation in Table 5 is 8.05 (at learning rate 5e-4), not 5.09; please report the full range in the text.
[Code availability] The code link is given as 'available at hadamRNN' without a full URL; please provide the complete repository address.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the Hadamard construction is derived from standard matrix theory and validated on external benchmarks; self-citations are comparison-only.

full rationale

The central derivation is self-contained. The recurrent weight family is explicitly defined as W(u) = dh^{-1/2} diag(u) S_{2^k} (Eq. 3) and W(u) = 2^{-k/2} diag(u)(I_q ⊗ S_{2^k}) (Eq. 4), and orthogonality is proved directly from Sylvester/Hadamard properties in Appendix B.4, without assuming the target result. The copy-task, pMNIST, sMNIST, IMDB, GLUE, and IoT results are external benchmarks; no constants are fitted to the reported cross-entropy or accuracy numbers. The only data-dependent calibration is the fixed-point activation scaling factor in Appendix I.5, chosen from training and validation hidden-state ranges, and it does not support the paper's scientific claims about binary recurrent weights. Self-citations to Foucault et al. (2024) identify prior quantized-ORNN work and provide comparison baselines, but the construction, the orthogonality proofs, and the empirical evaluation do not reduce to those citations. No circular step is present.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new physical entities or hidden quantities. Its main assumptions are standard Hadamard-matrix mathematics, the STE optimization heuristic, and the empirical expressiveness of the restricted binary-orthogonal family. The only fitted calibration constant is the activation scaling factor used for fixed-point inference.

free parameters (1)

Activation scaling factor alpha_h = 2^{n_h} / alpha_W, with n_h chosen as the smallest integer satisfying 2^{n_h} >= max_h * alpha_W
Appendix I.5 selects alpha_h from the maximum full-precision hidden state over training and validation data to fit the 12-bit fixed-point range. This is a calibration constant for quantized inference, not a scientific parameter of the core method.

assumptions (5)

standard math Sylvester matrices S_{2^k} are Hadamard, and multiplying rows by signs preserves the Hadamard property.
Propositions 3.2 and 3.3 with proofs in Appendix B.2 and B.3; external mathematical facts, not assumed from the target result.
standard math The Kronecker product I_q ⊗ S_{2^k} has pairwise orthogonal rows.
Proposition B.2 and the proof in Appendix B.4 establish orthogonality of the sparse ternary Block-HadamRNN matrices.
domain assumption The straight-through estimator provides a valid gradient approximation for optimizing the binary sign vector u.
Appendix C.1 states that the true gradient is undefined or zero and STE replaces it with the identity; there is no convergence guarantee, and the paper relies on empirical success.
ad hoc to paper The restricted family of sign-flipped Sylvester matrices is expressive enough for the evaluated tasks.
Section 3.2 says experiments confirm sufficient expressiveness, but no theoretical expressivity bound or capacity analysis is provided.
domain assumption Post-training quantization of activations to 12 bits does not degrade long-term memorization.
Appendix I describes the fixed-point recurrence, and Table 2 shows comparable results with and without activation quantization, but there is no formal error bound for the quantized recurrence.

how reviews work

0 comments

Cite this review

Pith. "Pith review of HadamRNN: Binary and Sparse Ternary Orthogonal RNNs." pith.science (2026). https://pith.science/paper/JSHHBK6W

@misc{pith2026250200047,
  author       = {Pith},
  title        = {Pith review of: HadamRNN: Binary and Sparse Ternary Orthogonal RNNs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JSHHBK6W}},
  note         = {Machine review of arXiv:2502.00047}
}

read the original abstract

Binary and sparse ternary weights in neural networks enable faster computations and lighter representations, facilitating their use on edge devices with limited computational power. Meanwhile, vanilla RNNs are highly sensitive to changes in their recurrent weights, making the binarization and ternarization of these weights inherently challenging. To date, no method has successfully achieved binarization or ternarization of vanilla RNN weights. We present a new approach leveraging the properties of Hadamard matrices to parameterize a subset of binary and sparse ternary orthogonal matrices. This method enables the training of orthogonal RNNs (ORNNs) with binary and sparse ternary recurrent weights, effectively creating a specific class of binary and sparse ternary vanilla RNNs. The resulting ORNNs, called HadamRNN and Block-HadamRNN, are evaluated on benchmarks such as the copy task, permuted and sequential MNIST tasks, the IMDB dataset, two GLUE benchmarks, and two IoT benchmarks. Despite binarization or sparse ternarization, these RNNs maintain performance levels comparable to state-of-the-art full-precision models, highlighting the effectiveness of our approach. Notably, our approach is the first solution with binary recurrent weights capable of tackling the copy task over 1000 timesteps.

Figures

Figures reproduced from arXiv: 2502.00047 by the authors.

**Figure 2.** Position of HadamRNN and Block-HadamRNN models for different values of [PITH_FULL_IMAGE:figures/full_fig_p022_2.png] view at source ↗

**Figure 3.** Top: 𝐞 𝑙𝑖𝑛 and 𝐟 𝑙𝑖𝑛, bottom: 𝐞 𝑅𝑒𝐿𝑈 and 𝐟 𝑅𝑒𝐿𝑈 , see (10) and (11). Conclusion Linear-ORNNs have better memorization property than ReLU-ORNNs, and are able to retain information over arbitrarily long delays. This information may be lost after a certain delay with a ReLU-ORNN. H SWITCHING COLUMNS SIGNS IS NOT USEFUL H.1 SWITCHING COLUMNS SIGNS DOES NOT PROVIDE MORE EXPRESSIVENESS In this section, we first provide a … view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

29 extracted references · 20 canonical work pages

[1]

Abreu, J

S. Abreu, J. E. Pedersen, K. M. Heckel, and A. Pierro. Q-s5: Towards quantized state space models. arXiv preprint arXiv:2406.09477,

arXiv
[2]

This visualization highlights that, for the Copy task,𝑞 can be tuned to balance the trade-off between complexity and performance. 21 Published as a conference paper at ICLR 2025 103 104 Complexity (number of fixed-point precision additions) - log scale 10 6 10 5 10 4 10 3 Cross-entropy - log scale BHRNNq = 32 BHRNN8q = 8 BHRNNq = 2 HRNNp = 4 Figure 2: Pos...

work page 2025
[3]

20 Published as a conference paper at ICLR 2025 Table 5: HadamRNN’s training variability over 5 experiments on the IMDB dataset, for each combination of bitwidth𝑝, for the matrices𝑈 and𝑉 , and initial learning rate. U & V Initial Average Standard bitwidth learning rate performance deviation 2 1.e-2 83.28 1.00 5.e-3 83.83 1.36 1.e-3 80.84 5.09 5.e-4 80.75 ...

work page 2025
[4]

Experiments where done on a NVIDIA GeForce RTX 3080 GPU

As the training times for the Block-HadamRNN are comparable, they are not explicitly reported here. Experiments where done on a NVIDIA GeForce RTX 3080 GPU. For comparison, FastGRNN Kusupati et al. (2018) requires 16.97 hours of training time on the MNIST dataset. The training times for the HAR-2 and DSA-19 datasets, along with a comparison to those of Fa...

work page 2018
[6]

Quantized Approximately Orthogonal Recurrent Neural Networks

A. Foucault, F. Mamalet, and F. Malgouyres. Quantized approximately orthogonal recurrent neural networks. arXiv preprint arXiv:2402.04012,

work page Pith review arXiv
[7]

The experiments were conducted using the IMDB dataset, with similar trends observed across the other benchmarks, supporting the consistency of these conclusions

Table 4: Training times, in hours dataset Copy task pMNIST / sMNIST IMDB SST-2 QQP # epochs 10 200 30 60 50 training time (hr) 13 13 0.5 0.36 7.5 E.2 T RAINING STABILITY In this section, we examine the variability of the trained HadamRNN model based on different random initializations and randomness in the stochastic algorithm (Adam). The experiments were...

work page 2010
[8]

This is because HadamRNN better memorize than HadamRNN-ReLU

It can be observed that the HadamRNN- ReLU fails to learn the copy task, and obtains poor results on the other benchmarks, as compared to the HadamRNN. This is because HadamRNN better memorize than HadamRNN-ReLU. This phenomenon is analyzed in Appendix G.2. Table 8: Ablation study: performance comparison between HadamRNNs with linear recurrent units and w...

work page 2025
[9]

This finding, along with Appendix H.1, supports our choice to learn row switches only

The standard version consistently outperforms the variant, except in the copy task. This finding, along with Appendix H.1, supports our choice to learn row switches only. I P OST TRAINING QUANTIZATION OF ACTIVATIONS I.1 R EMINDERS ON FIXED -POINT ARITHMETIC We follow conventional notations for fixed-point arithmetic: For integers𝑞 ≥0 and𝑝 ≥1, the set of 𝑝...

work page 2025

Show all 29 references

[13]

13 Published as a conference paper at ICLR 2025 T. K. Rusch and S. Mishra. Unicornn: A recurrent model for learning very long time dependencies. In International Conference on Machine Learning,

2025
[14]

Zhang, Q

14 Published as a conference paper at ICLR 2025 J. Zhang, Q. Lei, and I. Dhillon. Stabilizing gradients for deep neural networks via efficient svd parameterization. In International Conference on Machine Learning,

2025
[15]

S. Zhou, Y . Wu, Z. Ni, X. Zhou, H. Wen, and Y . Zou. DoReFa-net: Training low bitwidth convolu- tional neural networks with low bitwidth gradients. arXiv preprint arXiv:1606.06160,

arXiv
[16]

Unitary Recurrent Neural Networks (URNNs) were introduced in Arjovsky et al

15 Published as a conference paper at ICLR 2025 A B IBLIOGRAPHY ON ORNN S This appendix provides a detailed bibliography on unitary and orthogonal recurrent neural networks. Unitary Recurrent Neural Networks (URNNs) were introduced in Arjovsky et al. (2016) to capture long-ter...

2016
[21]

F M ORE BENCHMARKS In Appendix F.1, we provide a comparison of HadamRNN and quantized versions of BERT on the SST-2 and QQP benchmarks from GLUE (Wang et al., 2019)

The bitwidth of the quantized matrices𝑈 and𝑉 is set to𝑝=4 . F M ORE BENCHMARKS In Appendix F.1, we provide a comparison of HadamRNN and quantized versions of BERT on the SST-2 and QQP benchmarks from GLUE (Wang et al., 2019). In Appendix F.2, we provide results on IoT task ben...

2018
[23]

(2022)) 88.7 13 400 BiBERT (Qin et al.,

89.9 13 400 BiBERT (Qin et al. (2022)) 88.7 13 400 BiBERT (Qin et al.,

2022
[24]

This result surpasses the smallest BiBERT, despite our model being more than 130 times smaller

78.2 4 400 Block-HadamRNN (ours,𝑑𝑖=256 ,𝑑ℎ=512 ,𝑞=128 ) 82.1 33 For the QQP task, using a Block-HadamRNN network with a hidden size of 512 and a block size of 128, we obtained an accuracy of82.1%. This result surpasses the smallest BiBERT, despite our model being more than 130...

2018
[25]

For both benchmarks, the hyperparameter𝑞 in Block-HadamRNN (detailed in Section 3.4) was optimized using grid search. Human Activity Recognition (HAR-2) The Human Activity Recognition (HAR) dataset consists of human motion data captured using an accelerometer and gyroscope emb...

2018
[27]

Conversely, in the ReLU-ORNN case, the norm increases, possibly approaching0

As predicted by the analysis of Appendix G.2.1, for the linear-ORNN, the magnitude of the difference in hidden states remains constant, whether the perturbation is applied along the first or second axis. Conversely, in the ReLU-ORNN case, the norm increases, possibly approachi...

2021
[29]

29 Published as a conference paper at ICLR 2025 Table 10: Value of𝛼𝑊 and𝛼ℎ for activation quantification across the datasets and bitwidth

In the following sections, we use this idea and, instead of applying the network with quantized weights(𝑞1(𝑊),𝑏𝑖,𝑞𝑝(𝑈),𝑞𝑝(𝑉),𝑏𝑜) = (𝛼𝑊̃𝑊,𝑏 𝑖,𝛼𝑈̃𝑈,𝛼 𝑉 ̃𝑉,𝑏 𝑜), we rescale with 𝜆 = 1 𝛼𝑖𝛼𝑈 and apply the network with the parameters(𝛼𝑊̃𝑊, 𝑏𝑖 𝛼𝑖𝛼𝑈 , 1 𝛼𝑖 ̃𝑈,𝛼 𝑖𝛼𝑈𝛼𝑉 ̃𝑉,𝑏 𝑜). 29 Publi...

2025
[256]

In comparison, the smallest BiBERT model, distilled from a BERT pre-trained on a large dataset and with a network size 550 times larger, achieves85.4%

For the SST-2 task, the best results were achieved with an HadamRNN network having a hidden size of 128, resulting in a total model size of approximately 8 kB and an accuracy of 81.9%. In comparison, the smallest BiBERT model, distilled from a BERT pre-trained on a large datas...

2025
[1893]

Q. He, H. Wen, S. Zhou, Y . Wu, C. Yao, X. Zhou, and Y . Zou. Effective quantization methods for recurrent neural networks. arXiv preprint arXiv:1611.10176,

arXiv
[1978]

Helfrich, D

K. Helfrich, D. Willmott, and Q. Ye. Orthogonal recurrent neural networks with scaled Cayley transform. In International Conference on Machine Learning, pages 1969–1978. PMLR,

1969
[2001]

J. Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805,

arXiv
[2015]

Courbariaux, I

3https://www.deel.ai/ 11 Published as a conference paper at ICLR 2025 M. Courbariaux, I. Hubara, D. Soudry, R. El-Yaniv, and Y . Bengio. Binarized neural networks: Training deep neural networks with weights and activations constrained to +1 or -1. arXiv preprint arXiv:1602.02830,

2025 arXiv
[2017]

12 Published as a conference paper at ICLR 2025 C. Jose, M. Cissé, and F. Fleuret. Kronecker recurrent units. In International Conference on Machine Learning,

2025
[2019]

Z. Lin, M. Courbariaux, R. Memisevic, and Y . Bengio. Neural networks with few multiplications. arXiv preprint arXiv:1510.03009,

arXiv
[2021]

Bengio, N

Y . Bengio, N. Léonard, and A. Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432,

arXiv
[2022]

Z. Liu, B. Oguz, C. Zhao, E. Chang, P. Stock, Y . Mehdad, Y . Shi, R. Krishnamoorthi, and V . Chandra. LLM-QAT: Data-free quantization aware training for large language models. arXiv preprint arXiv:2305.17888,

arXiv
[2023]

J. Ott, Z. Lin, Y . Zhang, S.-C. Liu, and Y . Bengio. Recurrent neural networks with limited numerical precision. arXiv preprint arXiv:1608.06902,

arXiv
[2024]

M. Z. Alom, A. T. Moody, N. Maruyama, B. C. V . Essen, and T. M. Taha. Effective quantization ap- proaches for recurrent neural networks. 2018 International Joint Conference on Neural Networks (IJCNN), pages 1–8,

2018

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

Abreu, J

S. Abreu, J. E. Pedersen, K. M. Heckel, and A. Pierro. Q-s5: Towards quantized state space models. arXiv preprint arXiv:2406.09477,

arXiv

[2] [2]

This visualization highlights that, for the Copy task,𝑞 can be tuned to balance the trade-off between complexity and performance. 21 Published as a conference paper at ICLR 2025 103 104 Complexity (number of fixed-point precision additions) - log scale 10 6 10 5 10 4 10 3 Cross-entropy - log scale BHRNNq = 32 BHRNN8q = 8 BHRNNq = 2 HRNNp = 4 Figure 2: Pos...

work page 2025

[3] [3]

20 Published as a conference paper at ICLR 2025 Table 5: HadamRNN’s training variability over 5 experiments on the IMDB dataset, for each combination of bitwidth𝑝, for the matrices𝑈 and𝑉 , and initial learning rate. U & V Initial Average Standard bitwidth learning rate performance deviation 2 1.e-2 83.28 1.00 5.e-3 83.83 1.36 1.e-3 80.84 5.09 5.e-4 80.75 ...

work page 2025

[4] [4]

Experiments where done on a NVIDIA GeForce RTX 3080 GPU

As the training times for the Block-HadamRNN are comparable, they are not explicitly reported here. Experiments where done on a NVIDIA GeForce RTX 3080 GPU. For comparison, FastGRNN Kusupati et al. (2018) requires 16.97 hours of training time on the MNIST dataset. The training times for the HAR-2 and DSA-19 datasets, along with a comparison to those of Fa...

work page 2018

[5] [6]

Quantized Approximately Orthogonal Recurrent Neural Networks

A. Foucault, F. Mamalet, and F. Malgouyres. Quantized approximately orthogonal recurrent neural networks. arXiv preprint arXiv:2402.04012,

work page Pith review arXiv

[6] [7]

The experiments were conducted using the IMDB dataset, with similar trends observed across the other benchmarks, supporting the consistency of these conclusions

Table 4: Training times, in hours dataset Copy task pMNIST / sMNIST IMDB SST-2 QQP # epochs 10 200 30 60 50 training time (hr) 13 13 0.5 0.36 7.5 E.2 T RAINING STABILITY In this section, we examine the variability of the trained HadamRNN model based on different random initializations and randomness in the stochastic algorithm (Adam). The experiments were...

work page 2010

[7] [8]

This is because HadamRNN better memorize than HadamRNN-ReLU

It can be observed that the HadamRNN- ReLU fails to learn the copy task, and obtains poor results on the other benchmarks, as compared to the HadamRNN. This is because HadamRNN better memorize than HadamRNN-ReLU. This phenomenon is analyzed in Appendix G.2. Table 8: Ablation study: performance comparison between HadamRNNs with linear recurrent units and w...

work page 2025

[8] [9]

This finding, along with Appendix H.1, supports our choice to learn row switches only

The standard version consistently outperforms the variant, except in the copy task. This finding, along with Appendix H.1, supports our choice to learn row switches only. I P OST TRAINING QUANTIZATION OF ACTIVATIONS I.1 R EMINDERS ON FIXED -POINT ARITHMETIC We follow conventional notations for fixed-point arithmetic: For integers𝑞 ≥0 and𝑝 ≥1, the set of 𝑝...

work page 2025