REVIEW 3 major objections 6 minor 33 references
An Efficient Attention Mechanism for Sequential Recommendation Tasks: HydraRec
T0 review · 3 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read HydraRec replaces pairwise attention with a global context vector and a filter, claiming linear-time attention that matches or beats dot-product models for next-item prediction.
desk verdict A faithful Hydra-attention-for-recommendation port with plausible causal-mask results, but the paper overclaims the linear-in-d regime and its own Table 4 undermines the no-accuracy-loss efficiency story. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The mechanism is Hydra attention, given in Equation (6) as $$\mathrm{Hydra}(Q,K,V,\varphi)=\varphi(Q)\sum_{i=0}^{N}\varphi(K)^T V^T.$$ It collapses the entire sequence into one global context vector through the summation, then multiplies by $\varphi(Q)$ to filter that context for each item. Because the expensive matrix product is computed first as $\varphi(K)^T V$, the complexity becomes $O(Nd(d/H))$, and setting the number of heads $H$ equal to the embedding dimension $d$ yields $O(Nd)$ time and space complexity.
What would settle it
A controlled run on ML-20m with maximum sequence length 200, heads fixed at 8, comparing HydraRecUni against dot-product attention under causal masking would settle the claim: if NDCG@10 falls below the dot-product baseline as sequences lengthen, the global-context filter is losing the pairwise information the method depends on.
Extended reading notes
Core claim
The central claim is that efficient attention for sequential recommendation does not require a pairwise attention matrix. HydraRec computes one global vector from the whole item sequence and then filters it per item, using $$\mathrm{Hydra}(Q,K,V,\varphi)=\varphi(Q)\sum_{i=0}^{N}\varphi(K)^T V^T,$$ where the sum is elementwise and $\varphi$ is a decomposable kernel. This avoids the $N\times N$ attention matrix, with complexity $O(Nd(d/H))$, and $O(Nd)$ when the number of heads $H$ equals the embedding dimension $d$. The paper reports that under causal masking this formulation outperforms dot-product attention and three linear-attention baselines in NDCG@10 and Hit@10 across ML-1m, ML-20m, and Beauty, and that in bidirectional mode it reaches accuracy comparable to BERT4Rec with lower training time.
Load-bearing premise
The load-bearing assumption is that a single global context vector, filtered per item, preserves enough item-to-item interaction information for next-item prediction, and that the head-count-equals-embedding configuration that gives full linearity is a practical setting.
Editorial extensions
If this is right
- Long user histories can be processed at cost that grows linearly with sequence length, making transformer recommenders feasible where quadratic attention would be prohibitive.
- In causal-mask next-item prediction, the efficiency gain does not come at an accuracy cost; the paper reports higher NDCG@10 and Hit@10 than dot-product attention on all three datasets.
- When the number of heads equals the embedding dimension, the same model also removes the quadratic dependence on embedding size, which matters as catalogues and item features grow.
- Because HydraRec changes only the attention computation, existing transformer-based recommender architectures can adopt it without redesigning embeddings, training objective, or masking strategy.
Reading between the lines
- A natural next test is to run the main causal-mask comparisons with heads equal to the embedding dimension, since the fully linear $O(Nd)$ regime is claimed but the headline experiments use eight heads.
- The global-vector formulation discards pairwise interaction structure, so its relative advantage plausibly depends on sequence length and sparsity; datasets with longer dense histories might favor dot-product attention, and the paper's reported decline in bidirectional hit rate on longer sequences points in that direction.
- The same attention swap could be tested in decoder-only generative recommenders or language models, since causal masking is already standard there and the complexity saving would compound with longer contexts.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper adapts Hydra attention, a many-head linear attention mechanism from vision transformers, to the BERT4Rec encoder architecture for sequential recommendation, producing two variants: HydraRecUni (causal masking) and HydraRecBi (bidirectional masking). The authors claim that HydraRec reduces attention complexity to O(Nd) in time and space when the number of heads equals the embedding dimension d, and they report accuracy comparisons against dot-product BERT4Rec and three linear-attention baselines on ML-1m, ML-20m, and Beauty, together with runtime comparisons. The central empirical claim is that HydraRec outperforms dot-product attention under causal masking and is comparable to BERT4Rec bidirectionally while being substantially faster.
Significance. If fully supported, the result would be practically useful: HydraRec is a simple drop-in replacement for the attention module, it is built on a published mechanism, and the paper evaluates it on standard datasets against relevant linear and dot-product baselines while holding the rest of the architecture fixed. The runtime measurements are a genuine strength, and the comparison design is reasonable in that only the attention calculation changes across models. However, the paper does not currently establish the advertised O(Nd) regime in the settings where accuracy is reported, and the head-count ablation in Table 4 appears to contradict the 'no accuracy loss' narrative. The central significance therefore rests on additional experiments that are not reported, or on a substantial narrowing of the claims. The paper also provides no code, no error bars, and no significance tests, which further limits the strength of the empirical evidence.
major comments (3)
- [Section 4.4 and Table 4 (RQ4)] The central efficiency-accuracy claim is not supported by the reported experiments. The O(Nd) regime requires h=d, but the accuracy comparisons in Tables 2 and 3 use h=8 as stated in Section 4.4, so those results correspond to O(Nd^2/8), which is quadratic in d. In the only head-count ablation (Table 4, ML-1m), the h=d configuration is tested only at d=64; it gives NDCG@10 0.2922 versus 0.3028 for h=8,d=64, and the higher-dimensional rows (h=64,d=128/256/512) are not h=d. Moreover, NDCG@10 drops from 0.4113 (h=8,d=256) to 0.3659 (h=64,d=256) and from 0.4875 (h=8,d=512) to 0.3568 (h=64,d=512), so the statement in the Discussion that 'NDCG@10 was comparable for all cases' is contradicted by the paper's own numbers. The authors should report h=d results for the causal-mask setting and for all datasets, and either demonstrate no accuracy loss or qualify the headline claim.
- [Table 3, Beauty rows] The BERT4Rec baseline on Amazon Beauty reports HIT@10 values of 0.3421 and 0.3498, which are far below every other model in the same table, including HydraRecBi at about 0.76, while the NDCG@10 differences are much smaller. This large and unexplained discrepancy suggests an implementation or evaluation issue with the baseline and directly undermines the claim that HydraRecBi is comparable to BERT4Rec. In addition, the paper reports only the best validation-set result with no error bars, no random seeds, and no significance tests, so small differences (for example ML-20m NDCG@10 0.7553 vs 0.7561) cannot be meaningfully interpreted.
- [Section 3.2.2, Equation (6)] The definition of Hydra attention is not self-contained. The expression Hydra(Q,K,V,φ) = φ(Q)∑_{i=0}^N φ(K)^T V^T is dimensionally unclear for the stated shapes, and the text that 'the multiplication φ(K)^T V^T is elementwise' does not resolve the ambiguity. Because this equation is the basis for the complexity calculation and for the claimed difference from dot-product attention, a precise per-head derivation, or a faithful reproduction of the definition in reference [11], is needed before the theoretical claims can be audited.
minor comments (6)
- [Section 3.1 and Equation (6)] The summation in Equation (6) runs from i=0 to N, but item sequences in Section 3.1 are indexed from 1 to n_u; please use a consistent indexing convention.
- [Table 3] The Beauty HIT@10 row lists '10' in the Epochs column for the second result; this is presumably a typo for '100' and should be corrected.
- [Section 4.4] The paper says 'We report the best results on a validation set' but does not give the exact hyperparameters (embedding dimension, sequence length, batch size, masking probability) that produced each row of Tables 2 and 3; please provide the configuration for reproducibility.
- [Section 4.5] The statement that 'HydraRecUni outperforms every model' is too strong for Table 2, where LM1 is close on ML-1m (NDCG@10 0.4755 vs 0.4899) and LM2/LM3 are within small margins on several rows; please qualify the claim to the datasets and metrics where the difference is consistent.
- [Figures 2 and 3] The figure captions do not specify which dataset and sequence length were used, and the y-axis label 'TIME IN SECS' is inconsistent with the axis title 'EPOCHS' shown in the figure; please clarify.
- [Throughout] There is no statement about code or data availability, nor any report of random seeds or multiple runs; adding this would materially improve reproducibility.
Circularity Check
No circular derivation: HydraRec's efficiency claim is inherited algebra from Hydra attention and its accuracy claims are empirical comparisons against external baselines.
full rationale
The paper's central efficiency claim reduces to the algebraic complexity formula in Equation (6), O(Nd(d/H)), which becomes O(Nd) when H=d. This is a mathematical consequence of the cited Hydra attention construction, not a fitted parameter renamed as a prediction; the paper explicitly builds on the existing Hydra attention model rather than claiming to derive it from first principles. The accuracy claims are supported by direct empirical comparisons against independently implemented baselines (BERT4Rec, LinRec, and other linear attention models) on three datasets, with results reported as validation metrics under stated hyperparameter searches. There are no load-bearing self-citations: the author has no prior work cited as the source of the central result, and no uniqueness theorem is imported to forbid alternatives. The main weakness is that the advertised O(Nd) regime (h=d) is not the configuration used in the principal causal-mask experiments, and Table 4 shows noticeable NDCG@10 degradation when heads are increased toward d, contradicting the paper's assertion that NDCG@10 was comparable. That is a correctness or support concern, not a circularity: the reported numbers are not constructed from the claim they are meant to prove. Consequently, no step in the derivation chain reduces to its own inputs by definition or by fitted-value equivalence.
Assumptions & free parameters
free parameters (5)
- Best hyperparameter configuration per dataset =
Not fully reported, selected as best on validation split
- Number of attention heads h =
8 in main experiments; 8, 16, 64 in Table 4
- Embedding dimension d =
16, 32, 64, 128 in main experiments; 64, 128, 256, 512 in Table 4
- Masking probability =
10 percent
- Kernel choice =
Cosine similarity kernel only
assumptions (5)
- standard math A decomposable kernel phi allows rewriting attention as phi(Q)(phi(K)^T V) using associativity (Equation 3).
- domain assumption With H attention heads, the total complexity of linear attention is O(H N (d/H)^2) = O(N d^2 / H).
- domain assumption The global-then-filter formulation of Hydra attention preserves enough temporal context for sequential recommendation.
- domain assumption BERT4Rec's cloze training with masked item prediction and leave-one-out evaluation is a valid setup for sequential recommendation.
- domain assumption The cosine similarity kernel is a valid decomposable similarity function for attention.
Cite this review
Pith. "Pith review of An Efficient Attention Mechanism for Sequential Recommendation Tasks: HydraRec." pith.science (2026). https://pith.science/paper/MM3E4XZ4
@misc{pith2026250101242,
author = {Pith},
title = {Pith review of: An Efficient Attention Mechanism for Sequential Recommendation Tasks: HydraRec},
year = {2026},
howpublished = {\url{https://pith.science/paper/MM3E4XZ4}},
note = {Machine review of arXiv:2501.01242}
}
read the original abstract
Transformer based models are increasingly being used in various domains including recommender systems (RS). Pretrained transformer models such as BERT have shown good performance at language modelling. With the greater ability to model sequential tasks, variants of Encoder-only models (like BERT4Rec, SASRec etc.) have found success in sequential RS problems. Computing dot-product attention in traditional transformer models has quadratic complexity in sequence length. This is a bigger problem with RS because unlike language models, new items are added to the catalogue every day. User buying history is a dynamic sequence which depends on multiple factors. Recently, various linear attention models have tried to solve this problem by making the model linear in sequence length (token dimensions). Hydra attention is one such linear complexity model proposed for vision transformers which reduces the complexity of attention for both the number of tokens as well as model embedding dimensions. Building on the idea of Hydra attention, we introduce an efficient Transformer based Sequential RS (HydraRec) which significantly improves theoretical complexity of computing attention for longer sequences and bigger datasets while preserving the temporal context. Extensive experiments are conducted to evaluate other linear transformer-based RS models and compared with HydraRec across various evaluation metrics. HydraRec outperforms other linear attention-based models as well as dot-product based attention models when used with causal masking for sequential recommendation next item prediction tasks. For bi-directional models its performance is comparable to the BERT4Rec model with an improvement in running time.
Reference graph
Works this paper leans on
-
[13]
Bert4rec: Sequential recommendation with bidirectional encoder representations from transformer,
F. Sun et al. , “Bert4rec: Sequential recommendation with bidirectional encoder representations from transformer,” Int. Conf. Inf. Knowl. Manag. Proc., pp. 1441–1450, 2019, doi: 10.1145/3357384.3357895. An Efficient Attention Mechanism for Sequential Recommendation Tasks: HydraRec 17
arXiv 2019
-
[11]
Hydra Attention: Efficient Attention with Many Heads,
D. Bolya, C. Y. Fu, X. Dai, P. Zhang, and J. Hoffman, “Hydra Attention: Efficient Attention with Many Heads,” Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics) , vol. 13807 LNCS, pp. 35 –49, 2023, doi: 10.1007/978-3-031-25082-8_3
-
[1]
Neural attentive session -based recommendation,
J. Li, P. Ren, Z. Chen, Z. Ren, T. Lian, and J. Ma, “Neural attentive session -based recommendation,” in Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, 2017, pp. 1419–1428
work page 2017
-
[2]
Self-attentive sequential recommendation,
W.-C. Kang and J. McAuley, “Self-attentive sequential recommendation,” in 2018 IEEE international conference on data mining (ICDM), 2018, pp. 197–206
work page 2018
-
[3]
Pre-training of Context-aware Item Representation for Next Basket Recommendation
J. Yang, J. Xu, J. Tong, S. Gao, J. Guo, and J. Wen, “Pre-training of Context-aware Item Representation for Next Basket Recommendation.”
-
[4]
A. Vaswani et al., “Attention is all you need,” Adv. Neural Inf. Process. Syst., vol. 2017- Decem, no. Nips, pp. 5999–6009, 2017
work page 2017
-
[5]
BERT: Pre -training of deep bidirectional transformers for language understanding,
J. Devlin, M. W. Chang, K. Lee, and K. Toutanova, “BERT: Pre -training of deep bidirectional transformers for language understanding,” NAACL HLT 2019 - 2019 Conf. North Am. Chapter Assoc. Comput. Linguist. Hum. Lang. Technol. - Proc. Conf., vol. 1, no. Mlm, pp. 4171–4186, 2019
work page 2019
-
[6]
A survey on efficient training of transformers,
B. Zhuang, J. Liu, Z. Pan, H. He, Y. Weng, and C. Shen, “A survey on efficient training of transformers,” arXiv Prepr. arXiv2302.01107, 2023
arXiv 2023
Show all 33 references
-
[7]
Linformer: Self -Attention with Linear Complexity,
S. Wang, B. Z. Li, M. Khabsa, H. Fang, and H. Ma, “Linformer: Self -Attention with Linear Complexity,” vol. 2048, no. 2019, 2020, [Online]. Available: http://arxiv.org/abs/2006.04768
2019 arXiv
-
[8]
Big bird: Transformers for longer sequences,
M. Zaheer et al. , “Big bird: Transformers for longer sequences,” Adv. Neural Inf. Process. Syst., vol. 33, pp. 17283–17297, 2020
2020
-
[9]
BasisFormer: Attention -based Time Series Forecasting with Learnable and Interpretable Basis,
Z. Ni, H. Yu, S. Liu, J. Li, and W. Lin, “BasisFormer: Attention -based Time Series Forecasting with Learnable and Interpretable Basis,” Adv. Neural Inf. Process. Syst. , vol. 36, no. NeurIPS, 2023
2023
-
[10]
LinRec: Linear Attention Mechanism for Long -term Sequential Recommender Systems,
L. Liu et al. , “LinRec: Linear Attention Mechanism for Long -term Sequential Recommender Systems,” SIGIR 2023 - Proc. 46th Int. ACM SIGIR Conf. Res. Dev. Inf. Retr., pp. 289–299, 2023, doi: 10.1145/3539618.3591717
2023
-
[12]
Faster causal attention over large sequences through sparse flash attention,
M. Pagliardini, D. Paliotta, M. Jaggi, and F. Fleuret, “Faster causal attention over large sequences through sparse flash attention,” arXiv Prepr. arXiv2306.01160, 2023
2023 arXiv
-
[14]
An attention free transformer,
S. Zhai et al., “An attention free transformer,” arXiv Prepr. arXiv2105.14103, 2021
2021 arXiv
-
[15]
Training data-efficient image transformers & distillation through attention,
H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou, “Training data-efficient image transformers & distillation through attention,” in International conference on machine learning, 2021, pp. 10347–10357
2021
-
[16]
An MDP -based recommender system.,
G. Shani, D. Heckerman, R. I. Brafman, and C. Boutilier, “An MDP -based recommender system.,” J. Mach. Learn. Res., vol. 6, no. 9, 2005
2005
-
[17]
A Markov chain model for integrating context in recommender systems,
F. Mourchid, J. Ben Othman, A. Kobbane, E. Sabir, and M. El Koutbi, “A Markov chain model for integrating context in recommender systems,” in 2016 IEEE global communications conference (GLOBECOM), 2016, pp. 1–6
2016
-
[18]
A hybrid recommender system for sequential recommendation: combining similarity models with markov chains,
Y. Yang, H. -J. Jang, and B. Kim, “A hybrid recommender system for sequential recommendation: combining similarity models with markov chains,” IEEE Access, vol. 8, pp. 190136–190146, 2020
2020
-
[19]
Long short -term memory,
A. Graves and A. Graves, “Long short -term memory,” Supervised Seq. Label. with Recurr. neural networks, pp. 37–45, 2012
2012
-
[20]
Recurrent recommender networks,
C.-Y. Wu, A. Ahmed, A. Beutel, A. J. Smola, and H. Jing, “Recurrent recommender networks,” in Proceedings of the tenth ACM international conference on web search and data mining, 2017, pp. 495–503
2017
-
[21]
A social hybrid recommendation system using LSTM and CNN,
H. Daneshvar and R. Ravanmehr, “A social hybrid recommendation system using LSTM and CNN,” Concurr. Comput. Pract. Exp., vol. 34, no. 18, p. e7015, 2022
2022
-
[22]
Sequential user -based recurrent neural network recommendations,
T. Donkers, B. Loepp, and J. Ziegler, “Sequential user -based recurrent neural network recommendations,” in Proceedings of the eleventh ACM conference on recommender systems, 2017, pp. 152–160
2017
-
[23]
Recurrent neural networks with top-k gains for session- based recommendations,
B. Hidasi and A. Karatzoglou, “Recurrent neural networks with top-k gains for session- based recommendations,” in Proceedings of the 27th ACM international conference on information and knowledge management, 2018, pp. 843–852
2018
-
[24]
Personalized top-n sequential recommendation via convolutional sequence embedding,
J. Tang and K. Wang, “Personalized top-n sequential recommendation via convolutional sequence embedding,” in Proceedings of the eleventh ACM international conference on web search and data mining, 2018, pp. 565–573
2018
-
[25]
Deep learning based recommender system: A survey and new perspectives,
S. Zhang, L. Yao, A. Sun, and Y. Tay, “Deep learning based recommender system: A survey and new perspectives,” ACM Comput. Surv., vol. 52, no. 1, pp. 1–38, 2019
2019
-
[26]
Neural machine translation by jointly learning to align and translate,
D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” arXiv Prepr. arXiv1409.0473, 2014
2014 arXiv
-
[27]
How good are gpt models at machine translation? a comprehensive evaluation,
A. Hendy et al., “How good are gpt models at machine translation? a comprehensive evaluation,” arXiv Prepr. arXiv2302.09210, 2023
2023 arXiv
-
[28]
SSE-PT: Sequential recommendation via personalized transformer,
L. Wu, S. Li, C.-J. Hsieh, and J. Sharpnack, “SSE-PT: Sequential recommendation via personalized transformer,” in Proceedings of the 14th ACM conference on recommender systems, 2020, pp. 328–337
2020
-
[29]
Recent advances in vision transformer: A survey and outlook of recent work,
K. Islam, “Recent advances in vision transformer: A survey and outlook of recent work,” arXiv Prepr. arXiv2203.01536, 2022
2022 arXiv
-
[30]
Toward the next generation of recommender systems: A survey of the state-of-the-art and possible extensions,
G. Adomavicius and A. Tuzhilin, “Toward the next generation of recommender systems: A survey of the state-of-the-art and possible extensions,” IEEE Trans. Knowl. Data Eng., vol. 17, no. 6, pp. 734–749, 2005
2005
-
[31]
Core: simple and effective session -based recommendation within consistent representation space,
Y. Hou, B. Hu, Z. Zhang, and W. X. Zhao, “Core: simple and effective session -based recommendation within consistent representation space,” in Proceedings of the 45th international ACM SIGIR conference on research and development in information 18 Uzma Mushtaque. retrieval, 20...
2022
-
[32]
Efficient attention: Attention with linear complexities,
Z. Shen, M. Zhang, H. Zhao, S. Yi, and H. Li, “Efficient attention: Attention with linear complexities,” in Proceedings of the IEEE/CVF winter conference on applications of computer vision, 2021, pp. 3531–3539
2021
-
[33]
Transformers are rnns: Fast autoregressive transformers with linear attention,
A. Katharopoulos, A. Vyas, N. Pappas, and F. Fleuret, “Transformers are rnns: Fast autoregressive transformers with linear attention,” in International conference on machine learning, 2020, pp. 5156–5165
2020
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.