REVIEW 4 major objections 4 minor 1 cited by
User Long-Term Multi-Interest Retrieval Model for Recommendation
T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read ULIM enables retrieval-stage models to consume two years of user behavior history, beating deployed baselines and lifting Taobao mini-app orders by 11.01%.
desk verdict A credible industrial push toward long-sequence retrieval, with a real online lift, but the offline evidence doesn't yet pin the gain on the architecture. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Category-Aware Hierarchical Dual-Interest Learning is the training-side mechanism: it clusters the long behavior sequence by category into subsequences, reducing per-sequence complexity from $O(L)$ to $O(L/N)$, uses the pooled short-term sequence as the query for target-attention over the matching long-term subsequence, and optimizes a weighted sum of long- and short-term sampled-softmax losses. Pointer-Enhanced Cascaded Category-to-Item Retrieval is the serving-side mechanism: a Pointer-Generator Interest Network blends a pointer distribution over the user's own past categories with a generator distribution over all categories to predict the top K categories, then parallel ANN searches using both long-term category-specific embeddings and a global short-term embedding retrieve items only from those categories. The category constraint keeps offline training and online serving in the same distribution and is what makes thousand-scale sequences computationally tractable at retrieval time.
What would settle it
Run the YouTube-DNN or MIND baseline with the same two-year user history and the same category-homogeneous negative sampling, then compare HR@2000; if either reaches about 16.55%, the claimed architectural contribution is not what drives the result.
Extended reading notes
Core claim
The paper's central claim is that retrieval-stage models can consume two years of user behavior and outperform a deployed MIND-based channel, if the long sequence is organized by category rather than processed as one flat list. ULIM partitions behavior into category-homogeneous subsequences aligned to the ranking stage's categories, trains separate long-term interest embeddings per subsequence and one short-term embedding, and redefines the training objective so positives and negatives come from the same category subspace. At serving, a Pointer-Generator Interest Network predicts the top K categories, and each long-term embedding searches only its own category's candidate pool while the short-term embedding searches everything. The paper reports offline HR@2000 of 16.55% versus 15.15% for the MIND variant and 14.93% for the YouTube-DNN variant, and online lifts of 5.54% clicks, 11.01% orders, and 4.03% GMV on Taobaomiaosha.
Load-bearing premise
The experiments assume the reported gains come from ULIM's architecture rather than from the fact that ULIM alone is given two years of behavior history and category-constrained negatives; the paper never runs a baseline with the same history length.
Editorial extensions
If this is right
- Retrieval-stage models can be upgraded from hundreds to thousands of past actions without breaking the latency budget, since category prediction narrows the search space before item embedding search.
- Multi-interest representations and target-aware attention, previously confined to ranking, can be moved into candidate retrieval.
- Aligning the category granularity of retrieval with ranking creates a consistent feature space across the two stages, which should make downstream ranking easier.
- The 15 ms online latency increase is small enough that similar two-stage cascades are plausible in other large-scale industrial recommenders.
Reading between the lines
- The paper's own ablation only halves ULIM's sequence and swaps target-attention for self-attention; it never tests a simple long-history baseline, so the cleanest test of the paper's thesis would be an apples-to-apples history-length comparison.
- If the category-to-item cascade is the real driver, then the same pointer-generator idea could extend to other structured taxonomies, such as brand or store hierarchies, wherever retrieval can be decomposed into a coarse prediction followed by a narrow search.
- Because the short-term embedding still searches the whole pool, the architecture still retains a global fallback; a natural extension would be to learn when to skip the global search entirely to save latency.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes ULIM, a retrieval-stage model for Taobao's mini-app that consumes up to two years of user behavior by (i) clustering long-term behaviors into category-aware subsequences and learning separate long- and short-term interest embeddings with a composite sampled-softmax loss, and (ii) serving via a Pointer-Generator Interest Network that predicts top-K categories followed by parallel ANN retrieval within those categories. The authors report offline HR@500/1000/2000 gains over YouTube-DNN and MIND variants on a Taobao dataset and an online A/B lift of 5.54% clicks, 11.01% orders, and 4.03% GMV. The core design is plausible and practical, but the evaluation currently does not isolate the architecture's contribution from the effects of longer input history and category-constrained evaluation, and the training objective as written is incomplete.
Significance. If the reported results hold, this is a valuable industrial contribution: it demonstrates that retrieval-stage models can exploit thousand-scale behavior sequences with acceptable latency through a category-cascaded design, and the positive online A/B results on a live Taobao mini-app provide real-world evidence of deployability. The paper also ships a useful practical recipe—category-aware sub-sequence clustering, dual long/short-term interest training, and pointer-based category prediction—that could transfer to other e-commerce retrieval systems. However, the significance claims rest on comparisons that are confounded by input-length and evaluation-protocol differences; the paper would be materially stronger if the ablations and baselines were matched on these axes.
major comments (4)
- [Section 3.1.1, Table 1, Table 2] The offline comparison does not control for sequence length. Section 3.1.1 states that ULIM uses two years of user historical behavior, but the text does not state that the YouTube-DNN and MIND baselines are given the same two-year history; it only mentions matching embedding dimension, number of retrieved items, and optimizer. The half-sequence ablation in Table 2 drops HR@2000 from 16.55% to 13.36%, showing that input length is a major driver of the metric. Therefore, the reported margin over the baselines could be due to the longer input rather than to ULIM's architecture. Please add a baseline (e.g., a YouTube-DNN or MIND variant) trained and evaluated with the same two-year sequence, or explicitly state and justify any shorter history used for baselines.
- [Section 2.2.2, Table 1] It is unclear whether the baselines are evaluated under the same category-constrained candidate set. Section 2.2.2 says ULIM's long-term embeddings retrieve only from their corresponding category's candidates, while the short-term embedding searches the entire pool. If the YouTube-DNN and MIND baselines search the full item pool for all embeddings, their HR values are computed over a different (usually much larger) candidate space, making the absolute HR numbers incomparable. The paper must disclose the exact candidate pool construction for each method and either apply the same constraint to all methods or report a pooled evaluation protocol.
- [Eq. (2)] Equation (2) is not a valid sampled softmax as written. The denominator sums only over j in Ineg (the negative set) and does not include the positive item's exponent term, so the quotient is not a normalized probability distribution over the sampled items. With this loss, increasing the scale of all dot products can reduce the loss even when the positive item is not ranked first, which would lead to training collapse. Please clarify whether the positive item is included in Ineg; if it is, write the denominator as exp(v_u^T e_i) + sum_{j in Ineg} exp(v_u^T e_j), or define Ineg to include the positive sample.
- [Section 3.2 and Section 3.1.2] The paper reports offline HR numbers without standard deviations or significance tests and online A/B percentage lifts without confidence intervals or p-values. For example, the online claim of 11.01% orders lift and the offline HR@2000 difference of 15.15% vs. 16.55% are presented as point estimates with no uncertainty quantification. Given that the A/B test adds ULIM as an independent retrieval channel, the lifts conflate the effect of adding a new channel with the effect of ULIM's architecture. Please report confidence intervals and, where feasible, significance levels for both offline and online metrics, and state whether the online test was a holdout/gray-release design.
minor comments (4)
- [Section 3.3, Table 2] The text says 'Three variants are proposed' and 'ULIM significantly outperforms the other three variants', but Table 2 lists only two variant rows (ULIM-half-sequence and ULIM-self-attention) plus the full ULIM. Please correct the count or add the missing variant.
- [Section 3.1.3, Figure 3] The text says greater K achieves better HR@2000 while exhibiting 'marginal effect', but Figure 3 shows a monotonic increase from 6.10% at K=1 to 16.55% at K=13. The word 'marginal' is misleading; it appears each additional category adds a smaller increment, not that the overall effect is marginal. Please rephrase.
- [Conclusion] There are typos in the final paragraph: 'brides' should be 'bridges' and 'stags' should be 'stages'. The paper would benefit from a careful proofread.
- [Section 3.1.1] Please specify the exact history length used for each baseline and for ULIM, and state whether the two-year window is the same for offline training and online serving. The current sentence only says ULIM uses two years; it does not say what the baselines use.
Circularity Check
No circularity: ULIM's claimed gains rest on external baselines and a live A/B test, with no derivation that reduces to its own inputs.
full rationale
The paper is an empirical systems paper. Its central claims—that ULIM achieves state-of-the-art retrieval performance and lifts Taobaomiaosha business metrics—are supported by offline comparisons against YouTube-DNN and MIND variants and by a three-week online A/B test. The training objectives in Eq. (1)-(2) are standard sampled-softmax losses supervised by external click labels; PGIN's blended category distribution is trained by cross-entropy against true next-category labels. The category-constrained serving rule in Sec. 2.2.2 is an explicit train/serve alignment choice, not a quantity fitted to the evaluation metric. The paper contains no self-citation chain, no imported uniqueness theorem, and no equation in which an output variable is defined as the input variable. The concerns raised by a skeptical reader—two-year history used only for ULIM, and the candidate-pool protocol for HR not fully disclosed—are experimental-fairness issues that could affect correctness or generalizability, but they are not circular reductions under the definitions used here. Therefore no circular step is identified.
Assumptions & free parameters
free parameters (5)
- Loss weights alpha and beta =
not reported
- Category count K =
not reported (Fig. 3 explores 1 to 13)
- Short-term sequence length (100 actions) =
100
- Long-term behavior window =
two years
- In-batch negative sampling configuration =
not reported
assumptions (5)
- domain assumption A fixed category taxonomy aligned with the ranking stage is available.
- domain assumption User behavior sequences are chronologically ordered and carry category labels.
- domain assumption In-batch sampled softmax with shared negatives approximates the full softmax sufficiently.
- domain assumption Category-level prediction is a useful intermediate for item retrieval.
- domain assumption ANN search returns high-recall nearest neighbors within each category subspace.
Cite this review
Pith. "Pith review of User Long-Term Multi-Interest Retrieval Model for Recommendation." pith.science (2026). https://pith.science/paper/IKOLFBYY
@misc{pith2026250710097,
author = {Pith},
title = {Pith review of: User Long-Term Multi-Interest Retrieval Model for Recommendation},
year = {2026},
howpublished = {\url{https://pith.science/paper/IKOLFBYY}},
note = {Machine review of arXiv:2507.10097}
}
read the original abstract
User behavior sequence modeling, which captures user interest from rich historical interactions, is pivotal for industrial recommendation systems. Despite breakthroughs in ranking-stage models capable of leveraging ultra-long behavior sequences with length scaling up to thousands, existing retrieval models remain constrained to sequences of hundreds of behaviors due to two main challenges. One is strict latency budget imposed by real-time service over large-scale candidate pool. The other is the absence of target-aware mechanisms and cross-interaction architectures, which prevent utilizing ranking-like techniques to simplify long sequence modeling. To address these limitations, we propose a new framework named User Long-term Multi-Interest Retrieval Model(ULIM), which enables thousand-scale behavior modeling in retrieval stages. ULIM includes two novel components: 1)Category-Aware Hierarchical Dual-Interest Learning partitions long behavior sequences into multiple category-aware subsequences representing multi-interest and jointly optimizes long-term and short-term interests within specific interest cluster. 2)Pointer-Enhanced Cascaded Category-to-Item Retrieval introduces Pointer-Generator Interest Network(PGIN) for next-category prediction, followed by next-item retrieval upon the top-K predicted categories. Comprehensive experiments on Taobao dataset show that ULIM achieves substantial improvement over state-of-the-art methods, and brings 5.54% clicks, 11.01% orders and 4.03% GMV lift for Taobaomiaosha, a notable mini-app of Taobao.
Figures
Forward citations
Cited by 1 Pith paper
-
Coarse-to-Fine Long-term Interest Modeling for Generative Recommendation
GLASS extends generative retrieval with a tiered long-term interest vector and a first-SID-keyed search of long histories, reporting consistent gains over Tiger and DualGR on two public datasets.
Reference graph
Works this paper leans on
-
[1]
Yukuo Cen, Jianwei Zhang, Xu Zou, Chang Zhou, Hongxia Yang, and Jie Tang
-
[2]
Qiwei Chen, Changhua Pei, Shanshan Lv, Chao Li, Junfeng Ge, and Wenwu Ou
-
[3]
Xu Chen, Hongteng Xu, Yongfeng Zhang, Jiaxi Tang, Yixin Cao, Zheng Qin, and Hongyuan Zha. 2018. Sequential Recommendation with User Memory Networks. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, WSDM 2018, Marina Del Rey, CA, USA, February 5-9, 2018, Yi Chang, Chengxiang Zhai, Yan Liu, and Yoelle Maarek (Eds.)....
arXiv 2018
-
[4]
Paul Covington, Jay Adams, and Emre Sargin. 2016. Deep Neural Networks for YouTube Recommendations. In Proceedings of the 10th ACM Conference on Recommender Systems, Boston, MA, USA, September 15-19, 2016, Shilad Sen, Werner Geyer, Jill Freyne, and Pablo Castells (Eds.). ACM, 191–198. https://doi.org/10. 1145/2959100.2959190
arXiv 2016
-
[5]
Hinton, Alex Krizhevsky, and Sida D
Geoffrey E. Hinton, Alex Krizhevsky, and Sida D. Wang. 2011. Transforming Auto-Encoders. In Artificial Neural Networks and Machine Learning - ICANN 2011 - 21st International Conference on Artificial Neural Networks, Espoo, Finland, June 14-17, 2011, Proceedings, Part I (Lecture Notes in Computer Science, Vol. 6791) , Timo Honkela, Wlodzislaw Duch, Mark A....
-
[6]
Hervé Jégou, Matthijs Douze, and Cordelia Schmid. 2011. Product Quantization for Nearest Neighbor Search. IEEE Trans. Pattern Anal. Mach. Intell. 33, 1 (2011), 117–128. https://doi.org/10.1109/TPAMI.2010.57
-
[7]
George Karypis. 2001. Evaluation of Item-Based Top-N Recommendation Al- gorithms. In Proceedings of the 2001 ACM CIKM International Conference on Information and Knowledge Management, Atlanta, Georgia, USA, November 5-10,
work page 2001
-
[9]
Chenliang Li, Cong Quan, Li Peng, Yunwei Qi, Yuming Deng, and Libing Wu
Show all 25 references
-
[10]
Sen Li, Fuyu Lv, Taiwei Jin, Guli Lin, Keping Yang, Xiaoyi Zeng, Xiao-Ming Wu, and Qianli Ma. 2021. Embedding-based Product Retrieval in Taobao Search. In KDD ’21: The 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, Singapore, August 14-18, 20...
2021 doi
-
[11]
Fuyu Lv, Taiwei Jin, Changlong Yu, Fei Sun, Quan Lin, Keping Yang, and Wilfred Ng. 2019. SDM: Sequential Deep Matching Model for Online Large-scale Rec- ommender System. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, CIKM 2019,...
2019
-
[12]
Malkov and Dmitry A
Yury A. Malkov and Dmitry A. Yashunin. 2020. Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs.IEEE Trans. Pattern Anal. Mach. Intell. 42, 4 (2020), 824–836. https://doi.org/10.1109/ TPAMI.2018.2889473
2020
-
[13]
Qi Pi, Weijie Bian, Guorui Zhou, Xiaoqiang Zhu, and Kun Gai. 2019. Practice on Long Sequential User Behavior Modeling for Click-Through Rate Prediction. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2019, Anchorage, AK...
2019
-
[14]
Qi Pi, Guorui Zhou, Yujing Zhang, Zhe Wang, Lejian Ren, Ying Fan, Xiaoqiang Zhu, and Kun Gai. 2020. Search-based User Interest Modeling with Lifelong Sequential Behavior Data for Click-Through Rate Prediction. In CIKM ’20: The 29th ACM International Conference on Information a...
2020
-
[15]
Sara Sabour, Nicholas Frosst, and Geoffrey E. Hinton. 2017. Dynamic Rout- ing Between Capsules. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, Decem- ber 4-9, 2017, Long Beach, CA, USA , Isabelle Guyon,...
2017
-
[16]
Qiaoyu Tan, Jianwei Zhang, Jiangchao Yao, Ninghao Liu, Jingren Zhou, Hongxia Yang, and Xia Hu. 2021. Sparse-Interest Network for Sequential Recommendation. In WSDM ’21, The Fourteenth ACM International Conference on Web Search and Data Mining, Virtual Event, Israel, March 8-12...
2021
-
[17]
Gomez, Lukasz Kaiser, and Illia Polosukhin
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Advances in Neural Information Processing Systems 30: An- nual Conference on Neural Information Processing Systems ...
2017
-
[18]
Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015. Pointer Networks. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, Corinna Cortes, Neil D. Lawrence...
2015
-
[19]
Yueqi Xie, Jingqi Gao, Peilin Zhou, Qichen Ye, Yining Hua, Jae Boum Kim, Fangzhao Wu, and Sunghun Kim. 2023. Rethinking Multi-Interest Learning for Candidate Matching in Recommender Systems. In Proceedings of the 17th ACM Conference on Recommender Systems, RecSys 2023, Singapo...
2023
-
[20]
Jiaqi Zhai, Lucy Liao, Xing Liu, Yueming Wang, Rui Li, Xuan Cao, Leon Gao, Zhaojie Gong, Fangda Gu, Jiayuan He, Yinghai Lu, and Yu Shi. 2024. Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations. In Forty-first International...
2024
-
[21]
Han Zhang, Songlin Wang, Kang Zhang, Zhiling Tang, Yunjiang Jiang, Yun Xiao, Weipeng Yan, and Wenyun Yang. 2020. Towards Personalized and Se- mantic Retrieval: An End-to-End Solution for E-commerce Search via Em- bedding Learning. In Proceedings of the 43rd International ACM S...
2020
-
[22]
Guorui Zhou, Xiaoqiang Zhu, Chengru Song, Ying Fan, Han Zhu, Xiao Ma, Yanghui Yan, Junqi Jin, Han Li, and Kun Gai. 2018. Deep Interest Network for Click-Through Rate Prediction. In Proceedings of the 24th ACM SIGKDD Inter- national Conference on Knowledge Discovery & Data Mini...
2018
-
[2001]
https://doi.org/10.1145/502585.502627
ACM, 247–254. https://doi.org/10.1145/502585.502627
-
[2019]
A Capsule Network for Recommendation and Explaining What You Like and Dislike. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2019, Paris, France, July 21-25, 2019, Benjamin Piwowarski, Max Chevalier, É...
2019
-
[2020]
In KDD ’20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, CA, USA, August 23-27, 2020 , Rajesh Gupta, Yan Liu, Jiliang Tang, and B
Controllable Multi-Interest Framework for Recommendation. In KDD ’20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, CA, USA, August 23-27, 2020 , Rajesh Gupta, Yan Liu, Jiliang Tang, and B. Aditya Prakash (Eds.). ACM, 2942–2951. https://...
2020 doi
-
[2021]
CoRR abs/2108.04468 (2021)
End-to-End User Behavior Retrieval in Click-Through RatePrediction Model. CoRR abs/2108.04468 (2021). arXiv:2108.04468 https://arxiv.org/abs/2108.04468
2021 arXiv
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.