REVIEW 4 major objections 4 minor 35 references
USD: A User-Intent-Driven Sampling and Dual-Debiasing Framework for Large-Scale Homepage Recommendations
T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read A user-intent-driven sampling and dual-debiasing framework raises Taobao homepage click-through rates by 35.4% and 14.5% in online tests.
desk verdict Plausible industrial recipe undercut by an unstated train/serve gap and a mislabeled IPS step; worth refereeing with a request for clarification. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the sampling rule of Eq. (1), $U' = \{u \in U \mid y^p_u = 1\}$, which replaces random or click-based negative sampling with 'keep only users who visited the marketing portal today.' That rule is supported by the User Intent Extraction Module (UIEM), a Transformer decoder with causal masking that reads a month of user behavior and outputs two calibrated probabilities, $\hat{y}^p_u$ (portal intent) and $\hat{y}^b_u$ (block-click intent), via two MLP heads. The dual-debiasing loss of Eq. (7) uses these probabilities as inverse-propensity weights, $1/(1-\hat{y}^p_u)$ for portal visitors and $1/\hat{y}^b_u$ for block clickers, so the CTR model is trained on a confidence-weighted, intent-corrected version of the exposure log; the final loss adds the UIEM's two binary cross-entropy terms with small weights $\alpha, \beta = 0.0001$.
What would settle it
Run the same week-long online A/B test but compute the sampling condition at request time from the UIEM's predicted $\hat{y}^p_u$ instead of the same-day label $y^p_u$; if the UCTR deltas of 35.4% and 14.5% collapse or disappear, the headline gain depends on a label that is unavailable when the model must serve.
Extended reading notes
Core claim
The central claim is that pseudo-negatives and pseudo-positives in homepage recommendation are not separate problems: both come from the same gap between what a user is shown and what the user intended to do. The paper's USD framework closes that gap with two coupled modules. A user intent-aware sampler restricts training to users who visited the marketing portal on the given day, treating that visit as a signal that their exposures carry real intent; then the User Intent Extraction Module (UIEM) predicts two probabilities for each user—intent to visit the portal and intent to click the marketing block—from a month of behavior sequences. The causal-based dual-debiasing module uses these two probabilities as inverse-propensity weights in the CTR loss: portal visitors get weight $1/(1-\hat{y}^p_u)$ so that non-clicks from strongly portal-intent users count more, and block-click users get weight $1/\hat{y}^b_u$ so clicks that are really portal-entrance behavior are downweighted. The paper reports that this joint design, trained on 1.45 billion samples and evaluated on 97 million, improves GAUC over its base model and beats several debiasing baselines, and that the fully deployed system raised UCTR by 35.4% on Baiyibutie and 14.5% on Taobaomiaosha.
Load-bearing premise
The load-bearing premise is that a user's same-day portal visit (the sampling label $y^p_u=1$) is known at serving time, or that a predicted proxy from the UIEM can replace it without loss; if that label is only available after the fact, the training sampler leaks information and the reported online gains are not explained by the method as written.
Editorial extensions
If this is right
- If USD is correct, homepage recommenders can drop random negative sampling in favor of a behavior-defined confidence set, because the users who visit the marketing portal are the ones whose exposures carry interpretable intent.
- The two learned intent probabilities make a single CTR model apply two opposite propensity corrections—downweighting portal-entrance clicks and upweighting non-clicks from portal-intent users—so one model handles both pseudo-positives and pseudo-negatives.
- The reported ablations, with USD gaining 2.55%, 0.65%, and 0.82% GAUCavg over removing sampling, portal-debiasing, and block-debiasing respectively, indicate each component contributes a separate part of the offline lift.
- The full deployment on Taobao's Baiyibutie and Taobaomiaosha blocks shows that the framework's added complexity—two auxiliary heads and a reweighted loss—can be served at the scale of tens of millions of daily users.
Reading between the lines
- One extension the paper leaves implicit is that the same-day portal-visit signal could serve as a reusable, self-supervised intent label for other entry-point surfaces, such as push notifications or banner widgets, wherever a destination visit defines the user's goal.
- If the train/serve mismatch over $y^p_u$ is real, the open question is whether the UIEM's predicted $\hat{y}^p_u$ alone can reproduce the reported gains; the paper does not report that ablation, and it is the cheapest experiment that would settle the mechanism.
- The framework's success suggests a two-stage view of biased implicit feedback—remove invalid exposures from the training set first, then reweight the survivors—which could be tested against end-to-end propensity models on the same data.
- A targeted offline check would stratify the evaluation by whether the user's portal visit came through the marketing block or another channel, since the dual-debiasing loss predicts these two groups receive opposite weight treatments.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes USD, a framework for large-scale homepage recommendation that combines (i) a user-intent negative-sampling module that filters the training population to users who visited the marketing portal on the current day (Eq. 1), and (ii) a dual-debiasing module in which a User Intent Extraction Module (UIEM) predicts portal-visit and block-click probabilities and uses these to reweight the CTR loss (Eq. 7). The authors report offline GAUC improvements over several baselines on a 1.45-billion-sample Taobao dataset and online A/B gains in UCTR of 35.4% and 14.5% on two homepage marketing blocks. The central claim is that this joint sampling-and-debiasing design is responsible for the production gains.
Significance. If the results are valid, the paper would make a practically important industrial contribution: it addresses a real failure mode of homepage recommendation (clicks that are merely portal-seeking rather than item-interest signals) and proposes a joint treatment of invalid exposure and selection bias. The paper's strengths are its production scale, the use of behavioral sequences to model two kinds of intent, and the inclusion of both offline and online industrial evaluations. However, the significance is currently not assessable because the method is underspecified at two load-bearing points: the same-day portal-visit label in Eq. (1) is not available at serving time, and the weights in Eq. (7) are not derived as inverse propensities of the actual sampling rule. These issues affect every reported result, including the headline online gains.
major comments (4)
- [Sec. 2.2, Eq. (1)] The sampling rule U' = {u | y_u^p = 1} uses the current-day portal-visit label y_u^p, which is not known at the moment a homepage impression is served. The paper never states whether the deployed system applies this filter only during training, uses a predicted proxy for y_u^p, or gates serving by the same-day label. All offline baselines in Table 1 are trained on this filtered set, so the comparison measures performance on the subpopulation with y_u^p = 1, not on the all-user population implied by Sec. 3.3. The 35.4% and 14.5% online UCTR claims therefore cannot be evaluated or reproduced from the paper as written, and if the filter is training-only the model is subject to an undeclared covariate shift from P(x | y_p = 1) to P(x).
- [Sec. 2.4, Eq. (7)] The module is described as inverse propensity scoring (IPS), but the weights in Eq. (7) are not the inverse propensities of the sampling rule in Eq. (1). If \hat{y}_u^p estimates P(y_u^p = 1 | x), the IPS correction for selecting U' would be 1/\hat{y}_u^p, whereas Eq. (7) uses 1/(1 - \hat{y}_u^p). Moreover, the statement '1/(1 - \hat{y}_u^p) ∝ \hat{y}_u^p' is mathematically false; the function is monotone increasing but not proportional. No derivation is given to show that these particular weights remove exposure bias or click bias, so the 'causal-based dual-debiasing' interpretation is unsupported.
- [Sec. 2.3 and Eq. (7)] The UIEM is trained to predict the same current-day labels y_u^p and y_u^b that define the sampling set U' and its partition into U'_portal and U'_block. Using fitted predictions of the selection variable to reweight samples selected by that same variable is circular: the weights inherit any prediction error, and for users in U'_portal (where y_u^p = 1 by construction) the denominator 1 - \hat{y}_u^p becomes unstable when the predictor is confident. The paper provides no calibration analysis or sensitivity study for these propensity-style weights beyond the arbitrary clip to [1, 15], so the claimed debiasing effect is not established.
- [Sec. 3.3] The online A/B test reporting is too incomplete to support the headline claim. No confidence intervals, significance tests, traffic allocation ratios, or guardrail metrics are reported, and the paper does not explain how the Eq. (1) sampling filter is implemented online. Given the train/serve mismatch identified in the first comment, the reported UCTR gains cannot be attributed to the described method without this information.
minor comments (4)
- [Sec. 2.4] The phrase '1/(1 - \hat{y}_u^p) ∝ \hat{y}_u^p' should be replaced by a statement that the weight increases monotonically with \hat{y}_u^p; the current wording is mathematically incorrect.
- [References] References [14] and [15] are duplicate entries for the same Liang et al. (2016) paper; one should be removed.
- [Sec. 2.4 and Fig. 1] The notation U'_portal and U'_block in the text is not aligned with the labels \mathcal{U}_{portal} and \mathcal{U}_{block} in Figure 1(b); please unify the notation.
- [Sec. 3.1.3] The hyperparameters α = β = 0.0001 and the weight clip range [1, 15] are reported without sensitivity analysis, although the clip range directly controls the magnitude of the debiasing weights in Eq. (7) and could dominate the observed effect.
Circularity Check
No significant circularity: online A/B test and offline GAUC provide external checks; the debiasing weights are intermediate learned predictions, not the prediction target.
full rationale
The derivation chain is not circular. Eq. (1) defines the sampled training set via the same-day portal-visit label y_u^p; Eqs. (2)-(6) train the UIEM to estimate y_hat_u^p and y_hat_u^b from historical behavior sequences; Eq. (7) uses those estimates as per-user loss weights for the CTR model; Eq. (8) sums the losses. The target quantity y_hat_{u,i} (click probability for a user-item pair) is not defined in terms of y_hat_u^p or y_hat_u^b, and the reported effectiveness is not a re-statement of Eq. (1) or Eq. (7). Instead, the paper's central evidence is an external comparison: 'Online A/B tests show 35.4% and 14.5% UCTR gains for Taobao's Baiyibutie and Taobaomiaosha homepage blocks' (Sec. 3.3), plus offline GAUC comparisons on production data (Table 1). Those evaluations compare USD against BASE and other published methods on held-out/online traffic, so the result is not forced by construction. No load-bearing self-citation is present: the cited debiasing and baseline works (ETA, ESMM, ESCM2, DCMT, NISE, IPS) are authored by other groups. The paper's statement that '1/(1 - y_hat_u^p) is proportional to y_hat_u^p' is an interpretation of the chosen weighting, not a derivation of the outcome from that weighting. The possible unavailability of same-day portal labels at serving time is a train/serve consistency concern, not a circularity.
Assumptions & free parameters
free parameters (5)
- α (portal loss weight) =
0.0001
- β (block loss weight) =
0.0001
- Debiasing weight clip range =
[1, 15]
- UIEM training window =
past week
- Behavior sequence length =
past month
assumptions (4)
- domain assumption Users with same-day portal visits have intent aligned with the marketing block (Eq. 1 filter)
- domain assumption Same-day portal engagements correlate more with intent than cross-day ones
- domain assumption UIEM outputs ŷ_u^p and ŷ_u^b are valid propensity weights for debiasing
- standard math Cross-entropy loss and IPS framework from Schnabel et al. [20]
Cite this review
Pith. "Pith review of USD: A User-Intent-Driven Sampling and Dual-Debiasing Framework for Large-Scale Homepage Recommendations." pith.science (2026). https://pith.science/paper/7SLOBNXT
@misc{pith2026250706503,
author = {Pith},
title = {Pith review of: USD: A User-Intent-Driven Sampling and Dual-Debiasing Framework for Large-Scale Homepage Recommendations},
year = {2026},
howpublished = {\url{https://pith.science/paper/7SLOBNXT}},
note = {Machine review of arXiv:2507.06503}
}
read the original abstract
Large-scale homepage recommendations face critical challenges from pseudo-negative samples caused by exposure bias, where non-clicks may indicate inattention rather than disinterest. Existing work lacks thorough analysis of invalid exposures and typically addresses isolated aspects (e.g., sampling strategies), overlooking the critical impact of pseudo-positive samples - such as homepage clicks merely to visit marketing portals. We propose a unified framework for large-scale homepage recommendation sampling and debiasing. Our framework consists of two key components: (1) a user intent-aware negative sampling module to filter invalid exposure samples, and (2) an intent-driven dual-debiasing module that jointly corrects exposure bias and click bias. Extensive online experiments on Taobao demonstrate the efficacy of our framework, achieving significant improvements in user click-through rates (UCTR) by 35.4% and 14.5% in two variants of the marketing block on the Taobao homepage, Baiyibutie and Taobaomiaosha.
Figures
Reference graph
Works this paper leans on
-
[1]
Jiawei Chen, Hande Dong, Xiang Wang, Fuli Feng, Meng Wang, and Xiangnan He
-
[2]
Jiawei Chen, Can Wang, Sheng Zhou, Qihao Shi, Jingbang Chen, Yan Feng, and Chun Chen. 2020. Fast adaptively weighted matrix factorization for recommen- dation with implicit feedback. In Proceedings of the AAAI Conference on artificial intelligence, Vol. 34. 3470–3477
work page 2020
-
[3]
Jiawei Chen, Can Wang, Sheng Zhou, Qihao Shi, Yan Feng, and Chun Chen. 2019. SamWalker: Social Recommendation with Informative Sampling Strategy. In The World Wide Web Conference. 228–239
work page 2019
-
[4]
Qiwei Chen, Changhua Pei, Shanshan Lv, Chao Li, Junfeng Ge, and Wenwu Ou. 2021. End-to-End User Behavior Retrieval in Click-Through RatePrediction Model. ArXiv abs/2108.04468 (2021)
arXiv 2021
-
[5]
Shuai Chen and Zhoujun Li. 2024. Intent-Enhanced Data Augmentation for Sequential Recommendation. arXiv:2410.08583 https://arxiv.org/abs/2410.08583
work page Pith review arXiv 2024
-
[6]
Xiao Chen, Wenqi Fan, Jingfan Chen, Haochen Liu, Zitao Liu, Zhaoxiang Zhang, and Qing Li. 2023. Fairly Adaptive Negative Sampling for Recommendations. In Proceedings of the ACM Web Conference 2023, WWW 2023 . 3723–3733
work page 2023
-
[7]
Jingtao Ding, Yuhan Quan, Quanming Yao, Yong Li, and Depeng Jin. 2020. Sim- plify and Robustify Negative Sampling for Implicit Collaborative Filtering. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual
work page 2020
-
[8]
Tom Fawcett. 2006. An introduction to ROC analysis. Pattern Recognition Letters 27, 8 (2006), 861–874
work page 2006
Show all 35 references
-
[9]
Balázs Hidasi and Alexandros Karatzoglou. 2018. Recurrent Neural Networks with Top-k Gains for Session-based Recommendations. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management, CIKM 2018, Torino, Italy, October 22-26, 2018 . 843–852
2018
-
[10]
Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk
-
[11]
Jiahui Huang, Lan Zhang, Junhao Wang, Shanyang Jiang, Dongbo Huang, Cheng Ding, and Lan Xu. 2024. Utilizing Non-click Samples via Semi-supervised Learn- ing for Conversion Rate Prediction. In Proceedings of the 18th ACM Conference on Recommender Systems. 350–359
2024
-
[12]
Riwei Lai, Li Chen, Yuhan Zhao, Rui Chen, and Qilong Han. 2023. Disentangled Negative Sampling for Collaborative Filtering. In Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining . 96–104
2023
-
[13]
Defu Lian, Yongji Wu, Yong Ge, Xing Xie, and Enhong Chen. 2020. Geography- Aware Sequential Location Recommendation. In KDD ’20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, CA, USA, August 23-27, 2020. 2009–2019
2020
-
[14]
Dawen Liang, Laurent Charlin, James McInerney, and David M. Blei. 2016. Mod- eling User Exposure in Recommendation. In Proceedings of the 25th International Conference on World Wide Web. 951–961
2016
-
[15]
Dawen Liang, Laurent Charlin, James McInerney, and David M. Blei. 2016. Mod- eling User Exposure in Recommendation. In Proceedings of the 25th International Conference on World Wide Web, WWW 2016, Montreal, Canada, April 11 - 15, 2016 . 951–961
2016
-
[16]
Xiaoyang Liu, Chong Liu, Pinzheng Wang, Rongqin Zheng, Lixin Zhang, Leyu Lin, Zhijun Chen, and Liangliang Fu. 2023. UFNRec: Utilizing False Negative Samples for Sequential Recommendation. In Proceedings of the 2023 SIAM International Conference on Data Mining, SDM 2023, Minnea...
2023
-
[17]
Haokai Ma, Ruobing Xie, Lei Meng, Xin Chen, Xu Zhang, Leyu Lin, and Jie Zhou
-
[18]
Haokai Ma, Ruobing Xie, Lei Meng, Fuli Feng, Xiaoyu Du, Xingwu Sun, Zhanhui Kang, and Xiangxu Meng. 2024. Negative Sampling in Recommendation: A Survey and Future Directions. arXiv preprint arXiv:2409.07237 (2024)
2024 arXiv
-
[19]
Xiao Ma, Liqin Zhao, Guan Huang, Zhi Wang, Zelin Hu, Xiaoqiang Zhu, and Kun Gai. 2018. Entire Space Multi-Task Model: An Effective Approach for Estimating Post-Click Conversion Rate. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrie...
2018
-
[20]
In Proceedings of the 17th ACM Conference on Recommender Systems
Exploring False Hard Negative Sample in Cross-Domain Recommendation. In Proceedings of the 17th ACM Conference on Recommender Systems . 502–514
-
[21]
Qijie Shen, Wanjie Tao, Jing Zhang, Hong Wen, Zulong Chen, and Quan Lu
-
[22]
Gomez, Łukasz Kaiser, and Illia Polosukhin
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems. 6000–6010
2017
-
[23]
Tobias Schnabel, Adith Swaminathan, Ashudeep Singh, Navin Chandak, and Thorsten Joachims. 2016. Recommendations as treatments: debiasing learning and evaluation. In Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48 . 1670–1679
2016
-
[24]
Hao Wang, Tai-Wei Chang, Tianqiao Liu, Jianmin Huang, Zhichao Chen, Chao Yu, Ruopeng Li, and Wei Chu. 2022. ESCM2: Entire Space Counterfactual Multi- Task Model for Post-Click Conversion Rate Estimation. In Proceedings of the 45th International ACM SIGIR Conference on Research...
2022
-
[25]
Jinpeng Wang, Jieming Zhu, and Xiuqiang He. 2021. Cross-Batch Negative Sampling for Training Two-Tower Recommenders. Proceedings of the 44th In- ternational ACM SIGIR Conference on Research and Development in Information Retrieval (2021), 1632–1636
2021
-
[26]
Li, Simon Xiaoming Wang, Taibai Xu, and Ed H
Ji Yang, Xinyang Yi, Derek Zhiyuan Cheng, Lichan Hong, Y. Li, Simon Xiaoming Wang, Taibai Xu, and Ed H. Chi. 2020. Mixed Negative Sampling for Learning Two-tower Neural Networks in Recommendations. Companion Proceedings of the Web Conference 2020 (2020), 441–447. https://api.s...
2020
-
[27]
Can Wang, Jiawei Chen, Sheng Zhou, Qihao Shi, Yan Feng, and Chun Chen
-
[28]
Yu Zheng, Chen Gao, Xiang Li, Xiangnan He, Yong Li, and Depeng Jin. 2021. Disentangling User Interest and Conformity for Recommendation with Causal Embedding. In WWW ’21: The Web Conference 2021, Virtual Event / Ljubljana, Slovenia, April 19-23, 2021. 2980–2991
2021
-
[29]
Guorui Zhou, Xiaoqiang Zhu, Chenru Song, Ying Fan, Han Zhu, Xiao Ma, Yanghui Yan, Junqi Jin, Han Li, and Kun Gai. 2018. Deep Interest Network for Click- Through Rate Prediction. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining...
2018
-
[30]
Feng Zhu, Mingjie Zhong, Xinxing Yang, Longfei Li, Lu Yu, Tiehua Zhang, Jun Zhou, Chaochao Chen, Fei Wu, Guanfeng Liu, and Yan Wang. 2023. DCMT: A Direct Entire-Space Causal Multi-Task Framework for Post-Click Conversion Estimation. In 39th IEEE International Conference on Dat...
2023
-
[32]
Ji Yang, Xinyang Yi, Derek Zhiyuan Cheng, Lichan Hong, Yang Li, Simon Xiaom- ing Wang, Taibai Xu, and Ed H. Chi. 2020. Mixed Negative Sampling for Learning Two-tower Neural Networks in Recommendations. In Companion Proceedings of the Web Conference 2020. 441–447
2020
-
[2016]
In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings
Session-based Recommendations with Recurrent Neural Networks. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings
2016
-
[2020]
IEEE Transactions on Knowledge and Data Engineering 35 (2020), 2004–2018
SamWalker++: Recommendation With Informative Sampling Strategy. IEEE Transactions on Knowledge and Data Engineering 35 (2020), 2004–2018. https://api.semanticscholar.org/CorpusID:226965628
2020
-
[2021]
In Proceedings of the 30th ACM International Conference on Information & Knowledge Management
SAR-Net: A Scenario-Aware Ranking Network for Personalized Fair Rec- ommendation in Hundreds of Travel Scenarios. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management . 4094–4103
-
[2023]
ACM Trans
Bias and Debias in Recommender System: A Survey and Future Directions. ACM Trans. Inf. Syst. 41, 3 (2023), 67:1–67:39
2023
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.