Pith. sign in

REVIEW 5 major objections 6 minor 50 references

Bootstrapping your behavior: a new pretraining strategy for user behavior sequence data

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A new pretraining strategy for user behavior sequences predicts a single embedding that summarizes all behaviors in a future window, removing the need for a manually selected behavior vocabulary and delivering better downstream accuracy…

desk verdict A plausible BYOL-style UBS pretraining method, but the headline AUC and throughput gains are confounded by the day-level pooling that only BYB uses. read the letter →

arxiv 2506.11053 v1 pith:VTQBVLAS submitted 2025-05-22 cs.LG cs.AI

Weichang Wu , Xiaolu Zhang , Jun Zhou , Yuchen Li , Wenwen Xia This is my paper
classification cs.LGcs.AI
keywords userbehaviorsequencepretrainingself-supervisedlearningembeddingpredictionstudent-teacherencoderpoolingfinancialrisk
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

User behavior sequence (UBS) pretraining has relied on predicting which behaviors from a manually selected vocabulary will occur next; choosing that vocabulary is labor-intensive, biased, and caps generalization. BYB removes that step: the model is pretrained to predict a single supervision embedding that averages the embeddings of all behaviors in a future time window. A student-teacher encoder with a stop-gradient and exponential moving average supplies stable supervision without any labels. On two industrial datasets and eight downstream tasks the pretrained representation improves average AUC by 3.9% over the best frozen pretrained baselines and raises pretraining throughput by about 98.9%, and a two-month online deployment improved Kolmogorov–Smirnov scores by 2.7% and 7.1% for two overdue-risk tasks. The paper argues this shows vocabulary-free embedding prediction is a more general and efficient pretraining objective for UBS data.

What carries the argument

The central mechanism is a student-teacher embedding-prediction scheme: the teacher encoder builds the supervision as the mean-pooled embedding of all behaviors in a future window (Eq. 7) with stop-gradient; the student encodes the observed window, pools behaviors by day to shorten the sequence, and a predictor maps the sequence representation to that supervision embedding; the teacher is updated from the student by an exponential moving average. This single-embedding target is what removes the need for a behavior vocabulary, and the day-level behavior pooling is what gives the large training-throughput gain.

What would settle it

Construct a downstream task whose label is determined by a rare behavior (e.g., a single high-risk financial action) and pretrain BYB with that behavior present in the prediction windows; if removing that behavior from the future window does not change the finetuned performance, or if BYB does not beat a vocabulary-based baseline that includes it, then the claim that the mean-pooled supervision preserves long-tail behavior information would be falsified.

Watch

Extended reading notes

Core claim

The paper claims that predicting a single automatically constructed embedding of all future behaviors is a better pretraining objective for user behavior sequences than predicting a distribution over a fixed, manually selected vocabulary. The supervision embedding is produced by a teacher behavior encoder that mean-pools the embeddings of every behavior in the prediction window, with gradients stopped; the student side predicts this embedding from an observation window, and the teacher is updated by EMA so the supervision stays stable. Because the objective does not depend on any vocabulary, it covers frequent and long-tail behaviors alike and avoids the bias and capacity limits of selected behavior sets. The authors report that BYB achieves an average 3.9% AUC improvement over the strongest frozen pretrained baselines across eight tasks, roughly doubles pretraining throughput (98.9% average gain), scales with model size, and shows meaningful attention and clustering without labels; in online financial-risk use it lifted KS by about 2.7% and 7.1% over the MSDP baseline.

Load-bearing premise

The load-bearing premise is that a single mean-pooled embedding of all future behaviors is a rich enough teaching signal—in particular, that rare but important behaviors are not washed out by frequent ones by the averaging—so that the learned representation transfers to downstream tasks.

Editorial extensions

If this is right

  • Pretrained user behavior models can be built without any hand-curated behavior vocabulary, removing a costly, bias-prone manual step.
  • Pretraining cost no longer scales with vocabulary size, so modeling long user sequences becomes substantially cheaper; the paper reports about a 98.9% throughput gain over the second-best method.
  • A single pretrained encoder can transfer to many downstream tasks, including classification and financial risk prediction, with only a small finetuning head.
  • The learned representations show structure (attention decay, periodicity, clusters) that matches known properties of user behavior, suggesting the objective captures meaningful signal without labels.
  • The method scales with model size, with deeper architectures helping more than wider ones on the tested tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The choice of mean pooling for the supervision embedding is not analyzed for long-tail retention; a weighted or attention-based pool might preserve rare-behavior signal better, and that variant is a natural test of the paper's 'captures long-tail behaviors smoothly' claim.
  • The same 'predict a summary embedding of the future' objective could transfer to other sequential domains with large or open-vocabulary event spaces, such as clickstreams, IoT logs, or health-event records, where vocabulary selection is also impractical.
  • Because the paper's throughput advantage largely comes from day-level pooling, a fair comparison with a pooled MSDP baseline would isolate how much of the AUC gain comes from the embedding-prediction objective itself rather than from sequence shortening.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes Bootstrapping Your Behavior (BYB), a self-supervised pretraining strategy for user behavior sequences that predicts a single supervision embedding summarizing all behaviors in a future time window, thereby eliminating manual behavior-vocabulary selection. A student-teacher encoder scheme with an EMA-updated teacher and a stop-gradient operation constructs the supervision; the pretraining objective is cross-entropy or MSE between the predictor output and the supervision embedding. Experiments on the Tmall and Alipay Mobile App datasets across eight downstream tasks report an average 3.9% AUC improvement over prior UBS pretraining baselines, a 98.9% pretraining throughput gain, and an online deployment over two months improving KS by 2.7% and 7.1% on two financial overdue-risk tasks.

Significance. The method is practically motivated: it targets a real bottleneck in industrial UBS pretraining, the manual selection of a behavior vocabulary, and it pairs this with a computationally efficient pooling design. The paper supplies a concrete algorithm, ablations of the EMA/loss/predictor components, a scaling study, and an online deployment evaluation, which are notable strengths. However, the headline claims are not established by the current experiments: the comparison against baselines is confounded by BYB's day-level pooling, and the dataset splits appear to temporally overlap, undermining the reported downstream gains. The long-tail motivation, central to the paper's narrative, is not backed by analysis. If the confounds are resolved, the work could be a useful contribution to industrial UBS pretraining, but substantial experimental revisions are required.

major comments (5)
  1. [Sec. 5.1.1, Tab. 8] The dataset splits are overlapping: the Tmall pretraining window (2013.04–2013.07) contains the finetuning window (2013.05–2013.06) and overlaps the test window (2013.07–2013.08), so the pretrained model has seen the behavior sequences used for downstream finetuning and evaluation. This leakage invalidates the reported downstream improvements for the Tmall tasks and needs to be resolved by re-splitting the data into strictly disjoint time periods.
  2. [Sec. 4.2, Eq. (4); Sec. 5.2, Tab. 2; Sec. 5.3, Tab. 3] The comparison against MBM, NBP, CTS, and MSDP does not control for the day-level behavior pooling applied only to BYB. Behavior pooling shrinks the sequence from roughly 2000 to 180 on the Mobile dataset, which changes both the input distribution and the self-attention cost; the 98.9% throughput gain and the AUC gains in Tab. 2 may therefore be attributed to pooling rather than to the vocabulary-free embedding prediction. The authors should either equip all baselines with the same pooling or provide a BYB-no-pooling ablation.
  3. [Sec. 5.6, Tabs. 6 and 7] The ablation tables report a 'Base' model whose numbers do not match the corresponding BYB(freeze) results in Tab. 2 (e.g., Active-30d: 0.9096 vs. 0.9134; Tmall 5-day: 0.5073 vs. 0.5779). This inconsistency suggests unreported differences in setup or data, and it prevents the reader from verifying the ablation conclusions. Please reconcile all reported numbers across tables.
  4. [Sec. 4.3, Eq. (12); Sec. 4.2, Eq. (7)] The supervision embedding is generated by a teacher encoder that is an EMA copy of the student (Eq. 12), so the predictor learns to match the student's own evolving representation. This is a self-bootstrapping objective without an independent target, and the paper offers no analysis of possible representation collapse or a comparison against a fixed target. Given that the supervision is further mean-pooled over all behaviors in the future window (Eq. 7), the reader needs evidence that the learned representation retains task-relevant information rather than converging to a trivial average.
  5. [Sec. 1; Sec. 4.2, Eq. (7)] The central motivation is that the single supervision embedding 'captures both frequent and long-tail behaviors smoothly,' but mean pooling assigns equal weight 1/N to each behavior in the window, so rare behaviors are necessarily diluted. The paper provides no analysis of long-tail retention (e.g., how rare behaviors affect the supervision embedding or downstream performance), which undermines the claim of eliminating the vocabulary limitation. Please provide quantitative evidence.
minor comments (6)
  1. [Sec. 5.4, Appendix C] The paper contains unresolved references 'Tab. ??' in Sec. 5.4 and Appendix C; these need to be filled in.
  2. [Sec. 5.2] The abstract and Sec. 5.2 report '3.9% improvement' without clarifying whether this is an absolute percentage-point difference or a relative improvement; please define the metric.
  3. [Sec. 5.3, Fig. 3] The epoch-time and memory figures have inconsistent formatting, and the units on the y-axis of (a) are unclear; please label axes.
  4. [Sec. 5.7, Fig. 5] The periodic attention pattern claim would be stronger with error bars or a statistical test, since the weights are only averaged over many samples.
  5. [Sec. 5.8] The online deployment results report KS improvements without sample sizes, confidence intervals, or a description of how the comparison to MSDP was implemented; please add details.
  6. [Throughout] There are several typos and grammatical errors, e.g., 'pretraing' in Sec. 2.2 and 'merge lager' in Sec. 4.2, which should be corrected.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: self-bootstrapped EMA target is a standard SSL design, and the main claims rest on external downstream labels and throughput measurements, with only a minor non-load-bearing self-citation.

full rationale

The paper's pretraining target is self-referential in the sense that the teacher encoder that produces the supervision embedding (Eq. 7) is updated by EMA from the student encoder (Eq. 12). However, this is an explicit and standard self-distillation design, not a hidden equivalence between an input and a claimed prediction. The target is computed from a future time window, which is genuinely different from the observation window, so the prediction task has temporal content; the loss value is not used as evidence of success. The paper's central claims are downstream AUC gains, training throughput, and online KS improvements, all of which are evaluated against external task labels and measured runtime, not derived from the pretext loss. The MSDP baseline and online comparison method [15] is the authors' prior work, but this self-citation is not load-bearing: even if MSDP were not SOTA, the comparison would still be a reasonable baseline and does not by construction force the reported improvements. The paper also honestly attributes the throughput advantage to behavior pooling rather than to the supervision objective. Potential experimental confounds, such as not applying the same day-level pooling to all pretraining baselines, are real threats to the validity of the comparison, but they are correctness/control issues rather than circularity. No step in the derivation reduces to its own inputs by definition, and no fitted parameter is renamed as a prediction.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The method introduces no new physical entities. The free parameters are the pretraining hyperparameters chosen by hand. The main axioms are domain assumptions about UBS structure and the effectiveness of self-bootstrapped pooled embeddings. The temporal split assumption is effectively violated, which weakens the empirical claims.

free parameters (6)
  • behavior pooling window size Delta_T1 = 1 day
    Hand-chosen default based on the claim that user behaviors are stable at one-day granularity (Sec. 4.2). No sweep is reported for this value.
  • prediction window size Delta_T2 = 1 day
    Default setting; ablation only on Tmall with 1, 3, 5, and 10 days (Sec. 5.5). Performance varies with this value.
  • EMA momentum m = 0.995
    Momentum for teacher update (Eq. 12). Chosen by hand, not ablated in the paper.
  • softmax temperature tau = 0.1
    Temperature for the cross-entropy loss (Eq. 10). Chosen by hand, no sensitivity analysis.
  • pretraining loss function = cross-entropy (CE) by default
    CE vs MSE changes results across datasets (Tabs. 6-7); the choice is a free design parameter affecting the central results.
  • pooling operator = mean
    Mean pooling chosen after 'preliminary experiments' (Sec. 4.2); max, sum, and RNN alternatives are left unexplored.
assumptions (5)
  • domain assumption User behavior sequences exhibit local randomness and global orderliness, so pooling behaviors within a day preserves task-relevant information.
    Invoked in Sec. 1 and Sec. 4.2 to justify mean behavior pooling and the day-level granularity. If intraday order or exact timing matters for a downstream task, this assumption fails.
  • domain assumption One-day granularity is a stable and informative time scale for industrial user behaviors.
    Stated in the clarification in Sec. 4.2 and used to set Delta_T1 and stride. No experiment compares alternative pooling granularities.
  • ad hoc to paper The EMA teacher with stop-gradient and a predictor prevents representation collapse and yields stable supervision.
    Borrowed from BYOL (Grill et al. [16]) and applied in Sec. 4.3. The paper provides no collapse analysis or variance diagnostics for the teacher targets.
  • ad hoc to paper A single mean-pooled embedding of all future behaviors is a sufficient supervision signal for diverse downstream tasks.
    Eq. 7 defines the supervision as BehPool over the entire prediction window. The paper assumes this aggregate retains enough information; long-tail behaviors are diluted by averaging.
  • domain assumption The evaluation protocol is leakage-free: pretraining data do not overlap with data used in finetune/test sequences.
    Table 8 shows Tmall pretraining 2013.04-2013.07 and test split 2013.07-2013.08, and Mobile pretraining 2023.03-2023.07 and finetune 2023.07-2023.08. This assumption is violated by construction, inflating the reported gains.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Bootstrapping your behavior: a new pretraining strategy for user behavior sequence data." pith.science (2026). https://pith.science/paper/VTQBVLAS

@misc{pith2026250611053,
  author       = {Pith},
  title        = {Pith review of: Bootstrapping your behavior: a new pretraining strategy for user behavior sequence data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VTQBVLAS}},
  note         = {Machine review of arXiv:2506.11053}
}
read the original abstract

User Behavior Sequence (UBS) modeling is crucial in industrial applications. As data scale and task diversity grow, UBS pretraining methods have become increasingly pivotal. State-of-the-art UBS pretraining methods rely on predicting behavior distributions. The key step in these methods is constructing a selected behavior vocabulary. However, this manual step is labor-intensive and prone to bias. The limitation of vocabulary capacity also directly affects models' generalization ability. In this paper, we introduce Bootstrapping Your Behavior (\model{}), a novel UBS pretraining strategy that predicts an automatically constructed supervision embedding summarizing all behaviors' information within a future time window, eliminating the manual behavior vocabulary selection. In implementation, we incorporate a student-teacher encoder scheme to construct the pretraining supervision effectively. Experiments on two real-world industrial datasets and eight downstream tasks demonstrate that \model{} achieves an average improvement of 3.9\% in AUC and 98.9\% in training throughput. Notably, the model exhibits meaningful attention patterns and cluster representations during pretraining without any label supervision. In our online deployment over two months, the pretrained model improves the KS by about 2.7\% and 7.1\% over the baseline model for two financial overdue risk prediction tasks in the Alipay mobile application, which reduces bad debt risk by millions of dollars for Ant group.

Figures

Figures reproduced from arXiv: 2506.11053 by the authors.

Figure 1
Figure 1. Conceptual comparison between our pretraining strategy and previous pretraining strategies for UBS data. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Detailed implementation of BYB. Note that we stop the gradients of the supervision embedding be￾cause we expect the teacher behavior encoder to only provide the supervision, instead of optimizing its parameters. ■Finally ( [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 4
Figure 4. Prediction window size effect on the Tmall dataset. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figures from the paper (2 more)
Figure 6
Figure 6. Figure 6: Representation visualization on the Mobile app [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 5
Figure 5. Figure 5: Attention patterns of the pretrained BYB. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

50 extracted references · 17 canonical work pages

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

    arXiv 2023

  2. [2]

    Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. 2023. Self-supervised learning from images with a joint-embedding predictive architecture. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 15619–15629

    work page 2023

  3. [3]

    Fabrício Benevenuto, Tiago Rodrigues, Meeyoung Cha, and Virgílio Almeida

    work page

  4. [4]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901

    2020

  5. [5]

    Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. 2018. Deep clustering for unsupervised learning of visual features. In Proceedings of the European conference on computer vision (ECCV) . 132–149

    work page 2018

  6. [6]

    Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. 2020. Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems 33 (2020), 9912–9924

    2020

  7. [7]

    Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. 2021. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision. 9650–9660

    2021

  8. [8]

    Qiwei Chen, Huan Zhao, Wei Li, Pipei Huang, and Wenwu Ou. 2019. Behavior sequence transformer for e-commerce recommendation in alibaba. InProceedings of the 1st international workshop on deep learning practice for high-dimensional sparse data. 1–4

    2019

Show all 50 references
  1. [9]

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In Interna- tional conference on machine learning . PMLR, 1597–1607

    2020

  2. [10]

    Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. 2020. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297 (2020)

    2020 arXiv

  3. [11]

    Xinlei Chen and Kaiming He. 2021. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 15750–15758

    2021

  4. [12]

    Xu Chen, Hongteng Xu, Yongfeng Zhang, Jiaxi Tang, Yixin Cao, Zheng Qin, and Hongyuan Zha. 2018. Sequential recommendation with user memory networks. In Proceedings of the eleventh ACM international conference on web search and data mining. 108–116

    2018

  5. [13]

    Mingyue Cheng, Fajie Yuan, Qi Liu, Xin Xin, and Enhong Chen. 2021. Learning transferable user representations with sequential behaviors via contrastive pre- training. In 2021 IEEE International Conference on Data Mining (ICDM) . IEEE, 51–60

    2021

  6. [14]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding.arXiv preprint arXiv:1810.04805 (2018)

    2018 arXiv

  7. [15]

    Chilin Fu, Weichang Wu, Xiaolu Zhang, Jun Hu, Jing Wang, and Jun Zhou

  8. [16]

    Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. 2020. Bootstrap your own latent-a new approach to self-supervised learning. Advances in ne...

    2020

  9. [17]

    Jie Gu, Feng Wang, Qinghui Sun, Zhiquan Ye, Xiaoxiao Xu, Jingmin Chen, and Jun Zhang. 2021. Exploiting behavioral consistence for universal user representation. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35. 4063–4071

    2021

  10. [18]

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick

  11. [19]

    Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020. Mo- mentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 9729–9738

    2020

  12. [20]

    Shigeki Karita, Nanxin Chen, Tomoki Hayashi, Takaaki Hori, Hirofumi Inaguma, Ziyan Jiang, Masao Someki, Nelson Yalta, Ryuichi Yamamoto, Xiao fei Wang, Shinji Watanabe, Takenori Yoshimura, and Wangyou Zhang. 2019. A Comparative Study on Transformer vs RNN in Speech Applications...

    2019

  13. [21]

    Runshi Liu, Pengda Qin, Yuhong Li, Weigao Wen, Dong Li, Kefeng Deng, and Qiang Wu. 2021. InfoBehavior: Self-supervised Representation Learning for Ultra-long Behavior Sequence via Hierarchical Grouping. arXiv preprint arXiv:2106.06905 (2021)

    2021 arXiv

  14. [22]

    Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Tim- othy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. 2016. Asyn- chronous methods for deep reinforcement learning. In International conference on machine learning. PMLR, 1928–1937

    2016

  15. [23]

    Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. 2015. Human-level control through deep reinforcement learning. nature 518, 7540 (2015), 529–533

    2015

  16. [24]

    Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El- Nouby, et al. 2023. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 (2023)

    2023 arXiv

  17. [25]

    Nikil Pancha, Andrew Zhai, Jure Leskovec, and Charles Rosenberg. 2022. Pinner- former: Sequence modeling for user representation at pinterest. In Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining . 3702–3712

    2022

  18. [26]

    Qi Pi, Weijie Bian, Guorui Zhou, Xiaoqiang Zhu, and Kun Gai. 2019. Practice on long sequential user behavior modeling for click-through rate prediction. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining . 2671–2679

    2019

  19. [27]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learni...

    2021

  20. [28]

    Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al . 2018. Improving language understanding by generative pre-training. (2018)

    2018

  21. [29]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog 1, 8 (2019), 9

    2019

  22. [30]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research 21, 140 (2020), 1–67

    2020

  23. [31]

    Kyuyong Shin, Hanock Kwak, Kyung-Min Kim, Minkyu Kim, Young-Jin Park, Jisu Jeong, and Seungjae Jung. 2021. One4all user representation for recommender systems in e-commerce. arXiv preprint arXiv:2106.00573 (2021)

    2021 arXiv

  24. [32]

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 (2023)

    2023 arXiv

  25. [33]

    Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. 2022. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training. Advances in neural information processing systems 35 (2022), 10078–10093

    2022

  26. [34]

    Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of machine learning research 9, 11 (2008)

    2008

  27. [35]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017)

    2017

  28. [36]

    Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao

  29. [37]

    Chuhan Wu, Fangzhao Wu, Tao Qi, and Yongfeng Huang. 2022. Userbert: Pre- training user model with contrastive self-supervision. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2087–2092

    2022

  30. [38]

    Chuhan Wu, Fangzhao Wu, Tao Qi, Jianxun Lian, Yongfeng Huang, and Xing Xie. 2020. PTUM: Pre-training user model from unlabeled user behaviors via self-supervision. arXiv preprint arXiv:2010.01494 (2020)

    2020 arXiv

  31. [39]

    Xiang Zhang, Hans-Frederick Brown, and Anil Shankar. 2016. Data-driven personas: Constructing archetypal users with clickstreams and user telemetry. In Proceedings of the 2016 CHI conference on human factors in computing systems . 5350–5359

    2016

  32. [40]

    Yuyu Zhang, Liang Pang, Lei Shi, and Bin Wang. 2014. Large scale purchase prediction with historical user actions on B2C online retail platform. arXiv preprint arXiv:1408.6515 (2014). xx, xxx, 2025 Xia et al. Algorithm 1: PyTorch-like pseudocode forBYB pretraining Input: stude...

    2014 arXiv

  33. [41]

    Qihang Zhao. 2022. RESETBERT4Rec: A pre-training model integrating time and user historical behavior for sequential recommendation. InProceedings of the 45th international ACM SIGIR conference on research and development in information retrieval. 1812–1816

    2022

  34. [42]

    Wenliang Zhong, Rong Jin, Cheng Yang, Xiaowei Yan, Qi Zhang, and Qiang Li

  35. [43]

    Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. 2021. ibot: Image bert pre-training with online tokenizer.arXiv preprint arXiv:2111.07832 (2021)

    2021 arXiv

  36. [44]

    Yongchun Zhu, Dongbo Xi, Bowen Song, Fuzhen Zhuang, Shuai Chen, Xi Gu, and Qing He. 2020. Modeling users’ behavior sequences with hierarchical explainable network for cross-domain fraud detection. In Proceedings of The Web Conference

    2020

  37. [2009]

    In Proceedings of the 9th ACM SIGCOMM Conference on Internet Measurement

    Characterizing user behavior in online social networks. In Proceedings of the 9th ACM SIGCOMM Conference on Internet Measurement . 49–62

  38. [2015]

    In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

    Stock constrained recommendation in tmall. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining . 2287–2296

  39. [2020]

    A Datasets and preprocessing details In this section, we report details of the datasets we utilized for pretraining and downstream finetuning for diverse tasks

    928–938. A Datasets and preprocessing details In this section, we report details of the datasets we utilized for pretraining and downstream finetuning for diverse tasks. The com- prehensive statistics of datasets are shown in Tab. 8. The Tmall dataset contains about 3.9 millio...

    2013

  40. [2021]

    arXiv preprint arXiv:2108.10904 (2021)

    Simvlm: Simple visual language model pretraining with weak supervision. arXiv preprint arXiv:2108.10904 (2021)

    2021 arXiv

  41. [2022]

    In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 16000–16009

  42. [2023]

    In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management

    Robust user behavioral sequence representation via multi-scale stochastic distribution prediction. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management . 4567–4573

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.