Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

FuXi-\beta: Towards a Lightweight and Fast Large-Scale Generative Recommendation Model

T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A generative recommender can drop its query-key attention map and replace bucketed temporal bias with a learnable power function, matching or beating prior models while training faster.

desk verdict Useful architecture work with real public speedups, but the industrial NDCG gains are not yet attributable to the design; needs revision before the headline claims are trusted. read the letter →

arxiv 2508.10615 v1 pith:XUUATKIZ submitted 2025-08-14 cs.IR

Yufei Ye , Wei Guo , Hao Wang , Hong Zhu , Yuyang Ye , Yong Liu , Huifeng Guo , Ruiming Tang
show 2 more authors
Defu Lian Enhong Chen
This is my paper · ORCID
classification cs.IR
keywords generativerecommendationsequentialefficienttransformerrelativeattentionbiasquery-keyattention-freetokenmixerscalinglawlarge-scale
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that Transformer-like generative recommenders can be made both faster and more accurate by subtracting components, not adding them. It claims that the query-key attention map in self-attention is redundant for recommendation and often harmful, and that the relative attention bias—positional plus temporal—can carry the entire inter-item interaction. To replace the inefficient bucketed temporal bias used in earlier models, it introduces Functional Relative Attention Bias, a learnable power function $f(x)=a(1+x)^{-b}$, and to remove the query-key map it designs an Attention-Free Token Mixer. Applied to the previous FuXi-$\alpha$ model, the new FuXi-$\beta$ matches or slightly beats it on public benchmarks while training faster, and reports 27% to 47% higher NDCG@10 on two industrial-scale datasets, with the paper attributing that gain to reduced overfitting.

What carries the argument

The central mechanism is the Attention-Free Token Mixer (AFTM): an inter-item interaction layer that deletes the query and key matrices and instead multiplies SiLU-activated value projections by two attention maps—the positional Relative Attention Bias (RAB) and the new Functional Relative Attention Bias (FRAB). FRAB is the second load-bearing piece: a learnable power function $f(x)=a(1+x)^{-b}$ that replaces the bucketed temporal indexing of HSTU and FuXi-$\alpha$, eliminating non-contiguous memory operations while providing a monotonically decaying recency weight.

What would settle it

On the same industrial datasets, retrain FuXi-β with the query-key attention map added back (keeping FRAB), and retrain FuXi-α with per-model tuned hyperparameters and matched compute. If FuXi-β-with-QK performs as well as FuXi-β, or if a tuned FuXi-α matches FuXi-β, then the removal is not the cause. Additionally, plot train/validation NDCG curves for both models on industrial data: the overfitting story predicts FuXi-α's validation curve pulls away from training while FuXi-β's stays close.

Watch

Extended reading notes

Core claim

On the paper's own terms: the query-key attention map is not only unnecessary but counterproductive in autoregressive recommendation, because the relative attention bias terms already act as attention maps. Removing the query and key projections and mixing tokens through two bias maps—positional RAB and temporal FRAB—keeps or improves accuracy while cutting roughly half of the self-attention computation. FRAB replaces the bucketed, index-heavy temporal bias of HSTU and FuXi-$\alpha$ with a scalar function $f(x)=a(1+x)^{-b}$ with learnable $a$ and $b$, avoiding non-contiguous memory access. The resulting model FuXi-$\beta$ is at parity with FuXi-$\alpha$ on MovieLens-1M and MovieLens-20M (e.g

Load-bearing premise

The load-bearing premise is that the 27-47% industrial NDCG gains are caused by removing the query-key attention map itself, not by the comparison setup—the paper applies identical hyperparameters to all baselines and offers no industrial ablation or training curves to demonstrate the overfitting mechanism; if FuXi-α is simply undertuned for industrial data, the headline superiority claim collapses.

Editorial extensions

If this is right

  • If the central claim holds, the default design of Transformer-like generative recommenders can be simplified: query-key attention can be removed entirely, and temporal bias can be a cheap scalar function, cutting self-attention cost roughly in half.
  • The speedups scale with model size: on MovieLens-20M the 8-layer FuXi-$\beta$ trains 27% faster than FuXi-$\alpha$, making larger (and better) models more affordable under fixed latency budgets.
  • The framework transfers: applying the same surgery to LLaMa-style and HSTU-style blocks reduces their training time on public datasets, though the paper reports HSTU-$\beta$ loses accuracy, indicating the query-key map's role varies across architectures.
  • The scaling-law result on industrial data implies that removing attention components does not cap model growth; FuXi-$\beta$ continues to improve in NDCG@10 as layers increase to 32.
  • On industrial-scale data, dropping the query-key map can yield large accuracy gains (27-47% NDCG@10), not just efficiency gains, if the paper's overfitting explanation is correct.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if query-key attention is systematically redundant in recommendation, the same removal could be applied to other sequential models (e.g., two-tower or LLM-based recommenders) to obtain similar speedups; the paper's own compatibility results make this a testable extension rather than a demonstrated result.
  • Editorial inference: the paper's overfitting explanation for the industrial gains is a post hoc hypothesis—no industrial ablations or training curves are shown. A direct test would be comparing train/validation gaps of FuXi-$\alpha$ and FuXi-$\beta$ on those datasets; a smaller gap for FuXi-$\beta$ would support the claim, equal gaps would refute it.
  • Editorial inference: the function $f(x)=a(1+x)^{-b}$ is a power-law decay kernel, and the paper's own function sweep shows that only monotone-decaying functions (pow, exp) work well; this suggests the essential inductive bias is recency decay, linking this design to classical time-decay models of user interest.
  • Editorial inference: because public-dataset gains are near zero, the claimed superiority of removing the query-key map may be data-dependent; the public/industrial gap itself (data scale, sequence length, noise) is a natural target for follow-up study.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a lightweight framework for Transformer-like generative recommender models, with two main modifications: (1) Functional Relative Attention Bias (FRAB), replacing bucketed relative temporal attention bias with a learnable power function f(x)=a(1+x)^{-b}, and (2) an Attention-Free Token Mixer (AFTM) that removes the query-key attention map and relies on relative-position and relative-time bias maps as attention. The framework is instantiated as FuXi-β by modifying FuXi-α. Experiments on MovieLens-1M, MovieLens-20M, and two private industrial datasets report public-dataset parity with FuXi-α, training-time reductions, scaling-law behavior, and 27–47% NDCG@10 improvements over FuXi-α on the industrial datasets. The authors attribute the industrial gains primarily to reduced overfitting from removing the query-key attention map.

Significance. If the central claims hold, the paper provides a useful simplification result: query-key attention may be unnecessary in generative recommendation, and a parametric temporal bias can replace bucketed lookup with lower cost. The strengths include public code, a concrete complexity analysis (Table 1), compatibility experiments on LLaMa and HSTU, and scaling experiments on industrial data. The main significance, however, rests on the industrial NDCG gains and the causal attribution to architecture; as reported, that attribution is not directly supported by the experiments. The public-dataset ablations are mixed, the industrial datasets are private, and the baseline tuning protocol is not described. The central architectural claim is plausible but needs stronger evidence before the 27–47% improvement can be accepted as architecture-driven.

major comments (4)
  1. [Section 5.3, Eq. (14)] Equation (14) is dimensionally inconsistent as written. W_u and W_v are defined in R^{d x 2d}, so U = φ(XW_u) and V = φ(XW_v) are n x 2d. Then concat(BV, B_t V) is n x 4d, and the elementwise product U ⊙ concat(...) is invalid. The AFTM formulation must be corrected, e.g., by projecting to separate d-dimensional gates/values or by splitting U into two components. As written, the central module cannot be implemented reproducibly.
  2. [Section 5.3 / Eq. (14)] The causal mask is not specified for the bias-only attention maps. In standard self-attention the mask is applied before normalization; here B and B_t serve directly as attention maps, but the paper does not state that their upper-triangular entries are masked or otherwise constrained. Without a causal mask, future items leak into earlier positions, which would break the autoregressive training objective defined in Section 5.5. If masking is applied in code, it must be stated explicitly in Eq. (14) or the surrounding text.
  3. [Section 6.2.2, Table 4] The 27–47% NDCG@10 improvements on industrial datasets are attributed to 'reduced overfitting' from removing the query-key map, but no industrial ablation or training/validation curve is provided. The public ablations do not consistently support the mechanism: Table 5 shows that removing the query-key map alone degrades NDCG@10 on ML-1M (0.1797 vs. 0.1835), and the full FuXi-β is only at parity on both public datasets. The metric pattern in Table 4 is also anomalous: on Daily Recommendations, HR@50 changes by only 0.7% (0.6449 to 0.6497) while NDCG@10 changes by 47%, suggesting a top-rank redistribution that is not explained. Without industrial ablations or curves, the mechanism remains a post-hoc explanation.
  4. [Section 6.1.4 / Section 6.2.2] The paper states that identical hyperparameters (embedding size 256, 4 layers) were used for all models on industrial data, but this does not establish a fair comparison. If FuXi-α, HSTU, or other baselines have different optimal learning rates, dropout, or schedules on these datasets, the gap could be a tuning artifact. The tuning protocol, search space, and final hyperparameters for baselines must be reported before the headline industrial superiority claim can be accepted.
minor comments (5)
  1. [Section 6.2.1 / Table 3] The text says public performance results are presented in Table 2, but Table 2 is dataset statistics; the performance numbers are in Table 3. Similarly, Section 6.3.1 refers to 'Table 3 presents total training time', but Table 3 is the performance table. Please renumber tables or fix cross-references.
  2. [Eq. (12)] f(x) = a(1+x)^{-b} is monotonically decreasing only when b > 0. The text calls it 'a monotonically function' without specifying the parameter sign. Clarify the parameterization or the intended monotonicity constraint.
  3. [Section 6.4.2 / Table 6] In Table 6, the 'bucket' row is compared against 'pow' on ML-1M/ML-20M. It would help to specify whether 'bucket' uses the same number of buckets and the same bucketing function as HSTU/FuXi-α; otherwise the comparison is ambiguous.
  4. [Figure 3] Panel (c) shows attention weight magnitudes up to 400 on ML-20M for 'linear' and 'mixed', while other variants are near zero. Please explain the scale or whether this is due to the learned parameters in those functions, since it affects the qualitative conclusions about suitable functions.
  5. [Section 6.4.3 / Table 7] The terms 'query-key attention map', 'positional attention map', and 'temporal attention map' should be defined in the caption or text. In particular, the 'positional' and 'temporal' maps correspond to B and B_t in Eq. (14), and their precise construction under FRAB should be stated.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: design choices are empirically ablated on public benchmarks and industrial gains are reported as comparisons, not derived from fitted parameters.

full rationale

The paper's chain is empirical rather than definitional. The two architectural interventions (FRAB and removal of the query-key attention map) are motivated by profiling operator runtimes and by external results (Synthesizer) showing alternative attention maps can work. The power-function form f(x)=a(1+x)^{-b} is not derived from the target results; it is one entry in a function-family comparison (Table 6) and is selected on MovieLens validation metrics before being applied to industrial data. This is ordinary model selection, not a predicted quantity equal to a fitted parameter. The removal of the query-key map is supported by ablations on public datasets (Tables 5 and 7), and the industrial gains are presented as experimental measurements over baselines, including FuXi-α, which is re-run under the stated protocol rather than accepted on faith from the citation. No equation in the paper reconstructs its output from its own inputs, and no uniqueness/authority argument is imported from the authors' prior work. Self-citations to FuXi-α and to scaling-law papers are normal background and are not load-bearing in the sense of substituting for evidence: the baseline numbers come from the present experiments. The post-hoc overfitting explanation for the large industrial gains is a validity/threats concern (the mechanism is not directly evidenced by industrial ablations or training curves), but that is not circularity. Hence steps are empty and score is 0.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The paper's central claims are empirical. The main learned quantities the model depends on are the FRAB coefficients (a,b) and the inherited bucketed positional bias parameters; both are fitted during training and not reported. The functional form choice (power) was made after comparing alternatives on public validation data, so performance claims on industrial data rely on transfer of that model selection. No new entities are postulated; AFTM and FRAB are notational modules, not extra physical or mathematical objects.

free parameters (2)
  • FRAB learnable scale a and exponent b (per layer) = not reported (learned)
    Define the temporal attention map in Eq. (12); performance depends on their learned values, and the power family was selected by comparing alternatives on public validation sets (Table 6).
  • T5/HSTU bucketed positional bias parameters β = not reported (learned)
    The positional attention map B in AFTM is inherited from the bucketed RAB and remains a learnable attention map; it is load-bearing for position information after the query-key map is removed.
assumptions (4)
  • domain assumption The query-key derived attention map is unnecessary for item feature interaction in sequential recommendation; relative-bias maps alone suffice.
    Stated in Sec 5.3 and tested in Sec 6.4.3; results are mixed across datasets, so it functions as an unproven premise for the architecture.
  • ad hoc to paper The functional form f(x)=a(1+x)^{-b} is expressive enough to encode temporal decay for recommendation.
    Chosen in Sec 5.2 and justified post hoc in Sec 6.4.2 by decay-rate arguments; no a priori derivation from data properties.
  • domain assumption Interaction timestamps are non-negative and sequences are chronological, so x=t_j-t_i>=0 and x+1 is always positive.
    Sec 5.2 explicitly assumes x is non-negative; equal-timestamp and unordered cases are not handled.
  • domain assumption Causal masking is preserved when the query-key map is removed.
    Original attention had an explicit mask function (Eq. 4); Eq. (14) shows no mask, so this behavior is assumed without being specified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FuXi-\beta: Towards a Lightweight and Fast Large-Scale Generative Recommendation Model." pith.science (2026). https://pith.science/paper/XUUATKIZ

@misc{pith2026250810615,
  author       = {Pith},
  title        = {Pith review of: FuXi-\beta: Towards a Lightweight and Fast Large-Scale Generative Recommendation Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XUUATKIZ}},
  note         = {Machine review of arXiv:2508.10615}
}
abstract

Scaling laws for autoregressive generative recommenders reveal potential for larger, more versatile systems but mean greater latency and training costs. To accelerate training and inference, we investigated the recent generative recommendation models HSTU and FuXi-$\alpha$, identifying two efficiency bottlenecks: the indexing operations in relative temporal attention bias and the computation of the query-key attention map. Additionally, we observed that relative attention bias in self-attention mechanisms can also serve as attention maps. Previous works like Synthesizer have shown that alternative forms of attention maps can achieve similar performance, naturally raising the question of whether some attention maps are redundant. Through empirical experiments, we discovered that using the query-key attention map might degrade the model's performance in recommendation tasks. To address these bottlenecks, we propose a new framework applicable to Transformer-like recommendation models. On one hand, we introduce Functional Relative Attention Bias, which avoids the time-consuming operations of the original relative attention bias, thereby accelerating the process. On the other hand, we remove the query-key attention map from the original self-attention layer and design a new Attention-Free Token Mixer module. Furthermore, by applying this framework to FuXi-$\alpha$, we introduce a new model, FuXi-$\beta$. Experiments across multiple datasets demonstrate that FuXi-$\beta$ outperforms previous state-of-the-art models and achieves significant acceleration compared to FuXi-$\alpha$, while also adhering to the scaling law. Notably, FuXi-$\beta$ shows an improvement of 27% to 47% in the NDCG@10 metric on large-scale industrial datasets compared to FuXi-$\alpha$. Our code is available in a public repository: https://github.com/USTC-StarTeam/FuXi-beta

Figures

Figures reproduced from arXiv: 2508.10615 by the authors.

Figure 1
Figure 1. The overview of the proposed model FuXi- [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Scaling of FuXi-𝛽 on Industrial Dataset. excellent performance on large-scale datasets while still adher￾ing to the scaling law, which states that the model’s performance continues to improve with the increase in data, model size, and computational resource. 6.3 Efficiency Comparison 6.3.1 Public Datasets [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Variation of attention weights produced by the FRAB module with respect to relative time using different functions. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Noise is not always detrimental: the capacity of quantum batteries is enhanced in black holes

    quant-ph 2026-04 unverdicted novelty 5.0 of 10

    Hawking radiation is claimed to enhance quantum battery capacity for bipartite mixed states, while environmental noise generally degrades it in type-dependent ways.

  2. Why Thinking Hurts: Diagnosing and Rectifying Linguistic Inertia in Large Language Models for Recommendation

    cs.IR 2026-02 conditional novelty 5.0 of 10

    Chain-of-thought reasoning degrades semantic-ID recommendation accuracy through 'linguistic inertia,' and a training-free compression-plus-contrastive decoding fix restores and often improves accuracy.

Reference graph

Works this paper leans on

85 extracted references · 28 canonical work pages · cited by 2 Pith papers

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

    arXiv 2023

  2. [2]

    Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. 2023. Gqa: Training generalized multi-query trans- former models from multi-head checkpoints. arXiv preprint arXiv:2305.13245 (2023)

    arXiv 2023

  3. [3]

    Newsha Ardalani, Carole-Jean Wu, Zeliang Chen, Bhargav Bhushanam, and Adnan Aziz. 2022. Understanding scaling laws for recommendation models. arXiv preprint arXiv:2208.08489 (2022)

    arXiv 2022

  4. [4]

    Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long- document transformer. arXiv preprint arXiv:2004.05150 (2020)

    arXiv 2020

  5. [5]

    Sung Min Cho, Eunhyeok Park, and Sungjoo Yoo. 2020. MEANTIME: Mixture of attention mechanisms with multi-temporal embeddings for sequential recom- mendation. In Proceedings of the 14th ACM Conference on recommender systems . 515–520

    2020

  6. [6]

    Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. 2020. Rethinking attention with performers. arXiv preprint arXiv:2009.14794 (2020)

    arXiv 2020

  7. [7]

    Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y Wu, et al. 2024. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models. arXiv preprint arXiv:2401.06066 (2024)

    arXiv 2024

  8. [8]

    Zihang Dai. 2019. Transformer-xl: Attentive language models beyond a fixed- length context. arXiv preprint arXiv:1901.02860 (2019)

    arXiv 2019

Show all 85 references
  1. [9]

    Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems 35 (2022), 30318–30332

    2022

  2. [10]

    Alexey Dosovitskiy. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)

    2020 arXiv

  3. [11]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024)

    2024 arXiv

  4. [12]

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2022. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323 (2022)

    2022 arXiv

  5. [13]

    Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin

  6. [14]

    Wei Guo, Hao Wang, Luankang Zhang, Jin Yao Chin, Zhongzhou Liu, Kai Cheng, Qiushi Pan, Yi Quan Lee, Wanqi Xue, Tingjia Shen, et al . 2024. Scal- ing New Frontiers: Insights into Large Recommendation Models. arXiv preprint arXiv:2412.00714 (2024)

    2024 arXiv

  7. [15]

    Xingzhuo Guo, Junwei Pan, Ximei Wang, Baixu Chen, Jie Jiang, and Mingsheng Long. 2023. On the Embedding Collapse when Scaling up Recommendation Models. arXiv preprint arXiv:2310.04400 (2023)

    2023 arXiv

  8. [16]

    Song Han, Jeff Pool, John Tran, and William Dally. 2015. Learning both weights and connections for efficient neural network. Advances in neural information processing systems 28 (2015)

    2015

  9. [17]

    Yongqiang Han, Hao Wang, Kefan Wang, Likang Wu, Zhi Li, Wei Guo, Yong Liu, Defu Lian, and Enhong Chen. 2024. Efficient noise-decoupling for multi-behavior sequential recommendation. In Proceedings of the ACM Web Conference 2024 . 3297–3306

    2024

  10. [18]

    Yongqiang Han, Likang Wu, Hao Wang, Guifeng Wang, Mengdi Zhang, Zhi Li, Defu Lian, and Enhong Chen. 2023. Guesr: A global unsupervised data- enhancement with bucket-cluster sampling for sequential recommendation. In International conference on database systems for advanced app...

    2023

  11. [19]

    Bobby He and Thomas Hofmann. 2023. Simplifying transformer blocks. arXiv preprint arXiv:2311.01906 (2023)

    2023 arXiv

  12. [20]

    Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020. De- berta: Decoding-enhanced bert with disentangled attention. arXiv preprint arXiv:2006.03654 (2020)

    2020 arXiv

  13. [21]

    Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Ja- cob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al

  14. [22]

    B Hidasi. 2015. Session-based Recommendations with Recurrent Neural Networks. arXiv preprint arXiv:1511.06939 (2015)

    2015 arXiv

  15. [23]

    Geoffrey Hinton. 2015. Distilling the Knowledge in a Neural Network. arXiv preprint arXiv:1503.02531 (2015)

    2015 arXiv

  16. [24]

    Weizhe Hua, Zihang Dai, Hanxiao Liu, and Quoc Le. 2022. Transformer quality in linear time. In International conference on machine learning. PMLR, 9099–9117

    2022

  17. [25]

    Zhiheng Huang, Davis Liang, Peng Xu, and Bing Xiang. 2020. Improve transformer models with better relative position embeddings. arXiv preprint arXiv:2009.13658 (2020)

    2020 arXiv

  18. [26]

    Wang-Cheng Kang and Julian McAuley. 2018. Self-attentive sequential recom- mendation. In 2018 IEEE international conference on data mining (ICDM) . IEEE, 197–206

    2018

  19. [27]

    Guolin Ke, Di He, and Tie-Yan Liu. 2020. Rethinking positional encoding in language pre-training. arXiv preprint arXiv:2006.15595 (2020)

    2020 arXiv

  20. [28]

    Anton Klenitskiy and Alexey Vasilev. 2023. Turning Dross Into Gold Loss: is BERT4Rec really better than SASRec?. In Proceedings of the 17th ACM Conference on Recommender Systems (RecSys ’23) . ACM, 1120–1125. doi:10.1145/3604915. 3610644

    2023 doi

  21. [29]

    John P Kotter. 2012. Accelerate. Harvard business review 90, 11 (2012), 45–58

    2012

  22. [30]

    Zhenzhong Lan. 2019. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942 (2019)

    2019 arXiv

  23. [31]

    Jing Li, Pengjie Ren, Zhumin Chen, Zhaochun Ren, Tao Lian, and Jun Ma. 2017. Neural attentive session-based recommendation. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management . 1419–1428

    2017

  24. [32]

    Jiacheng Li, Yujie Wang, and Julian McAuley. 2020. Time interval aware self- attention for sequential recommendation. In Proceedings of the 13th international conference on web search and data mining . 322–330. Conference’17, July 2017, Washington, DC, USA Yufei Ye, Wei Guo, H...

    2020

  25. [33]

    Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al . 2024. Deepseek- v2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434 (2024)

    2024 arXiv

  26. [34]

    Weiwen Liu, Wei Guo, Yong Liu, Ruiming Tang, and Hao Wang. 2023. User Behavior Modeling with Deep Learning for Recommendation: Recent Advances. In Proceedings of the 17th ACM Conference on Recommender Systems . 1286–1287

    2023

  27. [35]

    Chengqiang Lu, Jianwei Zhang, Yunfei Chu, Zhengyu Chen, Jingren Zhou, Fei Wu, Haiqing Chen, and Hongxia Yang. 2022. Knowledge distillation of transformer- based language models revisited. arXiv preprint arXiv:2206.14366 (2022)

    2022 arXiv

  28. [36]

    Yingtao Luo, Qiang Liu, and Zhaocheng Liu. 2021. Stan: Spatio-temporal attention network for next location recommendation. In Proceedings of the web conference

    2021

  29. [37]

    Xinyin Ma, Gongfan Fang, and Xinchao Wang. 2023. Llm-pruner: On the struc- tural pruning of large language models.Advances in neural information processing systems 36 (2023), 21702–21720

    2023

  30. [38]

    Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. 2017. Mixed precision training. arXiv preprint arXiv:1710.03740 (2017)

    2017 arXiv

  31. [39]

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing system...

    2019

  32. [40]

    Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, et al. 2023. Rwkv: Reinventing rnns for the transformer era. arXiv preprint arXiv:2305.13048 (2023)

    2023 arXiv

  33. [41]

    Bo Peng, Daniel Goldstein, Quentin Anthony, Alon Albalak, Eric Alcaide, Stella Biderman, Eugene Cheah, Xingjian Du, Teddy Ferdinan, Haowen Hou, et al. 2024. Eagle and finch: Rwkv with matrix-valued states and dynamic recurrence. arXiv preprint arXiv:2404.05892 (2024)

    2024 arXiv

  34. [42]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research 21, 140 (2020), 1–67

    2020

  35. [43]

    Alessandro Raganato, Yves Scherrer, and Jörg Tiedemann. 2020. Fixed encoder self-attention patterns in transformer-based machine translation. arXiv preprint arXiv:2002.10260 (2020)

    2020 arXiv

  36. [44]

    Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme

  37. [45]

    Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. Self-attention with relative position representations. arXiv preprint arXiv:1803.02155 (2018)

    2018 arXiv

  38. [46]

    Noam Shazeer. 2019. Fast transformer decoding: One write-head is all you need. arXiv preprint arXiv:1911.02150 (2019)

    2019 arXiv

  39. [47]

    Noam Shazeer. 2020. Glu variants improve transformer. arXiv preprint arXiv:2002.05202 (2020)

    2020 arXiv

  40. [48]

    Tingjia Shen, Hao Wang, Chuhan Wu, Jin Yao Chin, Wei Guo, Yong Liu, Huifeng Guo, Defu Lian, Ruiming Tang, and Enhong Chen. 2025. Optimizing Sequen- tial Recommendation Models with Scaling Laws and Approximate Entropy. arXiv:2412.00430 [cs.AI] https://arxiv.org/abs/2412.00430

    2025 arXiv

  41. [49]

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing 568 (2024), 127063

    2024

  42. [50]

    Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. 2023. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306.11695 (2023)

    2023 arXiv

  43. [51]

    Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jiany- ong Wang, and Furu Wei. 2023. Retentive network: A successor to transformer for large language models. arXiv preprint arXiv:2307.08621 (2023)

    2023 arXiv

  44. [52]

    Yi Tay, Dara Bahri, Donald Metzler, Da-Cheng Juan, Zhe Zhao, and Che Zheng

  45. [53]

    Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. 2024. Visual autoregressive modeling: Scalable image generation via next-scale prediction. arXiv preprint arXiv:2404.02905 (2024)

    2024 arXiv

  46. [54]

    Zhen Tian, Ting Bai, Zibin Zhang, Zhiyuan Xu, Kangyi Lin, Ji-Rong Wen, and Wayne Xin Zhao. 2023. Directed acyclic graph factorization machines for CTR prediction via knowledge distillation. In Proceedings of the Sixteenth ACM Inter- national Conference on Web Search and Data M...

    2023

  47. [55]

    Junxiong Tong, Mingjia Yin, Hao Wang, Qiushi Pan, Defu Lian, and Enhong Chen

  48. [56]

    A Vaswani. 2017. Attention is all you need. Advances in Neural Information Processing Systems (2017)

    2017

  49. [57]

    Hao Wang, Wei Guo, Luankang Zhang, Jin Yao Chin, Yufei Ye, Huifeng Guo, Yong Liu, Defu Lian, Ruiming Tang, and Enhong Chen. 2025. Generative Large Recommendation Models: Emerging Trends in LLMs for Recommendation. arXiv preprint arXiv:2502.13783 (2025)

    2025 arXiv

  50. [58]

    Hao Wang, Yongqiang Han, Kefan Wang, Kai Cheng, Zhen Wang, Wei Guo, Yong Liu, Defu Lian, and Enhong Chen. 2024. Denoising Pre-Training and Customized Prompt Learning for Efficient Multi-Behavior Sequential Recommendation.arXiv preprint arXiv:2408.11372 (2024)

    2024 arXiv

  51. [59]

    Hao Wang, Defu Lian, Hanghang Tong, Qi Liu, Zhenya Huang, and Enhong Chen. 2021. Decoupled representation learning for attributed networks. IEEE Transactions on Knowledge and Data Engineering 35, 3 (2021), 2430–2444

    2021

  52. [60]

    Hao Wang, Tong Xu, Qi Liu, Defu Lian, Enhong Chen, Dongfang Du, Han Wu, and Wen Su. 2019. MCNE: An end-to-end framework for learning multiple conditional network representations of social network. In Proceedings of the 25th ACM SIGKDD international conference on knowledge disc...

    2019

  53. [61]

    Hao Wang, Mingjia Yin, Luankang Zhang, Sirui Zhao, and Enhong Chen. 2025. MF-GSLAE: A Multi-Factor User Representation Pre-training Framework for Dual-Target Cross-Domain Recommendation. ACM Transactions on Information Systems 43, 2 (2025), 1–28

    2025

  54. [62]

    Kefan Wang, Hao Wang, Kenan Song, Wei Guo, Kai Cheng, Zhi Li, Yong Liu, Defu Lian, and Enhong Chen. 2025. A Universal Framework for Compressing Embeddings in CTR Prediction. arXiv preprint arXiv:2502.15355 (2025)

    2025 arXiv

  55. [63]

    Likang Wu, Zhi Zheng, Zhaopeng Qiu, Hao Wang, Hongchao Gu, Tingjia Shen, Chuan Qin, Chen Zhu, Hengshu Zhu, Qi Liu, et al . 2024. A survey on large language models for recommendation. World Wide Web 27, 5 (2024), 60

    2024

  56. [64]

    Wenjia Xie, Hao Wang, Luankang Zhang, Rui Zhou, Defu Lian, and Enhong Chen

  57. [65]

    Wenjia Xie, Rui Zhou, Hao Wang, Tingjia Shen, and Enhong Chen. 2024. Bridging User Dynamics: Transforming Sequential Recommendations with Schrödinger Bridge and Diffusion Models. In Proceedings of the 33rd ACM International Con- ference on Information and Knowledge Management ...

    2024

  58. [66]

    Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. 2020. On layer normalization in the transformer architecture. In International Conference on Machine Learning. PMLR, 10524–10533

    2020

  59. [67]

    Xiang Xu, Hao Wang, Wei Guo, Luankang Zhang, Wanshan Yang, Runlong Yu, Yong Liu, Defu Lian, and Enhong Chen. 2024. Multi-granularity Interest Retrieval and Refinement Network for Long-Term User Behavior Modeling in CTR Prediction. arXiv preprint arXiv:2411.15005 (2024)

    2024 arXiv

  60. [68]

    Yufei Ye, Wei Guo, Jin Yao Chin, Hao Wang, Hong Zhu, Xi Lin, Yuyang Ye, Yong Liu, Ruiming Tang, Defu Lian, and Enhong Chen. 2025. FuXi- 𝛼: Scal- ing Recommendation Model with Feature Interaction Enhanced Transformer. arXiv:2502.03036 [cs.IR] https://arxiv.org/abs/2502.03036

    2025 arXiv

  61. [69]

    Mingjia Yin, Hao Wang, Wei Guo, Yong Liu, Zhi Li, Sirui Zhao, Zhen Wang, Defu Lian, and Enhong Chen. 2024. Learning partially aligned item representation for cross-domain sequential recommendation. arXiv preprint arXiv:2405.12473 (2024)

    2024 arXiv

  62. [70]

    Advances in Neural Information Processing Systems 37 (2024), 22720–22744

    Breaking determinism: Fuzzy modeling of sequential recommendation using discrete state space diffusion model. Advances in Neural Information Processing Systems 37 (2024), 22720–22744

    2024

  63. [71]

    Mingjia Yin, Hao Wang, Xiang Xu, Likang Wu, Sirui Zhao, Wei Guo, Yong Liu, Ruiming Tang, Defu Lian, and Enhong Chen. 2023. Apgl4sr: A generic framework with adaptive and personalized global collaborative information in sequential recommendation. In Proceedings of the 32nd ACM ...

    2023

  64. [72]

    Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. 2020. Big bird: Transformers for longer sequences. Advances in neural information processing systems 33 (2020), 17283–17297

    2020

  65. [73]

    Jiaqi Zhai, Lucy Liao, Xing Liu, Yueming Wang, Rui Li, Xuan Cao, Leon Gao, Zhao- jie Gong, Fangda Gu, Michael He, et al. 2024. Actions speak louder than words: Trillion-parameter sequential transducers for generative recommendations.arXiv preprint arXiv:2402.17152 (2024)

    2024 arXiv

  66. [74]

    Biao Zhang and Rico Sennrich. 2019. Root Mean Square Layer Normaliza- tion. In Advances in Neural Information Processing Systems 32: Annual Con- ference on Neural Information Processing Systems 2019, NeurIPS 2019, Decem- ber 8-14, 2019, Vancouver, BC, Canada , Hanna M. Wallach...

    2019

  67. [75]

    Gaowei Zhang, Yupeng Hou, Hongyu Lu, Yu Chen, Wayne Xin Zhao, and Ji-Rong Wen. 2023. Scaling Law of Large Sequential Recommendation Models. arXiv preprint arXiv:2311.11351 (2023). FuXi-𝛽: Towards a Lightweight and Fast Large-Scale Generative Recommendation Model Conference’17,...

    2023 arXiv

  68. [76]

    Mingjia Yin, Hao Wang, Wei Guo, Yong Liu, Suojuan Zhang, Sirui Zhao, Defu Lian, and Enhong Chen. 2024. Dataset regeneration for sequential recommendation. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 3954–3965

    2024

  69. [77]

    Luankang Zhang, Kenan Song, Yi Quan Lee, Wei Guo, Hao Wang, Yawen Li, Huifeng Guo, Yong Liu, Defu Lian, and Enhong Chen. 2025. Killing Two Birds with One Stone: Unifying Retrieval and Ranking with a Single Generative Rec- ommendation Model. arXiv:2504.16454 [cs.IR] https://arx...

    2025 arXiv

  70. [78]

    Luankang Zhang, Hao Wang, Suojuan Zhang, Mingjia Yin, Yongqiang Han, Jiaqing Zhang, Defu Lian, and Enhong Chen. 2024. A Unified Framework for Adaptive Representation Enhancement and Inversed Learning in Cross-Domain Recommendation. In International Conference on Database Syste...

    2024

  71. [79]

    Guorui Zhou, Na Mou, Ying Fan, Qi Pi, Weijie Bian, Chang Zhou, Xiaoqiang Zhu, and Kun Gai. 2019. Deep interest evolution network for click-through rate prediction. In Proceedings of the AAAI conference on artificial intelligence , Vol. 33. 5941–5948

    2019

  72. [82]

    Jiaqing Zhang, Mingjia Yin, Hao Wang, Yawen Li, Yuyang Ye, Xingyu Lou, Junping Du, and Enhong Chen. 2025. TD3: Tucker Decomposition Based Dataset Distilla- tion Method for Sequential Recommendation. arXiv preprint arXiv:2502.02854 (2025)

    2025 arXiv

  73. [2012]

    arXiv preprint arXiv:1205.2618 (2012)

    BPR: Bayesian personalized ranking from implicit feedback. arXiv preprint arXiv:1205.2618 (2012)

    2012 arXiv

  74. [2017]

    In International conference on machine learning

    Convolutional sequence to sequence learning. In International conference on machine learning. PMLR, 1243–1252

  75. [2020]

    arXiv preprint arXiv:2010.14701 (2020)

    Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010.14701 (2020)

    2010 arXiv

  76. [2021]

    In Interna- tional conference on machine learning

    Synthesizer: Rethinking self-attention for transformer models. In Interna- tional conference on machine learning . PMLR, 10183–10192

  77. [2024]

    In International Conference on Web Information Systems Engineering

    MDAP: A Multi-view Disentangled and Adaptive Preference Learning Framework for Cross-Domain Recommendation. In International Conference on Web Information Systems Engineering. Springer, 164–178

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.