Pith. sign in

REVIEW 3 major objections 5 minor 47 references

Sona Technical Report

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A single jointly trained transformer, Sona, replaces the entire multi-stage recommendation cascade in Yandex Music and improves engagement in a live A/B test.

desk verdict A credible single-model recommender that replaces a 15-component cascade, but the headline A/B is one short surface with no CIs—promising, not settled. read the letter →

arxiv 2608.11015 v2 pith:IJHMC4R3 submitted 2026-08-11 cs.IR

Sona Team: Alexandr Udeneev , Aleksei Krasilnikov , Alexey Nadtochiy , Andrey Semenov , Andrey Tsyrkunov , Anna Krivonos , Anna Lipkina , Artem Matveev
show 26 more authors
Daniil Burlakov Daniil Leshchev Daria Tikhonovich Denis Burshtein Ekaterina Dmitrieva Eugene Krofto Grigorii Khlystov Ilya Murzin Kirill Golovko Ksenia Sycheva Leonid Dmitriev Mariia Rozaeva Mariia Ulianova Mikhail Sandul Nikolai Savushkin Oleg Sorokin Roman Odobesku Semyon Panenko Sergei Liamaev Sergei Makeev Vadim Shilov Veronika Ivanova Viktor Yanush Vladimir Baikalov Vladislav Dodonov Vladislav Tytskiy
This is my paper · ORCID
classification cs.IR
keywords generativerecommendationsingle-modelrecommendersemanticIDscandidategenerationrankingknowledgedistillationmusicstreamingonlineA/Btesting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Sona claims that a mature industrial recommendation pipeline—more than fifteen candidate generators followed by pre-ranking and ranking models consuming hundreds of features—can be replaced by one transformer trained and served end to end. In a seven-day A/B test on Yandex Music's My Vibe smart-speaker surface, the single model raised Active Users by 4.53%, Total Listening Time by 6.30%, and Likes by 11.42% relative to the production control. The architecture couples candidate generation and ranking by having an autoregressive Semantic-ID decoder and a Ranking Module consume the same encoder states, supervised jointly by next-token prediction and distillation from a larger frozen Teacher Ranker. If the result holds, it matters because it suggests the dominant cost structure of industrial recommendation—maintaining a stack of separately trained models—can be collapsed into one model without sacrificing quality.

What carries the argument

The load-bearing mechanism is the shared encoder memory K: the encoder runs once per request and its hidden states feed both the Semantic-ID decoder and the Ranking Module, so generation and ranking are trained and served by one model. History Compression splits the 8,192-event history into a 2,048-event recent block that receives a deep seven-layer stack and a long-term block that is mixed once across the full history, cutting attention cost to roughly half of a full transformer while retaining most of its offline accuracy. The Semantic tokenizer (three residual 32,000-entry codebooks) keeps the output space small enough for autoregressive generation, and Rollout Distillation—beam-search candidates from the current decoder scored by the frozen teacher—makes the ranking signal follow the candidate distribution the model actually produces. The Teacher Ranker is what distills a year of engagement history into dense per-candidate targets; at serving it is gone, leaving only the encoder, decoder, and Ranking Module.

What would settle it

Run the full Sona configuration on 100% of My Vibe traffic for a multi-month experiment and compare Active Users, listening time, likes, and catalog coverage against the production cascade; the central claim fails if the 7-day uplift reverses or if the gap closes once catalog coverage is matched.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is this: a single generative recommender can outperform a mature cascade on live traffic. Sona's served model is a transformer over the user's chronological engagement events; a beam-search decoder emits each recommendation as a three-code Semantic ID tuple, the tuple expands to all catalog tracks sharing it, and a Ranking Module scores those candidates against the same encoder memory used for generation. Nothing in the served path uses hand-engineered features. Training couples the decoder's next-token objective with a distillation objective in which a frozen Teacher Ranker—itself a transformer trained on a year of logs by next-item prediction followed by ranking fine-tuning—supplies per-head scores for decoder rollouts and logged impressions. In the final configuration the teacher is absent from serving. Experiment 5 reports statistically significant control-relative gains of +4.53% Active Users, +6.30% Total Listening Time, +11.42% Likes, +17.99% Repeat Commands, and +7.37% Deeply Engaged Users, with the Active Users uplift being 2.35 times the increment of the strongest previously deployed model on this surface.

Load-bearing premise

The paper treats a 7-day A/B test on 15% of users of one smart-speaker surface as sufficient evidence that replacing the entire production cascade with Sona improves engagement, despite Section 8 noting that full-traffic deployment, multi-month validation, and catalog-coverage parity are still pending.

Editorial extensions

If this is right

  • If the result holds, other industrial recommenders can treat the multi-stage cascade as optional: a single jointly trained model can handle candidate generation and ranking with no hand-engineered features.
  • The gain is additive to prior deployments, and on Active Users it is 2.35 times the increment of the strongest previously deployed model, suggesting the single-model approach is not merely competitive but better on the primary metric.
  • Since the decoder and Ranking Module share the encoder and both losses update it, improvements from either objective propagate to the other; the paper's joint training is the mechanism that makes unified generation and ranking work.
  • With History Compression, the 8,192-event history is affordable at serving, and offline ablations show longer histories materially improve Teacher Recall, so the final system's gains depend on keeping long user context, not just on distillation.
  • The online-training loop (45-minute median event-to-model latency, 10-minute weight sync) shows the deployed model can continuously adapt, making the single-model stack a live system rather than a batch-trained artifact.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • I would expect the 7-day gains to be partly driven by improved exploitation of familiar tracks; the paper reports no novelty or catalog-diversity metric, and its own limitation notes lower catalog coverage, so a multi-month test could show a different long-term balance.
  • A natural controlled extension would isolate the 8k History Compression contribution online by serving the Experiment 4 model (2k history) and the Experiment 5 model in the same experiment; the two online numbers currently confound history length with the full recipe.
  • The recipe suggests the operational bottleneck shifts from maintaining many generators and rankers to maintaining the tokenizer, teacher, and continuous distillation pipeline; teams adopting it would likely invest in data plumbing rather than feature engineering.
  • If the results replicate on other surfaces, it would imply that music recommendation's passive-listening, repeat-friendly feedback does not require surface-specific feature stacks—the same event fields suffice across contexts.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. SONA is a single-model generative recommender for Yandex Music that aims to replace the production multi-stage cascade (candidate generation, pre-ranking, ranking) with one served transformer. The model couples an autoregressive Semantic-ID decoder for candidate generation and a Ranking Module, both sharing a user-event encoder; a larger Teacher Ranker provides distillation targets during training and is absent from serving. The report describes the tokenizer, architecture, History Compression, training objectives, and the online-training infrastructure, and evaluates the system in offline ablations and five online A/B experiments on the My Vibe smart-speaker surface. The headline result is Experiment 5: over 7 days on 15% of users per split, the full SONA model produced relative uplifts of +4.53% Active Users, +6.30% Total Listening Time, and +11.42% Likes versus the production control, with unmarked deltas reported as significant at p<0.01. Section 8 states that full-traffic deployment and multi-month validation are pending, other surfaces are not yet validated, and SONA's catalog coverage is lower than the production stack.

Significance. If the online effect persists beyond one week and generalizes to other surfaces, this is a notable industrial result: it would show that a single jointly trained model can replace a system of more than 15 candidate generators plus pre-ranking and ranking models while improving engagement. The design is clearly explained and the online A/B test against production traffic provides non-circular external evidence; the report is also refreshingly explicit about its own limitations, including the non-identification of the history-length contribution. The main gap is that the abstract's claim 'can be replaced' is stronger than the evidence, which covers one surface and a 7-day window and is subject to the limitations in Section 8. The report would be strengthened by tighter statistical reporting and by aligning the conclusions with the actual experimental scope.

major comments (3)
  1. [Section 7.5, Table 7.12; Section 8] The central claim of the paper rests on a single 7-day A/B experiment on the My Vibe smart-speaker surface, with 15% of randomly selected users per split. The report's own Section 8 concedes that full-traffic deployment, multi-month validation, and validation on other surfaces are pending, and that SONA's catalog coverage is lower than the production stack. Under these conditions, the observed engagement lift could partly reflect novelty or a shift toward more popular, well-covered items. The abstract and conclusion should therefore either be explicitly scoped to a 7-day, single-surface controlled experiment, or be backed by confidence intervals, a time-course analysis of the treatment effect, and longer-horizon data.
  2. [Section 7.5, Experiments 4 and 5; Table 7.7] The paragraph following Table 7.12 attributes an important part of SONA's online gain to extending history from 2k to 8k events with History Compression. However, the two online experiments were run separately, and Table 7.7 shows only modest offline differences between 2k and 8k full attention (Target Recall@1000 +0.0066, Teacher Recall@10 +0.0581, WPA +0.0092). The paper correctly notes that the difference does not isolate history length, but the subsequent text still describes longer context as 'an important part' of the configuration. Please add a same-experiment 2k-vs-8k arm or explicitly state that the comparison between Experiments 4 and 5 is confounded by other changes.
  3. [Section 7.5, general statistical reporting] The convention 'Unless marked †, reported deltas are significant at p<0.01' is insufficient for the key evidence. No confidence intervals, standard errors, or per-metric sample sizes are given, and with five experiments and roughly thirty metrics there is no control for multiple comparisons. For the primary metrics of Experiment 5, please report confidence intervals (and, if feasible, the p-values per metric), or at minimum state the number of users per arm and the standard deviation of the estimator.
minor comments (5)
  1. [Section 7.3, Table 7.5; Section 7.4, Tables 7.6-7.7] Many offline comparisons differ by less than 0.005 in WPA or Teacher Recall and are reported without error bars; please add uncertainty quantification or note that these differences may be within noise.
  2. [Section 7.5, Experiment 3, Table 7.10] The Likes rows are marked as not significant, yet the text says the Teacher Ranker 'can successfully replace the production ranker'; please restrict the claim to the metrics that are statistically significant.
  3. [Section 3.1, Table 3.2] The two collaborative-pair streams use different mining windows (3 weeks vs 3 months) and are concatenated; please clarify how the imbalance is handled in training and why the windows differ.
  4. [Appendix B, Table B.2] 'Per-level SID embeddings 32001×128' needs a one-sentence explanation of the vocabulary size (32,000 codes plus a BOS or padding token) and how it relates to the two hash embedding tables for semantic prefixes.
  5. [Figure 1] The Likes panel shows 0.00 for V0 and V2; if these are exactly zero increments, please say so, and if they are rounded, add a note in the caption.

Circularity Check

1 steps flagged · score 2.0 of 10

Central online A/B claim is independent of training objectives; only the offline Teacher Recall metric is self-referential, measuring agreement with the very teacher that supplies the distillation targets.

  1. self definitional [Section 7.1 (Teacher Recall definition) and Section 4.2 (distillation loss, L_rollout)]
    "Teacher Recall@k evaluates ranking fidelity conditional on candidate generation: for a held-out request r, the decoder produces a candidate pool C_r, and the Ranking Module and teacher rank the same pool. ... the teacher exposes n engagement scores per candidate, and the Ranking Module regresses a matching head onto each with an element-wise mean absolute error."

    Teacher Recall is the top-k overlap between Ranking Module scores and teacher scores on the same generated candidate pool, while the Ranking Module's only ranking supervision is MAE regression onto those same teacher scores. Consequently, high Teacher Recall is largely a restatement of the training objective: the metric measures agreement with the very scorer that provides the targets, so it cannot serve as independent external evidence of ranking quality. The paper uses it as a principal offline signal for the 'unified-model transfer' evaluation question and for selecting distillation configurations (Tables 7.1 and 7.6). This is a self-consistency check, not a prediction from first principles; it does not affect the online A/B result, which is measured against the production control.

full rationale

The paper's central claim—that one served model can replace the full production cascade—rests on Experiment 5, a live A/B against the production control on My Vibe smart-speaker traffic. That comparison is external to Sona's own training losses and therefore not circular. The offline Target-track Recall and WPA metrics are also independent of the final model's objectives. The one circularity-adjacent element is Teacher Recall@k: because the Ranking Module is trained to regress the frozen teacher's scores, its agreement with that teacher is partly built into the objective. The report is transparent that this metric measures ranking fidelity, and it uses it for architecture selection rather than as a claim of independent quality, so this is a minor self-referential metric rather than a load-bearing circular step. Self-citations to Argus [12] and Gryphon [28] exist and overlap with the Sona Team, but they are baseline/comparison and adopted-architecture attributions, not substitutes for the reported A/B evidence. Section 8 explicitly qualifies the central claim by stating that full-traffic deployment, a multi-month validation, and validation on other surfaces are pending, and that Sona's catalog coverage is lower than the production stack; these are validity/risk limitations, not circularity. Concerns about the 7-day duration and missing confidence intervals are similarly evidence-strength issues, not derivation-circularity. Weighing all of this, no prediction in the paper reduces by construction to its inputs; the only notable self-reference is the Teacher Recall fidelity metric, giving a low circularity score.

Assumptions & free parameters 7 free parameters · 4 assumptions · 0 invented entities

The system is built from standard ML components; the load-bearing assumptions are about the trustworthiness of the production logs and the validity of the engagement-grade ordering, plus the representativeness of the single-surface A/B experiment. No new physical or conceptual entities are introduced.

free parameters (7)
  • Tokenizer codebook sizes = 3 levels x 32,000
    Selected by ablation (Table 7.2); 3x32k gives Recall@1000 0.8111 vs 0.8036 for 3x8192.
  • InfoNCE temperature tau = 0.1
    Listed in model card B.1; not ablated.
  • Alignment weight lambda_align = 0.1
    Set by hand in Section 3.1; not ablated.
  • History Compression split N/nr = 8192/2048
    Chosen in Section 3.3 to balance cost and recall; Table 7.7 shows close offline values vs full 8k attention.
  • Distillation loss weights NTP:rollout:impression = 1:1:1
    Set in model card B.3; no ablation reported.
  • Training rollout beam size = 32
    Retained as lower-cost reference in Table 7.6; beam 128 gives higher Teacher Recall point estimates but quadruples training cost.
  • Serving beam width = 1024
    Fixed for evaluation in Section 7.4; not tuned at serving.
assumptions (4)
  • domain assumption Logged engagement events and their feedback are a faithful signal of user preference.
    The whole training pipeline optimizes on these logs; Section 3.2 and 4.1. If logs are biased by the production system, the model inherits the bias.
  • domain assumption The graded engagement ordering like > play > skip > dislike used to supervise the teacher is a valid proxy for recommendation quality.
    Equation 5.1 in Section 5.3 constructs pairwise labels from this order; the online improvements depend on this ordering being meaningful.
  • domain assumption Semantic ID tuples with shared prefixes support knowledge transfer across similar tracks and generalize to new items.
    Adopted from prior work (Rajput et al. [22]) and used as the basis of decoding; Section 3.1 and 3.4.
  • domain assumption The A/B test protocol (7-day, 15% users per split) is sufficient to estimate the true treatment effect of replacing the entire cascade.
    Section 7.5 Experiment 5; the authors themselves note the need for a multi-month experiment in Section 8.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Sona Technical Report." pith.science (2026). https://pith.science/paper/IJHMC4R3

@misc{pith2026260811015,
  author       = {Pith},
  title        = {Pith review of: Sona Technical Report},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IJHMC4R3}},
  note         = {Machine review of arXiv:2608.11015}
}
read the original abstract

We introduce Sona, a single-model generative recommender for Yandex Music. In an online A/B test, Sona replaced the entire production cascade, comprising more than 15 candidate generators followed by pre-ranking and ranking models that consume hundreds of features, including signals from large transformer models such as Argus and target-attention scorers, while significantly improving key engagement metrics. The architecture of Sona unifies candidate generation and ranking around a shared user representation. Its encoder transforms the user's chronological sequence of logged engagement events into hidden states consumed by both the autoregressive decoder and the Ranking Module. The next-token-prediction and distillation objectives jointly update the encoder, coupling generation and ranking through the same user state. Neither Sona nor its Teacher Ranker uses hand-engineered features; both operate on logged event fields and learned item representations. In the final Sona configuration, the larger teacher supplies ranking targets during training but is absent from serving, leaving the encoder, decoder, and Ranking Module as a single deployed model. We evaluate Sona in an online A/B experiment using live traffic from My Vibe on smart speakers, one of Yandex Music's largest recommendation surfaces. Relative to the production control, Sona produced statistically significant uplifts of 4.53% in Active Users, the primary metric, 6.30% in Total Listening Time, and 11.42% in Likes. These effects were incremental to improvements retained from preceding deployments. The Active Users uplift was 2.35 times the increment previously delivered by Argus, the strongest model deployed on this surface before Sona. These results show that a single jointly trained model can replace a mature multi-stage recommendation cascade while improving recommendation quality on live traffic.

Figures

Figures reproduced from arXiv: 2608.11015 by the authors.

Figure 2.1
Figure 2.1. summarizes SONA’s serving path and training signals. At serving time, the encoder processes the user’s chronological event history once, producing a representation shared by the decoder and Ranking Module. Following the generative-recommendation paradigm, the decoder autoregressively produces a beam of short Semantic ID tuples [8, 18, 22, 39, 41]. A catalog index maps each tuple to one or more tracks, and the Rankin… view at source ↗
Figure 3.1
Figure 3.1. Tokenization pipeline. The frozen Qwen2.5-Omni runs in prefill-only mode — no decoding. with cosine similarity sim and temperature τ . The alignment term keeps each item embedding close to h¯ a, the mean of the same track’s content features, Lalign = 1 2m X 2m a=1 [PITH_FULL_IMAGE:figures/full_fig_p007_3_1.png] view at source ↗
Figure 3.2
Figure 3.2. History Compression, read bottom-up. Dark: the long-term block O (no events); light: the recent block R (nr events). 3.3 Encoder For a full encoder, let H denote the hidden-state matrix computed from the event sequence X of Section 3.2, H = Encθ(X), H ∈ R (T +1)×d . The decoder (Section 3.4) and Ranking Module (Section 3.5) consume a shared encoder memory K, computed once per request. The symbols distinguish the enc… view at source ↗
Figures from the paper (5 more)
Figure 3.3
Figure 3.3. Figure 3.3: Unified generation-and-ranking architecture: the decoder and the Ranking Module attend to the same encoded user history. 4 SONA Training 4.1 Dataset Design The dataset is request-level: its unit is a single served recommendation request. A sample carries the user’s c…
Figure 4.1
Figure 4.1. Figure 4.1: Joint training of the decoder (LNTP) and the Ranking Module (LDistillation, regression toward the frozen Teacher Ranker) on the shared encoder. scoring and loss. The rollout set is complemented with the request’s logged impressions I, which broaden teacher supervisio…
Figure 5.1
Figure 5.1. Figure 5.1: Teacher Ranker architecture: a history encoder over the typed event sequence (left) and a candidate scorer that cross-attends to its hidden states (right). 5 Teacher Ranker 5.1 Architecture The teacher ranker is a large ranker without hand-engineered features, used a…
Figure 6.1
Figure 6.1. Figure 6.1: Online-training infrastructure for the final SONA configuration. In the diagram, “Teacher Ranker” is the frozen distillation teacher, “Train Teacher Ranker” its training pipeline, “Training Model” / “Serving Model” the trainer and serving copies of SONA, and “Inferen…
Figure 7.1
Figure 7.1. Figure 7.1: Train NTP loss of the Small, Medium, and 2× Medium configurations at matched cumulative packed-target exposure. Increasing capacity consistently lowers train NTP loss over the measured exposure range. Training-data volume. We next fix the Medium backbone, 2k-event hi…

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

47 extracted references · 29 linked inside Pith

  1. [1]

    Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862, 2022

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862, 2022

    arXiv 2022

  2. [2]

    From features to transformers: Redefining ranking for scalable impact.arXiv preprint arXiv:2502.03417, 2025

    Fedor Borisyuk, Lars Hertel, Ganesh Parameswaran, Gaurav Srivastava, Sudarshan Srinivasa Ra- manujam, Borja Ocejo, Peng Du, Andrei Akterskii, Neil Daftary, Shao Tang, Daqi Sun, Qiang Charles Xiao, Deepesh Nathani, Mohit Kothari, Yun Dai, Guoyao Li, and Aman Gupta. From features to transformers: Redefining ranking for scalable impact.arXiv preprint arXiv:2...

    arXiv 2025

  3. [3]

    Chi, and Derek Zhiyuan Cheng

    Benjamin Coleman, Wang-Cheng Kang, Matthew Fahrbach, Ruoxi Wang, Lichan Hong, Ed H. Chi, and Derek Zhiyuan Cheng. Unified embedding: Battle-tested feature representations for web-scale ml systems. InAdvances in Neural Information Processing Systems, 2023. URL https://arxiv.org/abs/ 2305.12102

    arXiv 2023

  4. [4]

    Fu, Stefano Ermon, Atri Rudra, and Christopher Ré

    Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness.arXiv preprint arXiv:2205.14135, 2022. URL https: //arxiv.org/abs/2205.14135. 24 Sona Technical Report

    arXiv 2022

  5. [5]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2021. UR...

    arXiv 2021

  6. [6]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    arXiv 2025

  7. [8]

    Oxygenrec: An instruction-following generative framework for e-commerce recommendation.arXiv preprint arXiv:2512.22386, 2025

    Xuegang Hao, Ming Zhang, Alex Li, et al. Oxygenrec: An instruction-following generative framework for e-commerce recommendation.arXiv preprint arXiv:2512.22386, 2025. URL https://arxiv.org/ abs/2512.22386

    arXiv 2025

  8. [9]

    Chi, Cristos Goodrow, Ningren Han, He Ma, Romer Rosales, Abby Van Soest, Su-Lin Wu, Weilong Yang, and Yilin Zheng

    Ruining He, Lukasz Heldt, Lichan Hong, Raghunandan Keshavan, Shifan Mao, Nikhil Mehta, Zhengyang Su, Alicia Tsai, Yueqi Wang, Shao-Chuan Wang, Xinyang Yi, Lexi Baugher, Baykal Cakici, Ed H. Chi, Cristos Goodrow, Ningren Han, He Ma, Romer Rosales, Abby Van Soest, Su-Lin Wu, Weilong Yang, and Yilin Zheng. Plum: Adapting pre-trained language models for indus...

    arXiv

Show all 47 references
  1. [10]

    Query-key normalization for transformers.Findings of EMNLP, 2020

    Alex Henry, Prudhvi Raj Dachapally, Shubham Pawar, and Yuxuan Chen. Query-key normalization for transformers.Findings of EMNLP, 2020. URLhttps://arxiv.org/abs/2010.04245

    2020 arXiv

  2. [11]

    Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models.arXiv preprint arXiv:2001.08361, 2020. URLhttps://arxiv.org/abs/2001.08361

    2001 arXiv

  3. [12]

    Scaling recommender transformers to one billion parameters

    Kirill Khrylchenko, Artem Matveev, Sergei Makeev, and Vladimir Baikalov. Scaling recommender transformers to one billion parameters. InProceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), 2026. URLhttps://arxiv.org/abs/2507.15994

    2026

  4. [13]

    Unipinrec: Unifying generative retrieval and ranking at pinterest scale.arXiv preprint arXiv:2606.00422, 2026

    Hanyu Li, Yi-Ping Hsu, Aditya Mantha, Prabhat Agarwal, Laksh Bhasin, Jialu Wang, Hongtao Lin, Bella Huang, Yaxin Li, Xinyi Li, et al. Unipinrec: Unifying generative retrieval and ranking at pinterest scale.arXiv preprint arXiv:2606.00422, 2026

    2026 arXiv

  5. [14]

    Harmonizing generative retrieval and ranking in chain-of-recommendation

    Yu Liu and Jiangxia Cao. Harmonizing generative retrieval and ranking in chain-of-recommendation. arXiv preprint arXiv:2604.25787, 2026. URLhttps://arxiv.org/abs/2604.25787

    2026 arXiv

  6. [15]

    Qarm: Quantitative alignment multi-modal recommendation at kuaishou.arXiv preprint arXiv:2411.11739, 2024

    Xinchen Luo, Jiangxia Cao, et al. Qarm: Quantitative alignment multi-modal recommendation at kuaishou.arXiv preprint arXiv:2411.11739, 2024

    2024 arXiv

  7. [16]

    Gated bidirectional linear attention for generative retrieval.arXiv preprint arXiv:2606.07317, 2026

    Artem Matveev, Vladislav Tytskiy, Sergei Makeev, and Sergei Liamaev. Gated bidirectional linear attention for generative retrieval.arXiv preprint arXiv:2606.07317, 2026. URL https://arxiv.org/ abs/2606.07317

    2026 arXiv

  8. [17]

    Webgpt: Browser-assisted question-answering with human feedback.arXiv preprint arXiv:2112.09332, 2021

    Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback.arXiv preprint arXiv:2112.09332, 2021

    2021 arXiv

  9. [18]

    Onerec technical report.arXiv preprint arXiv:2506.13695, 2025

    OneRec Team. Onerec technical report.arXiv preprint arXiv:2506.13695, 2025. URL https://arxiv. org/abs/2506.13695

    2025

  10. [19]

    Training language models to follow instructions with human feedback.Advances in Neural Information Processing Systems, 2022

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback.Advances in Neural Information Processing Systems, 2022. UR...

    2022 arXiv

  11. [20]

    Ega-v1: Unifying online advertising with end-to-end learning.arXiv preprint arXiv:2505.19755, 2025

    Junyan Qiu et al. Ega-v1: Unifying online advertising with end-to-end learning.arXiv preprint arXiv:2505.19755, 2025

    2025 arXiv

  12. [21]

    Qwen2.5-omni technical report.arXiv preprint arXiv:2503.20215, 2025

    Qwen Team. Qwen2.5-omni technical report.arXiv preprint arXiv:2503.20215, 2025. URL https: //arxiv.org/abs/2503.20215

    2025 arXiv

  13. [22]

    Recommender systems with generative 25 Sona Technical Report retrieval.Advances in Neural Information Processing Systems, 2023

    Shashank Rajput, Nikhil Mehta, Anima Singh, Raghunandan Hulikal Keshavan, Trung Vu, Lukasz Heldt, Lichan Hong, Yi Tay, Vinh Tran, Jonah Samost, et al. Recommender systems with generative 25 Sona Technical Report retrieval.Advances in Neural Information Processing Systems, 2023...

    2023 arXiv

  14. [23]

    Glu variants improve transformer.arXiv preprint arXiv:2002.05202, 2020

    Noam Shazeer. Glu variants improve transformer.arXiv preprint arXiv:2002.05202, 2020

    2002 arXiv

  15. [24]

    Contrastive learning of musical representations.arXiv preprint arXiv:2103.09410, 2021

    Janne Spijkervet and John Ashley Burgoyne. Contrastive learning of musical representations.arXiv preprint arXiv:2103.09410, 2021. URLhttps://arxiv.org/abs/2103.09410

    2021 arXiv

  16. [25]

    Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano

    Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback.Advances in Neural Information Processing Systems, 2020. URLhttps://arxiv.org/abs/2009.01325

    2020 arXiv

  17. [26]

    Roformer: Enhanced transformer with rotary position embedding.arXiv preprint arXiv:2104.09864, 2021

    Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding.arXiv preprint arXiv:2104.09864, 2021

    2021 arXiv

  18. [27]

    Grank: Towards target-aware and streamlined industrial retrieval with a generate-rank framework

    Yijia Sun, Shanshan Huang, Zhiyuan Guan, Qiang Luo, Ruiming Tang, Kun Gai, and Guorui Zhou. Grank: Towards target-aware and streamlined industrial retrieval with a generate-rank framework. In Proceedings of the ACM Web Conference 2026 (WWW ’26), 2026. URL https://arxiv.org/abs...

    2026

  19. [28]

    Gryphon: A unified architecture for semantic-id generation and item-level scoring in industrial recommendations

    Daria Tikhonovich, Oleg Sorokin, Vladislav Dodonov, Mariia Ulianova, and Ilya Murzin. Gryphon: A unified architecture for semantic-id generation and item-level scoring in industrial recommendations. arXiv preprint arXiv:2606.08604, 2026

    2026 arXiv

  20. [29]

    Philippe Tillet, H. T. Kung, and David Cox. Triton: An intermediate language and compiler for tiled neural network computations. InProceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages (MAPL), 2019. URL https://doi.org/10.1145/...

    2019 doi

  21. [30]

    Scaling transformers for discriminative recommendation via generative pretraining.arXiv preprint arXiv:2506.03699, 2025

    Chunqi Wang, Bingchao Wu, Zheng Chen, Lei Shen, Bing Wang, and Xiaoyi Zeng. Scaling transformers for discriminative recommendation via generative pretraining.arXiv preprint arXiv:2506.03699, 2025. URLhttps://arxiv.org/abs/2506.03699

    2025 arXiv

  22. [31]

    Onelive: Dynamically unified generative framework for live-streaming recommenda- tion.arXiv preprint arXiv:2602.08612, 2026

    Shen Wang et al. Onelive: Dynamically unified generative framework for live-streaming recommenda- tion.arXiv preprint arXiv:2602.08612, 2026

    2026

  23. [32]

    Learnable item tokenization for generative recommendation

    Wenjie Wang, Honghui Bao, Xinyu Lin, Jizhi Zhang, Yongqi Li, Fuli Feng, See-Kiong Ng, and Tat-Seng Chua. Learnable item tokenization for generative recommendation. InProceedings of the 33rd ACM International Conference on Information and Knowledge Management, 2024. URL https:/...

    2024

  24. [33]

    Oneloc: Geo-aware generative recommender systems for local life service.arXiv preprint arXiv:2508.14646, 2025

    Zhipeng Wei et al. Oneloc: Geo-aware generative recommender systems for local life service.arXiv preprint arXiv:2508.14646, 2025

    2025 arXiv

  25. [35]

    On layer normalization in the transformer architecture

    Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tie-Yan Liu. On layer normalization in the transformer architecture. InProceedings of the 37th International Conference on Machine Learning, 2020. URL https://pr...

    2020

  26. [36]

    On layer normalization in the transformer architecture

    Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. On layer normalization in the transformer architecture. InInternational conference on machine learning, pages 10524–10533. PMLR, 2020

    2020

  27. [37]

    Flashinfer: Efficient and customizable attention engine for LLM inference serving

    Zihao Ye, Lequn Chen, Ruihang Lai, Wuwei Lin, Yineng Zhang, Stephanie Wang, Tianqi Chen, Baris Kasikci, Vinod Grover, Arvind Krishnamurthy, and Luis Ceze. Flashinfer: Efficient and customizable attention engine for LLM inference serving. InProceedings of Machine Learning and S...

    2025

  28. [38]

    Sampling-bias-corrected neural modeling for large corpus item recommendations

    Xinyang Yi, Ji Yang, Lichan Hong, Derek Zhiyuan Cheng, Lukasz Heldt, Aditee Kumthekar, Zhe Zhao, Li Wei, and Ed Chi. Sampling-bias-corrected neural modeling for large corpus item recommendations. InRecSys, 2019

    2019

  29. [39]

    Actions speak louder than words: Trillion-parameter sequential transducers for generative recommendations.arXiv preprint arXiv:2402.17152, 2024

    Jiaqi Zhai et al. Actions speak louder than words: Trillion-parameter sequential transducers for generative recommendations.arXiv preprint arXiv:2402.17152, 2024. 26 Sona Technical Report

    2024 arXiv

  30. [40]

    Root mean square layer normalization

    Biao Zhang and Rico Sennrich. Root mean square layer normalization. InAdvances in Neural Information Processing Systems, 2019. URLhttps://arxiv.org/abs/1910.07467

    2019 arXiv

  31. [41]

    Gpr: Towards a generative pre-trained one-model paradigm for large-scale advertising recommendation.arXiv preprint arXiv:2511.10138, 2025

    Jun Zhang, Yi Li, Yue Liu, et al. Gpr: Towards a generative pre-trained one-model paradigm for large-scale advertising recommendation.arXiv preprint arXiv:2511.10138, 2025. URL https://arxiv. org/abs/2511.10138

    2025

  32. [42]

    Onemall: One architecture, more scenarios — end-to-end generative recommender family at kuaishou e-commerce.arXiv preprint arXiv:2601.21770, 2026

    Kun Zhang et al. Onemall: One architecture, more scenarios — end-to-end generative recommender family at kuaishou e-commerce.arXiv preprint arXiv:2601.21770, 2026

    2026

  33. [43]

    Scaling user modeling: Large-scale online user representations for ads personalization in meta

    Wei Zhang, Dai Li, Chen Liang, Fang Zhou, Zhongke Zhang, Xuewei Wang, Ru Li, Yi Zhou, Yaning Huang, Dong Liang, et al. Scaling user modeling: Large-scale online user representations for ads personalization in meta. InCompanion Proceedings of the ACM Web Conference (WWW ’24 Companion),

  34. [44]

    Pytorch fsdp: Experiences on scaling fully sharded data parallel.arXiv preprint arXiv:2304.11277, 2023

    Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, et al. Pytorch fsdp: Experiences on scaling fully sharded data parallel.arXiv preprint arXiv:2304.11277, 2023

    2023 arXiv

  35. [45]

    Enhancing embedding representation stability in recom- mendation systems with semantic id

    Carolina Zheng, Minhui Huang, Dmitrii Pedchenko, Kaushik Rangadurai, Siyu Wang, Fan Xia, Gaby Nahum, Jie Lei, Yang Yang, Tao Liu, et al. Enhancing embedding representation stability in recom- mendation systems with semantic id. InProceedings of the Nineteenth ACM Conference on...

    2025

  36. [46]

    Ega-v2: An end-to-end generative framework for industrial advertising.arXiv preprint arXiv:2505.17549, 2025

    Zuowu Zheng et al. Ega-v2: An end-to-end generative framework for industrial advertising.arXiv preprint arXiv:2505.17549, 2025. A Contributions and Acknowledgments Contributors are listed alphabetically by first name. * marks contributors who have since departed from the team....

    2025 arXiv

  37. [2024]

    URLhttps://arxiv.org/abs/2311.09544

    arXiv

  38. [2025]

    URLhttps://arxiv.org/abs/2505.18654

    arXiv

  39. [2026]

    URLhttps://arxiv.org/abs/2601.20083

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.