Pith. sign in

REVIEW 3 major objections 6 minor 43 references

Short-video search can rank for long-term satisfaction instead of just clicks, and a new reinforcement-learning fusion framework shows how.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A reinforcement-learning fusion framework using query reformulation, session gaps, and next-day retention as satisfaction rewards improved short-video search ranking and retention in Kuaishou's online A/B test.

T0 review reviewed 2026-08-03 challenge →

load-bearing objection A credible industrial method, but the offline 'satisfaction' and 'retention' metrics are the system's own outputs, so the SOTA claim rides almost entirely on an online A/B against a single baseline. the 3 major comments →

arxiv 2603.19585 v2 pith:XH2NQCKE submitted 2026-03-20 cs.IR

SaFRO: Satisfaction-Aware Fusion via Dual-Relative Policy Optimization for Short-Video Search

Renzhe Zhou , Songyang Li , Feiran Zhu , Chenglei Dai , Yi Zhang , Yi Wang , Jingwei Zhuo This is my paper
classification cs.IR
keywords multi-task fusionshort-video searchuser satisfactionreinforcement learningpolicy optimizationquery reformulationsession gapuser retention
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the multi-task fusion step in short-video search—the component that combines predicted click, like, and relevance scores into one ranking score—can be trained to optimize long-term user satisfaction rather than just immediate engagement. It introduces SaFRO, a framework that learns a fusion policy via reinforcement learning, using a reward model built from query-level behavioral proxies: whether the user reformulates their query, how the session gap compares with the user's own history, and next-day retention. The policy is trained with a dual-relative advantage estimator that normalizes rewards both within a group of candidate fusion weights for one query and across a batch of queries, which the paper says stabilizes learning under search's sparse feedback. Offline experiments on a large industrial dataset and a 60-day online A/B test on a major short-video platform report improvements in ranking quality, satisfaction score, and retention. The central claim is that long-term satisfaction can be directly optimized in search without giving up short-term ranking quality.

Core claim

The paper's central claim is that the fusion policy that turns multiple predicted scores into a final short-video search ranking can be optimized directly for long-term user satisfaction, and that this is feasible despite the sparsity and intent constraints that distinguish search from recommendation feeds. The core discovery is a reward signal that encodes satisfaction at the query level: it gives zero credit to sessions where the user reformulates the query, and otherwise combines a user-normalized session gap with a binary next-day retention indicator. This hand-crafted reward is distilled into a learned list-level reward model, which then guides a policy trained with Dual-Relative Policy

What carries the argument

The central machinery is the satisfaction-aware reward model and the dual-relative advantage. The reward is r_sat = (1 - I_reform) * [alpha * exp(-session_gap/(user_quantile_baseline * T)) + (1-alpha)*I_retention], combining a query-reformulation penalty, a user-normalized exponential-decayed session gap, and a next-day retention term; this reward is then learned by a list-level model R_phi via importance-weighted regression. The dual-relative advantage decomposes into A_group + A_batch = (r - mu_i)/sigma_i + (mu_i - mu_batch)/sigma_batch, which is plugged into a GRPO-style clipped objective. This decomposition yields two gradient terms: Term I preserves fine-grained within-query action rank

Load-bearing premise

The load-bearing premise is that the hand-crafted satisfaction reward r_sat—built from query reformulation, user-normalized session gap, and next-day retention—and the learned reward model R_phi trained on it truly capture long-term user satisfaction, rather than merely encoding a proxy that the policy then learns to optimize.

What would settle it

A decisive test would be to deploy the same policy with the satisfaction-aware reward replaced by a pure engagement reward in a blind online experiment, and measure actual next-week or next-month retention per user cohort; if retention gains disappear or reverse while proxy scores remain high, the proxies are the driver rather than true satisfaction. Alternatively, a controlled user study testing whether sessions flagged as 'dissatisfied' (reformulation, long gaps) actually match users' self-reported dissatisfaction would directly validate the reward's foundation.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Short-video search ranking can be aligned with long-term satisfaction without sacrificing short-term ranking quality; the reported NDCG and retention gains indicate the two are not inherently in conflict.
  • Query-level behavioral signals—query reformulation and session gap—can serve as dense training signals for long-term objectives in sparse search interaction settings, mitigating the attribution problem of delayed rewards.
  • Adding batch-relative normalization to group-relative advantage preserves within-query action ordering while dynamically adjusting for query-level quality, which stabilizes policy learning under sparse feedback.
  • Explicitly modeling interdependencies among objectives via a task-relation-aware fusion module outperforms independent or cross-feature fusion, yielding context-adaptive weights that better balance competing signals.
  • The online A/B test shows retention gains that widen over 60 days, suggesting the policy is driving real long-term behavior rather than only optimizing immediate proxy metrics.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The choice of query reformulation and session gap as satisfaction proxies is one specific design; alternative proxies—such as explicit feedback, video dwell time, or subjective ratings—could be tested to see if the benefit generalizes beyond these signals.
  • The dual-relative advantage can be seen as a general credit-assignment technique for any RL setting where samples are grouped, so it may transfer to other sparse-reward domains such as conversational recommendation or task-oriented dialogue where group quality varies.
  • The offline 'Satisfaction Score' is the reward model's own prediction, so some of the offline retention gains may partly reflect the model's internal consistency; external validation of the proxies against observed long-term retention would strengthen the causal claim.
  • The task-relation matrix produced by the TRAF module could be analyzed as an interpretable structure, revealing which objectives the system considers complementary or competing, which might inform product decisions beyond ranking.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. SaFRO is a multi-task fusion framework for short-video search that combines (i) a satisfaction-aware reward model built from query reformulation, user-normalized session gap, and next-day retention; (ii) Dual-Relative Policy Optimization (DRPO), which adds a batch-relative advantage term to GRPO's group-relative advantage; and (iii) a Task-Relation-Aware Fusion module that adaptively reweights task logits via pairwise task interactions. The paper reports large offline gains over LTR, black-box, and RL baselines on Kuaishou data, plus a 60-day online A/B test with a positive retention gap. The mathematical derivation of DRPO is clear and correct, but the empirical support for the long-term claims is weakened by circular offline metrics and a narrow online comparison.

Significance. If the long-term claims held, SaFRO would be a practical contribution: the DRPO advantage decomposition is clean, the synthetic illustration in Fig. 5 is helpful, and the two-tier normalization is a sensible way to inject global context into group-relative policy optimization. The online A/B test is a genuine strength, as is the explicit treatment of format and relevance constraints in the reward. However, the central 'state-of-the-art long-term retention' claim is not yet established: the offline Satisfaction Score is produced by the same reward model used for training, the offline User Retention column is generated by an unvalidated simulator, and the online test only compares against a single engagement-optimized DDPG baseline. These issues are load-bearing and require substantial additional evidence before the headline claim can be accepted.

major comments (3)
  1. [§5.1.2, Table 1, Eqs. (8)-(9), Eq. (16)] The offline Satisfaction Score is R_phi's own prediction. R_phi is trained on r_sat (Eq. 8) and the same r_sat appears as the satisfaction term in the composite training reward (Eq. 16). Evaluating SaFRO on this metric is circular: the policy is scored by the exact proxy it was trained to maximize. No external validation of R_phi is provided against held-out observed next-day return, explicit satisfaction labels, or reformulation outcomes. The choice of alpha=0.5 in Sec. 5.3.3 is made by maximizing the Pearson correlation of r_sat with its own two constituent terms (Fig. 7), which is tautological. Please report observed long-term outcomes or validate R_phi externally before using this column as evidence.
  2. [§5.1.2, Table 1 'User Retention'] The retention rates in Table 1 are not observed behavior; they come from a 'retention simulator' trained on the same dataset. The paper gives no architecture, training targets, feature set, calibration, or validation for this simulator. Without evidence that simulator predictions track true next-day return, the retention column cannot support the long-term claim. Please describe the simulator in detail and validate it (e.g., AUC/calibration on held-out users, or agreement with observed retention in the online A/B period).
  3. [§5.4, Fig. 8, Table 3] The online A/B compares SaFRO only against a DDPG baseline optimized for engagement. The retention-oriented SOTA baselines from the offline table (BatchRL-MTF, RLUR, AURO) are not included in the online experiment. Consequently, the abstract's claim of outperforming state-of-the-art baselines in long-term user retention is supported online only against a non-retention baseline, while the offline SOTA comparison relies on the circular metrics above. Add at least one online arm with a retention-oriented baseline, or restrict the online claim to 'improvement over an engagement-optimized DDPG baseline.'
minor comments (6)
  1. [Table 1 and text] The baseline is called 'BatchRL-MTF' in Sec. 5.1.3 and the references but 'Batch-MTF' in Table 1. Please use one consistent name.
  2. [Eq. (3)] The displayed definition of Afeas has an awkward line break before the absolute value; fix the formatting for readability.
  3. [Fig. 5] The right panel's arrows and annotations are very small and hard to read. Enlarge the figure and/or explain the arrows in the caption.
  4. [Title page] The header still reads 'Conference'17, July 2017' and the ACM DOI is a placeholder. Update these before submission.
  5. [Table 1] No variance or significance information is given for offline metrics. On a fixed industrial dataset, paired significance tests or standard deviations across seeds/date splits should be reported to show the gains are not noise.
  6. [Abstract and contributions] The phrase 'to the best of our knowledge, we are the first' is a broad novelty claim; consider softening or providing a more precise scope for what is new in short-video search specifically.

Circularity Check

2 steps flagged

Offline Satisfaction Score is R_phi's own prediction and training reward; offline User Retention is an unvalidated simulator output, so the long-term SOTA claim is only partially grounded by online A/B.

specific steps
  1. fitted input called prediction [§4.2 Eq. (9), §4.3 Eq. (16), §5.1.2 Metrics, Table 1]
    "We also report the satisfaction score predicted by the satisfaction-aware model. ... User satisfaction reward rsat: predicted by the satisfaction model in Sec 4.2, which takes the ranked list as input and outputs a scalar score in [0,1] reflecting holistic user satisfaction. ... r = reng + rsat + r_a^fmt + r_r^fmt (16)."

    The offline 'Satisfaction Score' column in Table 1 is computed by R_phi, the learned satisfaction-aware reward model. R_phi is trained by Eq. (9) to regress r_sat (Eq. 8), and the same R_phi output is inserted into the composite training reward as r_sat in Eq. (16). Thus the policy is trained to maximize exactly the function that is later reported as an independent offline satisfaction metric. No validation of R_phi against observed long-term outcomes (e.g., held-out next-day returns or explicit satisfaction labels) is provided, so SaFRO's high Satisfaction Score is forced by its own training objective rather than by an external criterion.

  2. other [§5.1.2 Metrics]
    "Furthermore, we assess user retention rate by building a retention simulator on the same dataset to predict users’ probability of returning the next day, which reflects the long-term effectiveness of each method."

    The 'User Retention' metric in Table 1 is not an observed outcome; it is a prediction produced by a retention simulator built on the same dataset used for training and evaluation. The paper does not describe the simulator's inputs, fitting target, or validation against real next-day retention. This is primarily a missing-evidence/validity concern rather than a full equation-level reduction, but it compounds the circularity: the only offline long-term metrics are model-generated rather than independently observed.

full rationale

The clearest circular step is the offline Satisfaction Score. §5.1.2 defines it as the output of the satisfaction-aware model, which is R_phi from Eq. (9); §4.3 uses the same model as the r_sat term in the composite training reward (Eq. 16). Evaluating a policy on a learned reward model that was its own training reward does not provide independent evidence of satisfaction, unless R_phi is separately validated against actual long-term user outcomes, which the paper does not report. The offline User Retention metric is also a simulator prediction on the same dataset, again lacking validation. The paper does have genuinely independent evidence: the NDCG@10 item-level metrics are standard and computed from observed interactions, and the online A/B test reports an observed retention gap over 60 days with p<0.05, although only against a DDPG engagement-optimized baseline rather than the retention-oriented SOTA baselines. There are no load-bearing self-citation chains or imported uniqueness theorems. The α=0.5 hyperparameter in §5.3.3 is chosen by maximizing r_sat's correlation with its own two constituent terms, which is an internal-consistency criterion rather than external validation, but it is not the main load-bearing circularity. Overall, the central long-term-satisfaction claim is partially circular: it rests on a fitted reward model used both as objective and as metric, with only the online A/B providing non-circular long-term grounding.

Axiom & Free-Parameter Ledger

7 free parameters · 6 axioms · 0 invented entities

The central claim depends on the behavioral-proxy reward being a true measure of satisfaction and on the offline long-term metrics not being the same fitted model used in training. The DRPO math itself is a small extension of GRPO and requires only standard RL assumptions.

free parameters (7)
  • alpha = 0.5
    Eq. 8 balances session-gap and next-day-retention terms; selected in Sec 5.3.3 by maximizing the sum of Pearson correlations of r_sat with its own two components, so the fit is partly circular.
  • temperature T = not reported
    Eq. 7 decay temperature for session gap; no value or sensitivity analysis is provided.
  • entropy coefficient beta = 0.05
    Sec 5.3.2 tuned on offline evaluation metrics; shows inverted-U behavior.
  • group size G = 32
    Sec 5.3.2 tuned; NDCG peaks at 16 but satisfaction and retention continue to improve up to 32.
  • session-gap quantile baseline and delta = 60% quantile; delta unspecified
    Sec 4.2 uses a personal historical beta-quantile baseline mu_u for session-gap normalization; the stability term delta is not specified.
  • action discretization bins B_j and tolerance xi = not reported
    Eq. 3 defines the feasible action space; these values are needed to implement the policy but are absent.
  • format-reward penalties and relevance threshold tau = -1, -2; tau not reported
    Eqs. 14-15 hand-set constraint penalties and a relevance threshold with no sensitivity analysis.
axioms (6)
  • standard math PPO-style clipped surrogate and importance-sampling estimators are valid
    Sec 3.2: DRPO inherits GRPO/PPO clipping; no proof is given but this is standard RL machinery.
  • domain assumption Query reformulation, user-normalized session gap, and next-day retention are valid satisfaction proxies
    Sec 4.2, Eq. 8: these define r_sat; if they are misaligned with true satisfaction, the entire reward signal is wrong.
  • domain assumption Learned reward model R_phi generalizes to true satisfaction
    Eq. 9: the same R_phi is used as the RL reward and as the offline evaluation metric, so generalization is assumed without independent validation.
  • ad hoc to paper Retention simulator accurately models next-day return probability
    Sec 5.1.2: built on the same dataset; no external validation of the simulator is provided.
  • domain assumption Discretized feasible action space with tolerance xi approximates continuous constrained weights
    Sec 3.1, Eq. 3: the policy is categorical over discretized bins; approximation error is unquantified.
  • ad hoc to paper Relevance constraint threshold tau is a valid business/fairness constraint
    Eq. 15: tau is not specified and the hard penalty -2 is chosen by hand.

reviewed 2026-08-03 · how reviews work

0 comments
Cite this review

Pith. "Pith review of SaFRO: Satisfaction-Aware Fusion via Dual-Relative Policy Optimization for Short-Video Search." pith.science (2026). https://pith.science/paper/XH2NQCKE

@misc{pith2026260319585,
  author       = {Pith},
  title        = {Pith review of: SaFRO: Satisfaction-Aware Fusion via Dual-Relative Policy Optimization for Short-Video Search},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XH2NQCKE}},
  note         = {Machine review of arXiv:2603.19585}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Multi-Task Fusion plays a pivotal role in industrial short-video search systems by aggregating heterogeneous prediction signals into a unified ranking score. However, existing approaches predominantly optimize for immediate engagement metrics, which often fail to align with long-term user satisfaction. While Reinforcement Learning (RL) offers a promising avenue for user satisfaction optimization, its direct application to search scenarios is non-trivial due to the inherent data sparsity and intent constraints compared to recommendation feeds. To this end, we propose SaFRO, a novel framework designed to optimize user satisfaction in short-video search. We first construct a satisfaction-aware reward model that utilizes query-level behavioral proxies to capture holistic user satisfaction beyond item-level interactions. Then we introduce Dual-Relative Policy Optimization (DRPO), an efficient policy learning method that updates the fusion policy through relative preference comparisons within groups and across batches. Furthermore, we design a Task-Relation-Aware Fusion module to explicitly model the interdependencies among different objectives, enabling context-sensitive weight adaptation. Extensive offline evaluations and large-scale online A/B tests on Kuaishou short-video search platform demonstrate that SaFRO significantly outperforms state-of-the-art baselines, delivering substantial gains in both short-term ranking quality and long-term user retention.

Figures

Figures reproduced from arXiv: 2603.19585 by ChengLei Dai, Feiran Zhu, Jingwei Zhuo, Renzhe Zhou, Songyang Li, Yi Wang, Yi Zhang.

Figure 1
Figure 1. Figure 1: Cascaded search system architecture. recommendation to search is non-trivial due to two fundamental differences: (i) Intent and relevance constraints. Search is driven by ex￾plicit queries that express well-defined user intent and impose a strong relevance constraint. Recommendation, by contrast, typi￾cally lacks an explicit intent signal and serves latent, exploratory preferences, where diversity and expl… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the SaFRO framework. After embedding input features into a state, the fusion policy employs relation [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Comparison of user behavioral patterns by reten [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: Comparison between GRPO and DRPO. Left: Advan [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗
Figure 4
Figure 4. Figure 4: An illustration of dual-relative advantage. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 6
Figure 6. Figure 6: Ablation results for key SaFRO modules. • BatchRL-MTF [38]: A Batch RL framework designed to optimize user satisfaction. • RLUR [2]: An Actor-Critic RL algorithm specifically tailored for retention optimization. • AURO [33]: An Actor-Critic RL algorithm for adaptive user retention optimization. 5.2 Overall Performance [PITH_FULL_IMAGE:figures/full_fig_p007_6.png] view at source ↗
Figure 8
Figure 8. Figure 8: Retention rate gap of SaFRO over the baseline model [PITH_FULL_IMAGE:figures/full_fig_p008_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

43 extracted references · 11 linked inside Pith

  1. [1]

    Christopher Burges, Robert Ragno, and Quoc Le. 2006. Learning to rank with nonsmooth cost functions.Advances in neural information processing systems19 (2006)

    2006

  2. [2]

    Qingpeng Cai, Shuchang Liu, Xueliang Wang, Tianyou Zuo, Wentao Xie, Bin Yang, Dong Zheng, Peng Jiang, and Kun Gai. 2023. Reinforcing user retention in a billion scale short video recommender system. InCompanion Proceedings of the ACM Web Conference 2023. 421–426

    2023

  3. [3]

    Xuezhi Cao, Sheng Zhu, Biao Tang, Rui Xie, Fuzheng Zhang, and Zhongyuan Wang. 2020. Ranking with Deep Multi-Objective Learning. InProceedings of the Deep Learning Practice for Industrial Problems. 1–4. doi:10.1145/3394486.3403288

    arXiv 2020

  4. [4]

    Rich Caruana. 1997. Multitask Learning.Machine Learning28, 1 (1997), 41–75

    1997

  5. [5]

    Xiaoshuang Chen, Gengrui Zhang, Yao Wang, Yulin Wu, Shuo Su, Kaiqiao Zhan, and Ben Wang. 2024. Cache-Aware Reinforcement Learning in Large-Scale Recommender Systems. InCompanion Proceedings of the ACM Web Conference

    2024

  6. [6]

    Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, and Andrew Rabinovich. 2018. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. InInternational conference on machine learning. PMLR, 794–803

    2018

  7. [7]

    Konstantina Christakopoulou, Can Xu, Sai Zhang, Sriraj Badam, Trevor Potter, Daniel Li, Hao Wan, Xinyang Yi, Ya Le, Chris Berg, Eric Bencomo Dixon, Ed H Chi, and Minmin Chen. 2022. Reward Shaping for User Satisfaction in a REINFORCE Recommender.arXiv preprint arXiv:2209.15166(2022)

    Pith/arXiv arXiv 2022

  8. [8]

    Rui Ding, Ruobing Xie, Xiaobo Hao, Xiaochun Yang, Kaikai Ge, Xu Zhang, Jie Zhou, and Leyu Lin. 2023. Interpretable User Retention Modeling in Recommen- dation. InProceedings of the 17th ACM Conference on Recommender Systems. ACM, 702–712

    2023

  9. [9]

    Scott Fujimoto, Herke Hoof, and David Meger. 2018. Addressing function ap- proximation error in actor-critic methods. InInternational conference on machine learning. PMLR, 1587–1596

    2018

  10. [10]

    Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. 2018. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. InInternational conference on machine learning. Pmlr, 1861–1870

    2018

  11. [11]

    Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. 2015. Continuous control with deep reinforcement learning.arXiv preprint arXiv:1509.02971(2015)

    Pith/arXiv arXiv 2015

  12. [12]

    Shih-Yang Liu, Xin Dong, Ximing Lu, Shizhe Diao, Peter Belcak, Mingjie Liu, Min-Hung Chen, Hongxu Yin, Yu-Chiang Frank Wang, Kwang-Ting Cheng, et al

  13. [13]

    Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. 2025. Understanding r1-zero-like training: A critical perspective.arXiv preprint arXiv:2503.20783(2025)

    Pith/arXiv arXiv 2025

  14. [14]

    Ziru Liu, Shuchang Liu, Bin Yang, Zhenghai Xue, Qingpeng Cai, Xiangyu Zhao, Zijian Zhang, Lantao Hu, Han Li, and Peng Jiang. 2024. Modeling User Retention through Generative Flow Networks. InProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. ACM, 5497–5508

    2024

  15. [15]

    Ziru Liu, Shuchang Liu, Zijian Zhang, Qingpeng Cai, Xiangyu Zhao, Kesen Zhao, Lantao Hu, Peng Jiang, and Kun Gai. 2024. Sequential Recommendation for Optimizing Both Immediate Feedback and Long-term Retention. InProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 1956–1966

    2024

  16. [16]

    Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H. Chi. 2018. Modeling Task Relationships in Multi-task Learning with Multi-gate Mixture- of-Experts. InProceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 1930–1939. doi:10.1145/3219819.3220007

    arXiv 2018

  17. [17]

    Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert. 2016. Cross-Stitch Networks for Multi-task Learning. InProceedings of the IEEE Confer- ence on Computer Vision and Pattern Recognition. 3994–4003

    2016

  18. [18]

    Jonas Močkus. 1974. On Bayesian methods for seeking the extremum. InIFIP Technical Conference on Optimization Techniques. Springer, 400–404

    1974

  19. [19]

    Michinari Momma, Chosen Dong, and Yetian Chen. 2022. Multi-objective ranking with directions of preferences. InProceedings of the ACM SIGIR Workshop on eCommerce. ACM, 1–5

    2022

  20. [20]

    Changhua Pei, Xinru Yang, Qing Cui, Xiao Lin, Fei Sun, Peng Jiang, Wenwu Ou, and Yongfeng Zhang. 2019. Value-Aware Recommendation Based on Reinforced Profit Maximization in E-commerce Systems. arXiv preprint (2019).arXiv preprint arXiv:1902.00851(2019)

    Pith/arXiv arXiv 2019

  21. [21]

    Przemysław Pobrotyn and Radosław Białobrzeski. 2021. Neuralndcg: Direct optimisation of a ranking metric via differentiable relaxation of sorting.arXiv preprint arXiv:2102.07831(2021)

    Pith/arXiv arXiv 2021

  22. [22]

    2004.The cross-entropy method: a unified approach to combinatorial optimization, Monte-Carlo simulation and machine learning

    Reuven Y Rubinstein and Dirk P Kroese. 2004.The cross-entropy method: a unified approach to combinatorial optimization, Monte-Carlo simulation and machine learning. Springer Science & Business Media

    2004

  23. [23]

    John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz

  24. [24]

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov

  25. [25]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300(2024)

    Pith/arXiv arXiv 2024

  26. [26]

    Vaishnavi Shrivastava, Ahmed Awadallah, Vidhisha Balachandran, Shivam Garg, Harkirat Behl, and Dimitris Papailiopoulos. 2025. Sample more to think less: Group filtered policy optimization for concise reasoning.arXiv preprint arXiv:2508.09726(2025)

    Pith/arXiv arXiv 2025

  27. [27]

    Liangcai Su, Junwei Pan, Ximei Wang, Xi Xiao, Shijie Quan, Xihua Chen, and Jie Jiang. 2024. STEM: unleashing the power of embeddings for multi-task recommendation. InProceedings of the AAAI conference on artificial intelligence, Vol. 38. 9002–9010

    2024

  28. [28]

    1998.Reinforcement learning: An intro- duction

    Richard S Sutton, Andrew G Barto, et al. 1998.Reinforcement learning: An intro- duction. Vol. 1. MIT press Cambridge

    1998

  29. [29]

    Hongyan Tang, Junning Liu, Ming Zhao, and Xudong Gong. 2020. Progres- sive Layered Extraction (PLE): A Novel Multi-Task Learning (MTL) Model for Personalized Recommendations. InProceedings of the 14th ACM Conference on Recommender Systems. 269–278. doi:10.1145/3383313.3412236

    arXiv 2020

  30. [30]

    Ruoxi Wang, Bin Fu, Gang Fu, and Mingliang Wang. 2017. Deep & cross network for ad click predictions. InProceedings of the ADKDD’17. 1–7

    2017

  31. [31]

    Xiaobei Wang, Shuchang Liu, Xueliang Wang, Qingpeng Cai, Lantao Hu, Han Li, Peng Jiang, and Guangming Xie. 2024. Future Impact Decomposition in Request- level Recommendations. InProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. ACM

    2024

  32. [32]

    Wanqi Xue, Qingpeng Cai, Zhenghai Xue, Shuo Sun, Shuchang Liu, Dong Zheng, Peng Jiang, Kun Gai, and Bo An. 2023. PrefRec: Recommender Systems with Human Preferences for Reinforcing Long-term User Engagement. InProceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. ACM, 2874–2884

    2023

  33. [33]

    Zhenghai Xue, Qingpeng Cai, Bin Yang, Lantao Hu, Peng Jiang, Kun Gai, and Bo An. 2025. AURO: Reinforcement learning for adaptive user retention optimization in recommender systems. InProceedings of the ACM on Web Conference 2025. 391–401

    2025

  34. [34]

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. 2025. Dapo: An open- source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476 (2025)

    Pith/arXiv arXiv 2025

  35. [35]

    Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. 2020. Gradient surgery for multi-task learning.Advances in neural information processing systems33 (2020), 5824–5836

    2020

  36. [36]

    Changshuo Zhang, Sirui Chen, Xiao Zhang, Sunhao Dai, Weijie Yu, and Jun Xu. 2024. Reinforcing Long-Term Performance in Recommender Systems with User-Oriented Exploration Policy. InProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 1850–1860

    2024

  37. [37]

    Gengrui Zhang, Yao Wang, Xiaoshuang Chen, Hongyi Qian, Kaiqiao Zhan, and Ben Wang. 2024. UNEX-RL: reinforcing long-term rewards in multi-stage rec- ommender systems with unidirectional execution. InProceedings of the AAAI Conference on Artificial Intelligence, Vol. 38. 9305–9313

    2024

  38. [38]

    Qihua Zhang, Junning Liu, Yuzhuo Dai, Yiyan Qi, Yifan Yuan, Kunlun Zheng, Fan Huang, and Xianfeng Tan. 2022. Multi-task fusion via reinforcement learning for long-term user satisfaction in recommender systems. InProceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining. 4510–4520

    2022

  39. [39]

    Kesen Zhao, Lixin Zou, Xiangyu Zhao, Maolin Wang, and Dawei Yin. 2023. User Retention-oriented Recommendation with Decision Transformer. InProceedings of the ACM Web Conference 2023. ACM, 1141–1149

    2023

  40. [40]

    Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al. 2025. Group sequence policy optimization.arXiv preprint arXiv:2507.18071(2025)

    Pith/arXiv arXiv 2025

  41. [2015]

    InInternational conference on machine learning

    Trust region policy optimization. InInternational conference on machine learning. PMLR, 1889–1897

  42. [2017]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347 (2017)

    Pith/arXiv arXiv 2017

  43. [2026]

    GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization.arXiv preprint arXiv:2601.05242(2026)

    Pith/arXiv arXiv 2026

This paper was first reviewed by deepseek-v4-flash on August 3, 2026.