Pith. sign in

REVIEW 4 major objections 5 minor 98 references

Multi-task Offline Reinforcement Learning for Online Advertising in Recommender Systems

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that one offline reinforcement learning model can jointly pick ad channels and allocate budgets, beating strong baselines on two public datasets and in a live A/B test.

desk verdict MTORL is a credible industrial recipe whose action-prediction results stand up, but the budget-allocation claims rest on a reward decoder that ignores the chosen action. read the letter →

arxiv 2506.23090 v2 pith:3BHLKV43 submitted 2025-06-29 cs.IR cs.LG

Langming Liu , Wanyu Wang , Chi Zhang , Bo Li , Hongzhi Yin , Xuetao Wei , Wenbo Su , Bo Zheng
show 1 more author
Xiangyu Zhao
This is my paper · ORCID
classification cs.IRcs.LG
keywords offlinereinforcementlearningonlineadvertisingchannelrecommendationbudgetallocationcausalstateencodingattentiondirectpreferenceoptimizationsequencemodeling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that one offline reinforcement learning model can jointly handle two advertising decisions that platforms usually make separately: which channel should serve a user's next ad exposure, and which users should get ads under a limited budget. The proposed model, MTORL, recasts both tasks as conditional sequence modeling over users' past exposure journeys, predicting the next channel action and a budget-penalized reward from a causal state representation. On two public benchmark datasets the model reports the best channel-prediction accuracy and the highest average reward under a fixed budget, compared with conservative value-based offline RL (CQL, IQL), constrained-RL methods, and transformer sequence models (Decision Transformer). In a live platform A/B test, integrating the learned user encoder into the deployed click-through-rate model yielded a 0.08 CTR gain and a 0.23% revenue-per-mille gain. If these results hold, offline-trained multi-task models of this kind could be a practical way to deploy budget-aware advertising policies without online exploration.

What carries the argument

The load-bearing component is the causal state encoder: a dilated causal temporal convolutional network that forces the hidden state at time $t$ to depend only on states at times $k \le t$, preserving temporal causality in the user sequence. Around this, the model stacks a causal attention module (masked self-attention in the style of a generative transformer) to emphasize relevant historical exposures, and two decoders: an action decoder that produces the channel policy, and a reward decoder that maps the causal states $\tilde{\mathbf{s}}_t$ to the penalized reward. The reward estimates feed the online advertising procedure, where they rank users (user-level budget allocation) and reweight channel policies in proportion to reliable conversion counts (channel-level allocation). A direct preference optimization loss over high-return versus low-return trajectories is added to the cross-entropy action loss and reward loss, giving the multi-task objective. The fixed penalty $s=0.5$ in the reward definition is what lets the budget constraint be folded into the reward signal.

What would settle it

On a held-out logged dataset, if the reward decoder's predictions become significantly more accurate when the actual channel of the next exposure is added to the causal-state input, the reward model is misspecified and the user-ranking signal is unjustified. In a live test, if users ranked in the top tier by predicted reward do not convert at a higher rate than randomly selected users with the same expected cost, the user-level budget allocation mechanism fails.

Watch

Extended reading notes

Core claim

The central claim is that the sparse-reward, budget-constrained advertising problem is better approached by supervised causal sequence modeling than by value-based offline reinforcement learning. Instead of estimating Q-values, MTORL treats the observed channel as the action label and the normalized penalized reward $r_t = \mathrm{MinMaxNorm}(g_t - s\,w_t)$ with $s=0.5$ as the reward label, and trains two decoders on top of a shared sequence encoder. The encoder first applies dilated causal convolutions to a user's exposure history to produce causal states $\tilde{\mathbf{s}}_t$ that depend only on past exposures, then applies causal attention to weigh relevant exposures. One decoder predicts the next channel; the other predicts the reward directly from the causal states, deliberately avoiding the two-hop path through predicted actions to prevent error accumulation. Those predicted rewards are then used to rank and filter target users and to reweight channels, turning budget allocation into an inference-time procedure rather than a separate optimization. The paper reports consistent gains over value-based offline RL and transformer sequence baselines on both benchmark datasets, and a live A/B test in which the encoder's user embeddings improved an existing CTR model. The intended conclusion is that a single multi-task offline RL model can form the core of an automated advertising pipeline.

Load-bearing premise

The result depends on the assumption that a user's next ad outcome can be predicted from her past exposure history alone, without knowing which channel will actually be served, and that the fixed cost penalty of 0.5 reflects the advertiser's true tradeoff.

Editorial extensions

If this is right

  • If MTORL's benchmark results hold, supervised causal sequence modeling can outperform value-based offline RL methods in reward-sparse advertising data.
  • Reward predictions can serve as a user-ranking signal, so channel recommendation and budget allocation can be learned jointly and executed at inference time under the budget.
  • The DPO loss provides a sparse-reward remedy that improves policy quality without explicit Q-value estimation.
  • The automated advertising procedure offers a deployment pattern in which offline pretraining, initial exploration with a blended policy, and rank-then-exploit serve for online use.
  • The reported online gains suggest that injecting an offline-pretrained user encoder into an existing CTR model can produce measurable lifts in click rate and revenue per mille.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The live experiment validates the learned user encoder inside an existing CTR model, but not the full channel-recommendation and budget-allocation loop; the end-to-end value of the complete MTORL pipeline remains an extrapolation.
  • The penalty term $s=0.5$ is taken as a constant without calibration; if an advertiser's true cost-gain tradeoff differs, the learned rewards would rank users incorrectly, so tuning $s$ against realized profit is a natural next test.
  • A direct ablation that feeds the actual next channel into the reward decoder would reveal whether the causal-state-only reward model is well specified; if accuracy improves substantially, the ranking signal should condition on the served channel.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes MTORL, a multi-task sequence-modeling method for online advertising. It formulates channel recommendation and budget allocation as a constrained MDP, encodes user journeys with a dilated causal convolution encoder and a causal attention module, and decodes channel actions and penalized rewards jointly. Training uses cross-entropy on logged actions, an auxiliary reward prediction loss, and a DPO loss. A deployment procedure ranks users by predicted reward and reweights channels by explicit and implicit CTR ratios to allocate budget. Experiments on KuaiRand-Pure and Criteo compare against DL, offline Q-learning, and Decision Transformer baselines on accuracy and average reward; an online A/B test on Taobao reports a 0.08 CTR and 0.23% RPM gain.

Significance. If the empirical claims are correct, MTORL provides a deployable, integrated solution for channel recommendation and budget allocation that outperforms a broad set of baselines on public benchmarks and in an online test. The paper's strengths are its released code, the breadth of baselines, the component ablation, and the inclusion of an online A/B experiment. However, the central validity of the budget-allocation claims depends on two assumptions that are not established: that rewards can be predicted without conditioning on the served channel, and that a fixed penalty strength s=0.5 enforces the budget constraint. The lack of standard deviations and a precise budget-allocation protocol also makes the reported average-reward superiority difficult to verify. These issues are fixable, but they require substantial additional analysis.

major comments (4)
  1. [3.5, Eq. (12); 3.7.2; Algorithm 1] The reward decoder in Eq. (12) predicts r_t from causal states \tilde{s}_t and deliberately omits the current action a_t, while the training reward in Eq. (3) is the realized outcome r_t = MinMaxNorm(g_t - s w_t) of the exposed channel. Consequently \hat{r}_t is an action-marginal estimate under the logging distribution, not an estimate of r(s,a). Algorithm 1 (line 6) ranks users by this marginal reward, and Section 3.7.2 uses it to compute per-channel implicit CTR ratios \hat{CTR}_j = \sum I(\hat{r}_t > \tau | a_j)/N_j; without action-conditioned reward predictions the condition '| a_j' cannot produce a channel-specific quantity. The reported average-reward gains in Table 3 therefore cannot be attributed to correct channel-level or user-level allocation. Please condition the reward decoder on the chosen action (or provide an ablation showing that the action-marginal reward suffices for ranking).
  2. [2.2.2, Eq. (2); 3.1.1, Eq. (3); Algorithm 1] The Lagrangian relaxation in Eq. (2) introduces a penalty strength s that should be adjusted to satisfy the budget constraint in Eq. (1), but Section 3.1.1 fixes s = 0.5 and no ablation or tuning of s is reported. As a result, the budget constraint is never enforced during policy optimization; Algorithm 1 only stops when W reaches zero at serving time. Please report how W and s are set, sweep s, and show constraint satisfaction (e.g., total cost versus W) to support the claim that MTORL performs budget-constrained optimization.
  3. [4.2, Tables 2 and 3] Table 2 claims 'All improvements are statistically significant (i.e., two-sided t-test with p < 0.05)', but no standard deviations, confidence intervals, or test statistics are reported anywhere. Table 3 reports average reward under 'the same budget' without specifying W, the top-N threshold, channel costs, or the exploration-phase budget; without this protocol the budget-fairness of the comparison cannot be verified. Please report means and variances over seeds and a complete budget-allocation evaluation protocol.
  4. [3.6, Eqs. (13)-(16)] The policy objective is cross-entropy against logged actions plus a DPO term; the model has no return-to-go conditioning, no value function, and no constraint-aware policy improvement. Thus MTORL is best described as a supervised sequence model with auxiliary reward prediction, and its comparison with CQL, IQL, and DT should be framed as such. Please either adopt an actual offline RL objective (e.g., return-conditioned or value-based) or moderate the claim that MTORL is an offline RL method.
minor comments (5)
  1. [3.6, Eq. (14)] The binary cross-entropy term should be (1-r_t)\log(1-\hat{r}_t); the printed (1-r_t)(1-\log(\hat{r}_t)) is incorrect.
  2. [3.7.2] The implicit budget ratio defines \hat{p}_j but uses p_j on the right-hand side; the notation should be made consistent.
  3. [3.6, Eq. (15)] The DPO loss omits the reference policy \pi_ref and does not explain how the preference pairs \{x_w, x_l\} are sampled; please clarify.
  4. [4.5] The online experiment reports +0.08 CTR and +0.23% RPM without confidence intervals or test details; the footnote that a 0.001-level gain is significant is not a statistical test. Also, the online system only uses the pretrained user encoder, not the full MTORL pipeline, so the online gain does not directly validate the budget-allocation modules.
  5. [4.1.1] The text says 'the video types and campaign categories in three datasets' but only two datasets are used; please correct the count.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: MTORL's claimed contributions are empirical and self-contained against public benchmarks and an independent online A/B test.

full rationale

The paper's derivation chain is not circular. The MDP/CMDP formulation (Eqs. 1–2) is standard; the reward definition r_t = MinMaxNorm(g_t − s w_t) in Eq. (3) is a preprocessing choice, not a result derived from the model. The action and reward decoders (Eqs. 9–12) are trained with supervised losses (Eqs. 13–14) on logged labels, and the DPO loss (Eq. 15) uses logged total rewards to form preference pairs; the model's outputs do not enter the training labels by construction. Algorithm 1 uses predicted rewards for user ranking and the predicted policy for channel recommendation, but the claimed superiority is evaluated against external benchmarks (KuaiRand-Pure, Criteo) and an independent online A/B test on Taobao (+0.08 CTR, +0.23% RPM). No load-bearing self-citation or imported uniqueness theorem is invoked; citations to the authors' prior work appear only as related work or standard building blocks. The skeptic's concern that the reward decoder in Eq. (12) omits the current action and therefore ranks users by an action-marginal quantity is a causal-identification or objective-mismatch concern, not a circularity: the predicted reward is not definitionally equal to the evaluation reward, and the paper does not state that Table 3's average reward is the model's own prediction. The unspecified evaluation protocol for Table 3 is a reproducibility gap rather than evidence of circularity, and no specific reduction to inputs can be exhibited.

Assumptions & free parameters 8 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical entity. Its 'causal state' is a learned representation, not a postulated entity. The central claim rests on a reward definition with fixed penalty strength, an unstated omission of the current action in reward prediction, and a heuristic budget-allocation procedure with several unreported parameters.

free parameters (8)
  • penalty strength s = 0.5 (default, not tuned)
    Defines the penalized reward r_t = MinMaxNorm(g_t - s*w_t) in Eq. (3). It sets the cost-gain trade-off for both training and evaluation; the paper defaults it to 0.5 citing [67,79] without testing alternatives.
  • DPO temperature beta = 0.1
    Used in Eq. (15) DPO-style preference loss; chosen as a default without sensitivity analysis in the main text.
  • loss weight mu = 0.08 on KuaiRand-Pure, 0.04 on Criteo
    Weights the reward learning auxiliary loss in Eq. (16); tuned on validation in Section 4.4.3.
  • loss weight lambda = 1.4
    Weights the DPO loss in Eq. (16); tuned in Section 4.4.2.
  • sequence length n = 20
    Truncation/padding length for user journeys; tuned in Section 4.4.1. Longer sequences hurt performance by adding noise.
  • allocation mixing coefficient alpha = not reported
    Mixes explicit CTR-based policy and implicit reward-threshold policy in Eq. (17) for the online initial policy; no value or tuning details are given.
  • reward threshold tau = not reported
    Used in the implicit CTR estimation \hat{CTR}_j = sum I(r_hat_t > tau | a_j) / N_j; no threshold value or selection procedure is specified.
  • top-N and weight-averaging factor eta = not reported
    Algorithm 1 uses top-N target users and eta for weight-averaging the policy; no values are specified, so the online advertising procedure is not fully reproducible.
assumptions (4)
  • domain assumption The logged user journeys from KuaiRand-Pure and Criteo constitute a valid offline RL dataset for advertising, with channels represented by video types and campaign categories.
    Section 4.1.1 treats video types and campaign categories as advertising channels following prior work, but this mapping may not reflect true advertising channel semantics, and the logged data may not cover the exploration needed for RL.
  • ad hoc to paper The advertising problem can be modeled as a CMDP with state s_t = Concat(q_t, f), action a_t = channel one-hot, and reward r_t = MinMaxNorm(g_t - s*w_t).
    Section 2.2 and Eq. (3). No verification is provided that this state definition is Markov or that the chosen reward matches the advertiser's true objective.
  • ad hoc to paper Reward at time t can be predicted from causal states \tilde{S}_t without conditioning on the action a_t chosen at time t.
    The reward decoder in Eq. (12) uses causal states produced from the sequence X_t = Concat(a_{t-1}, r_{t-1}, s_t), which excludes the current action. The paper does not justify this omission, and it is used for user-level allocation.
  • domain assumption The DPO-style loss in Eq. (15) provides a valid preference signal for maximizing cumulative reward in this offline setting.
    The loss treats sequences with larger summed logged reward as preferred and assumes the policy can be steered by pairwise action likelihood differences. This is a heuristic adaptation of DPO, not a proven objective for offline RL.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Multi-task Offline Reinforcement Learning for Online Advertising in Recommender Systems." pith.science (2026). https://pith.science/paper/3BHLKV43

@misc{pith2026250623090,
  author       = {Pith},
  title        = {Pith review of: Multi-task Offline Reinforcement Learning for Online Advertising in Recommender Systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3BHLKV43}},
  note         = {Machine review of arXiv:2506.23090}
}
read the original abstract

Online advertising in recommendation platforms has gained significant attention, with a predominant focus on channel recommendation and budget allocation strategies. However, current offline reinforcement learning (RL) methods face substantial challenges when applied to sparse advertising scenarios, primarily due to severe overestimation, distributional shifts, and overlooking budget constraints. To address these issues, we propose MTORL, a novel multi-task offline RL model that targets two key objectives. First, we establish a Markov Decision Process (MDP) framework specific to the nuances of advertising. Then, we develop a causal state encoder to capture dynamic user interests and temporal dependencies, facilitating offline RL through conditional sequence modeling. Causal attention mechanisms are introduced to enhance user sequence representations by identifying correlations among causal states. We employ multi-task learning to decode actions and rewards, simultaneously addressing channel recommendation and budget allocation. Notably, our framework includes an automated system for integrating these tasks into online advertising. Extensive experiments on offline and online environments demonstrate MTORL's superiority over state-of-the-art methods.

Figures

Figures reproduced from arXiv: 2506.23090 by the authors.

Figure 1
Figure 1. The overview of MTORL. Part (a) shows the network architecture, which inputs the historical journeys and outputs [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Impact of different components 4.3 Ablation Study (RQ2) To verify the contribution of each module in the main framework, we conduct the ablation study with four variants of MTORL, in￾cluding (1) w/o CausalState: without the causal state encoder, (2) w/o CausalAttn: without the causal attention module, (3)w/o Add&Norm: without the residual connection and layer normalization, and (4) w/o DPO: without the DPO loss. We … view at source ↗
Figure 3
Figure 3. Results of tuning parameters. 4.4.3 Tuning parameter 𝜇. Then, we consider the auxiliary task: tuning 𝜇 to find an optimal point that balances reward and action prediction performance. We tune 𝜇 in {0, 0.02, 0.04, · · · , 0.2} and record the Recall (of action prediction) and Accuracy (of reward prediction) of MTORL. The results are shown in [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

98 extracted references · 48 canonical work pages

  1. [1]

    Himan Abdollahpouri, Masoud Mansoury, Robin Burke, and Bamshad Mobasher

  2. [2]

    Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. 2017. Constrained policy optimization. In International conference on machine learning . PMLR, 22– 31

    2017

  3. [3]

    M Mehdi Afsar, Trafford Crump, and Behrouz Far. 2022. Reinforcement learning based recommender systems: A survey. Comput. Surveys 55, 7 (2022), 1–38

    2022

  4. [4]

    Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. 2020. An opti- mistic perspective on offline reinforcement learning. In International Conference on Machine Learning. PMLR, 104–114

    2020

  5. [5]

    Noga Alon, Iftah Gamzu, and Moshe Tennenholtz. 2012. Optimizing budget allocation among channels and influencers. InProceedings of the 21st international conference on World Wide Web. 381–388

    2012

  6. [6]

    Eitan Altman. 1999. Constrained Markov decision processes . Vol. 7. CRC press

    1999

  7. [7]

    Kai Arulkumaran, Marc Peter Deisenroth, Miles Brundage, and Anil Anthony Bharath. 2017. Deep reinforcement learning: A brief survey.IEEE Signal Processing Magazine 34, 6 (2017), 26–38

    2017

  8. [8]

    Vashist Avadhanula, Riccardo Colini Baldeschi, Stefano Leonardi, Karthik Abinav Sankararaman, and Okke Schrijvers. 2021. Stochastic bandits for multi-platform budget optimization in online advertising. In Proceedings of the Web Conference

    2021

Show all 98 references
  1. [9]

    Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normaliza- tion. arXiv preprint arXiv:1607.06450 (2016)

    2016 arXiv

  2. [10]

    Shaojie Bai, J Zico Kolter, and Vladlen Koltun. 2018. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.01271 (2018)

    2018 arXiv

  3. [11]

    Santiago Balseiro, Haihao Lu, and Vahab Mirrokni. 2020. Dual mirror descent for online allocation problems. In International Conference on Machine Learning . PMLR, 613–628

    2020

  4. [12]

    Qingpeng Cai, Zhenghai Xue, Chi Zhang, Wanqi Xue, Shuchang Liu, Ruohan Zhan, Xueliang Wang, Tianyou Zuo, Wentao Xie, Dong Zheng, et al. 2023. Two- Stage Constrained Actor-Critic for Short Video Recommendation. In Proceedings of the ACM Web Conference 2023 . 865–875

    2023

  5. [13]

    Tianchi Cai, Jiyan Jiang, Wenpeng Zhang, Shiji Zhou, Xierui Song, Li Yu, Lihong Gu, Xiaodong Zeng, Jinjie Gu, and Guannan Zhang. 2023. Marketing Budget Allocation with Offline Constrained Deep Reinforcement Learning. InProceedings of the Sixteenth ACM International Conference ...

    2023

  6. [14]

    Huiyuan Chen, Yusan Lin, Menghai Pan, Lan Wang, Chin-Chia Michael Yeh, Xiaoting Li, Yan Zheng, Fei Wang, and Hao Yang. 2022. Denoising self-attentive sequential recommendation. In Proceedings of the 16th ACM Conference on Rec- ommender Systems. 92–101

    2022

  7. [15]

    Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. 2021. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems 34 (2021), 15084–150...

    2021

  8. [16]

    Xiaocong Chen, Lina Yao, Julian McAuley, Guanglin Zhou, and Xianzhi Wang

  9. [17]

    Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, et al

  10. [18]

    Yinlam Chow, Ofir Nachum, Aleksandra Faust, Edgar Duenez-Guzman, and Mohammad Ghavamzadeh. 2019. Lyapunov-based safe policy optimization for continuous control. arXiv preprint arXiv:1901.10031 (2019)

    2019 arXiv

  11. [19]

    Peter J Danaher, Janghyuk Lee, and Laoucine Kerbache. 2010. Optimal internet media selection. Marketing Science 29, 2 (2010), 336–347

    2010

  12. [20]

    Evert De Haan, Thorsten Wiesel, and Koen Pauwels. 2016. The effectiveness of different forms of online advertising for purchase conversion in a multiple- channel attribution framework. International journal of research in marketing 33, 3 (2016), 491–507

    2016

  13. [21]

    Ming Fan, Subodha Kumar, and Andrew B Whinston. 2007. Selling or advertising: Strategies for providing digital media online. Journal of Management Information Systems 24, 3 (2007), 143–166

    2007

  14. [22]

    Chelsea Finn, Sergey Levine, and Pieter Abbeel. 2016. Guided cost learning: Deep inverse optimal control via policy optimization. In International conference on machine learning. PMLR, 49–58

    2016

  15. [23]

    Scott Fujimoto, David Meger, and Doina Precup. 2019. Off-policy deep rein- forcement learning without exploration. In International conference on machine learning. PMLR, 2052–2062

    2019

  16. [24]

    Jingtong Gao, Yewen Li, Shuai Mao, Peng Jiang, Nan Jiang, Yejing Wang, Qing- peng Cai, Fei Pan, Kun Gai, Bo An, et al. 2025. Generative Auto-Bidding with Value-Guided Explorations. arXiv preprint arXiv:2504.14587 (2025)

    2025

  17. [25]

    Yingqiang Ge, Shuchang Liu, Ruoyuan Gao, Yikun Xian, Yunqi Li, Xiangyu Zhao, Changhua Pei, Fei Sun, Junfeng Ge, Wenwu Ou, et al . 2021. Towards long- term fairness in recommendation. In Proceedings of the 14th ACM international conference on web search and data mining . 445–453

    2021

  18. [26]

    Qi Hao, Tianze Luo, and Guangda Huzhang. 2021. Re-ranking with constraints on diversified exposures for homepage recommender system. arXiv preprint arXiv:2112.07621 (2021)

    2021 arXiv

  19. [27]

    Xiaotian Hao, Zhaoqing Peng, Yi Ma, Guan Wang, Junqi Jin, Jianye Hao, Shan Chen, Rongquan Bai, Mingzhou Xie, Miao Xu, et al. 2020. Dynamic knapsack op- timization towards efficient multi-channel sequential advertising. InInternational Conference on Machine Learning . PMLR, 4060–4070

    2020

  20. [28]

    Elad Hazan et al. 2016. Introduction to online convex optimization. Foundations and Trends® in Optimization 2, 3-4 (2016), 157–325

    2016

  21. [29]

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition . 770–778

    2016

  22. [30]

    Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. 2018. Deep reinforcement learning that matters. In Proceedings of the AAAI conference on artificial intelligence , Vol. 32

    2018

  23. [31]

    Jonathan Ho and Stefano Ermon. 2016. Generative adversarial imitation learning. Advances in neural information processing systems 29 (2016)

    2016

  24. [32]

    Ahmed Hussein, Mohamed Medhat Gaber, Eyad Elyan, and Chrisina Jayne. 2017. Imitation learning: A survey of learning methods. ACM Computing Surveys (CSUR) 50, 2 (2017), 1–35

    2017

  25. [33]

    Wendi Ji and Xiaoling Wang. 2017. Additional multi-touch attribution for online advertising. InProceedings of the AAAI Conference on Artificial Intelligence, Vol. 31

    2017

  26. [34]

    Leslie Pack Kaelbling, Michael L Littman, and Andrew W Moore. 1996. Rein- forcement learning: A survey. Journal of artificial intelligence research 4 (1996), 237–285

    1996

  27. [35]

    Hyunjik Kim, George Papamakarios, and Andriy Mnih. 2021. The lipschitz constant of self-attention. InInternational Conference on Machine Learning. PMLR, 5562–5571

    2021

  28. [36]

    Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic opti- mization. arXiv preprint arXiv:1412.6980 (2014)

    2014 arXiv

  29. [37]

    Simon Kingsnorth. 2022. Digital marketing strategy: an integrated approach to online marketing. Kogan Page Publishers

    2022

  30. [38]

    Haruka Kiyohara, Kosuke Kawakami, and Yuta Saito. 2021. Accelerating offline reinforcement learning application in real-time bidding and recommendation: Potential use of simulation. arXiv preprint arXiv:2109.08331 (2021)

    2021 arXiv

  31. [39]

    Dmytro Korenkevych, Frank Cheng, Artsiom Balakir, Alex Nikulkov, Lingnan Gao, Zhihao Cen, Zuobing Xu, and Zheqing Zhu. 2024. Offline reinforcement learning for optimizing production bidding policies. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and D...

    2024

  32. [40]

    Ilya Kostrikov, Ashvin Nair, and Sergey Levine. 2021. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06169 (2021)

    2021 arXiv

  33. [41]

    Yueh-Ning Ku, Mikhail Kuznetsov, Shaunak Mishra, and Paloma de Juan. 2023. Staging e-commerce products for online advertising using retrieval assisted image generation. arXiv preprint arXiv:2307.15326 (2023)

    2023 arXiv

  34. [42]

    Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. 2019. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in Neural Information Processing Systems 32 (2019)

    2019

  35. [43]

    Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. 2020. Conserva- tive q-learning for offline reinforcement learning.Advances in Neural Information Processing Systems 33 (2020), 1179–1191

    2020

  36. [44]

    Sachin Kumar, Garima Gupta, Ranjitha Prasad, Arnab Chatterjee, Lovekesh Vig, and Gautam Shroff. 2020. Camta: Causal attention model for multi-touch attribution. In 2020 International Conference on Data Mining Workshops (ICDMW) . IEEE, 79–86

    2020

  37. [45]

    Kuang-chih Lee, Burkay Orten, Ali Dasdan, and Wentong Li. 2012. Estimating conversion rate in display advertising from past erformance data. In Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining. 768–776

    2012

  38. [46]

    Xiaopeng Li, Fan Yan, Xiangyu Zhao, Yichao Wang, Bo Chen, Huifeng Guo, and Ruiming Tang. 2023. Hamur: Hyper adapter for multi-domain recommendation. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management. 1268–1277

    2023

  39. [47]

    Guogang Liao, Ze Wang, Xiaoxu Wu, Xiaowen Shi, Chuheng Zhang, Yongkang Wang, Xingxing Wang, and Dong Wang. 2022. Cross dqn: Cross deep q network for ads allocation in feed. In Proceedings of the ACM Web Conference 2022 . 401– 409

    2022

  40. [48]

    Blerina Lika, Kostas Kolomvatsos, and Stathes Hadjiefthymiades. 2014. Facing the cold start problem in recommender systems. Expert systems with applications 41, 4 (2014), 2065–2073

    2014

  41. [49]

    Weilin Lin, Xiangyu Zhao, Yejing Wang, Yuanshao Zhu, and Wanyu Wang. 2023. Autodenoise: Automatic data instance denoising for recommendations. In Pro- ceedings of the ACM Web Conference 2023 . 1003–1011

    2023

  42. [50]

    Langming Liu, Liu Cai, Chi Zhang, Xiangyu Zhao, Jingtong Gao, Wanyu Wang, Yifu Lv, Wenqi Fan, Yiqi Wang, Ming He, et al. 2023. Linrec: Linear attention mechanism for long-term sequential recommender systems. In Proceedings of the 46th International ACM SIGIR Conference on Rese...

    2023

  43. [51]

    Tao Liu, Qi Xu, Wei Shi, Zhigang Hua, and Shuang Yang. 2025. Session-Level Dynamic Ad Load Optimization using Offline Robust Reinforcement Learning. arXiv preprint arXiv:2501.05591 (2025)

    2025 arXiv

  44. [52]

    Weiwen Liu, Yunjia Xi, Jiarui Qin, Fei Sun, Bo Chen, Weinan Zhang, Rui Zhang, and Ruiming Tang. 2022. Neural re-ranking in multi-stage recommender systems: A review. arXiv preprint arXiv:2202.06602 (2022)

    2022 arXiv

  45. [53]

    Yongshuai Liu, Avishai Halev, and Xin Liu. 2021. Policy learning with constraints in model-free reinforcement learning: A survey. In The 30th International Joint Conference on Artificial Intelligence (IJCAI)

    2021

  46. [54]

    Ziru Liu, Shuchang Liu, Bin Yang, Zhenghai Xue, Qingpeng Cai, Xiangyu Zhao, Zijian Zhang, Lantao Hu, Han Li, and Peng Jiang. 2024. Modeling User Retention through Generative Flow Networks. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining ...

    2024

  47. [55]

    Ziru Liu, Jiejie Tian, Qingpeng Cai, Xiangyu Zhao, Jingtong Gao, Shuchang Liu, Dayou Chen, Tonghao He, Dong Zheng, Peng Jiang, et al. 2023. Multi-task recommendations with reinforcement learning. In Proceedings of the ACM web conference 2023. 1273–1282

    2023

  48. [56]

    Andrew L Maas, Awni Y Hannun, Andrew Y Ng, et al. 2013. Rectifier nonlineari- ties improve neural network acoustic models. In Proc. icml, Vol. 30. Atlanta, GA, 3

    2013

  49. [57]

    Joseph Naor and David Wajc. 2018. Near-optimum online ad allocation for targeted advertising. ACM Transactions on Economics and Computation (TEAC) 6, 3-4 (2018), 1–20

    2018

  50. [58]

    Junwei Pan, Yizhi Mao, Alfonso Lobos Ruiz, Yu Sun, and Aaron Flores. 2019. Predicting different types of conversions with multi-task learning in online advertising. In Proceedings of the 25th acm sigkdd international conference on knowledge discovery & data mining . 2689–2697

    2019

  51. [59]

    Yuqi Qin, Pengfei Wang, and Chenliang Li. 2021. The world is binary: Contrastive learning for denoising next basket recommendation. In Proceedings of the 44th international ACM SIGIR conference on research and development in information retrieval. 859–868

    2021

  52. [60]

    Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al . 2018. Improving language understanding by generative pre-training. (2018)

    2018

  53. [61]

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems 36 (2024)

    2024

  54. [62]

    Kan Ren, Yuchen Fang, Weinan Zhang, Shuhao Liu, Jiajun Li, Ya Zhang, Yong Yu, and Jun Wang. 2018. Learning multi-touch conversion attribution with dual- attention mechanisms for online advertising. In Proceedings of the 27th ACM International Conference on Information and Know...

    2018

  55. [63]

    Tim Salimans and Durk P Kingma. 2016. Weight normalization: A simple repa- rameterization to accelerate training of deep neural networks.Advances in neural information processing systems 29 (2016)

    2016

  56. [64]

    Xiang-Rong Sheng, Liqin Zhao, Guorui Zhou, Xinyao Ding, Binding Dai, Qiang Luo, Siran Yang, Jingshan Lv, Chi Zhang, Hongbo Deng, et al. 2021. One model to serve all: Star topology adaptive recommender for multi-domain ctr prediction. In Multi-task Offline Reinforcement Learnin...

    2021

  57. [65]

    Jing-Cheng Shi, Yang Yu, Qing Da, Shi-Yong Chen, and An-Xiang Zeng. 2019. Virtual-taobao: Virtualizing real-world online retail environment for reinforce- ment learning. In Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 33. 4902–4909

    2019

  58. [66]

    Jie Tang, Sen Wu, Jimeng Sun, and Hang Su. 2012. Cross-domain collaboration recommendation. In Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining . 1285–1293

    2012

  59. [67]

    Chen Tessler, Daniel J Mankowitz, and Shie Mannor. 2018. Reward constrained policy optimization. arXiv preprint arXiv:1805.11074 (2018)

    2018 arXiv

  60. [68]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017)

    2017

  61. [69]

    Siyu Wang, Xiaocong Chen, Dietmar Jannach, and Lina Yao. 2023. Causal decision transformer for recommender systems via offline reinforcement learning. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval . 1599–1608

    2023

  62. [70]

    Yuhao Wang, Xiangyu Zhao, Bo Chen, Qidong Liu, Huifeng Guo, Huanshuo Liu, Yichao Wang, Rui Zhang, and Ruiming Tang. 2023. PLATE: A prompt-enhanced paradigm for multi-scenario recommendations. In Proceedings of the 46th In- ternational ACM SIGIR Conference on Research and Devel...

    2023

  63. [71]

    Marco A Wiering and Martijn Van Otterlo. 2012. Reinforcement learning. Adap- tation, learning, and optimization 12, 3 (2012), 729

    2012

  64. [72]

    Shuai Xiao, Le Guo, Zaifan Jiang, Lei Lv, Yuanbo Chen, Jun Zhu, and Shuang Yang. 2019. Model-based constrained MDP for budget allocation in sequential incentive marketing. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management . 971–980

    2019

  65. [73]

    Yue Xu, Qijie Shen, Jianwen Yin, Zengde Deng, Dimin Wang, Hao Chen, Lixi- ang Lai, Tao Zhuang, and Junfeng Ge. 2023. Multi-channel Integrated Recom- mendation with Exposure Constraints. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining . 5338–5349

    2023

  66. [74]

    Di Yao, Chang Gong, Lei Zhang, Sheng Chen, and Jingping Bi. 2022. CausalMTA: Eliminating the User Confounding Bias for Causal Multi-touch Attribution. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 4342–4352

    2022

  67. [75]

    Daniel Zantedeschi, Eleanor McDonnell Feit, and Eric T Bradlow. 2017. Measuring multichannel advertising response. Management Science 63, 8 (2017), 2706–2728

    2017

  68. [76]

    Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. 2023. Are transformers effective for time series forecasting?. In Proceedings of the AAAI conference on artificial intelligence, Vol. 37. 11121–11128

    2023

  69. [77]

    Weinan Zhang, Xiangyu Zhao, Li Zhao, Dawei Yin, Grace Hui Yang, and Alex Beutel. 2020. Deep reinforcement learning for information retrieval: Fundamen- tals and advances. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information R...

    2020

  70. [78]

    Xubing Zhang. 2009. Retailers’ multichannel and price advertising strategies. Marketing Science 28, 6 (2009), 1080–1094

    2009

  71. [79]

    Yang Zhang, Bo Tang, Qingyu Yang, Dou An, Hongyin Tang, Chenyang Xi, Xueying Li, and Feiyu Xiong. 2021. BCORLE ( 𝜆): An Offline Reinforcement Learning and Evaluation Framework for Coupons Allocation in E-commerce Market. Advances in Neural Information Processing Systems 34 (20...

    2021

  72. [80]

    Jun Zhao, Guang Qiu, Ziyu Guan, Wei Zhao, and Xiaofei He. 2018. Deep rein- forcement learning for sponsored search real-time bidding. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining . 1021–1030

    2018

  73. [81]

    Kesen Zhao, Lixin Zou, Xiangyu Zhao, Maolin Wang, and Dawei Yin. 2023. User retention-oriented recommendation with decision transformer. In Proceedings of the ACM Web Conference 2023. 1141–1149

    2023

  74. [82]

    Xiangyu Zhao. 2022. Adaptive and automated deep recommender systems. ACM SIGWEB Newsletter 2022, Spring (2022), 1–4

    2022

  75. [83]

    Xiangyu Zhao, Changsheng Gu, Haoshenglun Zhang, Xiwang Yang, Xiaobing Liu, Jiliang Tang, and Hui Liu. 2021. Dear: Deep reinforcement learning for online advertising impression in recommender systems. In Proceedings of the AAAI conference on artificial intelligence , Vol. 35. 750–758

    2021

  76. [84]

    Deep reinforce- ment learning for search, recommendation, and online advertising: a survey

    Xiangyu Zhao, Long Xia, Jiliang Tang, and Dawei Yin. 2019. " Deep reinforce- ment learning for search, recommendation, and online advertising: a survey" by Xiangyu Zhao, Long Xia, Jiliang Tang, and Dawei Yin with Martin Vesely as coordinator. ACM sigweb newsletter 2019, Spring...

    2019

  77. [85]

    Xiangyu Zhao, Long Xia, Liang Zhang, Zhuoye Ding, Dawei Yin, and Jiliang Tang. 2018. Deep reinforcement learning for page-wise recommendations. In Proceedings of the 12th ACM conference on recommender systems . 95–103

    2018

  78. [86]

    Xiangyu Zhao, Long Xia, Lixin Zou, Hui Liu, Dawei Yin, and Jiliang Tang. 2020. Whole-chain recommendations. In Proceedings of the 29th ACM international conference on information & knowledge management . 1883–1891

    2020

  79. [87]

    Xiangyu Zhao, Long Xia, Lixin Zou, Hui Liu, Dawei Yin, and Jiliang Tang. 2021. Usersim: User simulation via supervised generativeadversarial network. In Pro- ceedings of the Web Conference 2021 . 3582–3589

    2021

  80. [88]

    Xiangyu Zhao, Liang Zhang, Zhuoye Ding, Long Xia, Jiliang Tang, and Dawei Yin

  81. [89]

    Xiangyu Zhao, Liang Zhang, Long Xia, Zhuoye Ding, Dawei Yin, and Jiliang Tang. 2017. Deep reinforcement learning for list-wise recommendations. arXiv preprint arXiv:1801.00209 (2017)

    2017 arXiv

  82. [90]

    Xiangyu Zhao, Xudong Zheng, Xiwang Yang, Xiaobing Liu, and Jiliang Tang

  83. [91]

    Qinqing Zheng, Amy Zhang, and Aditya Grover. 2022. Online decision trans- former. In international conference on machine learning . PMLR, 27042–27059

    2022

  84. [92]

    Guorui Zhou, Xiaoqiang Zhu, Chenru Song, Ying Fan, Han Zhu, Xiao Ma, Yanghui Yan, Junqi Jin, Han Li, and Kun Gai. 2018. Deep interest network for click-through rate prediction. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining ...

    2018

  85. [93]

    Hao Zhou, Shaoming Li, Guibin Jiang, Jiaqi Zheng, and Dong Wang. 2023. Direct heterogeneous causal learning for resource allocation problems in marketing. In Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 37. 5446–5454. A Reward Construction A.1 Embedding...

    2023

  86. [2016]

    In Proceedings of the 1st workshop on deep learning for recommender systems

    Wide & deep learning for recommender systems. In Proceedings of the 1st workshop on deep learning for recommender systems . 7–10

  87. [2018]

    In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining

    Recommendations with negative feedback via pairwise deep reinforcement learning. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining . 1040–1048

  88. [2019]

    arXiv preprint arXiv:1907.13286 (2019)

    The unfairness of popularity bias in recommendation. arXiv preprint arXiv:1907.13286 (2019)

    2019 arXiv

  89. [2020]

    In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining

    Jointly learning to recommend and advertise. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining . 3319–3327

  90. [2021]

    arXiv preprint arXiv:2109.03540 (2021)

    A survey of deep reinforcement learning in recommender systems: A systematic review and future directions. arXiv preprint arXiv:2109.03540 (2021)

    2021 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.