Contrastive Representation for Interactive Recommendation

Dongxiao He; Guoli Wu; Hongqi Chen; Jingyu Li; Qinghang Gao; Zhiyong Feng

REVIEW 3 major objections 6 minor 60 references

Contrastive Representation for Interactive Recommendation

T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read An auxiliary preference-ranking contrastive loss lets a deep-reinforcement-learning recommender find a good cold-start policy by episode 8,000, where baselines lag past 20,000.

desk verdict A plausible contrastive trick for DRL recommenders, but the derivation is broken and the contrastive labels are partly self-confirming. read the letter →

arxiv 2412.18396 v3 pith:T3FDAEHD submitted 2024-12-24 cs.IR

Jingyu Li , Zhiyong Feng , Dongxiao He , Hongqi Chen , Qinghang Gao , Guoli Wu This is my paper · ORCID

classification cs.IR

keywords interactiverecommendationsampleefficiencydeepreinforcementlearningcontrastivepreferencerankingstaterepresentationcold-startInfoNCEloss

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that sample inefficiency in deep-reinforcement-learning-based interactive recommendation can be attacked at the representation level: instead of letting the RL loss alone shape user state representations, the agent runs an auxiliary contrastive task that pulls together behaviors the user paid most attention to and pushes apart behaviors the user cared less about. The proposed method, CRIR, uses the attention weights already computed by its state representation network to rank a user's past behaviors, then optimizes a Positional Weighted InfoNCE loss over pairs drawn from that ranking. The claim is a practical payoff: in cold-start Virtual-Taobao, CRIR reaches a good policy at roughly episode 8,000 while comparison methods have not reached that level within 20,000 episodes, and it also outperforms baselines on an ML-1M-based simulator with shifting interests. A careful reader would care because the fix is a cheap add-on, one auxiliary loss on top of an existing DRL agent, that could cut the online interaction budget before a recommender becomes useful.

What carries the argument

The load-bearing object is the interest-weight ranking produced by the attention-based state representation network, together with the PRCL loss built on it. For a user $u$ at time $t$, the network scores every past behavior $h_\tau$ with an activation unit $\Lambda(u_t, h_\tau)$; the scores serve two purposes at once, they form the weighted-sum part of the state representation in Eq. (1), and they define the contrastive pairs: the highest-scoring behavior stands in for the state, a randomly chosen behavior from the next $\lfloor n/2 \rfloor$ positions is the positive, and behaviors ranked below that are negatives. The Positional Weighted InfoNCE loss is the machinery's objective, with weight $1/\sqrt{R_u(h_k)}$ so that pairs closer to the top of the preference ranking contribute more. This design is what lets PRCL avoid computing over the full potential action set: the negative set comes from the user's own low-ranked history rather than from all candidate items.

What would settle it

Run the cold-start Virtual-Taobao experiment with the same CRIR code but permute the interest-weight rankings before building contrastive pairs, keeping every other detail, the loss, the weighting, the sampling, identical. If the roughly 8,000-episode lead over baselines survives the permutation, preference-ranking contrastive learning is not the operative cause of the sample-efficiency gain; if the lead disappears, the ranking mechanism is load-bearing.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that explicit interaction contains a latent preference ranking that can be extracted and turned into a contrastive objective without touching the large action space. The state representation network computes an interest weight $w_\tau$ for each behavior $h_\tau$ in the user's history through an attention activation unit; PRCL then ranks behaviors by these weights, takes the top-ranked behavior $h^* = h_i$ with $i = \arg\max_i w_i$ as a proxy for the user state, samples a positive from the upper half of the ranking and negatives from the lower half, and optimizes the Positional Weighted InfoNCE loss $-\frac{1}{\sqrt{R_u(h_k)}} \log \frac{\exp(h_k^\top h^*)}{\sum_{n} \exp(h_n^\top h^*)}$. The paper argues this works because the ranking is computed from the user's own history at each timestamp, so positive and negative pairs come for free, and because the auxiliary task is run separately from the RL update with a mixed sampling mechanism that also passes the DRL batch through PRCL. The empirical claim is that this representation-level addition, not any change to the RL algorithm, produces the sample-efficiency lead.

Load-bearing premise

The contrastive signal rests on the assumption that the single behavior with the highest attention weight faithfully represents the user's current state, and that lower-attended behaviors from the user's own history are valid negative examples for that state.

Editorial extensions

If this is right

Cold-start DRL recommenders can become usable with far fewer online interactions: on Virtual-Taobao, CRIR finds a good policy around episode 8,000 while the comparison methods have not reached that level within 20,000 episodes.
The same state representation network can serve both the RL state and the contrastive objective, so the sample-efficiency gain comes without a separate encoder or a large action-set computation.
The gain transfers beyond the main simulator: CRIR also outperforms baselines on the ML-1M-based simulator with shifting interest features, which suggests the mechanism is not tailored to Virtual-Taobao.
Increasing the frequency of the PRCL update improves sample efficiency, but with diminishing returns, giving a practical dial for trading compute against convergence speed.
The appendix's gradient comparison indicates that representation quality, not RL gradient magnitude, is what drives the early improvement: training the encoder with PRCL alone can keep up with, and eventually exceed, joint RL-plus-PRCL training.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper does not test what happens when the top-attention behavior is a poor state proxy; a natural extension would replace the single $h^*$ with a small weighted mixture of the top-ranked behaviors and compare sample efficiency, since the loss would stay almost unchanged.
The success of ranking-based positives and negatives suggests the recipe could transfer to other sequential interaction domains, dialogue, search, or control, wherever an attention-like score already orders past observations, not just to item recommendation.
Because the headline numbers come from simulators with synthetic rewards, the magnitude of the gain in live systems is an open question; a field deployment with noisy, delayed rewards would be the test that determines whether the 8,000-versus-20,000 gap survives.
The appendix's result that PRCL gradients do not disturb RL gradients implies a cheaper training scheme the paper leaves implicit: freeze the encoder during RL updates and update it only through PRCL, which the 'only PRCL' curve suggests should match or beat joint training.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Two things to know before you read this one. The empirical picture is stronger than the math, and the contrastive labels are generated by the same network they are supposed to improve. The core idea—use the attention weights in a DIN-style state encoder to rank behaviors, then run a weighted InfoNCE loss on the top versus bottom halves—is a reasonable way to add a cheap auxiliary task to a DRL recommender. That specific combination looks new, and the paper gives credit to CURL, ICL, and DIN appropriately.

What the paper does well is the experiments. Cold-start training on Virtual-Taobao and an ML-1M simulator, 95% confidence intervals from five runs, ablations on PRCL frequency, the positional coefficient, the sampling mechanism, and the auxiliary versus constrained training regime. The result is consistent: CRIR learns faster than the baselines, and the ablations mostly tell a coherent story about which pieces matter. Appendix D's gradient study is a nice extra, showing that the contrastive task alone can drive representation learning and that RL gradients can even interfere. That is the kind of evidence that makes the paper worth a look.

Now the soft spots, in order of severity. First, the derivation in Appendix B is not valid. Equation (18) is a squared-distance Gaussian kernel, and Equation (19) is a linear softmax. No equality, inequality, or approximation is supplied; they are not equivalent for any constant W. The paper claims PRCL optimizes the IR objective, but the chain from Eq. (3) to the deployed loss is broken. Second, the contrastive pairs come from the network's own interest weights. The positive pair is the max-attention behavior and a randomly chosen high-attention behavior; negatives are low-attention behaviors. There is no external check that max attention is a good proxy for user state, or that low attention means a true negative. A user can like an item without it being the top attention item at that moment. So the auxiliary task may just be making the encoder agree with its own attention—a smoothing or regularization effect—rather than learning preference ranking. The ablations vary the coefficient and sampling mechanism but never the label source, so the observed gain could be generic embedding smoothing. That is a real gap.

There are minor issues too: no code or data, the ML-1M simulator is only pseudo-coded and a bit ambiguous, and the text is riddled with typos and notation slips. None of those are fatal on their own.

Who should read this? People working on sample-efficient RL-based recommender systems, especially those interested in auxiliary self-supervised tasks. It is not a foundational result, but it is a plausible recipe with unusually thorough ablations for the subfield. The paper deserves a serious referee, but it needs major revision: fix or reframe the derivation, release the code and simulator details, and add an experiment that varies the label source (for example, using oracle user-satisfaction labels or a held-out validation of whether the contrastive pairs correlate with future reward).

Referee Report

3 major / 6 minor

Summary. The paper proposes Contrastive Representation for Interactive Recommendation (CRIR), an auxiliary representation-learning method for deep reinforcement learning based interactive recommender systems. CRIR couples a Deep Interest Network style state representation with a Preference Ranking Contrastive Learning (PRCL) loss. PRCL ranks the user's historical behaviors by the attention weights produced by the representation network, treats the top-ranked behavior as an anchor, and applies a position-weighted InfoNCE-style loss to pull higher-ranked behaviors closer to the anchor and push lower-ranked behaviors away. The authors claim this extracts latent preference-ranking features and improves sample efficiency without computing over the full action set. Experiments are run in cold-start settings on Virtual-Taobao and an ML-1M based simulator, with cumulative reward and CTR as metrics, and include ablations over PRCL frequency, the ranking coefficient, sampling mechanism, and training mechanism, plus a gradient-based study in Appendix D.

Significance. If the central claims hold, the paper offers a practically relevant recipe for improving sample efficiency of DRL-based interactive recommenders: an auxiliary, separately optimized contrastive objective that avoids expensive computation over large action sets. The empirical core is present: multiple baselines, two environments, 5 seeds, 95% confidence intervals, and ablations that isolate the PRCL frequency, the ranking coefficient, the sampling mechanism, and the auxiliary training mechanism. The gradient study in Appendix D is also a useful attempt to understand how the auxiliary loss interacts with RL gradients. However, the theoretical derivation of PRCL from the stated IR objective is not valid as written, and the contrastive labels are generated by the very network being trained, so the paper's conceptual claim that PRCL extracts latent preference features is not yet established.

major comments (3)

[Appendix B, Eqs. (18)-(19)] The derivation connecting the IR objective to PRCL is not valid. Eq. (18) defines the action distribution with a squared-distance Gaussian kernel, exp(-(a_{t,k}^T W_a - s_t^T W_s)^2), while Eq. (19) is a linear softmax, exp(a_{t,k}^T W s_t). No equality, inequality, or approximation is supplied that would justify replacing one with the other; a negative squared distance cannot be re-expressed as a linear dot product with a fixed matrix W. Since the paper states that PRCL optimizes Eq. (3)/(15), this is a load-bearing gap: as written, the deployed loss is not derived from the stated objective. In addition, the lower bound in Eq. (13) requires positive rewards, whereas the ML-1M simulator in Appendix C returns rewards in [-1,1].
[Section 3, Eq. (6) and Appendix B, Eq. (21)] The position weight 1/sqrt(R_u(h_k)) is introduced as a way to model the P(s_t) term in Eq. (16), but no derivation connects the ranking position of a behavior to the state distribution term. The text simply states that the ranking position 'can measure the importance of the contrastive pair' and that the coefficient is used 'to smooth the discrimination.' This coefficient is therefore an ad hoc weighting scheme rather than a consequence of the stated objective. The paper should either derive this weighting from a concrete probabilistic model or explicitly present it as a design choice, and it should be tested against alternative rank-weighting functions rather than only against a constant coefficient.
[Section 3, 'Ranking' and Eq. (5)-(6)] The contrastive labels are self-referential: the anchor h* = argmax_i w_i, the positive behavior h_k, and the negative set N_s_t are all determined by the attention weights w_i produced by the same state-representation network whose embeddings PRCL updates. The paper assumes without validation that the highest-attention behavior is a faithful proxy for the user state and that lower-attention behaviors from the user's own history are valid negatives. Low attention at a given moment does not imply absence of preference, and a behavior that is not the top-ranked item can still be positively preferred. The ablation in Figure 4(a) varies only the coefficient strategy, not the source of the labels or the split threshold floor(n/2). An experiment that replaces the attention-derived labels with alternative label sources (for example, the next interacted item as positive, or a random historical item as anchor) would directly test whether the observed gains come from extracting preference structure or from training the network to be consistent with its own attention weights.

minor comments (6)

[Section 3, Eq. (4)] The projection matrix W in Eq. (4) is used before it is defined; please define it at first use and clarify its dimensions relative to W_a and W_s in Appendix B.
[Appendix B, Eq. (17)] The notation u_t^T ⊗ h_tau in Eq. (17) is confusing: Eq. (1) uses u_t ⊗ h_tau for the outer product. Please unify the notation.
[Section 3, Eq. (6)] The denominator sum notation P^{N_s_t}_n exp(h_n^T h*) is ambiguous; please write the sum limits explicitly, e.g., sum_{n=1}^{|N_s_t|}.
[Section 1] There is a typo: 'an reasonable recommender workflow' should be 'a reasonable recommender workflow.'
[Section 2, References] The reference 'Ie and other 2019' is malformed and should be corrected to a standard author list citation.
[Appendix D, Figure 5] The claim that the state representation network 'doesn't need to be trained or fine-tuned by RL losses' is stronger than what the single environment in Figure 5 can support; please soften the wording or add a caveat about generalizability.

Circularity Check

1 steps flagged · score 6.0 of 10

PRCL's positive/negative pairs are generated from the same attention weights the loss trains, making the claimed extraction of preference ranking self-confirming; the sample-efficiency result itself is externally benchmarked.

self definitional [Section 3, 'Preference Ranking Contrastive Learning', Data Augmentation (iii) Ranking; Appendix B Eq. (20)-(21)]
"One single behavior with larger weight value means that the current user is predicted to pay more attention to the item in this behavior. These weights plays critical roles at the following (iii) Ranking step and Positional Weight InfoNCE Loss. ... In every interaction, the attention scores ranking the second to the the ⌊n/2⌋-th will be treated as candidate positive items. ... Every tuple like (h1, ht) where t ∈ {⌊n/2⌋+ 1, · · ·, n} is treated as negative pairs for this interaction."

The anchor h* = argmax_i w_i, the positive h_k, and the negatives are all selected by ranking the interest weights w_i output by the state-representation network whose embeddings PRCL updates (Eq. 21). The 'preference ranking' that PRCL claims to extract is therefore defined as the network's own current attention ranking. The loss only pulls the top-attention behavior closer to the second-ranked behavior and pushes lower-ranked behaviors away, so it enforces self-consistency of the network's scores rather than measuring any independent preference signal from the interaction. No external label (reward, rating, or held-out preference) enters the pair construction, so the claimed prediction of latent preference ranking reduces by construction to the network's own fitted weights.

full rationale

The headline sample-efficiency claim has independent empirical content: CRIR is compared against SAC, CRR, PPO, DRR, and NICF in cold-start Virtual-Taobao and ML-1M simulators, with ablations varying the PRCL frequency, coefficient strategy, sampling mechanism, and training mechanism. Those curves are not fitted to the reported rewards by the same constants, so the efficiency gain is not circular. The circular part is the method's self-description: PRCL purports to extract 'latent, high-level preference ranking features', but its positive/negative pairs are generated by ranking the network's own attention weights, and the loss trains that same network. The preference ranking is thus self-definitional; the auxiliary task can sharpen the model's internal scores without providing evidence that those scores represent true user preference. I do not score the Eq. (18) to Eq. (19) step as circularity: the Gaussian squared-distance form and the linear softmax are not equivalent, but that is a derivation gap, not an input/output reduction. No load-bearing self-citation appears; the only author-overlapping reference (Chen et al. 2022a) is used as related work. Overall, the empirical claim survives, but the central representational claim is partially circular, giving 6.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The central method rests on several hand-chosen design choices (split threshold, positional weight exponent, PRCL frequency) and on unvalidated modeling assumptions about how behaviors proxy states and actions. No new physical or latent entity is introduced.

free parameters (4)

positive/negative split threshold = floor(n/2)
Interactions ranked by interest weight; positions 2..floor(n/2) are positives and positions floor(n/2)+1..n are negatives. The split point is chosen by hand and determines all contrastive pairs.
positional weight function = 1/sqrt(Ru(hk))
The discrimination coefficient in Eq (6) is fixed as 1 over the square root of the ranking position. The exponent (1/2) is chosen by hand; no sensitivity analysis is reported.
PRCL frequency for main experiments = not specified
The paper studies frequencies {0, 0.25, 0.5, 0.75, 1.0} in Figure 3(e-f) but does not state which frequency is used for the main CRIR results in Figure 3(a-d).
max sequence length M = 50
The maximum interaction history length for state representation and the ML-1M simulator is set to 50, which controls the number of negatives in PRCL.

assumptions (6)

domain assumption DRL Representation Consensus: high-quality semantic representations lead to more sample-efficient DRL training.
Stated in Section 1 as the basis of the approach; the paper provides a supporting experiment in Appendix D but it is not a proof.
domain assumption The lower bound in Eq (13) can be split so that the ln P term can be optimized independently of the reward term (Eq 14).
Appendix B: the paper drops Eq (14) assuming the main DRL task covers it. No analysis shows this decomposition does not hurt the bound.
domain assumption The behavior with the largest interest weight h* represents the state s_t.
Appendix B, used to replace state with h* in the loss; no empirical verification of this representational equivalence.
domain assumption Negative behaviors from the user's own history can replace the full potential action set.
Appendix B, the paper calls this negative sampling; the bias introduced by using only user history items as negatives is not analyzed.
domain assumption The conditional action distribution P(a|s) is Gaussian around the state in a shared projection space.
Appendix B, Eq (18); this assumption is used to derive the InfoNCE form but is stated without justification.
ad hoc to paper The coefficient 1/sqrt(Ru) captures the importance of the state distribution term ln P(s_t).
Appendix B, added to the loss to model the dropped ln P(s_t) term; the functional form is asserted, not derived.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Contrastive Representation for Interactive Recommendation." pith.science (2026). https://pith.science/paper/T3FDAEHD

@misc{pith2026241218396,
  author       = {Pith},
  title        = {Pith review of: Contrastive Representation for Interactive Recommendation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/T3FDAEHD}},
  note         = {Machine review of arXiv:2412.18396}
}

read the original abstract

Interactive Recommendation (IR) has gained significant attention recently for its capability to quickly capture dynamic interest and optimize both short and long term objectives. IR agents are typically implemented through Deep Reinforcement Learning (DRL), because DRL is inherently compatible with the dynamic nature of IR. However, DRL is currently not perfect for IR. Due to the large action space and sample inefficiency problem, training DRL recommender agents is challenging. The key point is that useful features cannot be extracted as high-quality representations for the recommender agent to optimize its policy. To tackle this problem, we propose Contrastive Representation for Interactive Recommendation (CRIR). CRIR efficiently extracts latent, high-level preference ranking features from explicit interaction, and leverages the features to enhance users' representation. Specifically, the CRIR provides representation through one representation network, and refines it through our proposed Preference Ranking Contrastive Learning (PRCL). The key insight of PRCL is that it can perform contrastive learning without relying on computations involving high-level representations or large potential action sets. Furthermore, we also propose a data exploiting mechanism and an agent training mechanism to better adapt CRIR to the DRL backbone. Extensive experiments have been carried out to show our method's superior improvement on the sample efficiency while training an DRL-based IR agent.

Figures

Figures reproduced from arXiv: 2412.18396 by the authors.

**Figure 2.** Weighted sum part of the state representation net [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Performance for the proposed, ablation and baseline methods in cold-start setting. Each curve in the graph is repeated [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Study on coefficient strategy, data sampling and agent training mechanism of PRCL. Each curve is repeated for 5 [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

**Figure 5.** Figure 5: Strategies for different gradient updating policy in PRCL and gradient study. Logarithmic axis is used in (b) and (c) [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

60 extracted references · 43 canonical work pages

[1]

, " * write output.state after.block = add.period write newline

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Andrychowicz, M.; Wolski, F.; Ray, A.; Schneider, J.; Fong, R.; Welinder, P.; McGrew, B.; Tobin, J.; Pieter Abbeel, O.; and Zaremba, W. 2017. Hindsight experience replay. Advances in neural information processing systems, 30

2017
[4]

Cai, Q.; Liu, S.; Wang, X.; Zuo, T.; Xie, W.; Yang, B.; Zheng, D.; Jiang, P.; and Gai, K. 2023 a . Reinforcing User Retention in a Billion Scale Short Video Recommender System. In Companion Proceedings of the ACM Web Conference 2023, WWW '23 Companion, 421–426

work page 2023
[5]

Cai, T.; Bao, S.; Jiang, J.; Zhou, S.; Zhang, W.; Gu, L.; Gu, J.; and Zhang, G. 2023 b . Model-Free Reinforcement Learning with Stochastic Reward Stabilization for Recommender Systems. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '23, 2179–2183

work page 2023
[6]

Cai, X.; Huang, C.; Xia, L.; and Ren, X. 2023 c . LightGCL: Simple Yet Effective Graph Contrastive Learning for Recommendation. In The Eleventh International Conference on Learning Representations

work page 2023
[7]

Chen, H.; Dai, X.; Cai, H.; Zhang, W.; Wang, X.; Tang, R.; Zhang, Y.; and Yu, Y. 2019 a . Large-scale interactive recommendation with tree-structured policy gradient. In Proceedings of the AAAI conference on artificial intelligence, volume 33, 3312--3320

work page 2019
[8]

Chen, H.; Feng, Z.; Chen, S.; Xue, X.; Wu, H.; Sun, Y.; Xu, Y.; and Han, G. 2022 a . Capturing Users’ Fresh Interests via Evolving Session-Based Social Recommendation. In 2022 IEEE International Conference on Web Services (ICWS), 182--187

work page 2022

Show all 60 references

[9]

Chen, H.; Zhu, C.; Tang, R.; Zhang, W.; He, X.; and Yu, Y. 2023. Large-Scale Interactive Recommendation With Tree-Structured Reinforcement Learning. IEEE Transactions on Knowledge and Data Engineering, 35(4): 4018--4032

2023
[10]

Chen, M.; Beutel, A.; Covington, P.; Jain, S.; Belletti, F.; and Chi, E. H. 2019 b . Top-K Off-Policy Correction for a REINFORCE Recommender System. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, WSDM '19, 456–464

2019
[11]

Chen, T.; Kornblith, S.; Norouzi, M.; and Hinton, G. 2020. A Simple Framework for Contrastive Learning of Visual Representations. In Proceedings of the 37th International Conference on Machine Learning, ICML'20

2020
[12]

Chen, X.; Yao, L.; McAuley, J.; Guan, W.; Chang, X.; and Wang, X. 2022 b . Locality-Sensitive State-Guided Experience Replay Optimization for Sparse Rewards in Online Recommendation. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in I...

2022
[13]

Chen, X.; et al. 2021. A Survey of Deep Reinforcement Learning in Recommender Systems: A Systematic Review and Future Directions. ArXiv, abs/2109.03540

2021 arXiv
[14]

Chen, Y.; Liu, Z.; Li, J.; McAuley, J.; and Xiong, C. 2022 c . Intent contrastive learning for sequential recommendation. In Proceedings of the ACM Web Conference 2022, 2172--2182

2022
[15]

Chen, Y.; Liu, Z.; Li, J.; McAuley, J.; and Xiong, C. 2022 d . Intent Contrastive Learning for Sequential Recommendation. In WWW '22, 2172–2182

2022
[16]

Gao, C.; Huang, K.; Chen, J.; Zhang, Y.; Li, B.; Jiang, P.; Wang, S.; Zhang, Z.; and He, X. 2023 a . Alleviating Matthew Effect of Offline Reinforcement Learning in Interactive Recommendation. In Proceedings of the 46th International ACM SIGIR Conference on Research and Develo...

2023
[17]

Gao, C.; Li, S.; Lei, W.; Chen, J.; Li, B.; Jiang, P.; He, X.; Mao, J.; and Chua, T.-S. 2022 a . KuaiRec: A Fully-observed Dataset and Insights for Evaluating Recommender Systems. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, CI...

2022
[18]

Gao, C.; Li, S.; Zhang, Y.; Chen, J.; Li, B.; Lei, W.; Jiang, P.; and He, X. 2022 b . KuaiRand: An Unbiased Sequential Recommendation Dataset with Randomly Exposed Videos. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, CIKM '22, ...

2022
[19]

Gao, C.; Wang, S.; Li, S.; Chen, J.; He, X.; Lei, W.; Li, B.; Zhang, Y.; and Jiang, P. 2023 b . CIRS: Bursting filter bubbles by counterfactual interactive recommender system. ACM Transactions on Information Systems, 42(1): 1--27

2023
[20]

Gao, T.; Yao, X.; and Chen, D. 2021. S im CSE : Simple Contrastive Learning of Sentence Embeddings. In Moens, M.-F.; Huang, X.; Specia, L.; and Yih, S. W.-t., eds., Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 6894--6910

2021
[21]

Haarnoja; et al. 2018 a . Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905

2018 arXiv
[22]

Haarnoja, T.; Zhou, A.; Abbeel, P.; and Levine, S. 2018 b . Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, 1861--1870. PMLR

2018
[23]

He, K.; Fan, H.; Wu, Y.; Xie, S.; and Girshick, R. 2020 a . Momentum Contrast for Unsupervised Visual Representation Learning. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 9726--9735

2020
[24]

He, X.; Deng, K.; Wang, X.; Li, Y.; Zhang, Y.; and Wang, M. 2020 b . LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '20, 639–648

2020
[25]

Ie, E.; Jain, V.; Wang, J.; Narvekar, S.; Agarwal, R.; Wu, R.; Cheng, H.-T.; Chandra, T.; and Boutilier, C. 2019. SLATEQ: A Tractable Decomposition for Reinforcement Learning with Recommendation Sets. In Proceedings of the 28th International Joint Conference on Artificial Inte...

2019
[26]

Ie, E.; and other. 2019. Recsim: A configurable simulation platform for recommender systems. arXiv preprint arXiv:1909.04847

2019 arXiv
[27]

Isele, D.; and Cosgun, A. 2018. Selective experience replay for lifelong learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32

2018
[28]

Kaiser, L.; et al. 2024. Model-Based Reinforcement Learning for Atari. arXiv:1903.00374

2024 arXiv
[29]

M.; Ullman, T

Lake, B. M.; Ullman, T. D.; Tenenbaum, J. B.; and Gershman, S. J. 2017. Building machines that learn and think like people. Behavioral and brain sciences, 40: e253

2017
[30]

Laskin, M.; Srinivas, A.; and Abbeel, P. 2020. CURL: Contrastive Unsupervised Representations for Reinforcement Learning. In Proceedings of the 37th International Conference on Machine Learning, ICML'20

2020
[31]

P.; Hunt, J

Lillicrap, T. P.; Hunt, J. J.; Pritzel, A.; Heess, N.; Erez, T.; Tassa, Y.; Silver, D.; and Wierstra, D. 2016. Continuous control with deep reinforcement learning. In 4th International Conference on Learning Representations, ICLR

2016
[32]

Lin, Y.; Liu, Y.; Lin, F.; Zou, L.; Wu, P.; Zeng, W.; Chen, H.; and Miao, C. 2023. A Survey on Reinforcement Learning for Recommender Systems. IEEE Transactions on Neural Networks and Learning Systems, 1--21

2023
[33]

Lin, Z.; Tian, C.; Hou, Y.; and Zhao, W. X. 2022. Improving graph collaborative filtering with neighborhood-enriched contrastive learning. In Proceedings of the ACM web conference 2022, 2320--2329

2022
[34]

Liu, F.; Tang, R.; Li, X.; Zhang, W.; Ye, Y.; Chen, H.; Guo, H.; Zhang, Y.; and He, X. 2020. State representation modeling for deep reinforcement learning based recommendation. Knowledge-Based Systems, 205: 106170

2020
[35]

Luo, J.; and Li, H. 2020. Dynamic experience replay. In Conference on robot learning, 1191--1200. PMLR

2020
[36]

Mai, V.; Mani, K.; and Paull, L. 2022. Sample Efficient Deep Reinforcement Learning via Uncertainty Estimation. In International Conference on Learning Representations

2022
[37]

OpenAI. 2024. GPT-4 Technical Report. arXiv:2303.08774

2024 arXiv
[38]

Pan, F.; Cai, Q.; Tang, P.; Zhuang, F.; and He, Q. 2019. Policy Gradients for Contextual Recommendations. In The World Wide Web Conference, WWW '19, 1421–1431

2019
[39]

Paszke, A.; Gross, S.; Massa, F.; Lerer, A.; Bradbury, J.; Chanan, G.; Killeen, T.; Lin, Z.; Gimelshein, N.; Antiga, L.; et al. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32

2019
[40]

Rohde, D.; et al. 2018. RecoGym: A Reinforcement Learning Environment for the problem of Product Recommendation in Online Advertising. arXiv preprint arXiv:1808.00720

2018 arXiv
[41]

Schaul, T.; Quan, J.; Antonoglou, I.; and Silver, D. 2015. Prioritized experience replay. International Conference on Learning Representations

2015
[42]

Schulman; et al. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347

2017 arXiv
[43]

Shi, J.-C.; Yu, Y.; Da, Q.; Chen, S.-Y.; and Zeng, A.-X. 2019. Virtual-Taobao: Virtualizing Real-World Online Retail Environment for Reinforcement Learning. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01): 4902--4909

2019
[44]

Sun, P.; Zhou, W.; and Li, H. 2020. Attentive experience replay. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, 5900--5907

2020
[45]

N.; Kaiser, .; and Polosukhin, I

Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, .; and Polosukhin, I. 2017. Attention is all you need. Advances in neural information processing systems, 30

2017
[46]

Wang, X.; He, X.; Wang, M.; Feng, F.; and Chua, T.-S. 2019. Neural Graph Collaborative Filtering. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR'19, 165–174

2019
[47]

S.; Springenberg, J

Wang, Z.; Novikov, A.; Zolna, K.; Merel, J. S.; Springenberg, J. T.; Reed, S. E.; Shahriari, B.; Siegel, N.; Gulcehre, C.; Heess, N.; et al. 2020. Critic regularized regression. Advances in Neural Information Processing Systems, 33: 7768--7778

2020
[48]

Wu, J.; Wang, X.; Feng, F.; He, X.; Chen, L.; Lian, J.; and Xie, X. 2021. Self-supervised graph learning for recommendation. In Proceedings of the 44th international ACM SIGIR conference on research and development in information retrieval, 726--735

2021
[49]

Wu, J.; Xie, Z.; Yu, T.; Zhao, H.; Zhang, R.; and Li, S. 2022. Dynamics-aware adaptation for reinforcement learning based cross-domain interactive recommendation. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval...

2022
[50]

Xi, X.; Zhao, Y.; Liu, Q.; Ouyang, L.; and Wu, Y. 2023. Integrating Offline Reinforcement Learning with Transformers for Sequential Recommendation. In Proceedings of the 17th ACM Conference on Recommender Systems

2023
[51]

Xin, X.; Karatzoglou, A.; Arapakis, I.; and Jose, J. M. 2020. Self-Supervised Reinforcement Learning for Recommender Systems. Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval

2020
[52]

Yu, T.; Shen, Y.; and Jin, H. 2019. A Visual Dialog Augmented Interactive Recommender System. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD '19, 157–165

2019
[53]

Yu, Y. 2018. Towards Sample Efficient Reinforcement Learning. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18 , 5739--5743

2018
[54]

Zhang, A.; et al. 2020 a . Learning Invariant Representations for Reinforcement Learning without Reconstruction. ArXiv, abs/2006.10742

2020 arXiv
[55]

H.; and Bing, L

Zhang, Y.; He, R.; Liu, Z.; Lim, K. H.; and Bing, L. 2020 b . An Unsupervised Sentence Embedding Method by Mutual Information Maximization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 1601--1610

2020
[56]

Zhao, K.; Liu, S.; Cai, Q.; Zhao, X.; Liu, Z.; Zheng, D.; Jiang, P.; and Gai, K. 2023. KuaiSim: A Comprehensive Simulator for Recommender Systems. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track

2023
[57]

Zhou, G.; Zhu, X.; Song, C.; Fan, Y.; Zhu, H.; Ma, X.; Yan, Y.; Jin, J.; Li, H.; and Gai, K. 2018. Deep interest network for click-through rate prediction. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, 1059--1068

2018
[58]

Zhou, S.; Dai, X.; Chen, H.; Zhang, W.; Ren, K.; Tang, R.; He, X.; and Yu, Y. 2020. Interactive Recommender System via Knowledge Graph-Enhanced Reinforcement Learning. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retr...

2020
[59]

Zhu, Y.; et al. 2021. An Empirical Study of Graph Contrastive Learning. ArXiv, abs/2109.01116

2021 arXiv
[60]

X.; and Yin, D

Zou, L.; Xia, L.; Gu, Y.; Zhao, X.; Liu, W.; Huang, J. X.; and Yin, D. 2020. Neural Interactive Collaborative Filtering. In SIGIR '20, 749–758

2020

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

, " * write output.state after.block = add.period write newline

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Andrychowicz, M.; Wolski, F.; Ray, A.; Schneider, J.; Fong, R.; Welinder, P.; McGrew, B.; Tobin, J.; Pieter Abbeel, O.; and Zaremba, W. 2017. Hindsight experience replay. Advances in neural information processing systems, 30

2017

[4] [4]

Cai, Q.; Liu, S.; Wang, X.; Zuo, T.; Xie, W.; Yang, B.; Zheng, D.; Jiang, P.; and Gai, K. 2023 a . Reinforcing User Retention in a Billion Scale Short Video Recommender System. In Companion Proceedings of the ACM Web Conference 2023, WWW '23 Companion, 421–426

work page 2023

[5] [5]

Cai, T.; Bao, S.; Jiang, J.; Zhou, S.; Zhang, W.; Gu, L.; Gu, J.; and Zhang, G. 2023 b . Model-Free Reinforcement Learning with Stochastic Reward Stabilization for Recommender Systems. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '23, 2179–2183

work page 2023

[6] [6]

Cai, X.; Huang, C.; Xia, L.; and Ren, X. 2023 c . LightGCL: Simple Yet Effective Graph Contrastive Learning for Recommendation. In The Eleventh International Conference on Learning Representations

work page 2023

[7] [7]

Chen, H.; Dai, X.; Cai, H.; Zhang, W.; Wang, X.; Tang, R.; Zhang, Y.; and Yu, Y. 2019 a . Large-scale interactive recommendation with tree-structured policy gradient. In Proceedings of the AAAI conference on artificial intelligence, volume 33, 3312--3320

work page 2019

[8] [8]

Chen, H.; Feng, Z.; Chen, S.; Xue, X.; Wu, H.; Sun, Y.; Xu, Y.; and Han, G. 2022 a . Capturing Users’ Fresh Interests via Evolving Session-Based Social Recommendation. In 2022 IEEE International Conference on Web Services (ICWS), 182--187

work page 2022