Pith. sign in

REVIEW 5 major objections 3 minor 38 references

SalesLoop closes the loop between deployed lead rankings and real conversion outcomes, reporting +4.7% to +8.7% lift in a 160-day A/B test.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A monthly closed-loop RL system for ranking sales leads, using position- and speed-weighted conversion rewards and a listwise GRPO-style loss, reports +7.9% NDCG@K offline and +4.7%/+8.7% conversion lift in a 160-day production A/B test.

T0 review reviewed 2026-08-01 challenge →

load-bearing objection A plausible industrial closed-loop ranking system, but the A/B lift is confounded by self-reinforcement and the stats are too thin to trust the headline. the 5 major comments →

arxiv 2607.20655 v1 pith:CYZLVZBJ submitted 2026-07-22 cs.LG cs.AIcs.IR

SalesLoop: Reinforcement Learning from Performance Feedback for Sales Lead Ranking

Chenyu Zhang This is my paper
classification cs.LG cs.AIcs.IR
keywords lead rankingCRMreinforcement learningperformance feedbacklistwise optimizationGRPOA/B testingconversion prediction
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that high offline accuracy in lead-ranking models often fails to translate to production because of three structural mismatches: offline metrics do not reflect online conversion behavior, pointwise training does not optimize the listwise Top-K ranking that actually drives sales, and the lead distribution drifts over time. SalesLoop addresses all three by closing a feedback loop: deploy the ranker, observe which leads convert and how fast, convert those observations into a position- and velocity-weighted reward, and retrain the model monthly with a listwise objective. Offline, this yields +7.9% NDCG@K and +15.8% P@K over the strongest static baseline; in production, a 160-day A/B test across two provincial markets with 280 specialists and 16.5M leads shows cumulative lock-in lifts of +4.7% and +8.7%, both statistically significant, with the treatment advantage growing over time. A sympathetic reader would care because the result suggests a practical, general recipe for continuously improving ranking systems from deployment feedback rather than relying on static historical labels.

Core claim

SalesLoop establishes that a discriminative lead-ranking model can be continuously improved by optimizing a listwise objective against performance-aware rewards built from real conversion outcomes, instead of retraining on static labels. The reward R_i = y_i · g(r_i) · v(τ_i) gives credit only to converted leads, gives more credit to conversions that appear near the top of the deployed ranking, and gives more credit to faster conversions. Discriminative GRPO standardizes rewards within each training batch to group-relative advantages and minimizes the KL divergence between the advantage distribution and the model's score distribution, while a BCE regularization term preserves calibration. Th

What carries the argument

The central mechanism is the pair: (1) a performance-aware reward, R_i = y_i · g(r_i) · v(τ_i), where g(r_i) = 1/log2(r_i + 1) encodes logarithmic attention decay with rank and v(τ_i) = 1 - τ_i/T rewards faster conversions, turning sparse binary labels into a dense, listwise-relevant signal; and (2) Discriminative GRPO, which treats each training batch as a group, standardizes rewards into advantages A_i via Eq. 5, and optimizes L_list = D_KL(softmax(A/γ) || softmax(s)) via Eq. 6, coupling all leads in a batch so that changing one score shifts the relative probability mass. A BCE regularization term L_BCE(s, y) anchors calibration, and monthly warm-start iterations with a reduced learning ra

Load-bearing premise

The reward for each lead uses the rank that the deployed model itself assigned to that lead, so the training signal is partly a product of the model's own prior decisions; if this self-referential feedback is not separated from true lead quality, the claimed learning signal may partly be the model confirming its own ranking.

What would settle it

Run the same feedback loop in an offline replay where the ranks r_i are assigned randomly or held out instead of by the model f_θ; if SalesLoop's advantage over static baselines disappears or reverses under random rank assignment, the position-weighting term is exploiting the model's own decisions rather than learning lead quality. Alternatively, a production A/B test with outcome-only rewards (no g(r)) that matches the full-reward treatment's lift would show position weighting is unnecessary.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Offline, SalesLoop improves NDCG@K by +7.9% and P@K by +15.8% over the strongest static baseline at K = 10,000.
  • In the 160-day production A/B test, cumulative lock-in conversions rose by +4.7% (p = 0.047) in one market and +8.7% (p = 0.002) in the other, with per-specialist gains of +1.6 and +2.7 conversions.
  • The treatment-control gap grows roughly linearly after crossover, indicating a compounding, non-saturating advantage within the observation window.
  • The lift is largest when capacity is tight: +21.4% P@K at K = 1,000, diminishing to +1.9% at K = 100,000, showing the listwise objective matters most when only a small top fraction is surfaced.
  • The underlying ranker already delivers 44.1% Top-10% recall (4.4× random) and identifies incremental high-intent leads that convert at 2.3× the specialist-flagged baseline, with no degradation over 103 days.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The reward depends on the rank the deployed model itself assigned, so the training signal is partly self-referential: a lead under-ranked by the model gets less follow-up, which can suppress its conversion and lower its reward, potentially reinforcing the model's own errors. Removing or randomizing the rank weight would test whether the observed lift reflects true quality learning.
  • The roughly linear growth of the treatment-control gap could be a genuinely compounding learning effect, but it could also be a slowly converging treatment effect; longer observation or per-iteration learning curves would sharpen the distinction.
  • Because the position gain g(r_i) is tied to the operational capacity K, the reported quantitative lifts at K = 10,000 may not transfer to deployments with different capacity without re-tuning γ and α.
  • The authors propose faster feedback signals (test drives, showroom visits) as future work; a natural testable extension is whether weekly iterations using those intermediate signals outperform the monthly loop in both speed and final lift.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 3 minor

Summary. SalesLoop is a closed-loop learning framework for sales-lead ranking. It proposes a performance-aware reward (Eq. 2) that weights conversion outcomes by the rank assigned by the deployed model and by conversion latency, and a 'Discriminative GRPO' objective (Eqs. 5--6) that matches a reward-derived softmax target to the model's score softmax. The authors report offline gains over static baselines (NDCG@K +7.9%, P@K +15.8% at K=10,000) and a 160-day production A/B test across two provincial markets with 280 sales specialists, claiming +4.7% (p=0.047) and +8.7% (p=0.002) cumulative lift in lock-in conversions, with the gap widening over time. The paper interprets this as evidence of a compounding feedback loop that adapts to delayed, sparse conversion signals.

Significance. If the central claims hold, the contribution is practically significant: long-horizon production evidence of an online ranking loop with delayed feedback is rare in the learning-to-rank literature, and the proposed reward is simple and interpretable. The paper is also honest in listing limitations and in not over-claiming that the objective is a literal GRPO transplant. However, the paper's core interpretation—that the gains reflect improved ranking quality rather than a self-reinforcing loop—is not yet supported, and the offline protocol and statistical analysis are under-specified. The strengths are the real deployment scale and the clear separation of offline, ablation, and online evidence; those strengths are weakened by unresolved confounding and missing reproducibility details.

major comments (5)
  1. [§3.3, Eq. (2); §4.2; §4.3.2] The learning signal is endogenous to the model being trained. R_i = y_i·g(r_i)·v(τ_i) uses the rank r_i produced by the deployed f_{θ_{t-1}}, and y_i is only observed after follow-up effort that was allocated according to that same rank. A lead converts partly because the model surfaced it, so the reward rewards the model for repeating its own prior choices. The paper itself states Gap 1 — that historical labels are confounded by human effort — but neither the reward design nor the offline protocol applies inverse-propensity weighting, exploration, or a counterfactual analysis to break this loop. Consequently, the 'compounding' divergence in §4.3.2 and the offline gains in Table 2 are also consistent with a self-reinforcing feedback effect rather than an improved estimate of lead quality. Please provide at minimum an effort/propensity correction or a diagnostic that separates genuine qua
  2. [§4.2, Tables 2 and 5] The offline benchmark protocol is under-specified to the point of non-reproducibility. Table 2 reports SALESLOOP evaluated after '1 iteration using held-out feedback', and Table 5 reports an ablation 'offline evaluation after 4 iterations', but the paper never states how ranks r_i are replayed offline, how many leads have observed conversion labels after the Top-K cutoff, or how multiple iterations are simulated without new deployment data. The reader cannot tell whether P@K/NDCG are computed on all leads or only on those that received follow-up, and whether the 4-iteration result is a genuine multi-round simulation or repeated fitting of the same static labels. Please specify the exact data-generation procedure for each iteration, including how unobserved counterfactual outcomes for non-surfaced leads are handled.
  3. [§4.3.1, Table 3; §4.3.2–4.3.4] The statistical support for the headline lift is thin. The evidence is a weekly paired t-test on cumulative conversions (p=0.047 and p=0.002), with no confidence intervals, no adjustment for multiple weekly looks or autocorrelation in cumulative series, and no pre-registered analysis plan. Province A's p is marginal, and its monthly lifts are negative in Dec, Jan, and Feb and near zero in Apr; the 'compounding' claim rests on an eyeballed regime change in Fig. 4 rather than a formal trend or interaction test. Please report effect sizes with confidence intervals, a pre-specified or corrected test, and a formal test for divergence (e.g., treatment×time interaction or segmented regression).
  4. [§3.4, Eqs. (5)–(6)] The 'Discriminative GRPO' is not a policy-gradient objective in the usual sense: there is no sampling, no importance ratio, and no exploration term; Eq. (6) is a deterministic KL/cross-entropy matching between a reward-derived softmax target and the model's score softmax. This may be a valid listwise loss, but calling it RL and RLPF — and ascribing the online gains to reinforcement learning — requires a clearer statement of what is being optimized and why group-relative normalization is necessary. As written, the loss has no mechanism for exploring new rankings; this weakens the claim that the closed loop 'discovers' high-intent leads (Abstract, §4.5) rather than merely re-weighting the current ranker's outputs.
  5. [§4.3.1, A/B configuration] The A/B unit is the sales specialist, not the lead. The paper reports that specialists were randomly assigned, but it does not describe how incoming leads are routed to treatment vs. control specialists. If routing depends on territory, shift, or specialist load, the treatment and control groups may receive different lead streams; the shared seasonality in Fig. 5 does not establish balance. Please report lead volumes and lead-feature distributions per group, or describe the randomization/assignment mechanism.
minor comments (3)
  1. [§3.3, Eq. (2)] Please clarify the meaning of the rank r_i in a monthly-cycle, daily-deployment setting. Is it the lead's rank within the day it arrived, within the monthly batch, or the rank at the moment of conversion? This affects the reward magnitude and the offline replay.
  2. [§4.5, Table 6 vs. Fig. 7] There is an inconsistency in the reported 7-day Top-10% recall: Table 6 says mean 44.1%, while Fig. 7 shows mean 43.7%. Please harmonize.
  3. [Table 4] The column header 'June 2025' appears to be a typo; the test window is December 2025–June 2026.

Circularity Check

2 steps flagged

Reward in Eq. (2) uses the deployed model's own ranks, making the Eq. (6) target self-referential; Gap 1 admits the confound but no correction is applied.

specific steps
  1. self definitional [§3.1 Problem Formulation; §3.3 Eq. (2); §3.4 Eqs. (5)–(6)]
    "r_i ∈ {1, . . . , Nt}: rank assigned by the deployed model fθt−1 ... R_i = y_i · g(r_i) · v(τ_i), (2) ... A_i = (R_i − R¯)/(σ_R + ε) ... L_list = D_KL(softmax(A/γ) || softmax(s)), (6)"

    The target distribution in Eq. (6) is constructed from advantages A_i (Eq. 5), which are standardized rewards R_i (Eq. 2). R_i depends on r_i, the rank assigned by the deployed model f_{θ_{t-1}} being updated. Therefore the 'performance feedback' is a function of the model's own previous output, not an external benchmark. Training s_i = f_θ(x_i) to match this target makes the updated ranking a function of the old ranking; any claimed lift can be partly produced by reinforcing the old model's prior decisions. No propensity, position-bias, or counterfactual correction is applied, so the learning signal reduces to the model's own action-contingent labels.

  2. other [§1 Gap 1; §3.3 Eq. (2)]
    "historical conversion labels are themselves confounded by human effort: a lead converts partly because it was promising and partly because a specialist invested time in it. A model trained on these labels therefore learns a mixture of intent and effort ... R_i = y_i · g(r_i) · v(τ_i), (2)"

    The paper explicitly identifies the effort confound in historical labels, then defines the online reward R_i = y_i·g(r_i)·v(τ_i) using the same kind of confounded y_i, where specialist effort is allocated by the model's own Top-K ranking (Stage 1). Without exploration or inverse-propensity weighting, the reward is not an unbiased measure of lead quality. The offline gains and the A/B 'compounding' interpretation can therefore be explained as self-reinforcement of the deployed model's prior ranking rather than as evidence of improved lead-quality estimation, so the central empirical claim rests on a signal that the paper itself admits is confounded.

full rationale

The central learning signal is self-referential. Eq. (2) defines the reward using r_i, the rank assigned by the deployed model f_{θ_{t-1}}; Eq. (5) standardizes those rewards into advantages; Eq. (6) trains the updated model to match softmax(A/γ), a target built from the model's own previous output. Thus by the paper's own equations, the 'performance feedback' is a function of the model's prior decisions, not an external ground truth. The paper's Gap 1 admits that conversion labels are confounded by effort allocated according to rank, but the reward and training protocol never correct for this—no exploration, no propensity weighting, no counterfactual analysis. This makes the offline 'predictions' and the A/B 'compounding' partially forced by the reward construction rather than purely independent evidence. The A/B test is real, so the paper is not wholly circular; the self-citation to HPRO for architecture is not load-bearing. Score 6 reflects partial circularity in the reward-to-target chain, not full definitional equivalence.

Axiom & Free-Parameter Ledger

6 free parameters · 6 axioms · 0 invented entities

The paper introduces no new physical entities. The 'Discriminative GRPO' is a method name, not an entity. The free parameters γ and α are tuned on the validation set, and the reward functions g and v are chosen ad hoc. The central assumption of reward exogenous to the model's own ranking is unverified.

free parameters (6)
  • γ (temperature) = 0.1
    Set based on sensitivity analysis (Section 4.6).
  • α (BCE regularization weight) = 0.5
    Chosen via ablation studies (Table 5).
  • Conversion window T = 30 days
    Business-defined, but affects reward scaling.
  • Position decay function g(r) = 1/log2(r+1)
    Ad hoc choice based on position-based click model; no empirical justification given for lead ranking.
  • Velocity bonus v(τ) = 1 - τ/T
    Linear decay chosen without justification.
  • LoRA rank and alpha = r=16, α=32
    Standard choices; not claimed as tuned.
axioms (6)
  • domain assumption Logarithmic attention decay (position-based model) applies to sales follow-up behavior.
    Section 3.3, based on [6]; no empirical evidence that sales specialists follow this decay.
  • domain assumption Lock-in conversion within 30 days is a valid and sufficient proxy for business value.
    Used throughout; ignores long-term value and follow-up beyond 30 days.
  • standard math KL divergence against a softmax target is an appropriate surrogate for ranking metrics like P@K and NDCG@K.
    Common in listwise LTR (ListNet); not directly derived from the objective in Eq. 1.
  • domain assumption Within-batch standardization of rewards reduces variance in sparse-reward settings.
    Claimed in Section 3.4, but no theoretical or empirical support beyond the method's popularity.
  • domain assumption Random assignment of specialists to treatment/control yields exchangeable lead distributions.
    Needed for A/B test validity; not demonstrated with lead-level covariate balance.
  • domain assumption The conversion outcome is a function of lead quality and follow-up effort, and the rank position is an adequate proxy for effort.
    This is the key endogeneity assumption; the paper does not test it.

reviewed 2026-08-01 · how reviews work

0 comments
Cite this review

Pith. "Pith review of SalesLoop: Reinforcement Learning from Performance Feedback for Sales Lead Ranking." pith.science (2026). https://pith.science/paper/CYZLVZBJ

@misc{pith2026260720655,
  author       = {Pith},
  title        = {Pith review of: SalesLoop: Reinforcement Learning from Performance Feedback for Sales Lead Ranking},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CYZLVZBJ}},
  note         = {Machine review of arXiv:2607.20655}
}
Share X Bluesky LinkedIn Reddit HN
abstract

Lead ranking in Customer Relationship Management (CRM) systems faces a persistent challenge: models achieving high offline accuracy often underperform in production. We identify three fundamental gaps responsible for this disconnect: offline-online metric mismatch, pointwise-listwise objective misalignment, and temporal distribution drift. To address these gaps, we propose SalesLoop, a reinforcement learning framework that establishes a closed feedback loop between model predictions and real-world business outcomes. Our approach introduces (1) a performance-aware reward that encodes conversion outcomes weighted by ranking position and conversion velocity, and (2) Discriminative GRPO, a listwise optimization objective that adapts Group Relative Policy Optimization to discriminative ranking models. SalesLoop improves NDCG@K by +7.9\% and P@K by +15.8\% over the strongest static baseline. A 160-day production A/B test at a New Energy Vehicle manufacturer, spanning 16.5M leads and 280 sales specialists across two provincial markets, validates statistically significant cumulative lift of +4.7\% ($p=0.047$) and +8.7\% ($p=0.002$). In production, the ranking backbone achieves Top-10\% recall of 44.1\% and surfaces high-intent leads at $2.3\times$ the conversion rate of specialist baselines.

Figures

Figures reproduced from arXiv: 2607.20655 by Chenyu Zhang.

Figure 1
Figure 1. Figure 1: Overview of the SALESLOOP framework. In each monthly iteration, the deployed ranking model scores all leads and selects Top-K candidates for sales follow-up. After the 30-day conversion window, observed outcomes are converted into performance-aware rewards, which are used to update the model through a Discriminative GRPO objective. The updated model is then deployed in the next iteration, forming a closed … view at source ↗
Figure 2
Figure 2. Figure 2: Cumulative lock-in conversions over 160 days. Green shading indicates periods where treatment outperforms control; [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Monthly conversion lift (%) of treatment over con [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Cumulative conversion difference (Treatment [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Daily new conversions (7-day moving average) for treatment and control groups. Province A shows treatment [PITH_FULL_IMAGE:figures/full_fig_p014_5.png] view at source ↗
Figure 7
Figure 7. Figure 7: Top-10% recall stability over 103 days. The base [PITH_FULL_IMAGE:figures/full_fig_p014_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: 14-day conversion rate comparison: specialist [PITH_FULL_IMAGE:figures/full_fig_p015_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

38 extracted references · 7 linked inside Pith

  1. [1]

    Learning to rank using gradient descent

    Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hullender. Learning to rank using gradient descent. InProceedings of ICML, pages 89–96, 2005

    2005

  2. [2]

    Learning to rank: from pairwise approach to listwise approach

    Zhe Cao, Tao Qin, Tie-Yan Liu, Ming-Feng Tsai, and Hang Li. Learning to rank: from pairwise approach to listwise approach. InProceedings of ICML, pages 129– 136, 2007

    2007

  3. [3]

    Top-k off-policy correction for a reinforce recommender sys- tem

    Minmin Chen, Alex Beutel, Paul Covington, et al. Top-k off-policy correction for a reinforce recommender sys- tem. InProceedings of WSDM, pages 456–464, 2019. 11

    2019

  4. [4]

    Xgboost: A scalable tree boosting system

    Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. InProceedings of KDD, pages 785–794, 2016

    2016

  5. [5]

    Wide & deep learning for recom- mender systems

    Heng-Tze Cheng et al. Wide & deep learning for recom- mender systems. InProceedings of DLRS, pages 7–10, 2016

    2016

  6. [6]

    Click models for web search

    Aleksandr Chuklin, Ilya Markov, and Maarten de Rijke. Click models for web search. Morgan & Claypool Pub- lishers, 2015

    2015

  7. [7]

    A survey on con- cept drift adaptation.ACM Computing Surveys, 46(4):1– 37, 2014

    Jo ˜ao Gama, Indr ˙e ˇZliobait˙e, Albert Bifet, Mykola Pech- enizkiy, and Abdelhamid Bouchachia. A survey on con- cept drift adaptation.ACM Computing Surveys, 46(4):1– 37, 2014

    2014

  8. [8]

    Deepfm: a factorization-machine based neural network for ctr prediction

    Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. Deepfm: a factorization-machine based neural network for ctr prediction. InProceedings of IJCAI, pages 1725–1731, 2017

    2017

  9. [9]

    Lora: Low-rank adaptation of large language mod- els

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language mod- els. InProceedings of ICLR, 2022

    2022

  10. [10]

    Crmarena: Understand- ing the capacity of llm agents to perform professional crm tasks in realistic environments.arXiv preprint arXiv:2411.02305, 2024

    Kung-Hsiang Huang et al. Crmarena: Understand- ing the capacity of llm agents to perform professional crm tasks in realistic environments.arXiv preprint arXiv:2411.02305, 2024

    Pith/arXiv arXiv 2024

  11. [11]

    Slateq: A tractable decomposition for reinforcement learning with recommendation sets

    Eugene Ie, Vihan Jain, Jing Wang, et al. Slateq: A tractable decomposition for reinforcement learning with recommendation sets. InProceedings of IJCAI, pages 2592–2599, 2019

    2019

  12. [12]

    To model or to intervene: A comparison of counterfac- tual and online learning to rank from user interactions

    Rolf Jagerman, Harrie Oosterhuis, and Maarten de Rijke. To model or to intervene: A comparison of counterfac- tual and online learning to rank from user interactions. InProceedings of SIGIR, pages 15–24, 2019

    2019

  13. [13]

    Improving generative ad text on facebook using reinforcement learning.arXiv preprint arXiv:2507.21983, 2025

    Daniel R Jiang, Alex Nikulkov, Yu-Chia Chen, Yang Bai, and Zheqing Zhu. Improving generative ad text on facebook using reinforcement learning.arXiv preprint arXiv:2507.21983, 2025

    arXiv 2025

  14. [14]

    xdeepfm: Combining explicit and implicit feature interactions for recommender systems

    Jianxun Lian et al. xdeepfm: Combining explicit and implicit feature interactions for recommender systems. InProceedings of KDD, pages 1754–1763, 2018

    2018

  15. [15]

    Springer, 2009

    Tie-Yan Liu.Learning to rank for information retrieval. Springer, 2009

    2009

  16. [16]

    Decoupled weight de- cay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight de- cay regularization. InProceedings of ICLR, 2019

    2019

  17. [17]

    Salesrlagent: A re- inforcement learning approach for real-time sales con- version prediction and optimization.arXiv preprint arXiv:2503.23303, 2025

    Nandakishor Mukkunnoth et al. Salesrlagent: A re- inforcement learning approach for real-time sales con- version prediction and optimization.arXiv preprint arXiv:2503.23303, 2025

    Pith/arXiv arXiv 2025

  18. [18]

    Differentiable unbiased online learning to rank

    Harrie Oosterhuis and Maarten de Rijke. Differentiable unbiased online learning to rank. InProceedings of CIKM, pages 1293–1302, 2018

    2018

  19. [19]

    Training lan- guage models to follow instructions with human feed- back.Advances in Neural Information Processing Sys- tems, 35:27730–27744, 2022

    Long Ouyang, Jeffrey Wu, Xu Jiang, et al. Training lan- guage models to follow instructions with human feed- back.Advances in Neural Information Processing Sys- tems, 35:27730–27744, 2022

    2022

  20. [20]

    Continual lifelong learning with neural networks: A review.Neural Net- works, 113:54–71, 2019

    German I Parisi, Ronald Kemker, Jose L Part, Christo- pher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review.Neural Net- works, 113:54–71, 2019

    2019

  21. [21]

    A strategic framework for customer relationship management.Journal of Mar- keting, 69(4):167–176, 2005

    Adrian Payne and Pennie Frow. A strategic framework for customer relationship management.Journal of Mar- keting, 69(4):167–176, 2005

    2005

  22. [22]

    Neu- ralndcg: Direct optimisation of a ranking metric via differentiable relaxation of sorting.arXiv preprint arXiv:2102.07831, 2021

    Przemysław Pobrotyn and Radosław Białobrzeski. Neu- ralndcg: Direct optimisation of a ranking metric via differentiable relaxation of sorting.arXiv preprint arXiv:2102.07831, 2021

    Pith/arXiv arXiv 2021

  23. [23]

    Qwen2.5 technical report, 2025

    Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Day- iheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Ke- qin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao...

    2025

  24. [24]

    Di- rect preference optimization: Your language model is se- cretly a reward model.Advances in Neural Information Processing Systems, 36, 2024

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christo- pher D Manning, Stefano Ermon, and Chelsea Finn. Di- rect preference optimization: Your language model is se- cretly a reward model.Advances in Neural Information Processing Systems, 36, 2024

    2024

  25. [25]

    Recommenda- tions as treatments: Debiasing learning and evaluation

    Tobias Schnabel, Adith Swaminathan, Ashudeep Singh, Navin Chandak, and Thorsten Joachims. Recommenda- tions as treatments: Debiasing learning and evaluation. InProceedings of ICML, pages 1670–1679, 2016

    2016

  26. [26]

    Proximal policy optimiza- tion algorithms.arXiv preprint arXiv:1707.06347, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimiza- tion algorithms.arXiv preprint arXiv:1707.06347, 2017

    Pith/arXiv arXiv 2017

  27. [27]

    Online learning and online con- vex optimization.Foundations and Trends in Machine Learning, 4(2):107–194, 2012

    Shai Shalev-Shwartz. Online learning and online con- vex optimization.Foundations and Trends in Machine Learning, 4(2):107–194, 2012. 12

    2012

  28. [28]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    Zhihong Shao et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    Pith/arXiv arXiv 2024

  29. [29]

    Autoint: Automatic feature interac- tion learning via self-attentive neural networks

    Weiping Song et al. Autoint: Automatic feature interac- tion learning via self-attentive neural networks. InPro- ceedings of CIKM, pages 1161–1170, 2019

    2019

  30. [30]

    asllr: Llm based leads ranking in auto sales.arXiv preprint arXiv:2510.21713, 2025

    Yin Sun, Yiwen Liu, Junjie Song, Chenyu Zhang, Xinyuan Zhang, Lingjie Liu, Siqi Chen, and Yuji Cao. asllr: Llm based leads ranking in auto sales.arXiv preprint arXiv:2510.21713, 2025

    arXiv 2025

  31. [31]

    Dcn v2: Improved deep & cross net- work and practical lessons for web-scale learning to rank systems

    Ruoxi Wang et al. Dcn v2: Improved deep & cross net- work and practical lessons for web-scale learning to rank systems. InProceedings of WWW, pages 1785–1797, 2021

    2021

  32. [32]

    Deep & cross network for ad click predictions

    Ruoxi Wang, Bin Fu, Gang Fu, and Mingliang Wang. Deep & cross network for ad click predictions. InPro- ceedings of ADKDD, pages 1–7, 2017

    2017

  33. [33]

    Univer- sity of California at Berkeley, 2001

    Russell S Winer.Customer relationship management: A framework, research directions, and the future. Univer- sity of California at Berkeley, 2001

    2001

  34. [34]

    Listwise approach to learning to rank: theory and algorithm

    Fen Xia, Tie-Yan Liu, Jue Wang, Wensheng Zhang, and Hang Li. Listwise approach to learning to rank: theory and algorithm. InProceedings of ICML, pages 1192– 1199, 2008

    2008

  35. [35]

    An end-to-end multi-objective ensemble ranking frame- work for video recommendation.arXiv preprint arXiv:2508.05093, 2025

    Xiaoxiao Xu, Minzhi Xie, Runtong Li, et al. An end-to-end multi-objective ensemble ranking frame- work for video recommendation.arXiv preprint arXiv:2508.05093, 2025

    Pith/arXiv arXiv 2025

  36. [36]

    Auro: Reinforcement learning for adaptive user retention op- timization in recommender systems

    Zhenghai Xue, Qingpeng Cai, Tianyou Zuo, Bin Yang, Lantao Hu, Peng Jiang, Kun Gai, and Bo An. Auro: Reinforcement learning for adaptive user retention op- timization in recommender systems. InProceedings of WWW, 2025

    2025

  37. [37]

    Rethinking sales lead scoring with llm-based hierarchical preference rank- ing.arXiv preprint arXiv:2606.04387, 2026

    Chenyu Zhang, Yiwen Liu, Yin Sun, Xinyuan Zhang, Yuji Cao, Junming Jiao, and Juyi Qiao. Rethinking sales lead scoring with llm-based hierarchical preference rank- ing.arXiv preprint arXiv:2606.04387, 2026

    Pith/arXiv arXiv 2026

  38. [38]

    Towards disentangling relevance and bias in unbiased learning to rank

    Yunan Zhang, Le Yan, Zhen Qin, Honglei Zhuang, Ji- aming Shen, Xuanhui Wang, Michael Bendersky, and Marc Najork. Towards disentangling relevance and bias in unbiased learning to rank. InProceedings of KDD, 2023. A Additional Online Deployment Analysis This appendix provides supplementary analyses of the online deployment data to further characterize SALES...

    2023

This paper was first reviewed by deepseek-v4-flash on August 1, 2026.