A Point Process Model for Optimizing Repeated Personalized Action Delivery to Users

Alexander Merkov; David Rohde

REVIEW 3 major objections 5 minor 20 references

A Point Process Model for Optimizing Repeated Personalized Action Delivery to Users

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Repeated personalized action delivery is a Bayesian decision problem, and neural temporal point processes make it tractable.

desk verdict A clean, honest synthesis of Bayesian decision theory, marked TPPs, and REINFORCE for repeated personalized actions, but the central confounding assumption is asserted rather than tested and there are no experiments or new theorems. read the letter →

arxiv 2501.02961 v1 pith:5QRDU3AZ submitted 2025-01-06 stat.ML cs.LG

Alexander Merkov , David Rohde This is my paper · ORCID

classification stat.MLcs.LG

keywords temporalpointprocessesneuralBayesiandecisiontheorypolicygradientcausalinferencepersonalizedactiondeliverycomputationaladvertisingreinforcementlearning

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the problem of repeatedly delivering personalized actions to users—whether ads, recommendations, or reminders—is best understood as Bayesian decision theory applied to a marked temporal point process. In this view, each user's history is a time-ordered event stream, and the system's actions are appended to special 'request for action' events. If the conditional distribution of the next event's time delay and mark is modeled by a recurrent neural network, the paper shows that likelihood-based training, forward simulation, and policy-gradient optimization all become concrete algorithms. The payoff is a unified way to optimize both when to intervene and what action to deliver, using only logged data and without requiring the do-calculus or propensity-score adjustment. The authors further claim that online interactive systems whose actions are chosen by a known logged policy satisfy the no-unobserved-confounding assumption by construction, which would make the policy-gradient estimate of expected utility unbiased.

What carries the argument

The carrying mechanism is the conditional event distribution $P(\tau, m \mid \text{history}, \theta)$, factorized as $p(\tau \mid m, \theta) Q(m \mid \theta)$, with a recurrent network mapping each history to the parameter vector $\varphi$ of these distributions. The paper proposes a specific three-parameter heavy-tailed delay family (Equation 10) whose CDF and inverse CDF are elementary functions, so likelihood evaluation and inverse-CDF sampling are both cheap. On top of this generative model sits the policy $\pi_\xi(a_s \mid a_{1:s-1}, H_{0:s-1})$, which at each 'request for action' event selects an action; the REINFORCE estimator $\sum \nabla_\xi \log \pi_\xi(f_k \mid e_{1:k-1}) U(e)$ provides the gradient step for maximizing expected utility.

What would settle it

Find one deployed interactive system whose logging policy uses state not present in the dataset D, or show in logs that adding a previously unlogged covariate changes the estimated expected utility of a fixed policy; either would refute the claim that unobserved confounding is absent by construction.

Watch

Extended reading notes

Core claim

The central claim is that the expected utility of a personalized action policy πξ can be written as an expectation under the Bayesian posterior predictive distribution over user event sequences, and that this expectation can be maximized by a REINFORCE-style stochastic gradient ascent using sequences simulated from a neural temporal point process. The paper's formal contribution is to show that the abstract model of Section 2—products of conditional observation distributions and actions—specializes to marked temporal point processes in which each event's probability is the product of a delay distribution and a mark distribution, conditioned on the entire history through an RNN. Within this specialization, Algorithm 2 computes the likelihood of a logged sequence, Algorithm 4 simulates future sequences under any policy, and Algorithm 5 performs the gradient update that optimizes the policy's parameters.

Load-bearing premise

The argument assumes that every action in the logs was chosen with probabilities determined only by the recorded history, so no hidden variable influences both the action and the user's later behavior; the authors assert this holds by construction for online interactive systems, but if a hidden variable exists, the policy-gradient estimate is biased.

Editorial extensions

If this is right

Any repeated personalized intervention—ad display, recommendation, reminder—fits the same formalism, provided the user process is stationary, users do not interfere with each other, and the logging policy is known.
Offline policy evaluation and optimization can be carried out purely by simulating sequences from the trained point-process model, without online experimentation.
The closed-form delay distribution makes likelihood computation and event sampling computationally cheap, keeping both Bayesian posterior updating and policy-gradient iteration practical.
Under the no-unobserved-confounding assumption, propensity-score adjustment and do-calculus are unnecessary for this problem class, simplifying the estimation pipeline.
Timing and content of interventions are optimized jointly within one objective, rather than as separate subproblems.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural extension is to continuous or structured action spaces, such as bid amounts or message text, by reparameterizing the policy distribution while keeping the point-process model unchanged.
The no-unobserved-confounding claim is empirically testable: if production logs contain auxiliary features not used by the logging policy, computing estimated utility with and without those features and finding divergence would indicate hidden confounding.
Stationarity may be the more restrictive assumption in practice, since user behavior drifts; extending the framework to non-stationary processes or online weight adaptation would be needed for long-running systems.
The proposed delay family is a design choice; swapping in a more flexible distribution, such as a normalizing flow or learned mixture, should preserve every algorithm while potentially improving fit.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should read this as a position paper with explicit algorithms, not as a new result. The formalism is exactly what the title says: it puts repeated personalized action delivery into the language of Bayesian decision theory, specializes to marked temporal point processes, and then uses REINFORCE for policy optimization. There is no new theorem and no experiment. The novelty is in the assembly, and on that front it is mostly clean and readable.

What the paper does well: it spells out the likelihood with a censoring correction (Algorithm 2) carefully, including the no-event contribution. The heavy-tailed family in equations (10)-(11) is a simple piecewise power-law, but it is genuinely useful in this context because both the CDF and its inverse are elementary, so likelihood evaluation and sampling are cheap and exact. Algorithm 5 is a straightforward policy-gradient loop, and the paper is honest that many details — convergence, step size, posterior sampling — are left unspecified. In fact, the algorithms are littered with comments like "to be specified" and "requires some explanation", which is refreshing even if it undercuts the claim to a ready-to-use method.

The soft spot is the unobserved-confounding assumption. The paper asserts in Section 2 that online interactive systems satisfy conditional ignorability "by construction", and then footnote 3 immediately concedes that hierarchies of models with different features can create unobserved confounding. That is the exact condition on which Algorithm 5's unbiasedness rests. If actions depend on latent user state or on features not written to the log, then the learned point process is a confounded association, not an interventional model, and the policy gradient maximizes the wrong objective. The paper offers no synthetic or real check of this premise. That is not a fatal mathematical flaw, but for an applied position paper it is the load-bearing assumption, and it is waved through too quickly. I also agree with the reader that the heavy-tailed family is not justified on real delay data; the reference to [16] is suggestive but not evidence for this specific parameterization.

All that said, this paper deserves a serious referee. It addresses an important applied problem, the formalization is coherent, and the explicit algorithms give a referee something concrete to probe. My recommendation: send it to review, and push the authors either to add a small simulation study that varies the confounding strength and checks the policy-gradient bias, or to temper the "by construction" claim to a clearly stated assumption with conditions. As it stands, it is a promising framework missing its own validation.

Referee Report

3 major / 5 minor

Summary. The paper proposes a Bayesian decision-theoretic framework for optimizing repeated personalized actions to users from logged interaction data. Section 2 formulates the likelihood and posterior over an abstract parameter θ, defines a policy πξ, and derives REINFORCE-style expected-utility maximization. Section 3 specializes observations to marked temporal point processes, with actions triggered by events, and introduces a three-parameter piecewise power-law family for inter-event times (Eqs. 10–11). Section 4 proposes recurrent neural networks to parameterize the event distributions and gives algorithms for likelihood computation (Algorithm 2), event sampling (Algorithms 3–4), and policy gradient optimization (Algorithm 5). The paper contains no experimental evaluation.

Significance. If its correctness conditions hold, the paper is a useful conceptual bridge between Bayesian decision theory and neural temporal point processes for repeated interventions. The probability derivations in Eqs. (1)–(11) and Algorithms 2–3 are internally consistent, and the censoring correction in Algorithm 2 is carefully handled. The contribution is, however, purely formal: no data, experiments, or code are provided, and the central causal assumption (no unobserved confounding) is asserted rather than validated. As a result, the paper's usefulness for the stat.ML audience is not yet demonstrated.

major comments (3)

[Section 2, Eq. (15), Algorithm 5] The unbiasedness of the policy-gradient update in Algorithm 5 depends critically on conditional ignorability: actions must be independent of any unobserved variable that also affects future events, conditional on the logged history. The paper asserts this is satisfied 'by construction' for online interactive systems, but footnote 3 immediately concedes that hierarchies of models accessing different features produce unobserved confounding. Since Eq. (15) is estimated from observational logs, any such hidden variable biases both the learned event model and the expected-utility gradient, so Algorithm 5 maximises the wrong objective. The manuscript provides no diagnostic, sensitivity analysis, or experiment to check this premise. I ask the authors to state the exact recorded-variable condition under which ignorability holds and to demonstrate, at least in a synthetic experiment with a hidden confounder, that the proposed estimator behaves as claimed.
[Section 3, Eqs. (10)–(11)] The paper motivates the piecewise power-law family as a good model for real systems and as enabling tractable likelihood and sampling, but no real or synthetic data are used to support this modelling claim. The family is presented as a suggested example, yet the operational claims of the paper — that Algorithms 2–5 provide a practical solution — rest on this unvalidated choice. Please add at least one empirical demonstration, such as fitting the family to a standard point-process dataset or a simulation study with known ground truth, and discuss the non-smoothness and non-convexity that the authors themselves note.
[Section 2 vs Section 4] The Bayesian formalism (posterior P(θ|D), Eq. (2)) underpins Algorithm 1, but Algorithms 4–5 instead input a single 'trained RNN R', with no discussion of whether θ is a point estimate or a posterior sample. The paper should clarify the status of Bayesian uncertainty in the proposed implementation; as written, the central 'Bayesian decision theory' claim and the actually presented algorithms are disconnected. Without such clarification, the expected utility optimized by Algorithm 5 is not obviously the Bayesian expected utility defined in Eq. (4).

minor comments (5)

[Abstract] The phrase 'user-advertiser interaction in online advertiser' appears to be an editing error; 'online advertiser' should likely be 'online advertising', and the first sentence is grammatically incomplete.
[Section 2, first display] The distribution P(H_k|H_{0:k-1}, a_{1:k-1}) conditions on actions up to k−1, but the described sequence is H0, a1, H1, a2, H2, ..., so H_k is preceded by a_k; the conditioning set should be a_{1:k}, as is already used in Eq. (1) and the predictive distribution.
[Algorithm 4] The variable eB is used both for the previous event and for the newly sampled event, and it is initialized with B=0; renaming it to e_prev or e_{B-1} would avoid confusion.
[Section 3] After defining the mark m as the pair m=(v,x), the text reduces marks to integers {1,...,M}; please clarify the encoding of categorical marks and feature vectors into this finite index set.
[Algorithms 1 and 5] Algorithms 1 and 5 leave λ, initialization, and stopping criteria as placeholders; this is acceptable at a high level, but for reproducibility the authors should specify at least one concrete default choice in each case.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is a transparent application of Bayesian decision theory and neural temporal point processes.

full rationale

The paper's derivation chain is: (i) define a likelihood over user histories conditioned on actions (Eq. 1); (ii) place a prior and form the posterior (Eq. 2); (iii) introduce a policy and define expected utility (Eqs. 3-4); (iv) specialize to marked temporal point processes with a factored likelihood (Eqs. 7, 15) and an RNN parameterization (Algorithm 2); (v) use the learned RNN as a simulator and apply a REINFORCE-style policy gradient (Algorithm 5). Each step is either a definitional Bayesian or decision-theoretic identity, or a standard neural point-process likelihood construction. No parameter is fitted to a target quantity and then reported as a prediction; no uniqueness theorem is imported from the authors' prior work to force the model choice; and no ansatz is smuggled in solely via self-citation. The statements 'This formulation is simply Bayesian decision theory [3]' and 'causal inference is just inference [5, 11]' are presentational. The latter is a philosophical stance supported in part by the authors' own position papers, but the mathematical content of Sections 3 and 4 does not rely on it. The no-unobserved-confounding assumption is a substantive premise, explicitly stated as an assumption rather than an output of the derivation, and footnote 3 openly concedes realistic violations ('Poor practices such as having hierarchies of models accessing different features can and do cause such systems to be susceptible to unobserved confounding'). That concession is a validity threat to the unbiasedness of Algorithm 5, but it is not circularity: an untested or even violated assumption is not the same as a claim that reduces to its own inputs by construction. The likelihood and policy gradient are not equivalent to their inputs by definition unless the stated causal assumption is granted, and granting an assumption is not a circular step. Hence the paper contains no significant circularity.

Assumptions & free parameters 5 free parameters · 7 assumptions · 0 invented entities

The central method depends on a chain of modeling choices: stationarity, no interference between users, no unobserved confounding, finite mark space, and the proposed piecewise power-law family. The no-confounding assumption is the most fragile because it is asserted rather than derived and is disputed in the cited literature.

free parameters (5)

alpha_m, beta_m, tau*_m per mark m = to be learned
Parameters of the proposed heavy-tailed inter-event time distribution, Eq. (10), one set per event mark.
multinomial mark probabilities q_m = to be learned
Mark distribution Q_phi(m|...) in Section 3.1.
RNN weights theta = to be learned
Weights of the recurrent network mapping event histories to distribution parameters, Eqs. (13)-(14).
policy parameters xi = to be learned
Parameters of the action-delivery policy pi_xi optimized in Algorithm 5.
gradient step size lambda = not specified
Algorithm 1 comment: 'the gradient step size lambda > 0 should be defined somehow'.

assumptions (7)

domain assumption User histories are I independent stationary random processes with common conditional distributions P(H_k|H_0:k-1, a_1:k-1; theta).
Section 2, first paragraph; needed for pooling data across users and time.
domain assumption Actions only impact observations after they are delivered.
Section 2, paragraph 'the assumption that actions delivered to a users do not affect another users' includes the arrow-of-time statement.
domain assumption SUTVA: actions delivered to one user do not affect another user.
Section 2, paragraph assuming no cross-user interference.
domain assumption No unobserved confounding: actions depend only on information in the logs D.
Section 2, paragraph 'We will also assume that there is no unobserved confounding'; load-bearing for likelihood (1) and Algorithm 5.
domain assumption The space of features x is finite, so there are finitely many marks.
Section 3.1, 'For simplicity we suppose in this section that the space of features x is finite'.
ad hoc to paper Inter-event times follow the proposed piecewise power-law family (10)-(11).
Section 3.1, the 'Example'; this is the paper's proposed model, asserted without empirical justification.
domain assumption A recurrent neural network of unspecified structure can map event histories to distribution parameters.
Section 4, 'As the space Theta we suggest recurrent neural networks (RNNs) in a broad sense'; no architecture or training details given.

how reviews work

0 comments

Cite this review

Pith. "Pith review of A Point Process Model for Optimizing Repeated Personalized Action Delivery to Users." pith.science (2026). https://pith.science/paper/5QRDU3AZ

@misc{pith2026250102961,
  author       = {Pith},
  title        = {Pith review of: A Point Process Model for Optimizing Repeated Personalized Action Delivery to Users},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5QRDU3AZ}},
  note         = {Machine review of arXiv:2501.02961}
}

read the original abstract

This paper provides a formalism for an important class of causal inference problems inspired by user-advertiser interaction in online advertiser. Then this formalism is specialized to an extension of temporal marked point processes and the neural point processes are suggested as practical solutions to some interesting special cases.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 16 canonical work pages

[16]

Int ensity-free learning of temporal point processes

Oleksandr Shchur, Marin Biloˇ s, and Stephan G¨ unnemann. Int ensity-free learning of temporal point processes. arXiv preprint arXiv:1909.12127 , 2019

arXiv 1909
[1]

The likelihood principle

James O Berger and Robert L Wolpert. The likelihood principle . IMS, 1988

work page 1988
[2]

Charles, D

L´ eon Bottou, Jonas Peters, Joaquin Qui˜ nonero-Candela, Denis X. Charles, D. Max Chickering, Elon Portugaly, Dipankar Ray, Patrice Simard, an d Ed Snelson. Counterfactual reasoning and learning systems: The exam- ple of computational advertising. Journal of Machine Learning Research , 14(101):3207–3260, 2013

work page 2013
[3]

de Finetti

B. de Finetti. Foresight: its logical laws, its subjective sources. In Break- throughs in Statistics: Foundations and Basic Theory , pages 127–133. Springer, 1992

work page 1992
[4]

Some applications of causal inference in the real world

Ciar´ an Gilligan-Lee. Some applications of causal inference in the real world. YouTube, 2024

work page 2024
[5]

Replacing the do-calculus with Bayes rule

Finnian Lattimore and David Rohde. Replacing the do-calculus with B ayes rule. arXiv preprint arXiv:1906.07125 , 2019

work page Pith review arXiv 1906
[6]

Incrementality bidding and attrib ution

Randall Lewis and Jeﬀrey Wong. Incrementality bidding and attrib ution. arXiv preprint arXiv:2208.12809 , 2022. 12In fact, they have no need to be ﬁxed and might be changed on the run. 12

arXiv 2022
[7]

A survey on causal inference for recom men- dation

Huishi Luo, Fuzhen Zhuang, Ruobing Xie, Hengshu Zhu, Deqing Wa ng, Zhulin An, and Yongjun Xu. A survey on causal inference for recom men- dation. The Innovation , 2024

work page 2024

Show all 20 references

[8]

Causal diagrams for empirical research

Judea Pearl. Causal diagrams for empirical research. Biometrika, 82(4):669–688, 1995

1995
[9]

Robins and Wasserman respond to a nobel prize win ner, 2012

James Robins. Robins and Wasserman respond to a nobel prize win ner, 2012

2012
[10]

Toward a curse of dimensionalit y ap- propriate (coda) asymptotic theory for semi-parametric models

James M Robins and Ya’acov Ritov. Toward a curse of dimensionalit y ap- propriate (coda) asymptotic theory for semi-parametric models. Statistics in medicine, 16(3):285–319, 1997

1997
[11]

Causal inference, is just inference: A beautifully simple idea that not everyone accepts

David Rohde. Causal inference, is just inference: A beautifully simple idea that not everyone accepts. In I (Still) Can’t Believe It’s Not Better! Workshop at NeurIPS 2021 , pages 75–79. PMLR, 2022

2021
[12]

Position paper: Why the shooting in the dark metho d domi- nates recommender systems practice; a call to abandon anti-uto pian think- ing

David Rohde. Position paper: Why the shooting in the dark metho d domi- nates recommender systems practice; a call to abandon anti-uto pian think- ing. arXiv preprint arXiv:2402.02152 , 2024

2024 arXiv
[13]

The central role of the pr opensity score in observational studies for causal eﬀects

Paul R Rosenbaum and Donald B Rubin. The central role of the pr opensity score in observational studies for causal eﬀects. Biometrika, 70(1):41–55, 1983

1983
[14]

Comment: Which ifs have causal answers

Donald B Rubin. Comment: Which ifs have causal answers. Journal of the American statistical association , 81(396):961–962, 1986

1986
[15]

Fathi M. Salem. Recurrent Neural Networks . Springer, 2022

2022
[17]

Neural temporal point processes: A review

Oleksandr Shchur, Ali Caner T¨ urkmen, Tim Januschowski, and Stephan G¨ unnemann. Neural temporal point processes: A review. arXiv preprint arXiv:2104.03528, 2021

2021 arXiv
[18]

On an example of Larry Wasserman

Christopher Sims. On an example of Larry Wasserman. online manuscript, available from Link , 2(10), 2006

2006
[19]

Causal inference for recommender systems

Yixin Wang, Dawen Liang, Laurent Charlin, and David M Blei. Causal inference for recommender systems. In Proceedings of the 14th ACM Con- ference on Recommender Systems , pages 426–431, 2020

2020
[20]

Simple statistical gradient-following algorithms fo r connectionist reinforcement learning

Ronald J Williams. Simple statistical gradient-following algorithms fo r connectionist reinforcement learning. Machine learning, 8:229–256, 1992. 13

1992

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [16]

Int ensity-free learning of temporal point processes

Oleksandr Shchur, Marin Biloˇ s, and Stephan G¨ unnemann. Int ensity-free learning of temporal point processes. arXiv preprint arXiv:1909.12127 , 2019

arXiv 1909

[2] [1]

The likelihood principle

James O Berger and Robert L Wolpert. The likelihood principle . IMS, 1988

work page 1988

[3] [2]

Charles, D

L´ eon Bottou, Jonas Peters, Joaquin Qui˜ nonero-Candela, Denis X. Charles, D. Max Chickering, Elon Portugaly, Dipankar Ray, Patrice Simard, an d Ed Snelson. Counterfactual reasoning and learning systems: The exam- ple of computational advertising. Journal of Machine Learning Research , 14(101):3207–3260, 2013

work page 2013

[4] [3]

de Finetti

B. de Finetti. Foresight: its logical laws, its subjective sources. In Break- throughs in Statistics: Foundations and Basic Theory , pages 127–133. Springer, 1992

work page 1992

[5] [4]

Some applications of causal inference in the real world

Ciar´ an Gilligan-Lee. Some applications of causal inference in the real world. YouTube, 2024

work page 2024

[6] [5]

Replacing the do-calculus with Bayes rule

Finnian Lattimore and David Rohde. Replacing the do-calculus with B ayes rule. arXiv preprint arXiv:1906.07125 , 2019

work page Pith review arXiv 1906

[7] [6]

Incrementality bidding and attrib ution

Randall Lewis and Jeﬀrey Wong. Incrementality bidding and attrib ution. arXiv preprint arXiv:2208.12809 , 2022. 12In fact, they have no need to be ﬁxed and might be changed on the run. 12

arXiv 2022

[8] [7]

A survey on causal inference for recom men- dation

Huishi Luo, Fuzhen Zhuang, Ruobing Xie, Hengshu Zhu, Deqing Wa ng, Zhulin An, and Yongjun Xu. A survey on causal inference for recom men- dation. The Innovation , 2024

work page 2024