REVIEW 3 major objections 6 minor 31 references
Policy Improvement with Style-Specific Demonstrations
T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read MPPO, a mixed online/offline PPO with positive-return demonstration filtering, improves suboptimal game agents to near-PPO proficiency while preserving their play styles.
desk verdict Useful empirical LfD method with a broken monotonic-improvement proof; the experiments are worth a look, the theory is not. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the mixed surrogate objective in Eq. (4), which estimates the performance difference $J_\Delta(\pi, \pi_k)$ by sampling states from a mixture $\beta\rho_D + (1-\beta)\rho_{\pi_k}$: the offline component draws states from the demonstration dataset and actions from the current policy, while the online component draws from the current policy's own state distribution. Each component is a clipped PPO surrogate, so the whole objective is a single loss applied uniformly to online and LfD actor samples. The style-preservation mechanism is implicit: demonstration trajectories are filtered to positive total return and replayed by LfD actors, biasing the empirical distribution of training samples toward the demonstrator; $D_{\mathrm{policy}}$, a total-variation distance between action distributions, is used to verify that the student stays close to the teacher.
What would settle it
Compute, for each state-action pair in the filtered demonstration sets, the teacher's advantage $A_{\pi_T}(s,a)$; if a large fraction are non-positive and $D_{\mathrm{policy}}$ still decreases during training, the stated precondition is not the operative mechanism. A controlled comparison would train MPPO with per-state positive-advantage filtering versus episode-return filtering and check whether style retention differs.
Extended reading notes
Core claim
The central claim is that proficiency and style are not competing objectives if demonstration data are treated as a soft prior rather than a hard imitation target. MPPO's objective is the PPO clipped surrogate applied to a mixture of state distributions: a fraction $\beta$ of states sampled from the demonstration dataset and $1-\beta$ from the policy's own rollouts. The offline component is justified by an offline policy-improvement result and the online component by the standard TRPO/PPO guarantees, giving the paper's claim of monotonic improvement; style preservation comes from filtering demonstrations to positive-return trajectories and replaying them through LfD actors, which shifts the empirical sample distribution toward the demonstrator without an explicit cloning loss. The paper measures the outcome with $D_{\mathrm{policy}}$, a total-variation distance between action distributions, and, in Mahjong, with a distribution distance over winning patterns. In all three testbeds MPPO improves over the demonstrator and matches or beats PPO, with $D_{\mathrm{policy}}$ values substantially lower than those of PPO.
Load-bearing premise
The proof that MPPO preserves style assumes every action in the kept demonstrations is better than average for its state (positive advantage), but the implementation keeps only entire trajectories whose total return is positive, which does not guarantee that per-action condition.
Editorial extensions
If this is right
- MPPO students surpass their demonstrators in all three environments and, in Mahjong, one MPPO agent overtakes the leaderboard's top-ranked bot despite starting from suboptimal demonstrations.
- Style retention shows up at both the action level ($D_{\mathrm{policy}}$) and strategy level (winning-pattern distribution in Mahjong), unlike PPO, which collapses pattern diversity.
- The seed-plus-action replay format reduces offline dataset storage by at least 98% in the reported environments while still supporting full-episode GAE advantage estimation.
- Only a small demonstration fraction, about 5% of incoming data, is needed for the effect, and the best fraction is environment-dependent, peaking at different ratios in Blackjack, Maze, and Mahjong.
- Because MPPO's loss is the same as PPO's, adopting the method is an actor-side data-collection change rather than a new learner.
Reading between the lines
- A mechanism-separating experiment: train MPPO with demonstrations filtered by per-state positive advantage instead of positive episode return; if style retention is unchanged, the theorem's condition is not the operative ingredient.
- The All Data ablation shows that dropping the filter eliminates style retention in every environment, so the filter threshold itself is an untested hyperparameter; sweeping it would map the style-versus-proficiency trade-off and likely find environment-specific optima, matching the observed ratio sensitivity.
- The recipe is demonstrated on discrete-action games, but the same mixed-objective idea naturally extends to continuous-control settings where 'style' is a movement signature; that would require replacing $D_{\mathrm{policy}}$ with a trajectory-level distance and testing whether replaying recorded actions still works when actions are continuous.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Mixed Proximal Policy Optimization (MPPO), a learning-from-demonstration method that continues PPO training while replaying demonstration trajectories from suboptimal, style-specific bots. The authors claim a monotonic improvement guarantee and a monotonic decrease in style distance to the demonstrator, and support these claims with experiments in Blackjack, Maze Navigation, and MCR Mahjong, using win rates, success rates, and the proposed Dpolicy/Dtarget style metrics. They also report a compact demonstration-storage format and an ablation/comparison against PPO, DQfD, GAIL, SAIL, and PPOfD.
Significance. MPPO addresses a practically relevant problem, and the empirical comparison is substantial: three environments, repeated runs with error bars, ablations over demonstration ratio and advantage estimator, and comparisons against several learning-from-demonstration baselines. The code release, benchmark environments, and the simple Dpolicy metric are useful contributions. However, the theoretical guarantees are advertised as a main contribution, and as presented they do not cover the implemented algorithm. The paper's current value therefore rests on the empirical results, which are promising but would need the theoretical claims to be corrected or removed.
major comments (3)
- [§4, Eq. (4) and Algorithm 1] The offline term in Eq. (4) is E_{s∼ρ_D, a∼π(·|s)}[A_{π_k}(s,a)], sampling actions from the student policy at demonstration states. Algorithm 1's LfD actors instead replay the demonstrator's recorded actions a_t (Algorithm 1 line 6), and the PPO ratio r=π_θ(a_t|s_t)/π_k(a_t|s_t) is a likelihood ratio for the recorded action, not an importance weight that corrects the mismatch between the replay distribution a∼π_T(·|s) and the target distribution a∼π(·|s). The objective actually optimized is therefore not Eq. (4), and the BPPO/TRPO monotonic-improvement arguments invoked in Section 4 do not transfer to the implemented loss. This is a load-bearing gap because the monotonic improvement claim is stated as a formal contribution.
- [§4, Theorem 2 and positive-return filter] Theorem 2 requires A_{π_T}(s_t,a_t)>0 for every (s_t,a_t) in each demonstration trajectory. Section 4 states that trajectories are filtered by positive total return, which does not imply per-step positive advantage; moreover, Algorithm 1 computes advantages with the student critic via GAE, not with the teacher policy π_T. Thus the theorem's precondition is neither enforced nor verifiable from the reported procedure. The 'All Data' ablation in Section 6 correctly notes that including all trajectories violates the theorem, but the positive-return filter used in the main experiments has the same logical gap.
- [§4, Theorem 2 proof] The proof of Theorem 2 assumes clipping is not activated, assumes a deterministic teacher (π_T(a|s_t)=1_{a=a_t}), and analyzes a hand-specified update π'_S(a_t|s_t)=π_S(a_t|s_t)+α∇π_S(a_t|s_t)π_T(a_t|s_t)A_{π_T}(s_t,a_t) rather than the clipped PPO update in Eq. (2). As printed, the gradient step is not a normalized policy-gradient update, and no argument connects it to the loss used by Algorithm 1. Consequently the claimed monotonic decrease of D_policy under MPPO's offline component is not established for the implemented method.
minor comments (6)
- [§3, Eq. (1)] Equation (1) should read E_{s∼ρ_π(·), a∼π'(·|s)}[A_π(s,a)]; the current notation E_{s∼π(·)} is undefined as written.
- [§4, PPO objective] In the displayed MPPO objective, the second clip term is printed as clip(r,1−ϵ,1+rM); it should presumably be clip(r,1−ϵ,1+ϵ).
- [§4, Theorem 2] The theorem statement should explicitly state the deterministic-teacher assumption and should define D_policy consistently with Eq. (3), including the 1/2 factor and the expectation over states; the proof appears to analyze a single state s_t.
- [§5.2] The text says 'as shown in Figure 2B' when reporting Maze D_policy results, but the results appear in Table 2B.
- [§4 and §5] The relationship between the sampling fraction β in Section 4 and the number of LfD actors used in Section 5 is not explicit; Section 5 says demonstration data accounts for approximately 5% via actor counts, while Section 4 defines β as a sampling fraction. Clarify whether these are the same mechanism.
- [§5.3] The D_policy computation for Mahjong demonstrators uses p(a|s)=1_{a=a_i} from 100 held-out trajectories, which treats the demonstrator as deterministic and may affect the measured distances; this limitation should be stated when interpreting Table 4B.
Circularity Check
No significant circularity: MPPO's style/proficiency claims are checked against external baselines and external theory, with the style metric used only evaluatively.
full rationale
MPPO's central proficiency and style-preservation claims are not reductions of fitted inputs. The performance-difference decomposition in Eq. (4) is explicitly built on Kakade-Langford, TRPO, and BPPO, all external results, and the paper says it 'linearly combin[es] the theoretical results from previous works' to obtain the LfD objective; this is an imported theoretical scaffold rather than a self-referential derivation. The style metric Dpolicy in Eq. (3) is defined as an evaluation instrument and is not used as a training loss or tuned to produce the reported numbers; the beta mixing ratio is hand-set and explicitly ablated in Section 6 (2x Demo and 0.5x Demo), and the positive-return trajectory filter is a method design choice, not a parameter fitted to a subset and then re-reported as a prediction. The only self-citations are contextual: the Mahjong complexity statement cites (Lu, Li, and Li 2023), and Botzone is cited as the benchmark platform; neither carries the load-bearing claim that MPPO improves suboptimal agents while retaining style. The genuine weaknesses are non-circular correctness concerns: Theorem 2 requires per-state positive advantage while Section 4 filters only on positive total episode returns, and Eq. (4) samples actions from the new policy pi(·|s) while Algorithm 1 replays recorded demonstrator actions, so the formal guarantee may not cover the implemented loss. These are fidelity gaps between theory and implementation, not cases where an input is renamed as a discovery or where the conclusion is forced by self-citation.
Assumptions & free parameters
free parameters (3)
- beta (demonstration data fraction) =
approximately 5% of incoming data (set via number of LfD actors)
- positive-return filter threshold =
0 (trajectories with total return > 0 are kept)
- demonstration dataset size =
approximately 30K trajectories per demonstrator
assumptions (6)
- standard math Performance difference lemma (Kakade and Langford 2002) as stated in Theorem 1.
- domain assumption TRPO and BPPO each provide monotonic improvement guarantees for their respective online and offline objectives.
- domain assumption Demonstrator policy πT is deterministic on demonstration states, so πT(a|st)=1_{a=at}.
- ad hoc to paper Positive total return of a demonstration trajectory implies AπT(st,at)>0 for every (s,a) in that trajectory.
- domain assumption The student policy's state distribution can be approximated by βρD+(1−β)ρπk.
- ad hoc to paper The PPO importance ratio r=πθ(a|s)/π_old(a|s) is valid for offline samples whose actions come from the demonstrator.
Cite this review
Pith. "Pith review of Policy Improvement with Style-Specific Demonstrations." pith.science (2026). https://pith.science/paper/23DD2OXY
@misc{pith2026250616995,
author = {Pith},
title = {Pith review of: Policy Improvement with Style-Specific Demonstrations},
year = {2026},
howpublished = {\url{https://pith.science/paper/23DD2OXY}},
note = {Machine review of arXiv:2506.16995}
}
read the original abstract
Proficient game agents with diverse play styles enrich the gaming experience and enhance the replay value of games. However, recent advancements in game AI based on reinforcement learning have predominantly focused on improving proficiency, whereas methods based on evolution algorithms generate agents with diverse play styles but exhibit subpar performance compared to RL methods. To address this gap, this paper proposes Mixed Proximal Policy Optimization (MPPO), a method designed to improve the proficiency of existing suboptimal agents while retaining their distinct styles. MPPO unifies loss objectives for both online and offline samples and introduces an implicit constraint to approximate demonstrator policies by adjusting the empirical distribution of samples. Empirical results across environments of varying scales demonstrate that MPPO achieves proficiency levels comparable to, or even superior to, pure online algorithms while preserving demonstrators' play styles. This work presents an effective approach for generating highly proficient and diverse game agents, ultimately contributing to more engaging gameplay experiences.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
Dota 2 with large scale deep reinforcement learning
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Christopher Hesse, Rafal J \' o zefowicz, Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique Pond \' e de Oliveira Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon...
arXiv 1912
-
[2]
Superhuman ai for multiplayer poker
Noam Brown and Tuomas Sandholm. Superhuman ai for multiplayer poker. Science , 365(6456):885--890, 2019
work page 2019
-
[3]
Nvidia redefines game ai with ace autonomous game characters, 2025
Andrew Burnes, Seth Schneider, and Jason Paul. Nvidia redefines game ai with ace autonomous game characters, 2025
work page 2025
-
[4]
IMPALA: scalable distributed deep-rl with importance weighted actor-learner architectures
Lasse Espeholt, Hubert Soyer, R \' e mi Munos, Karen Simonyan, Volodymyr Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, Shane Legg, and Koray Kavukcuoglu. IMPALA: scalable distributed deep-rl with importance weighted actor-learner architectures. CoRR , abs/1802.01561, 2018
arXiv 2018
-
[5]
Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks, 2014
2014
-
[6]
Todd Hester, Matej Vecer \' k, Olivier Pietquin, Marc Lanctot, Tom Schaul, Bilal Piot, Andrew Sendonaris, Gabriel Dulac - Arnold, Ian Osband, John P. Agapiou, Joel Z. Leibo, and Audrunas Gruslys. Learning from demonstrations for real world reinforcement learning. CoRR , abs/1704.03732, 2017
arXiv 2017
-
[7]
Generative adversarial imitation learning
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. CoRR , abs/1606.03476, 2016
arXiv 2016
-
[8]
Approximately optimal approximate reinforcement learning
Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In Proceedings of the Nineteenth International Conference on Machine Learning , ICML '02, page 267–274, San Francisco, CA, USA, 2002. Morgan Kaufmann Publishers Inc
work page 2002
Show all 31 references
-
[9]
Policy optimization with demonstrations
Bingyi Kang, Zequn Jie, and Jiashi Feng. Policy optimization with demonstrations. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pages 2469--2478. PMLR, ...
2018
-
[10]
Conservative q-learning for offline reinforcement learning
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. CoRR , abs/2006.04779, 2020
2006 arXiv
-
[11]
Method for constructing artificial intelligence player with abstractions to markov decision processes in multiplayer game of mahjong
Moyuru Kurita and Kunihito Hoki. Method for constructing artificial intelligence player with abstractions to markov decision processes in multiplayer game of mahjong. IEEE Transactions on Games , 13(1):99--110, 2021
2021
-
[12]
A unified game-theoretic approach to multiagent reinforcement learning
Marc Lanctot, Vinicius Zambaldi, Audr\= u nas Gruslys, Angeliki Lazaridou, Karl Tuyls, Julien P\' e rolat, David Silver, and Thore Graepel. A unified game-theoretic approach to multiagent reinforcement learning. In Proceedings of the 31st International Conference on Neural Inf...
2017
-
[13]
Suphx: Mastering mahjong with deep reinforcement learning
Junjie Li, Sotetsu Koyamada, Qiwei Ye, Guoqing Liu, Chao Wang, Ruihan Yang, Li Zhao, Tao Qin, Tie - Yan Liu, and Hsiao - Wuen Hon. Suphx: Mastering mahjong with deep reinforcement learning. CoRR , abs/2003.13590, 2020
2003 arXiv
-
[14]
Official international mahjong: A new playground for ai research
Yunlong Lu, Wenxin Li, and Wenlong Li. Official international mahjong: A new playground for ai research. Algorithms , 16(5), 2023
2023
-
[15]
Building a computer mahjong player based on monte carlo simulation and opponent models
Naoki Mizukami and Yoshimasa Tsuruoka. Building a computer mahjong player based on monte carlo simulation and opponent models. In 2015 IEEE Conference on Computational Intelligence and Games (CIG) , pages 275--283, 2015
2015
-
[16]
Overcoming exploration in reinforcement learning with demonstrations
Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Overcoming exploration in reinforcement learning with demonstrations. CoRR , abs/1709.10089, 2017
2017 arXiv
-
[17]
Handbook on mahjong competition rules, 2016
Vitaly Novikov. Handbook on mahjong competition rules, 2016. Accessed: 2025-01-17
2016
-
[18]
Philipp Rohlfshagen, Jialin Liu, Diego Perez-Liebana, and Simon M. Lucas. Pac-man conquers academia: Two decades of research using a classic arcade game. IEEE Transactions on Games , 10(3):233--256, 2018
2018
-
[19]
Jordan, and P
John Schulman, Philipp Moritz, Sergey Levine, Michael I. Jordan, and P. Abbeel. High-dimensional continuous control using generalized advantage estimation. CoRR , abs/1506.02438, 2015
2015 arXiv
-
[20]
Jordan, and Pieter Abbeel
John Schulman, Sergey Levine, Philipp Moritz, Michael I. Jordan, and Pieter Abbeel. Trust region policy optimization, 2017
2017
-
[21]
Proximal policy optimization algorithms
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR , abs/1707.06347, 2017
2017 arXiv
-
[22]
Lillicrap, Karen Simonyan, and Demis Hassabis
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy P. Lillicrap, Karen Simonyan, and Demis Hassabis. Mastering chess and shogi by self-play with a general reinfo...
2017 arXiv
-
[23]
Riedmiller
Matej Vecer \' k, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pietquin, Bilal Piot, Nicolas Heess, Thomas Roth \" o rl, Thomas Lampe, and Martin A. Riedmiller. Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards. CoRR , abs/...
2017 arXiv
-
[24]
Czarnecki, Micha \"e l Mathieu, Andrew Dudzik, Junyoung Chung, David H
Oriol Vinyals, Igor Babuschkin, Wojciech M. Czarnecki, Micha \"e l Mathieu, Andrew Dudzik, Junyoung Chung, David H. Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang, Laurent Sifre, Trevor Cai, John P. Agapiou, M...
2019
-
[25]
Perfectdou: Dominating doudizhu with perfect information distillation, 2024
Guan Yang, Minghuan Liu, Weijun Hong, Weinan Zhang, Fei Fang, Guangjun Zeng, and Yue Lin. Perfectdou: Dominating doudizhu with perfect information distillation, 2024
2024
-
[26]
Towards playing full MOBA games with deep reinforcement learning
Deheng Ye, Guibin Chen, Wen Zhang, Sheng Chen, Bo Yuan, Bo Liu, Jia Chen, Zhao Liu, Fuhao Qiu, Hongsheng Yu, Yinyuting Yin, Bei Shi, Liang Wang, Tengfei Shi, Qiang Fu, Wei Yang, Lanxiao Huang, and Wei Liu. Towards playing full MOBA games with deep reinforcement learning. CoRR ...
2011 arXiv
-
[27]
Xiangyu Zhao and Sean B. Holden. Building a 3-player mahjong ai using deep reinforcement learning, 2022
2022
-
[28]
Botzone: an online multi-agent competitive platform for ai education
Haoyu Zhou, Haifeng Zhang, Yushan Zhou, Xinchao Wang, and Wenxin Li. Botzone: an online multi-agent competitive platform for ai education. In Proceedings of the 23rd Annual ACM Conference on Innovation and Technology in Computer Science Education , ITiCSE 2018, page 33–38, New...
2018
-
[29]
Learning sparse rewarded tasks from sub-optimal demonstrations
Zhuangdi Zhu, Kaixiang Lin, Bo Dai, and Jiayu Zhou. Learning sparse rewarded tasks from sub-optimal demonstrations. CoRR , abs/2004.00530, 2020
2004 arXiv
-
[30]
Behavior proximal policy optimization, 2023
Zifeng Zhuang, Kun Lei, Jinxin Liu, Donglin Wang, and Yilang Guo. Behavior proximal policy optimization, 2023
2023
-
[31]
write newline
" write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence '...
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.