Self-Play Reinforcement Learning under Imperfect Information in Big 2

Aalok Patwa

arxiv: 2605.28863 · v2 · pith:M4NAEV5Onew · submitted 2026-05-21 · 💻 cs.LG · cs.AI

Self-Play Reinforcement Learning under Imperfect Information in Big 2

Aalok Patwa This is my paper

Pith reviewed 2026-06-30 16:38 UTC · model grok-4.3

classification 💻 cs.LG cs.AI

keywords reinforcement learningself-playimperfect informationBig 2PPOmultiplayer gamespolicy gradientcard games

0 comments

The pith

Under identical conditions, PPO outperforms Monte Carlo Q approximation, SARSA, and Q-learning in self-play training for the imperfect-information card game Big 2.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper builds a self-play reinforcement learning framework for Big 2 to enable direct comparisons of policy-gradient and value-based agents. It reports that PPO achieves higher performance than the other three methods when all share the same environment, input representation, training budget, and evaluation protocol. Moderate entropy regularization further improves PPO by keeping policies from turning overly deterministic. Current-policy self-play also supplies a stronger training signal than checkpoint or fixed-opponent variants within the given budget. The work positions Big 2 as a controlled testbed for deep RL issues such as hidden information, non-stationary opponents, and variable action sets.

Core claim

We develop a self-play RL framework for Big 2 that enables controlled comparisons between policy-gradient and value-approximating agents. Under a common environment, input representation, training budget, and evaluation protocol, PPO outperforms Monte Carlo Q approximation, SARSA, and Q-learning against random, greedy, and heuristic Big 2 opponents. We further find that moderate entropy regularization improves PPO by preventing the policy from becoming overly deterministic, and that current-policy self-play provides a stronger finite-budget curriculum than checkpoint self-play or fixed-opponent training. Together, these results show that Big 2 is a useful controlled setting for studying deep

What carries the argument

self-play RL framework that standardizes environment, input representation, training budget, and evaluation protocol for comparing PPO against value-based methods in Big 2

If this is right

PPO records higher win rates than the tested value-based methods against random, greedy, and heuristic opponents in Big 2.
Moderate entropy regularization reduces premature convergence to deterministic policies during PPO training.
Current-policy self-play yields stronger learning progress than checkpoint self-play or fixed-opponent training under limited budgets.
Big 2 functions as a repeatable benchmark for RL methods that must handle imperfect information, multiplayer non-stationarity, and variable action sets.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The same standardized comparison setup could be reused to test newer RL algorithms on Big 2 without re-deriving the environment.
Results may indicate that policy gradients handle sparse rewards and hidden information more readily than value approximation when action spaces change per state.
Longer training runs or different opponent curricula could be tested to check whether value-based methods close the gap once the policy stabilizes.

Load-bearing premise

The shared environment, input representation, training budget, and evaluation protocol contain no hidden implementation choices that systematically favor policy-gradient methods over value-based ones.

What would settle it

An independent reimplementation of the same framework and protocol that finds value-based methods matching or exceeding PPO win rates against the same opponent classes when only random seeds or minor encoding details are altered.

Figures

Figures reproduced from arXiv: 2605.28863 by Aalok Patwa.

**Figure 1.** Figure 1: Neural architectures for different learning algorithms. The PPO policy and Q-network share the same card-aware state encoder, action encoder, and dot product scorer. The PPO policy network includes a value head. The SARSA variant uses the one-step on-policy target, yt = rt + γQtarget(ot+1, at+1), where at+1 is the next action actually selected by the behavior policy at the next model-controlled decision p… view at source ↗

**Figure 2.** Figure 2: Learning curves for PPO, Monte Carlo Q, SARSA, and Q-learning in four-player Big 2. Each checkpoint is evaluated against fixed random, greedy, and smart heuristic opponent pools. We report win rate and average score for the model-controlled seat. 5.3. Current-policy self-play outperforms alternative curricula We next ablate the opponent distribution used during training. The default setting trains against… view at source ↗

**Figure 3.** Figure 3: Average policy entropy during PPO current self-play training for different entropy coefficients. Entropy is sampled every 100 training batches. Larger entropy coefficients keep the policy more stochastic over the course of training, while the run with no entropy bonus becomes increasingly deterministic. E. Heuristic Baselines E.1. Greedy Heuristic Baseline The greedy baseline is a deterministic rule-based … view at source ↗

read the original abstract

Imperfect-information multiplayer games test whether agents can act under hidden information, sparse rewards, and non-stationary opponents. We study these challenges in Big 2, a four-player imperfect-information card game. We develop a self-play RL framework for Big 2 that enables controlled comparisons between policy-gradient and value-approximating agents. Under a common environment, input representation, training budget, and evaluation protocol, PPO outperforms Monte Carlo Q approximation, SARSA, and Q-learning against random, greedy, and heuristic Big 2 opponents. We further find that moderate entropy regularization improves PPO by preventing the policy from becoming overly deterministic, and that current-policy self-play provides a stronger finite-budget curriculum than checkpoint self-play or fixed-opponent training. Together, these results show that Big 2 is a useful controlled setting for studying deep RL under imperfect information, multiplayer interaction, delayed rewards, and variable action sets.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

Big 2 gets a controlled self-play RL treatment with PPO coming out ahead, but the abstract supplies no numbers or implementation checks to back the claim.

read the letter

The paper applies existing self-play RL methods to Big 2 and reports that PPO beats Monte Carlo Q, SARSA, and Q-learning under a shared environment, representation, budget, and evaluation setup. It also notes that moderate entropy helps PPO and that current-policy self-play works better than checkpoints or fixed opponents for the training length they used.

Those two practical observations are the parts that could actually transfer to other imperfect-information card games. The controlled comparison across algorithm families is the main new element; Big 2 itself is presented as a useful testbed because of hidden cards, four players, variable action sets, and delayed rewards.

The soft spot is the missing evidence. The abstract states the outperformance without win rates, variances, or any hyperparameter or encoding details. The stress-test concern holds: nothing in the supplied text confirms that value-based agents got equivalent exploration, action masking, or reward scaling as PPO. If the full methods section shows the setups were literally identical, the result gains weight; if any component differed, the comparison does not isolate policy-gradient advantages.

This is for people already working on multi-agent RL in card games who need a new benchmark. A reader looking for incremental empirical results on self-play curricula could find it useful. The work is internally coherent and cites the relevant self-play literature, so it shows honest engagement.

I would bring it to a reading group as a maybe to look at the implementation choices. I would not cite it in the next year unless the numbers prove substantial. It deserves peer review so referees can verify whether the common protocol really was common.

Referee Report

2 major / 2 minor

Summary. The manuscript develops a self-play RL framework for the four-player imperfect-information card game Big 2. It performs controlled comparisons between PPO and value-based methods (Monte Carlo Q approximation, SARSA, Q-learning) under a common environment, input representation, training budget, and evaluation protocol. The results indicate that PPO outperforms the other methods against random, greedy, and heuristic opponents. Additional findings include benefits from moderate entropy regularization in PPO and the superiority of current-policy self-play over checkpoint self-play or fixed-opponent training.

Significance. If the experimental controls are as described, this work provides a valuable controlled testbed for studying deep RL challenges in imperfect information, multiplayer, sparse reward settings. The identification of PPO's advantages and the effects of entropy and self-play curricula could inform algorithm design in similar domains. The use of a standard game like Big 2 with variable action sets is a positive aspect for reproducibility and comparability.

major comments (2)

[Abstract] Abstract: the claim that PPO outperforms Monte Carlo Q approximation, SARSA, and Q-learning is stated without any numerical win rates, margins, variance estimates, or statistical tests. This makes the magnitude and reliability of the central comparative result impossible to assess from the provided text.
The central claim requires literally identical environment dynamics, state encoding, action masking, reward scaling, training steps, and opponent sampling across PPO and the value-based agents. No pseudocode, hyperparameter tables, or implementation notes are supplied to confirm equivalent exploration mechanisms or curricula, rendering the outperformance claim unverifiable and load-bearing for the paper's conclusions.

minor comments (2)

Add explicit variance estimates or confidence intervals to all reported performance comparisons.
Clarify the precise form of the input representation and any action masking used for all agents.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the detailed and constructive report. The two major comments highlight the need for quantitative support in the abstract and greater transparency in the experimental protocol. We agree that both points require revision and will incorporate the requested details in the next version of the manuscript.

read point-by-point responses

Referee: [Abstract] Abstract: the claim that PPO outperforms Monte Carlo Q approximation, SARSA, and Q-learning is stated without any numerical win rates, margins, variance estimates, or statistical tests. This makes the magnitude and reliability of the central comparative result impossible to assess from the provided text.

Authors: We agree that the abstract should report concrete performance numbers. In the revised manuscript we will add the observed win rates (with standard deviations across independent runs) against each opponent type, the performance margins relative to the value-based baselines, and a brief statement on the statistical significance of the differences. revision: yes
Referee: [—] The central claim requires literally identical environment dynamics, state encoding, action masking, reward scaling, training steps, and opponent sampling across PPO and the value-based agents. No pseudocode, hyperparameter tables, or implementation notes are supplied to confirm equivalent exploration mechanisms or curricula, rendering the outperformance claim unverifiable and load-bearing for the paper's conclusions.

Authors: The manuscript asserts a common environment and training budget, but we accept that the current text does not supply sufficient implementation artifacts to allow independent verification. We will add (i) a hyperparameter table listing learning rates, entropy coefficients, network architectures, and training step counts for all algorithms, (ii) pseudocode for the self-play loop and opponent sampling procedure, and (iii) explicit statements confirming that state encoding, action masking, and reward scaling were held identical across agents. These additions will make the controlled comparison reproducible. revision: yes

Circularity Check

0 steps flagged

No circularity: purely empirical performance comparisons

full rationale

The paper reports direct experimental results comparing PPO, Monte Carlo Q, SARSA, and Q-learning in a shared Big 2 environment. No equations, derivations, or predictions are present that reduce reported win rates or performance metrics to quantities defined by fitted parameters within the paper. The central claims rest on empirical measurements against fixed opponent types rather than any self-referential construction, self-citation load-bearing argument, or ansatz smuggled via prior work. This is the expected outcome for an empirical RL benchmark paper.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

The paper is an empirical RL study and introduces no new mathematical axioms, free parameters, or invented entities. All claims rest on standard RL assumptions (Markov decision process formulation, function approximation) that are inherited from prior literature.

pith-pipeline@v0.9.1-grok · 5676 in / 1193 out tokens · 35291 ms · 2026-06-30T16:38:49.168152+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

8 extracted references · 8 canonical work pages · 2 internal anchors

[1]

Application of Self-Play Reinforcement Learning to a Four-Player Game of Imperfect Information

URL https://arxiv.org/ abs/1808.10442. Chen, L.-W. and Lu, Y .-R. Challenging artificial intelligence with multiopponent and multimovement prediction for the card game Big2.IEEE Access, 10:40661–40676,

work page internal anchor Pith review Pith/arXiv arXiv
[2]

Chen, L.-W

doi: 10.1109/ACCESS.2022.3166932. Chen, L.-W. and Lu, Y .-R. Markov decision process-based artificial intelligence with card-playing strategy and free- playing right exploration for four-player card game Big2. IEEE Transactions on Games, 17(2):267–281,

work page doi:10.1109/access.2022.3166932 2022
[3]

Heinrich, J

doi: 10.1109/TG.2024.3424431. Heinrich, J. and Silver, D. Deep reinforcement learning from self-play in imperfect-information games,

work page doi:10.1109/tg.2024.3424431 2024
[4]

Deep Reinforcement Learning from Self-Play in Imperfect-Information Games

URL https://arxiv.org/abs/1603.01121. Li, B., Fang, Z., and Huang, L. RL-CFR: Improving ac- tion abstraction for imperfect information extensive-form games with reinforcement learning. InProceedings of the 41st International Conference on Machine Learning, vol- ume 235 ofProceedings of Machine Learning Research, pp. 27752–27770. PMLR,

work page internal anchor Pith review Pith/arXiv arXiv
[5]

Liu, M., Farina, G., and Ozdaglar, A

URLhttps://arxiv.org/abs/2003.13590. Liu, M., Farina, G., and Ozdaglar, A. E. A policy-gradient approach to solving imperfect-information games with best-iterate convergence. InInternational Conference on Learning Representations,

work page arXiv 2003
[6]

Moravcik, M., Schmid, M., Burch, N., Lisy, V ., Morrill, D., Bard, N., Davis, T., Waugh, K., Johanson, M., and Bowl- ing, M

doi: 10.1016/j.asoc.2024.111545. Moravcik, M., Schmid, M., Burch, N., Lisy, V ., Morrill, D., Bard, N., Davis, T., Waugh, K., Johanson, M., and Bowl- ing, M. Deepstack: Expert-level artificial intelligence in heads-up no-limit poker.Science, 356(6337):508–513,

work page doi:10.1016/j.asoc.2024.111545 2024
[7]

Schmid, M., Moravcik, M., Burch, N., Kadlec, R., David- son, J., Waugh, K., Bard, N., Timbers, F., Lanctot, M., Holland, G

doi: 10.1126/science.add4679. Schmid, M., Moravcik, M., Burch, N., Kadlec, R., David- son, J., Waugh, K., Bard, N., Timbers, F., Lanctot, M., Holland, G. Z., Davoodi, E., Christianson, A., and Bowling, M. Student of games: A unified learning algorithm for both perfect and imperfect information games.Science Advances, 9(46):eadg3256,

work page doi:10.1126/science.add4679
[8]

Schrittwieser, J., Antonoglou, I., Hubert, T., Simonyan, K., Sifre, L., Schmitt, S., Guez, A., Lockhart, E., Hassabis, D., Graepel, T., Lillicrap, T., and Silver, D

doi: 10.1126/sciadv.adg3256. Schrittwieser, J., Antonoglou, I., Hubert, T., Simonyan, K., Sifre, L., Schmitt, S., Guez, A., Lockhart, E., Hassabis, D., Graepel, T., Lillicrap, T., and Silver, D. Mastering atari, go, chess and shogi by planning with a learned model.Nature, 588(7839):604–609,

work page doi:10.1126/sciadv.adg3256

[1] [1]

Application of Self-Play Reinforcement Learning to a Four-Player Game of Imperfect Information

URL https://arxiv.org/ abs/1808.10442. Chen, L.-W. and Lu, Y .-R. Challenging artificial intelligence with multiopponent and multimovement prediction for the card game Big2.IEEE Access, 10:40661–40676,

work page internal anchor Pith review Pith/arXiv arXiv

[2] [2]

Chen, L.-W

doi: 10.1109/ACCESS.2022.3166932. Chen, L.-W. and Lu, Y .-R. Markov decision process-based artificial intelligence with card-playing strategy and free- playing right exploration for four-player card game Big2. IEEE Transactions on Games, 17(2):267–281,

work page doi:10.1109/access.2022.3166932 2022

[3] [3]

Heinrich, J

doi: 10.1109/TG.2024.3424431. Heinrich, J. and Silver, D. Deep reinforcement learning from self-play in imperfect-information games,

work page doi:10.1109/tg.2024.3424431 2024

[4] [4]

Deep Reinforcement Learning from Self-Play in Imperfect-Information Games

URL https://arxiv.org/abs/1603.01121. Li, B., Fang, Z., and Huang, L. RL-CFR: Improving ac- tion abstraction for imperfect information extensive-form games with reinforcement learning. InProceedings of the 41st International Conference on Machine Learning, vol- ume 235 ofProceedings of Machine Learning Research, pp. 27752–27770. PMLR,

work page internal anchor Pith review Pith/arXiv arXiv

[5] [5]

Liu, M., Farina, G., and Ozdaglar, A

URLhttps://arxiv.org/abs/2003.13590. Liu, M., Farina, G., and Ozdaglar, A. E. A policy-gradient approach to solving imperfect-information games with best-iterate convergence. InInternational Conference on Learning Representations,

work page arXiv 2003

[6] [6]

Moravcik, M., Schmid, M., Burch, N., Lisy, V ., Morrill, D., Bard, N., Davis, T., Waugh, K., Johanson, M., and Bowl- ing, M

doi: 10.1016/j.asoc.2024.111545. Moravcik, M., Schmid, M., Burch, N., Lisy, V ., Morrill, D., Bard, N., Davis, T., Waugh, K., Johanson, M., and Bowl- ing, M. Deepstack: Expert-level artificial intelligence in heads-up no-limit poker.Science, 356(6337):508–513,

work page doi:10.1016/j.asoc.2024.111545 2024

[7] [7]

Schmid, M., Moravcik, M., Burch, N., Kadlec, R., David- son, J., Waugh, K., Bard, N., Timbers, F., Lanctot, M., Holland, G

doi: 10.1126/science.add4679. Schmid, M., Moravcik, M., Burch, N., Kadlec, R., David- son, J., Waugh, K., Bard, N., Timbers, F., Lanctot, M., Holland, G. Z., Davoodi, E., Christianson, A., and Bowling, M. Student of games: A unified learning algorithm for both perfect and imperfect information games.Science Advances, 9(46):eadg3256,

work page doi:10.1126/science.add4679

[8] [8]

Schrittwieser, J., Antonoglou, I., Hubert, T., Simonyan, K., Sifre, L., Schmitt, S., Guez, A., Lockhart, E., Hassabis, D., Graepel, T., Lillicrap, T., and Silver, D

doi: 10.1126/sciadv.adg3256. Schrittwieser, J., Antonoglou, I., Hubert, T., Simonyan, K., Sifre, L., Schmitt, S., Guez, A., Lockhart, E., Hassabis, D., Graepel, T., Lillicrap, T., and Silver, D. Mastering atari, go, chess and shogi by planning with a learned model.Nature, 588(7839):604–609,

work page doi:10.1126/sciadv.adg3256