Mastering Da Vinci Code: A Comparative Study of Transformer, LLM, and PPO-based Agents

Haotian Shen; Lecheng Zhang; Xujie Wang; Yuanshi Wang

REVIEW 4 major objections 5 minor 17 references

Mastering Da Vinci Code: A Comparative Study of Transformer, LLM, and PPO-based Agents

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A PPO-based agent with a Transformer encoder over full game history wins 58.5% ± 1.0% against a baseline, surpassing prompted LLM agents in Da Vinci Code.

desk verdict The comparative PPO-vs-LLM claim doesn't follow from the data because all win rates are measured against a single uncalibrated baseline, but the paper's environment, code, and 10,000-game PPO evaluation are real and worth a serious referee's time. read the letter →

arxiv 2506.12801 v1 pith:M5FRZDZA submitted 2025-06-15 cs.AI

LeCheng Zhang , Yuanshi Wang , Haotian Shen , Xujie Wang This is my paper · ORCID

classification cs.AI

keywords DaVinciCodeimperfect-informationgamesreinforcementlearningProximalPolicyOptimizationlargelanguagemodelslogicaldeductiongame-playingagents

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that for the imperfect-information deduction game Da Vinci Code, a self-trained reinforcement-learning agent is the strongest of three agent families tested. The PPO-based agent, which encodes the full game history with a Transformer, wins 58.5% ± 1.0% of 10,000 games against a history-limited Transformer baseline. This is well above every prompted LLM agent, whose best rate is 38.9%, and it sits between the paper's median and high human-expert benchmarks of 50.0% and 64.0%. The authors read this as evidence that learning implicit deductive strategies from self-play outperforms prompting general-purpose language models, and that a history-aware state representation is a key ingredient.

What carries the argument

The central mechanism is a history-aware state encoder: the game state and chronological event log are serialized into a string, tokenized with a 64-token domain-specific vocabulary, and passed through a Transformer encoder whose first-token output becomes the state vector for the policy and value networks. Combined with the clipped PPO objective, generalized advantage estimation, legal-action masking, and a 339-action discrete space (one place action plus guesses over 13 opponent positions and 26 card values), this lets the agent learn from self-play while retaining long-range dependencies that the baseline's snapshot input lacks.

What would settle it

Run a round-robin tournament with enough games per pairing, for example 2,000 or more, in which the trained PPO agent plays directly against each LLM agent and against a random agent, and check whether the PPO agent's win rate stays above chance; a prompted LLM agent winning at or above 50% would contradict the paper's central comparative claim.

Watch

Extended reading notes

Core claim

The paper's discovery is that an agent trained entirely by self-play with Proximal Policy Optimization, using a Transformer to encode the full game history, becomes the strongest of the three agent families tested. Over 10,000 games against the same history-limited Transformer opponent, this agent wins 58.5% ± 1.0%, while the prompted LLM agents win between 17.0% and 38.9% depending on the model. The PPO agent approaches the paper's high-level human benchmark of 64.0% and sits above the median expert benchmark of 50.0%. The authors attribute the gap to the PPO agent's ability to learn implicit deductive strategies from experience and to track and update constraints over full game history, whereas the LLMs, despite structured prompts and a suggested minimum-candidate-set heuristic, sometimes issued logically inconsistent moves.

Load-bearing premise

The ranking rests on win rates against a single history-limited Transformer baseline, and the paper treats those win rates as a transitive measure of relative agent strength; if that baseline is unrepresentative or exploitable in a way the compared agents exploit unevenly, the ranking would not generalize.

Editorial extensions

If this is right

For deduction games with hidden information, a history-aware self-play reinforcement-learning agent is a more reliable path to strong play than prompting a general-purpose LLM.
The PPO agent's 58.5% win rate against the baseline, between the median and high human benchmarks, suggests expert-level play is reachable with this architecture and modest training.
The baseline Transformer's poorer showing indicates that reasoning from a state snapshot without full game history loses strategic information that matters in this game.
The occasional logically inconsistent outputs from some LLMs imply that rule adherence, not just reasoning ability, is a bottleneck for using LLMs as game agents.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the paper's ranking is correct, a direct head-to-head match between the PPO agent and the best LLM agent should replicate the margin; that match is the natural next experiment and would also control for baseline-specific exploitation.
The PPO agent's edge may depend on the paper's reward shaping, with +3.0 for a win, +0.2 for a correct guess, and −0.5 for an incorrect guess; ablating those weights would reveal whether the discovered strategy is robust or tuned to that objective.
The same history-encoding design should transfer to similar deduction games with hidden tiles, such as Mastermind variants; a positive transfer result would support the paper's broader claim about history-aware reinforcement learning.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The one thing you should know: the headline claim rests on a comparison that the experiments don't actually make. Every win rate in Table 1 is against the same history-limited Transformer baseline, and win rates against a common opponent are not transitive in a two-player zero-sum game. The paper says PPO "significantly outperforms" the LLMs, but there are no PPO-vs-LLM games anywhere. The stress-test note has it right. If the baseline is weak or exploitable in a way the LLMs can't exploit, PPO's 58.5% could just mean it learned to beat that specific model.

That said, the paper does real work. It builds a clean Da Vinci Code environment, implements a PPO agent with a Transformer encoder and a sensible action space, releases the code, and runs 10,000 games for PPO. That is a reproducible, well-specified evaluation pipeline. The LLM prompting setup is also carefully described, including the fallback mechanism for invalid JSON. As a first systematic comparison of these agent families on this game, it's a legitimate data point.

The soft spots are real but addressable. The baseline is never calibrated: no win rate against random play or against humans, and its action-selection rule is only partially specified. LLM sample sizes are tiny (29 to 100 games), so the confidence intervals are huge and the ranking among LLMs is noise. The prompt encodes a specific median-candidate heuristic, so the LLM results partly test that heuristic rather than the raw reasoning ability of the models. The human comparison uses 20 and 25 games. The manuscript itself admits to missing learning curves and game statistics, and Figure 1 is a placeholder. The "O3" model is never fully identified.

I'd send this to peer review, but with the clear expectation of major revision. The evaluation design needs to include head-to-head matches, a calibrated baseline, and larger LLM samples. The authors seem capable of doing this; the paper's own discussion acknowledges most of these gaps. As it stands, the empirical claim is plausible but not proven.

For a reader, the value is in the environment and the cautionary example: it shows how easy it is to overstate a comparative conclusion when all agents are measured against a single uncalibrated opponent. I wouldn't cite the comparative result, but I might cite the environment if I worked on small imperfect-information games.

Referee Report

4 major / 5 minor

Summary. The paper develops a Da Vinci Code game environment and implements three classes of agents: a history-limited Transformer baseline, several prompted LLM agents (Gemini, DeepSeek, GPT-4o, O3, Qwen), and a PPO-trained agent with a Transformer encoder. All agents are evaluated by win rate against the Transformer baseline over a varying number of games (Section 4, Table 1). The paper reports that the PPO agent achieves a 58.5% ± 1.0% win rate over 10,000 games, far exceeding all LLM agents, and interprets this as evidence that deep reinforcement learning with comprehensive history encoding outperforms prompted LLMs in this deductive game. The conclusion also states the PPO agent approaches the proficiency of skilled human players.

Significance. If the comparative claim were properly supported, this would be a useful empirical contribution: it provides an open-source game environment, benchmarks several state-of-the-art LLMs against a dedicated RL agent, and explores a relatively understudied imperfect-information deduction game. The authors are transparent about missing analyses (e.g., learning curves, game statistics) and make the code available. However, the central claim is currently supported only by win rates against a single, uncalibrated baseline, with no direct PPO-versus-LLM matches and very small LLM sample sizes; as it stands, the evidence does not justify the comparative conclusion. The manuscript also contains placeholder figures, which indicates it is not in a polished state.

major comments (4)

[Section 4, Table 1] The central claim that the PPO agent 'significantly outperforms all LLM agents' rests solely on win rates against the history-limited Transformer baseline. In a two-player zero-sum game, a high win rate against a common opponent does not imply pairwise superiority (rock-paper-scissors cycles are possible). No head-to-head PPO-versus-LLM games are reported anywhere in the manuscript, and the baseline's own strength is never calibrated against random play, an MCTS agent, or human players. To support the comparative claim, the authors need either direct pairwise matches or a calibrated common-opponent protocol that justifies transitivity.
[Section 3.2] The baseline model's action-selection rule is not specified. The text says it predicts opponent tiles 'token by token,' but it does not explain how these predictions translate into the environment's legal actions ('place' vs. 'guess' with a position and card value). Without knowing the baseline's policy, readers cannot assess whether it is a reasonable common opponent or a weak, deterministic policy that a PPO agent can exploit more easily than an LLM can. This missing detail is load-bearing for interpreting every win rate in Table 1.
[Section 4, Table 1] Several LLM results are based on very small samples: DeepSeek R1 has n=36 (CI ±15.9 percentage points) and Qwen3-235B-A22B has n=29 (CI ±16.3). The paper's ranking of LLMs and the claim that all are 'significantly' worse than the PPO agent would be more convincing with larger sample sizes and a proper statistical comparison that accounts for multiple testing and the transliteracy of the common-opponent design. The current evidence supports only a qualitative ordering, not a strong significance claim.
[Section 4 and Conclusion] The human-comparison claim is weakly supported. 'Human Expert (Median)' and 'Human Expert (High)' are based on 20 and 25 games, respectively, with 95% confidence intervals of ±21.9 and ±18.8 percentage points. The criteria for selecting these experts and the conditions of play are not described. The statement that the PPO agent 'approaches the proficiency of skilled human players' goes beyond what this small, poorly specified sample can establish.

minor comments (5)

[Table 1] The row group label 'LLMs - Ous Category Models' appears to be a typo; it should likely read 'Other Category Models' or 'OpenAI Category Models.'
[Section 4 and Appendix A.2] Figure 1 and Figure 2 are still placeholders ('Please replace this placeholder with figure1.png' and 'figure_2.png'). This makes the manuscript incomplete and prevents readers from seeing the intended visualizations.
[Section 3.3] The 'unspecified O3 model' is not described with a version, access date, or API details, which makes replication difficult. The same applies to the exact prompts for the other LLMs beyond the Gemini example.
[References] The Tree-of-Thought reference is listed as 'Shinn Yao et al.'; the correct first author for that paper is Shunyu Yao. The TRPO citation also lacks a complete author list and venue details.
[Section 6] The statement that 'A GUI for Transformer vs. human play was also partially developed' is a scope limitation that should be made explicit in the abstract or introduction, since the paper otherwise appears to present a complete evaluation environment.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the results are external empirical measurements against a common baseline, and no fitted parameter or self-citation forces the outcome.

full rationale

The paper's core claims are empirical win rates measured in a custom Da Vinci Code environment against a Transformer baseline. The PPO agent is trained by self-play, the LLM agents are prompted, and the baseline is a separately trained predictor; none of these components is defined in terms of the reported win rates. The 58.5% win rate is an external measurement, not a quantity recovered from the model definitions, and there is no fitted parameter that is later renamed as a prediction. The LLM prompt's 'median candidate' heuristic influences behavior but does not mathematically force the benchmark result. The main weakness—comparing all agents through a common opponent rather than head-to-head and treating common-opponent win rates as transitive—is a validity or generalizability concern, not a circularity, because the reported numbers are not equivalent to the inputs by construction. References are standard external works by other authors; no load-bearing self-citation appears. The paper is self-contained as an empirical comparative study, so the circularity score is 0.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The empirical comparison rests on the game environment's fidelity, the uncalibrated baseline, and the chosen state encoding; no new physical or conceptual entities are introduced.

free parameters (1)

Reward function weights (win +3.0, loss -3.0, correct guess +0.2, incorrect guess -0.5, invalid action -3.0)
Hand-specified in Section 3.1; the PPO agent is trained to optimize these values, and the final win rate is contingent on them. No tuning procedure or sensitivity analysis is reported.

assumptions (3)

domain assumption The DaVinciCodeGameEnvironment correctly implements official Da Vinci Code rules, including joker ordering via randomized float values.
All win-rate measurements are produced inside this environment; if the rules or the joker handling are wrong, the comparison is about a different game. Section 3.1 defines the environment.
ad hoc to paper Win rate against the history-limited Transformer baseline is a transitive and fair measure of relative agent strength across PPO and LLMs.
The paper draws its main comparative conclusion from PPO's 58.5% and LLMs' 17-39% win rates against the same baseline, without head-to-head matches or baseline calibration. Section 4, Table 1.
ad hoc to paper The BPE tokenizer trained on a domain-specific vocabulary and the fixed MAX_HISTORY_LEN=256 are sufficient to represent the game state for the PPO policy.
The state representation is a design choice; no ablation or analysis of information loss is provided. Section 3.4.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Mastering Da Vinci Code: A Comparative Study of Transformer, LLM, and PPO-based Agents." pith.science (2026). https://pith.science/paper/M5FRZDZA

@misc{pith2026250612801,
  author       = {Pith},
  title        = {Pith review of: Mastering Da Vinci Code: A Comparative Study of Transformer, LLM, and PPO-based Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/M5FRZDZA}},
  note         = {Machine review of arXiv:2506.12801}
}

abstract

The Da Vinci Code, a game of logical deduction and imperfect information, presents unique challenges for artificial intelligence, demanding nuanced reasoning beyond simple pattern recognition. This paper investigates the efficacy of various AI paradigms in mastering this game. We develop and evaluate three distinct agent architectures: a Transformer-based baseline model with limited historical context, several Large Language Model (LLM) agents (including Gemini, DeepSeek, and GPT variants) guided by structured prompts, and an agent based on Proximal Policy Optimization (PPO) employing a Transformer encoder for comprehensive game history processing. Performance is benchmarked against the baseline, with the PPO-based agent demonstrating superior win rates ($58.5\% \pm 1.0\%$), significantly outperforming the LLM counterparts. Our analysis highlights the strengths of deep reinforcement learning in policy refinement for complex deductive tasks, particularly in learning implicit strategies from self-play. We also examine the capabilities and inherent limitations of current LLMs in maintaining strict logical consistency and strategic depth over extended gameplay, despite sophisticated prompting. This study contributes to the broader understanding of AI in recreational games involving hidden information and multi-step logical reasoning, offering insights into effective agent design and the comparative advantages of different AI approaches.

Figures

Figures reproduced from arXiv: 2506.12801 by the authors.

**Figure 2.** GUI for Human-AI Da Vinci Code Gameplay. Please replace this placeholder with [PITH_FULL_IMAGE:figures/full_fig_p014_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 7 canonical work pages

[1]

Qwen technical report

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Xiancold Han, Feifei Huang, Bowen Huang, Jiangtao Huang, Zhigen Hu, Yanlong Huo, Chen Ji, Li Jiang, Shuo Jiang, Benfeng Jiang, Peng Jin, Yu Jin, Shijie Kou, Cheng Li, Chuanqi Li, Dongjing Li, Hongyi Li, Jia Li, Liang Li, Jing Li, Min Li, Qingliang Li, Shijia Li, Xing...

arXiv 2023
[2]

Gong, Shang Guo, Tao Han, Shan Hong, Lifu Hou, Jiamin Hu, Jing Hu, Xia Hu, Kaidong Huang, Yan Huang, Jiaman Li, Ya-Lin Li, Y

Sheng Bi, Zengzhi Chen, Guangsheng Chen, Hongbo Cheng, Panpan Dai, Kaige Deng, Dongkai Ding, Xu Dong, Chong Duan, Jie Feng, Jiaming Fu, Yao Fu, Y. Gong, Shang Guo, Tao Han, Shan Hong, Lifu Hou, Jiamin Hu, Jing Hu, Xia Hu, Kaidong Huang, Yan Huang, Jiaman Li, Ya-Lin Li, Y. Li, Bei Liu, Chao Liu, Haowei Liu, Hongyi Liu, Jian Liu, Jiaqi Liu, Kui Liu, Qian Lu...

arXiv 2024
[3]

Superhuman ai for heads-up no-limit poker: Libratus beats top professionals

Noam Brown and Tuomas Sandholm. Superhuman ai for heads-up no-limit poker: Libratus beats top professionals. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, pages 5225--5232, 2018

work page 2018
[4]

Superhuman ai for multiplayer poker

Noam Brown and Tuomas Sandholm. Superhuman ai for multiplayer poker. Science, 365 0 (6456): 0 885--890, 2019

2019
[5]

Adam: A method for stochastic optimization

Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

arXiv 2014
[6]

Suphx: Mastering mahjong with deep reinforcement learning

Junjie Li, Sotetsu Koyamada, Qingchao Wang, Ryuichiro Hataya, Yuichi Yoshida, Tuomas Sandholm, and Yukino Baba. Suphx: Mastering mahjong with deep reinforcement learning. CoRR, abs/2003.13590, 2020. URL https://arxiv.org/abs/2003.13590

arXiv 2003
[7]

Gpt-4 technical report

OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

arXiv 2023
[8]

Hello gpt-4o

OpenAI. Hello gpt-4o. https://openai.com/index/hello-gpt-4o/, 2024. Accessed: June 5, 2024

work page 2024

Show all 17 references

[9]

Mastering the game of guandan with deep reinforcement learning and behavior regulating

Hao Pan, Yifan Yanggong, Futong Bao, Guangzheng Chen, Wenxin Huang, Boyu Li, Shiyu Li, Yunsheng Li, Ziyue Liu, Chunfeng Lu, Zeren Peng, Siyu Shen, Jing Song, Wei Sun, Zeqi Sun, Yisong Tan, Haina Tao, Jiajin Wang, Mingyu Wang, Xi Wang, Yilin Wang, Chen Wu, Fan Wu, Jiaxin Wu, Ha...

2024 arXiv
[10]

Proximal policy optimization algorithms

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

2017 arXiv
[11]

Trust region policy optimization

John Schulman et al. Trust region policy optimization. ICML, 2015

2015
[12]

Mastering the game of go with deep neural networks and tree search

David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. Nature, 529 0 (7587): 0 484-...

2016
[13]

Gemini: A family of highly capable multimodal models

Gemini Team and Google. Gemini: A family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023

2023 arXiv
[14]

Td-gammon, a self-teaching backgammon program, achieves master-level play

Gerald Tesauro. Td-gammon, a self-teaching backgammon program, achieves master-level play. Neural computation, 6 0 (2): 0 215--219, 1995

1995
[15]

Gomez, Lukasz Kaiser, and Illia Polosukhin

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2017

2017
[16]

Tree-of-thought prompting for large language models

Shinn Yao et al. Tree-of-thought prompting for large language models. In NeurIPS, 2023

2023
[17]

Douzero: Mastering doudizhu with self-play deep reinforcement learning

David Daochen Zha, Jingru Xie, Wenye Ma, Shengchao Liu, Chen Zhang, Xiangru Lian, Xia Hu, and Kwei Geng. Douzero: Mastering doudizhu with self-play deep reinforcement learning. arXiv preprint arXiv:2106.06135, 2021

2021 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Qwen technical report

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Xiancold Han, Feifei Huang, Bowen Huang, Jiangtao Huang, Zhigen Hu, Yanlong Huo, Chen Ji, Li Jiang, Shuo Jiang, Benfeng Jiang, Peng Jin, Yu Jin, Shijie Kou, Cheng Li, Chuanqi Li, Dongjing Li, Hongyi Li, Jia Li, Liang Li, Jing Li, Min Li, Qingliang Li, Shijia Li, Xing...

arXiv 2023

[2] [2]

Gong, Shang Guo, Tao Han, Shan Hong, Lifu Hou, Jiamin Hu, Jing Hu, Xia Hu, Kaidong Huang, Yan Huang, Jiaman Li, Ya-Lin Li, Y

Sheng Bi, Zengzhi Chen, Guangsheng Chen, Hongbo Cheng, Panpan Dai, Kaige Deng, Dongkai Ding, Xu Dong, Chong Duan, Jie Feng, Jiaming Fu, Yao Fu, Y. Gong, Shang Guo, Tao Han, Shan Hong, Lifu Hou, Jiamin Hu, Jing Hu, Xia Hu, Kaidong Huang, Yan Huang, Jiaman Li, Ya-Lin Li, Y. Li, Bei Liu, Chao Liu, Haowei Liu, Hongyi Liu, Jian Liu, Jiaqi Liu, Kui Liu, Qian Lu...

arXiv 2024

[3] [3]

Superhuman ai for heads-up no-limit poker: Libratus beats top professionals

Noam Brown and Tuomas Sandholm. Superhuman ai for heads-up no-limit poker: Libratus beats top professionals. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, pages 5225--5232, 2018

work page 2018

[4] [4]

Superhuman ai for multiplayer poker

Noam Brown and Tuomas Sandholm. Superhuman ai for multiplayer poker. Science, 365 0 (6456): 0 885--890, 2019

2019

[5] [5]

Adam: A method for stochastic optimization

Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

arXiv 2014

[6] [6]

Suphx: Mastering mahjong with deep reinforcement learning

Junjie Li, Sotetsu Koyamada, Qingchao Wang, Ryuichiro Hataya, Yuichi Yoshida, Tuomas Sandholm, and Yukino Baba. Suphx: Mastering mahjong with deep reinforcement learning. CoRR, abs/2003.13590, 2020. URL https://arxiv.org/abs/2003.13590

arXiv 2003

[7] [7]

Gpt-4 technical report

OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

arXiv 2023

[8] [8]

Hello gpt-4o

OpenAI. Hello gpt-4o. https://openai.com/index/hello-gpt-4o/, 2024. Accessed: June 5, 2024

work page 2024