REVIEW 4 major objections 5 minor 9 references
Strategy Adaptation in Large Language Model Werewolf Agents
T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Switching between Support and Attack prompts, chosen from role estimates, raises Werewolf agents' win rates in the Werewolf game.
desk verdict A promising pilot on explicit strategy adaptation for LLM Werewolf agents, but 30-game win rates without error bars cannot support the headline claim. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The mechanism is a two-strategy prompt library plus an adaptation gate. Each agent first estimates every other player's role on a 0-4 scale for werewolf, villager, seer, and doctor, after stating its reasoning; then, at three moments per round (after night actions, after the day debate, and after voting), the LLM reads the conversation history and these role scores and picks either the Support or Attack prompt according to criteria such as 'use Support when you are being suspected' and 'use Attack when a likely werewolf is already under suspicion'. The chosen prompt then generates the agent's utterances for that phase, so the strategy is part of the context the model responds to, not a separate controller.
What would settle it
Run the same 30-match protocol with the true roles substituted for the LLM's role-estimation scores; if Villager win rates still fall below the 0.80 no-estimation baseline, the adaptation criteria themselves are the bottleneck, not the estimator. If they rise above it, the estimator is the bottleneck and better role-estimation prompts should transfer directly to higher win rates.
Extended reading notes
Core claim
The central claim is that dynamically switching between two predefined strategy prompts—Support and Attack—guided by a hand-written decision rule and by each agent's 0-4 role estimates for other players, raises the win rate of LLM Werewolf agents relative to fixed-strategy and implicit-strategy baselines. The authors find the effect is most consistent for the Werewolf side: adaptation produced the highest Werewolf win rate with both tested models, and the Attack strategy chosen under adaptation was followed by a smaller increase in how accurately others estimated the werewolves' roles, suggesting it helps them act without arousing suspicion. For Villager agents the result is mixed; GPT Villagers improved under adaptation, Gemini Villagers did not, and the ablation attributes part of the difficulty to low role-estimation accuracy rather than to the adaptation rule itself.
Load-bearing premise
The method assumes the LLM's 0-4 role estimates are accurate enough to choose between Support and Attack; the paper's own ablation shows Villagers win less with estimation (0.73 vs 0.80) and werewolves are estimated at only about 0.30, so this premise is not clearly satisfied.
Editorial extensions
If this is right
- Werewolf agents in other social-deduction games could adopt the same Support/Attack switch and expect the largest gains, since they know their teammates and can coordinate around the choice.
- The value of adaptation depends on role-estimation quality; improving the estimator should improve Villager performance, where the paper currently reports a drop from 0.80 to 0.73 when estimation is added.
- The three switching points (night, post-debate, post-vote) give a natural schedule for learning when to switch; the paper suggests reinforcement learning as a future avenue.
- Because the Attack strategy lowered how much the werewolves' roles were detected, adaptive deception can be measured directly through changes in role-estimation accuracy rather than only through final win rates.
Reading between the lines
- If role estimates were replaced with ground-truth roles, the Villager side would likely match or beat the no-estimation baseline; the paper's 0.30 estimate for werewolves suggests the current prompt is too weak to support Villager decisions.
- The adaptation rule may be overfit to the Werewolf side: the criteria were designed around suspicion and credibility, which are exactly the levers werewolves control, while villagers need a third option (revealing information as seer or doctor) that the Support/Attack dichotomy does not provide.
- A direct test of the method would randomize the order of the three adaptation moments or run adaptation only at one moment, isolating whether switching frequency or rule content drives the win-rate gain.
- The same two-prompt adaptation could be ported to other negotiation settings where agents must choose between cooperating with allies and attacking opponents, with role estimation replaced by trust estimation.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper proposes a strategy-adaptation mechanism for LLM-based Werewolf agents, in which agents switch between a Support strategy and an Attack strategy based on conversation context and explicit role estimates. The method is evaluated in the Werewolf Arena environment using GPT-4o-mini and Gemini-2.0-flash, comparing the adaptive agent against an implicit-strategy baseline and fixed Support/Attack prompts, plus an ablation that removes adaptation and/or role estimation. The paper reports win rates over 30 matches per condition and concludes that adaptation 'consistently outperforms' fixed strategies and prior methods for Werewolf agents, while acknowledging that Villager agents sometimes do worse with adaptation, especially for Gemini and in the role-estimation ablation.
Significance. If the claimed effect is real, the paper would provide a simple, prompt-only method for dynamic strategy selection in social-deduction games, a task that demands situational flexibility well beyond static prompting. The authors are transparent about their limitations, provide the full prompts in an appendix, and include an ablation study. However, the empirical support is currently too thin to establish the central claim: win rates are computed from only 30 games per condition without confidence intervals or significance tests, and the paper's own ablation shows that role estimation can hurt Villager performance. The conclusion's reference to outperforming 'prior methods' is also unsupported because no prior method is implemented in the experiments. With improved statistical analysis and more modest claims the work could be a useful contribution, but in its present form the evidence is not convincing.
major comments (4)
- [Section 6, Conclusion] The conclusion states that the proposal is 'consistently outperforming both fixed strategies and prior methods in win rates,' but the experiments in Section 4 only compare against the implicit Werewolf Arena baseline and the two fixed-strategy prompts (Support and Attack). No experiments against Kano et al. (2023), Xu et al. (2024a), or Wu et al. (2024) are reported, so the claim about prior methods is not supported by the presented data.
- [Section 5.1, Figure 2, and Table 2] The headline comparisons rely entirely on 30 games per condition with no error bars, confidence intervals, or significance tests. For the key Werewolf ablation result in Table 2, a win rate of 0.60 versus 0.40 corresponds to a two-sided Fisher exact p-value of roughly 0.18, so the observed 0.20 gap could easily arise from sampling noise; the 0.08 gap against the 0.52 condition is even weaker. The phrase 'consistently outperforming' is therefore not established by the reported statistics.
- [Section 5.2, Table 2] The ablation results are mixed and partly undermine the claim that role estimation helps. For Villagers, removing role estimation improves the win rate from 0.73 to 0.80, which the paper explains by low estimation accuracy (Est≈0.30). However, no statistical test is provided to show that the 0.73/0.80 difference is meaningful, and the text in Section 5.2 says role estimation 'had a significant impact' without a significance test to support that wording.
- [Section 5.3 and Appendix A.2] The role-estimation accuracy for Werewolves is reported only as Est≈0.30, which is barely above the chance level of 0.25 for four roles. Since strategy selection in the adaptation method relies on these role estimates (per the criteria in Appendix A.2), the authors should report the complete Est values and formally compare them against the chance baseline to justify that the estimates provide usable signal; as presented, the near-chance accuracy casts doubt on the proposed mechanism.
minor comments (5)
- [Section 1] The sentence 'Success in the game relies on The game requires both the ability of deductive reasoning...' contains a duplicated and garbled phrase; it should be corrected to a single coherent sentence.
- [Appendix A.4, Tables 7, 8, and 9] Table 7 and Table 9 headers misspell 'Attack' as 'Atack,' and Table 8's header misspells 'Werewolves' as 'Worewolves.'
- [Appendix A.4, Table 5] The role-estimation prompt includes the text 'num villagers' without substitution; since the game has four villagers, the prompt should state the actual number rather than leaving a placeholder.
- [Section 5.3, Table 3] The table reports average changes in Est but does not include standard deviations or any measure of variance, so it is difficult to assess whether the observed differences between adaptation and fixed strategies are meaningful.
- [Section 7, Limitations] The limitations section is honest about the small number of models and the lack of human evaluation, but it would be helpful to also state whether code and prompt templates will be released for reproducibility.
Circularity Check
No significant circularity: strategy adaptation is empirically evaluated against held-out 30-match win rates with no fitted parameter or self-citation chain forcing the outcome.
full rationale
The paper's load-bearing claim is that dynamically selecting between hand-crafted Support and Attack strategy prompts improves Werewolf-side win rates relative to fixed or implicit strategies. This claim is supported entirely by measured 30-match win rates (Figure 2 and Table 2) comparing the proposal, baselines, and ablations. No parameter is fitted to the reported outcomes and then renamed as a prediction; the strategy-selection criteria in Appendix A.2 are fixed, hand-written prompts, and the role-estimation scores are measured quantities (Equations 1 and 2) reported as analysis rather than used to mathematically derive the win rates. The ablation subtracts components one at a time, so the comparisons are direct empirical contrasts rather than identities by construction. There is no load-bearing self-citation: the prior work cited (Werewolf Arena, AIWolfDial, and earlier Werewolf-agent papers) is used as environment and baseline, not as an unverified premise that forces the conclusion. The admitted limitation that Villager performance sometimes decreases with adaptation and the absence of significance testing are statistical-strength concerns, not circularity concerns. Thus the derivation chain is self-contained as an empirical evaluation, and no circular step is present.
Assumptions & free parameters
assumptions (4)
- domain assumption Werewolf Arena rules and configuration are an appropriate testbed.
- domain assumption LLM agents execute the Support/Attack prompts and the adaptation criteria as intended.
- domain assumption Role estimation scores (0 to 4) reliably indicate role likelihood.
- domain assumption Win rates over 30 matches provide statistically meaningful comparison.
Cite this review
Pith. "Pith review of Strategy Adaptation in Large Language Model Werewolf Agents." pith.science (2026). https://pith.science/paper/SZXLDBYT
@misc{pith2026250712732,
author = {Pith},
title = {Pith review of: Strategy Adaptation in Large Language Model Werewolf Agents},
year = {2026},
howpublished = {\url{https://pith.science/paper/SZXLDBYT}},
note = {Machine review of arXiv:2507.12732}
}
read the original abstract
This study proposes a method to improve the performance of Werewolf agents by switching between predefined strategies based on the attitudes of other players and the context of conversations. While prior works of Werewolf agents using prompt engineering have employed methods where effective strategies are implicitly defined, they cannot adapt to changing situations. In this research, we propose a method that explicitly selects an appropriate strategy based on the game context and the estimated roles of other players. We compare the strategy adaptation Werewolf agents with baseline agents using implicit or fixed strategies and verify the effectiveness of our proposed method.
Figures
Reference graph
Works this paper leans on
-
[1]
Suma Bailis, Jane Friedhoff, and Feiyang Chen. 2024. https://arxiv.org/abs/2407.13943 Werewolf Arena: A Case Study in LLM Evaluation via Social Deduction . Preprint, arXiv:2407.13943
arXiv 2024
-
[2]
Yoshinobu Kano, Neo Watanabe, Kaito Kagaminuma, Claus Aranha, Jaewon Lee, Benedek Hauer, Hisaichi Shibata, Soichiro Miki, Yuta Nakamura, Takuya Okubo, Soga Shigemura, Rei Ito, Kazuki Takashima, Tomoki Fukuda, Masahiro Wakutani, Tomoya Hatanaka, Mami Uchida, Mikio Abe, Akihiro Mikami, Takashi Otsuki, Zhiyang Qi, Kei Harada, Michimasa Inaba, Daisuke Katagam...
work page 2023
-
[3]
Xuehong Tao, Zhiqi Shen, and Chunyan Miao. 2010. https://api.semanticscholar.org/CorpusID:14330304 A Cooperative-Competitive Negotiation Model
work page 2010
-
[4]
Tomer Ullman. 2023. https://arxiv.org/abs/2302.08399 Large Language Models Fail on Trivial Alterations to Theory-of-Mind Tasks . Preprint, arXiv:2302.08399
arXiv 2023
-
[5]
Shuang Wu, Liwen Zhu, Tao Yang, Shiwei Xu, Qiang Fu, Yang Wei, and Haobo Fu. 2024. https://arxiv.org/abs/2402.02330 Enhance Reasoning for Large Language Models in the Game Werewolf . Preprint, arXiv:2402.02330
arXiv 2024
-
[6]
Yuzhuang Xu, Shuo Wang, Peng Li, Fuwen Luo, Xiaolong Wang, Weidong Liu, and Yang Liu. 2024 a . https://arxiv.org/abs/2309.04658 Exploring Large Language Models for Communication Games: An Empirical Study on Werewolf . Preprint, arXiv:2309.04658
arXiv 2024
-
[7]
Zelai Xu, Chao Yu, Fei Fang, Yu Wang, and Yi Wu. 2024 b . https://arxiv.org/abs/2310.18940 Language Agents with Reinforcement Learning for Strategic Play in the Werewolf Game . Preprint, arXiv:2310.18940
arXiv 2024
-
[8]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
Show all 9 references
-
[9]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.