Safe Equilibrium Policy Optimization for Strategic Agent Policies

Amit Dhanda; Karthika Arumugam; Kiran Kumar Manku

arxiv: 2605.30854 · v1 · pith:EQ6MBZGB · submitted 2026-05-29 · cs.MA · cs.AI

Safe Equilibrium Policy Optimization for Strategic Agent Policies

Karthika Arumugam , Kiran Kumar Manku , Amit Dhanda This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel 2026-06-28 20:53 UTCgrok-4.3pith:EQ6MBZGB record.json open to challenge →

classification cs.MA cs.AI

keywords safe equilibrium policy optimizationstrategic agentslanguage modelsexploitabilitymulti-agent reinforcement learninggroup relative policy optimizationKuhn pokernegotiation safety

0 comments

The pith

Safe Equilibrium Policy Optimization adds penalties for exploitability, collusion, and externalities to language-model agent training.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper establishes that standard reinforcement learning on language models for strategic tasks ignores multi-agent structure, allowing exploitation of opponents, harmful collusion, and cost externalization through the language interface. SEPO augments the payoff objective with explicit penalties for these risks and uses the resulting reward in Group Relative Policy Optimization after supervised fine-tuning. Across Iterated Prisoner's Dilemma, repeated auctions, two negotiation settings, and Kuhn Poker, the approach produces agents with zero exploit-pool advantage in poker for both tested models and superior safety metrics in four domains. It also reverses the over-cooperative tendency introduced by supervised fine-tuning alone. Ablations show that per-rollout exploitability computation is required, as constant penalties cancel under advantage normalization.

Core claim

The paper claims that augmenting expected payoff with penalties for exploitability, collusion risk, and externality cost, then using this as the reward signal for GRPO on Gemma 4 E4B-it and Qwen 3.5-4B after SFT, yields agents that achieve zero exploit-pool advantage in Kuhn Poker for both models, outperform the base model on safety metrics in four of five domains, correct over-cooperative SFT behavior, and in negotiation produce the only positive-safety outcome with positive normalized relative advantage among tested configurations.

What carries the argument

Safe Equilibrium Policy Optimization (SEPO), the objective that augments expected payoff with explicit penalties for exploitability, collusion risk, and externality cost before serving as the reward for Group Relative Policy Optimization.

If this is right

SEPO training reaches zero exploit-pool advantage in Kuhn Poker for both Gemma and Qwen models.
The method improves safety metrics over the base model in four of the five tested domains.
SEPO reverses the over-cooperative behavior that supervised fine-tuning alone introduces.
In negotiation domains, SEPO is the only configuration that reaches a positive-safety outcome with positive normalized relative advantage.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The per-rollout exploit computation step could be ported to other advantage-normalized RL methods to avoid gradient cancellation on safety terms.
Releasing the code and SFT datasets allows direct testing of whether the same penalties reduce harmful equilibria in additional language-mediated games.
If the approach scales, agents trained this way could be deployed in real multi-party negotiations or auctions without requiring post-hoc safety filters.

Load-bearing premise

That exploitability computed separately on each rollout from trajectories supplies a non-vanishing gradient once GRPO normalizes advantages, unlike a shared constant penalty that cancels as a control variate.

What would settle it

A replication in which a shared constant penalty version of the same training loop produces equivalent safety gains and zero exploit advantage in Kuhn Poker would falsify the necessity of per-rollout computation.

Figures

Figures reproduced from arXiv: 2605.30854 by Amit Dhanda, Karthika Arumugam, Kiran Kumar Manku.

**Figure 1.** Figure 1: Safety improvement (∆ Safety = SEPO − Base) across all games. Left: Gemma 4. Right: Qwen 3.5-4B. Positive bars indicate SEPO improves over base. tions. Training proceeds in two stages: a supervised fine-tuning (SFT) warm-start on expert strategy traces, followed by SEPO optimization computed from fresh opponent interactions at every rollout. We evaluate on Gemma 4 E4B-it and Qwen 3.5-4B across five stra… view at source ↗

**Figure 2.** Figure 2: SEPO training pipeline. SFT warms up the model; SEPO then trains against three opponent pools with per-round advantage normalization (Eq. 4). mutual cooperation (3, 3), mutual defection (1, 1), unilateral defect (5, 0). Subgame-perfect equilibrium of the finite game is mutual defection; in practice LLMs exhibit bounded rationality and TFT is the cooperative benchmark.1 GrimTrigger cooperates until any d… view at source ↗

read the original abstract

Language models fine-tuned with reinforcement learning typically optimize for task reward, ignoring multi-agent strategic structure. Because these agents condition on natural language game-state descriptions and emit actions through free-form generation, strategic failure modes -- exploiting weaker opponents, coordinating on harmful equilibria, and externalizing costs are inseparable from the language interface itself. We propose Safe Equilibrium Policy Optimization (\sepo{}), a training objective that augments expected payoff with explicit penalties for exploitability, collusion risk, and externality cost. We implement \sepo{} as a reward signal for Group Relative Policy Optimization (GRPO), applied to Gemma~4 E4B-it and Qwen~3.5-4B after supervised fine-tuning (SFT). Evaluated across five strategic domains: Iterated Prisoner's Dilemma, repeated auctions, two negotiation variants, and Kuhn Poker. \sepo{} achieves zero exploit-pool advantage in Kuhn Poker for both models, outperforms the base model on safety in four domains, and corrects the over-cooperative behavior introduced by SFT. In negotiation, \sepo{} achieves a positive-safety outcome and only the positive normalized relative advantage of any negotiation configuration. Ablation experiments confirm that per-rollout exploit computation is necessary: a shared constant penalty cancels in GRPO advantage normalization (constant control-variate property), producing zero gradient. To support further research in strategic safety for agents, we release our \href{https://anonymous.4open.science/r/sepo-2668/README.md}{code} and SFT datasets.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

SEPO adds three per-rollout penalties to GRPO for LM agents and the ablation on why constants cancel is the clearest contribution.

read the letter

The main point is that this paper defines Safe Equilibrium Policy Optimization as an objective that adds explicit penalties for exploitability, collusion risk, and externality cost to the expected payoff, then trains language models with GRPO after SFT. They report zero exploit-pool advantage in Kuhn Poker and safety gains in four other domains while fixing SFT over-cooperation. The ablation directly tests that a shared constant penalty produces zero gradient under GRPO normalization while per-rollout computation does not.

The release of code and SFT datasets is useful, and the ablation gives concrete evidence that the mechanism is not trivial. The evaluation spans five domains with an explicit check on the computation choice, which addresses the gradient-vanishing concern head-on.

The soft spots are the missing details. The abstract supplies no equations for the penalty terms, no description of how exploitability is estimated from trajectories, no error bars, and no dataset or rollout sizes. Those gaps make it hard to judge robustness or sensitivity to the free weights on the three terms. The central claims rest on implementation choices that are not shown in the provided text.

This is for researchers working on safety interventions for language-model agents in negotiation, auctions, or repeated games. A reader who wants practical RL fixes and open artifacts will get value from the ablation and the released material.

It deserves peer review because the problem is scoped and relevant, the ablation is informative, and the artifacts allow checking, even though more method detail would help.

Referee Report

2 major / 1 minor

Summary. The paper proposes Safe Equilibrium Policy Optimization (SEPO), an objective that augments expected payoff with explicit penalties on exploitability, collusion risk, and externality cost for language-model agents in multi-agent settings. SEPO is implemented as a reward for Group Relative Policy Optimization (GRPO) on Gemma-4 E4B-it and Qwen-3.5-4B after SFT, and evaluated on five domains (Iterated Prisoner's Dilemma, repeated auctions, two negotiation variants, Kuhn Poker). The central claims are that SEPO achieves zero exploit-pool advantage in Kuhn Poker for both models, improves safety metrics over the base model in four domains, corrects SFT-induced over-cooperation, and that per-rollout exploitability computation is required because a shared constant penalty cancels under GRPO advantage normalization. Code and SFT datasets are released.

Significance. If the results hold, SEPO supplies a concrete, GRPO-compatible training signal for mitigating strategic failure modes that arise directly from the language interface in multi-agent LLM agents. The explicit ablation demonstrating that a constant penalty vanishes while per-rollout computation does not is a strength, as is the release of code and datasets, which enables direct reproduction and extension. The work therefore supplies both an empirical method and supporting artifacts for the emerging area of strategic safety in language-model agents.

major comments (2)

[Abstract and §3] Abstract and §3 (objective definition): the three penalty terms (exploitability, collusion risk, externality) are described only at the level of natural-language names; no explicit functional forms, scaling constants, or estimation procedures from trajectories are given. Because the central claim that per-rollout computation produces a non-vanishing gradient under GRPO normalization rests on these definitions, their absence is load-bearing for both the method and the ablation.
[Evaluation section] Evaluation section (Kuhn Poker and cross-domain results): zero exploit-pool advantage, safety outperformance, and positive normalized relative advantage are reported without error bars, number of rollouts, or the precise procedure used to estimate exploitability from finite trajectories. These omissions directly affect the strength of the empirical claims that constitute the paper's primary evidence.

minor comments (1)

[Abstract] The abstract contains unreplaced LaTeX macros (\sepo{}, \href) that should be rendered in the camera-ready version.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the positive assessment and for identifying specific areas where additional technical detail will improve clarity and reproducibility. We address each major comment below.

read point-by-point responses

Referee: [Abstract and §3] Abstract and §3 (objective definition): the three penalty terms (exploitability, collusion risk, externality) are described only at the level of natural-language names; no explicit functional forms, scaling constants, or estimation procedures from trajectories are given. Because the central claim that per-rollout computation produces a non-vanishing gradient under GRPO normalization rests on these definitions, their absence is load-bearing for both the method and the ablation.

Authors: We agree that the manuscript presents the penalty terms primarily through natural-language descriptions in the abstract and §3 without the explicit functional forms, scaling constants, or trajectory-based estimation procedures. This omission weakens the presentation of both the method and the ablation. In the revision we will expand §3 with the precise definitions (exploitability penalty as the estimated best-response value gap scaled by λ_e, collusion risk as a pairwise action-correlation term scaled by λ_c, externality as a third-party cost term scaled by λ_x) together with the Monte-Carlo estimation procedures used on sampled trajectories. The abstract will be updated to reference these definitions. revision: yes
Referee: [Evaluation section] Evaluation section (Kuhn Poker and cross-domain results): zero exploit-pool advantage, safety outperformance, and positive normalized relative advantage are reported without error bars, number of rollouts, or the precise procedure used to estimate exploitability from finite trajectories. These omissions directly affect the strength of the empirical claims that constitute the paper's primary evidence.

Authors: We acknowledge that the evaluation section reports the key metrics without accompanying error bars, rollout counts, or a detailed description of how exploitability is estimated from finite trajectories. These details are required to substantiate the primary empirical claims. In the revised manuscript we will add standard-error bars computed across independent runs, state the number of evaluation rollouts per domain, and specify the estimation procedures (exact best-response computation via the game tree for Kuhn Poker; Monte-Carlo best-response approximation for the remaining domains). revision: yes

Circularity Check

0 steps flagged

No significant circularity identified

full rationale

The paper presents SEPO as an explicit reward augmentation (penalties for exploitability, collusion, externality) applied to GRPO after SFT. Central claims are empirical: zero exploit-pool advantage in Kuhn Poker, safety gains across five domains, and correction of SFT over-cooperation. Ablations directly test the gradient-vanishing concern by comparing per-rollout vs. shared-constant penalties, confirming the former produces non-zero gradients. No equation reduces a reported metric to a fitted constant by construction, no self-citation chain bears the load of the main result, and no ansatz or uniqueness theorem is smuggled in. The derivation chain is therefore self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

1 free parameters · 1 axioms · 0 invented entities

The central claim depends on three tunable penalty coefficients whose values are not reported, on the assumption that exploitability is estimable from finite rollouts in each domain, and on the GRPO normalization mechanics that make constant penalties ineffective.

free parameters (1)

weights on exploitability, collusion, and externality penalty terms
Scaling factors that balance the safety penalties against task reward; their specific values are required for the reported outcomes but are not stated.

axioms (1)

domain assumption Exploitability, collusion risk, and externality cost can be computed from per-rollout trajectories in the five chosen domains
Required for the per-rollout penalty signal that the ablation shows is necessary.

pith-pipeline@v0.9.1-grok · 5807 in / 1378 out tokens · 22065 ms · 2026-06-28T20:53:09.971820+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

2 extracted references · 2 canonical work pages · 1 internal anchor

[1]

arXiv preprint arXiv:2305.10142 (2023)

Improving Language Model Negotiation with Self-Play and In-Context Learning from AI Feedback. arXiv preprint arXiv:2305.10142. Google DeepMind. 2026. Gemma 4: Byte for byte, the most capable open models. Model card: google/gemma-4-E4B-it. Jiaxian Guo, Bo Yang, Paul Yoo, Bill Yuchen Lin, Yusuke Iwasawa, and Yutaka Matsuo. 2023. Suspicion-Agent: Playing Imp...

work page arXiv 2026
[2]

Proximal Policy Optimization Algorithms

Direct Preference Optimization: Your Lan- guage Model is Secretly a Reward Model. InAd- vances in Neural Information Processing Systems (NeurIPS), volume 36. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal Policy Optimization Algorithms.arXiv preprint arXiv:1707.06347. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runx...

work page internal anchor Pith review Pith/arXiv arXiv 2017

[1] [1]

arXiv preprint arXiv:2305.10142 (2023)

Improving Language Model Negotiation with Self-Play and In-Context Learning from AI Feedback. arXiv preprint arXiv:2305.10142. Google DeepMind. 2026. Gemma 4: Byte for byte, the most capable open models. Model card: google/gemma-4-E4B-it. Jiaxian Guo, Bo Yang, Paul Yoo, Bill Yuchen Lin, Yusuke Iwasawa, and Yutaka Matsuo. 2023. Suspicion-Agent: Playing Imp...

work page arXiv 2026

[2] [2]

Proximal Policy Optimization Algorithms

Direct Preference Optimization: Your Lan- guage Model is Secretly a Reward Model. InAd- vances in Neural Information Processing Systems (NeurIPS), volume 36. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal Policy Optimization Algorithms.arXiv preprint arXiv:1707.06347. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runx...

work page internal anchor Pith review Pith/arXiv arXiv 2017