{"id":"dacc9a64-fbbc-47cf-a930-54db0099f33a","arxiv_id":"2504.20593","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"In performative Markov potential games, independent policy gradient and natural policy gradient algorithms are claimed to converge to an approximate performatively stable equilibrium, but the proofs contain a false equality between value differences and potential differences.","lead":"This paper brings performative effects, where deployed policies change the environment's rewards and transitions, into multi-agent Markov potential games. It defines a new equilibrium called the performatively stable equilibrium and claims that existing independent policy gradient algorithms converge to it.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Lemma 8 equates a single agent's joint-update value change with the potential change; MPG definition only supports unilateral deviations, so the INPG and regularized-INPG proofs rest on a false identity.","rationale":"The reader's weakest-assumption analysis identifies exactly the load-bearing flaw: Lemma 8 assumes that the change in a single agent's value under a joint policy update equals the change in the potential. This is not a consequence of the Markov potential game definition, which only enforces equality for unilateral deviations. I independently checked the algebra: in Lemma 8, the first equality writes V^{t+1}_{i,t}(rho)-V^t_{i,t}(rho) as the potential difference Phi^{t+1}_t(rho)-Phi^t_t(rho), and this is false in general. A simple one-state two-agent game with gamma=0 provides a concrete violation. This invalidates the potential-improvement argument used to prove Theorem 4, Theorem 5, and the last-iterate claim. The paper's experiments and the repeated-optimization section may be useful empirical material, but they do not repair the central theoretical guarantees. No additional concern is needed; the false identity is sufficient to sustain the reader's REJECT verdict.","tokens_in":43884,"tokens_out":4719,"duration_ms":46728,"concrete_test":"Re-derive Lemma 8 in the two-agent, one-state potential game with r1(a,b)=2a+b, r2(a,b)=a+3b, gamma=0, and Phi(a,b)=2a+3b. For pi^t=(0,0) and pi^{t+1}=(1,1), compute V^{pi^{t+1}}_{1,pi^t}(rho)-V^{pi^t}_{1,pi^t}(rho)=3 and Phi^{pi^{t+1}}_{pi^t}(rho)-Phi^{pi^t}_{pi^t}(rho)=5; the inequality of these numbers shows that the identity used in Lemma 8 fails. To make the test fully decisive, simulate the INPG update (5) in this game and check whether the claimed performative-regret bound from Theorem 4 holds; if the proof relies on Lemma 8, the bound is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central convergence results (Theorems 4 and 5, and the asymptotic last-iterate claim) all pass through Lemma 8 in Appendix A.6.1, whose proof uses the identity V^{t+1}_{i,t}(rho) - V^t_{i,t}(rho) = Phi^{t+1}_t(rho) - Phi^t_t(rho) for every agent i. But the MPG definition, Eq. (2), only guarantees potential-value equality for a unilateral deviation of a single agent while all other agents are held fixed. Under the simultaneous joint update pi^{t+1} = (pi^{t+1}_i, pi^{t+1}_{-i}), the left-hand side is the change in agent i's value when all agents change, whereas the right-hand side is the potential change of the full joint move. These two quantities generally differ; the potential change telescopes as a sum of value changes at intermediate unilateral profiles, not as a single player's value change. A concrete counterexample is a one-state, two-agent game with gamma = 0, rewards r1(a,b) = 2a + b and r2(a,b) = a + 3b, which satisfies the MPG condition with Phi(a,b) = 2a + 3b. Moving from (0,0) to (1,1) gives V1(1,1) - V1(0,0) = 3 but Phi(1,1) - Phi(0,0) = 5. Thus the identity asserted in Lemma 8 is false. Since the subsequent performative-regret bounds telescope Phi^{T+1}_{T+1} - Phi^1_1 and then transfer this bound to a per-agent value difference, Theorems 4 and 5 do not follow from the stated assumptions. Lemma 2 bounds the cross-game difference but cannot repair the missing per-agent joint-update equality.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces performative effects into Markov potential games (MPGs), defining a performatively stable equilibrium (PSE) and a performative regret measure. It claims that a PSE always exists under a Lipschitz sensitivity assumption, that independent policy gradient ascent (IPGA) and independent natural policy gradient (INPG) converge to an approximate PSE in the best-iterate sense, that unregularized INPG converges to a PSE in the last-iterate sense, and that a repeated retraining method achieves finite-time last-iterate convergence under agent-independent transitions. The proofs adapt prior MPG analyses by adding terms that account for the dependence of the game on the deployed policy. Supporting experiments on safe-distancing and stochastic congestion games are included with released code.","tokens_in":44289,"tokens_out":8637,"duration_ms":86893,"significance":"If the convergence theorems were correct, the paper would be a useful extension of performative RL to multi-agent settings: the PSE notion is natural, the performative regret measure is appropriate, the recovery of prior rates as the sensitivity parameters vanish is appealing, and the experiments together with the released code would support the empirical claims. However, the main convergence results are built on an invalid identity in Lemma 8 and on an unjustified cross-game potential bound in Lemma 5, so the central theoretical contribution is not established by the submitted proofs.","major_comments":[{"comment":"The first identity in the proof of Lemma 8, V^{t+1}_{i,t+1}(rho)-V^t_{i,t}(rho)=Phi^{t+1}_t(rho)-Phi^t_t(rho), asserts that a single agent's value change under a simultaneous joint policy update equals the potential change of that joint update. Equation (2) only guarantees this equality for unilateral deviations of one agent with all other agents held fixed. For a joint update, the potential change is a telescoping sum of value changes at intermediate unilateral profiles, not the value change of any one agent. A concrete counterexample is a one-state, two-agent game with gamma=0, r_1(a,b)=2a+b, r_2(a,b)=a+3b, and Phi(a,b)=2a+3b, which satisfies the MPG condition; moving from (0,0) to (1,1) gives V_1(1,1)-V_1(0,0)=3 but Phi(1,1)-Phi(0,0)=5. Thus Lemma 8 is false, and the proofs of Theorems 4 and 5, as well as the asymptotic last-iterate convergence statement, do not follow from the stated assumptions.","section":"Appendix A.6.1, Lemma 8; used in Theorems 4, 5, and the last-iterate claim"},{"comment":"The proof of Lemma 5 bounds |Phi^{t+1}_{t+1}-Phi^{t+1}_t| by invoking Lemma 2. Lemma 2 bounds |V^pi_{i,pi'}-V^pi_{i,pi''}|, i.e., a value-function difference between two underlying games, not a potential-function difference. No argument is given that the potentials for different underlying games can be chosen so that cross-game potential differences are controlled by delta_{r,p} ||pi^{t+1}-pi^t||. Since this cross-game potential bound produces the additive W_{r,p} term in Theorem 3, and the same argument is used in the oracle-free analysis leading to Theorem 6, these results also lack support.","section":"Appendix A.4, Lemma 5; also used in Theorem 6 via Lemma 6"},{"comment":"The proof of Lemma 1 asserts that Phi^{pi'}_pi is continuous in the second argument pi because of the (omega_r, omega_p)-sensitivity assumption. The definition in Eq. (2) only requires existence of a potential for each fixed game G(pi); it does not in itself guarantee that a selection of potentials can be made jointly continuous in pi. The Kakutani fixed-point argument for Theorem 1 requires this continuity, so the existence proof of a PSE is incomplete unless a canonical continuous construction of the potential is supplied.","section":"Appendix A.1, Lemma 1 and Theorem 1"}],"minor_comments":[{"comment":"The definition of W_{r,p} is inconsistent: Theorem 3 in the main text defines W_{r,p} := T*(n+1)*n^2*S*delta_{r,p}, while the proof in Appendix A.4 defines W_{r,p} = n(n+1)/2 * delta_{r,p} * ||pi^{t+1}-pi^t||_2, and the errata appendix says W_{r,p} does not depend on T. These should be reconciled in a revision.","section":"Table 1 and Section 5.1.1"},{"comment":"The displayed inequality in Assumption 5(iii) reads ||nabla_pi Phi^pi_{pi'}(rho) - nabla_pi Phi^pi_{pi'}(rho)||_2 <= beta ||pi-pi'||_2, which has the same expression on both sides and is therefore either a typo or vacuous; presumably one of the gradients should be evaluated at a different policy or game.","section":"Assumption 5(iii)"},{"comment":"The manuscript contains several typos and inconsistencies, including the use of 'Perform-Regret' in Theorem 5 instead of PReg, the notation for the horizon H in Section 5.2, and the appendix errata items for Table 1 and Theorem 3. A careful proofreading pass is needed.","section":"Throughout"}],"recommendation":"reject","confidential_remarks":"The main issue is mathematical rather than presentational: the invalid identity in Lemma 8 directly supports the paper's core convergence theorems, and the potential-continuity and cross-game-potential gaps affect the remaining results. Even though the writing is clear, the experimental section is useful, and the code is released, the theoretical claims as stated are not established by the submitted proofs."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper defines performative Markov potential games, introduces a sensible PSE notion, and then claims convergence for IPGA and INPG. The setting is new and the question is relevant, but the central proofs do not hold. Lemma 8 in Appendix A.6.1 uses an identity that is false under the paper's own definition: it equates a single agent's value change under a simultaneous joint update with the potential change of that joint move. The MPG condition in Eq. (2) only guarantees that relation for unilateral deviations. For a joint update, the potential change is a sum of intermediate unilateral value changes, not necessarily equal to any one agent's value change. A one-state, two-agent, gamma=0 example gives V1 change 3 and potential change 5, so the claim in Lemma 8 is simply wrong. Since Theorems 4 and 5 telescope the potential and then transfer it to a per-agent regret, those theorems do not follow from the stated assumptions.\n\nLemma 5 has a related problem. It bounds the cross-game potential difference |Phi^{t+1}_{t+1} - Phi^{t+1}_t| by applying Lemma 2, which bounds per-agent value differences. But the potential functions of different games are not tied to per-agent values in that way; without an additional assumption on how Phi changes with the game parameter, that bound is unsupported. This affects Theorem 3 as well. The existence proof for PSE also asserts without proof that the potential is continuous in the game parameter, which is not implied by the definition.\n\nWhat the paper does well: the formal extension of MPGs to policy-dependent rewards and transitions is a legitimate contribution, the PSE existence argument is plausible once continuity is established, and Section 6's repeated occupancy-measure optimization is a reasonable adaptation of Mandal et al. that may be salvageable. The experiments are honestly reported and show interesting trends, though they cannot compensate for the missing theory.\n\nThe paper deserves a serious referee, not a desk reject, because the problem is important and the framework is mostly sound. But the referee should require a major revision: either prove the potential-improvement lemmas through per-agent value differences that do not rely on the false joint-update identity, or state and prove the needed sensitivity properties of the potential. As submitted, the main theoretical claims are not proven.","headline":"The performative-MPG framework is new and worth studying, but the main convergence theorems are built on a false identity and need major repair.","tokens_in":44813,"tokens_out":3605,"would_cite":false,"duration_ms":40457,"reading_group":"yes","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"In performative Markov potential games, where deployed policies change the rewards and transitions, independent policy gradient and natural policy gradient learners converge to approximate performatively stable equilibria; natural…","keywords":["performative reinforcement learning","Markov potential games","multi-agent learning","performatively stable equilibrium","natural policy gradient","performative regret","independent policy gradient","Markov games"],"falsifier":"Along one INPG iteration in a two-agent, two-state performative Markov potential game with nonzero sensitivity, compute both sides of $V^{t+1}_{i,t}(\\rho)-V^{t}_{i,t}(\\rho)=\\Phi^{t+1}_t(\\rho)-\\Phi^{t}_t(\\rho)$; the Theorem 4 proof asserts these are equal, so showing they differ in a valid performative MPG would falsify the bound as stated.","tokens_in":1536,"feed_emoji":"🎲","tokens_out":2464,"duration_ms":85512,"temperature":0.7,"pith_summary":"This paper asks whether independent gradient-based learners still converge when the policies they deploy change the game itself: rewards and transitions shift with the joint policy. It defines a multi-agent solution concept, the performatively stable equilibrium, and proves one always exists under a Lipschitz sensitivity assumption. The paper claims that independent policy gradient ascent and independent natural policy gradient both converge to an approximate performatively stable equilibrium in the best-iterate sense, with natural policy gradient additionally achieving asymptotic last-iterate convergence. For a special class with agent-independent transitions, a repeated occupancy-measure optimization method attains finite-time last-iterate convergence. If these results hold, natural-gradient multi-agent learning remains dependable even when agents reshape their environment, with the degradation controlled by the strength of the performative effects.","feed_headline":"Natural gradients converge in performative multi-agent games","feed_subtitle":"Independent agents settle near stable equilibria even as their policies reshape the environment; natural gradient regret vanishes.","key_machinery":"The central object is the performative Markov potential game $G(\\bar{\\pi})$, whose potential $\\Phi^{\\pi}_{\\bar{\\pi}}$ depends on both the played policy $\\pi$ and the induced game $\\bar{\\pi}$, together with the performatively stable equilibrium defined through that game. The argument is carried by sensitivity-based improvement lemmas: Lemma 2 bounds value changes across different induced games by $\\delta_{r,p}\\|\\pi'-\\pi''\\|_2$, and this bound is inserted into the potential-improvement inequalities inherited from prior Markov potential game analyses. The natural-gradient analysis also depends on the log-barrier regularizer used in regularized INPG and, for the last-iterate result, on a contraction-mapping argument over state-action occupancy measures under the agent-independent-transitions assumption.","core_discovery":"The paper introduces performative Markov potential games, in which the deployed joint policy determines the reward function and transition kernel of the Markov game, and generalizes solution concepts to this setting. Its central claim is that a performatively stable equilibrium always exists under the sensitivity assumption, that every such equilibrium is a $\\delta_{r,p}$-approximate Nash equilibrium, and that independent gradient algorithms converge to approximate performatively stable equilibria in the best-iterate sense. For independent natural policy gradient, the paper proves performative regret of order $O\\!\\left(\\sqrt{\\tilde{\\kappa}_\\rho(\\sqrt{n}+M\\delta_{r,p})/(c(1-\\gamma)^3 T)}\\right)$; with a log-barrier regularizer it proves a finite-time best-iterate bound independent of the optimal-action probability $c$. It further shows asymptotic last-iterate convergence of unregularized natural policy gradient and finite-time last-iterate convergence for repeated independent optimization over occupancy measures when transitions are agent-independent. As the performative effects vanish, the rates recover prior Markov potential game guarantees.","pith_inferences":["Editorial: the equality $V^{t+1}_{i,t}(\\rho)-V^t_{i,t}(\\rho)=\\Phi^{t+1}_t(\\rho)-\\Phi^t_t(\\rho)$ used in Lemma 8 is not implied by the definition of a Markov potential game; if it cannot be derived for a restricted class of potentials, Theorem 4's guarantee would hold only for that class.","Editorial: a direct test is whether the same natural-policy-gradient rates hold with sample-based gradient estimates; the paper provides a sample-based guarantee only for IPGA, leaving a concrete open extension.","Editorial: because $\\mathrm{PReg}(T)$ averages over rounds, practical deployment of IPGA should track the best visited policy rather than relying on the final iterate; this guidance is implicit in the best-iterate statement."],"forward_implications":["When the sensitivity parameters $\\omega_r,\\omega_p$ tend to zero, the guaranteed rates reduce to the known Markov potential game rates for IPGA, INPG, and regularized INPG, so the performative analysis degrades gracefully.","Every performatively stable equilibrium is a $\\delta_{r,p}$-approximate Nash equilibrium, meaning small performative sensitivity makes stability and Nash play coincide up to a controllable gap.","IPGA's performative regret carries an additive term that does not vanish as $T\\to\\infty$, whereas unregularized INPG's performative contribution disappears asymptotically, so the two algorithms differ qualitatively under strong performativity.","For agent-independent transitions, repeated independent optimization over occupancy measures converges in finite time to an approximate performatively stable equilibrium in the last-iterate sense.","The averaged performative regret guarantees that at least one iterate is an approximate performatively stable equilibrium, not that the final policy is one, for the policy-gradient methods."],"supporting_citations":[{"why":"Defines Markov potential games and supplies the original independent policy gradient analysis that this paper extends to performative settings.","marker":"Leonardos et al. (2022)"},{"why":"Supplies the IPGA update rule, potential-improvement lemmas, and distribution-mismatch analysis adapted in Theorems 3 and 6.","marker":"Ding et al. (2022)"},{"why":"Provides the log-barrier regularized INPG analysis and the lemmas that Theorem 5 adapts with performative costs.","marker":"Zhang et al. (2022)"},{"why":"Provides the INPG policy-improvement and regret lemmas that Theorem 4 extends by adding a performative term.","marker":"Alatur et al. (2024a)"},{"why":"Establishes INPG convergence in Markov potential games and contributes the stochastic congestion game used in the experiments.","marker":"Fox et al. (2022)"},{"why":"Introduces performative reinforcement learning and the repeated retraining proof technique generalized to multi-agent MPGs in Theorem 7.","marker":"Mandal et al. (2023)"}],"fun_headline_variants":["Last-iterate convergence for natural gradient in performative games","Independent agents settle near stable equilibrium as policies reshape environment","Natural policy gradient reaches last-iterate stability in performative games","Best-iterate and last-iterate convergence for independent gradient play"],"cache_read_input_tokens":46848,"weakest_assumption_plain":"The load-bearing premise is that after a joint policy update, each agent's value change equals the change in the whole game's potential; the definition of a Markov potential game guarantees this only for unilateral deviations, not for simultaneous joint updates.","fun_headline_variants_meta":{"raw":{"variants":["Last-iterate convergence for natural gradient in performative games","Independent agents settle near stable equilibrium as policies reshape environment","Natural policy gradient reaches last-iterate stability in performative games","Best-iterate and last-iterate convergence for independent gradient play"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001405,"raw_usage":{"total_tokens":5674,"prompt_tokens":939,"completion_tokens":4735,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":555,"completion_tokens_details":{"reasoning_tokens":4667}},"tokens_in":555,"tokens_out":4735,"duration_ms":30509,"temperature":1.0,"reasoning_tokens":4667,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:26:14.783191+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Along one INPG iteration in a two-agent, two-state performative Markov potential game with nonzero sensitivity, compute both sides of $V^{t+1}_{i,t}(\\rho)-V^{t}_{i,t}(\\rho)=\\Phi^{t+1}_t(\\rho)-\\Phi^{t}_t(\\rho)$; the Theorem 4 proof asserts these are equal, so showing they differ in a valid performative MPG would falsify the bound as stated.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines Markov potential games and supplies the original independent policy gradient analysis that this paper extends to performative settings."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the IPGA update rule, potential-improvement lemmas, and distribution-mismatch analysis adapted in Theorems 3 and 6."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the log-barrier regularized INPG analysis and the lemmas that Theorem 5 adapts with performative costs."},{"cited_title":"M., Overman, W., and Panageas, I","cited_arxiv_id":null,"evidence_quote":"Establishes INPG convergence in Markov potential games and contributes the stochastic congestion game used in the experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces performative reinforcement learning and the repeated retraining proof technique generalized to multi-agent MPGs in Theorem 7."}],"review_version":1}