{"id":"f778ec2f-56b5-44b9-8047-dc9a67f78242","arxiv_id":"2607.13607","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Q-learners with constant exploration in the repeated prisoner's dilemma spend most of their time on cooperative win-stay/lose-shift play above a boundary derived from Q-value gaps, matching simulations.","lead":"Two Q-learning agents playing a repeated prisoner's dilemma with constant random exploration can spend most of their time cooperating rather than defecting. The paper derives a discount-factor boundary for when this happens and shows it predicts simulation outcomes well.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The derived boundary equates raw Q-value gaps and ignores differing fluctuation variances (footnote 6), so the stability ordering it predicts may not be the true one; a variance-corrected re-derivation and comparison to simulations is needed.","rationale":"The paper's main contribution is a parameter-free analytic boundary predicting when cooperation (non-defection) dominates under constant exploration. The derivation is transparent and the boundary is not fitted. The strong simulation evidence, including the perturbation analysis in Figure 6 showing local optimality, indicates the boundary is informative. However, the theoretical justification of the boundary as a stability comparison is incomplete. The stability of a stochastic equilibrium is governed by both the barrier height and the noise amplitude. The paper explicitly assumes the latter can be ignored (Section 4, footnote 6: fluctuations in Q-values also depend on rewards and continuation payoffs). This assumption is not tested. If fluctuation variances differ substantially between AD and WSLS, the exact equality in Eq. 4.8 does not correspond to equal exit rates, so the boundary's success in the simulations may be due to a correlation between the raw gap and the noise scale rather than the proposed mechanism. The proposed test—estimating σ_NE and comparing a ΔQ/σ-corrected boundary—directly settles whether this omission changes the predicted transition. The requested test is a natural extension of the paper's own simulation framework and does not require new theory. Until such a test is run, the honest assessment is that the boundary is a well-calibrated heuristic but the claimed stability interpretation remains unverified. This supports the reader's CONDITIONAL verdict: the paper is promising and likely correct in its empirical claims, but the central theoretical derivation rests on an acknowledged but unquantified simplification. I therefore recommend no change to the reader's verdict.","tokens_in":23014,"tokens_out":17510,"duration_ms":172216,"concrete_test":"For a grid of (R,P,δ,ε) points straddling the theoretical boundary, estimate the stationary standard deviation σ_NE of the Q-value difference in the vulnerable state for each NE by running single-agent Q-learning against a fixed opponent playing AD/WSLS with ε-greedy, initialized at Q_NE. Then compare two classifiers: the paper's boundary (equating ΔQ) and a variance-corrected boundary (equating ΔQ/σ). Compute macro F1 against the empirical occupation-time labels (as in Section 5.3). If the ΔQ/σ boundary yields materially higher F1 (e.g., >0.05 improvement) or shifts the predicted transition by more than the width of the 50% isoquant, the raw boundary is not the true stability condition and the mechanism needs revision. If the raw boundary remains at least as accurate, the fluctuation concern is not empirically load-bearing in the tested regime.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central boundary δ_c (Eq. 4.9) is derived by equating the smallest Q-value gaps min_s ΔQ_AD(s) and min_s ΔQ_WSLS(s) (Appendix B). This treats the barrier height as the sole determinant of equilibrium stability. But in a stochastic Q-learning process with constant α, the probability of a Q-value-order reversal in the vulnerable state depends on the gap relative to the local fluctuation scale: exit rate ~ exp(-ΔQ/σ) (Freidlin–Wentzell). The paper's own footnote 6 concedes that fluctuation sizes differ across equilibria because they depend on rewards and continuation payoffs. If σ_AD ≠ σ_WSLS, the condition for equal stability is ΔQ_AD/σ_AD = ΔQ_WSLS/σ_WSLS, not ΔQ_AD = ΔQ_WSLS. The simulations validate the raw-gap boundary with F1 0.85–0.95, but this does not discriminate between the two conditions because σ differences may be small or correlated with payoffs in the tested grid. The concern is load-bearing because the paper's central theoretical claim is that the boundary is derived from stability, and the derivation omits a component it concedes is relevant.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies two Q-learners playing the iterated prisoner's dilemma with one-period memory, constant exploration epsilon, and constant learning rate alpha. Because exploration never vanishes, the authors propose measuring cooperation by the long-run fraction of time spent in cooperative strategy profiles rather than by convergence to a cooperative equilibrium. They prove a positive-probability recurrence result for a set of best-response profiles (Theorem 4.1), conjecture uniqueness of a stationary distribution (Conjecture 4.2), and then derive a heuristic boundary delta_c (Eq. 4.9) by equating the minimal Q-value gaps of the All-Defect (AD) and Win-Stay-Lose-Shift (WSLS) equilibria. The paper evaluates this boundary on a large simulation grid, reporting macro F1 scores of 0.85-0.95 in the empirically stationary region, with additional state-space robustness checks and perturbation analyses showing local optimality of the boundary.","tokens_in":23351,"tokens_out":26259,"duration_ms":243920,"significance":"If the theoretical derivation were sound, the paper would offer a valuable new perspective on algorithmic collusion: instead of asking whether Q-learners converge to collusion, it asks how resilient collusive strategies are to persistent exploration noise. The manuscript has clear strengths: the boundary is derived analytically from Bellman equations rather than fitted, the simulation campaign is extensive (136,000 trajectories), and the authors include careful robustness checks (state-space outcomes, adversarial precision/recall, boundary perturbations). The distinction between convergence-based and time-averaged notions of collusion is well motivated and relevant to the algorithmic collusion debate. However, the theoretical core is heuristic and rests on a conjecture, and the derivation of the central boundary contains an internal inconsistency that must be resolved before the paper's main claim can be accepted as stated.","major_comments":[{"comment":"The quantity called Delta Q_WSLS in the derivation is not the max-min difference defined in Eq. (4.7). The formulas in Appendix B compute, up to the common factor 1/(1-delta), the signed difference Q(s, pi_WSLS(s)) - Q(s, other action), which coincides with Eq. (4.7) only when WSLS is a Nash equilibrium. The paper itself notes (footnote 7) that WSLS is an equilibrium only in part of the parameter space, yet the boundary is evaluated over the full (R,P) grid. In non-equilibrium regions the signed gap is negative, so equating it to the positive AD gap is not a meaningful stability condition. For example, with T=1, R=0.6, P=0.5, S=0, epsilon=0.2, delta=0.7, Eq. (B.10) gives -0.416 while the actual max-min gap from (B.3)-(B.4) is positive. The theoretical derivation should be restricted to the WSLS-NE region, or the heuristic should be reformulated using the actual max-min gaps, and the simu","section":"§4, Eq. (4.7)-(4.9) and Appendix B"},{"comment":"The stability criterion equates equilibrium stability with the smallest Q-value gap and explicitly ignores the size of fluctuations inside each equilibrium. Footnote 6 concedes that Q-value fluctuation sizes differ across equilibria because they depend on rewards and continuation payoffs. In a stochastic Q-learning process, the relevant exit rate is approximately exp(-Delta Q / sigma), so equal stability requires Delta Q_AD / sigma_AD = Delta Q_WSLS / sigma_WSLS, not Delta Q_AD = Delta Q_WSLS. The simulation F1 scores do not discriminate between these two conditions because sigma differences may be small or correlated with payoffs on the tested grid. The authors should either derive a variance-corrected boundary, or provide empirical estimates of sigma_AD and sigma_WSLS showing that the omitted term is negligible in the region where the boundary is claimed to hold.","section":"§4, after Conjecture 4.2, and footnote 6"},{"comment":"The central theoretical framing relies on the existence of a unique stationary distribution of the hidden Markov process, but Conjecture 4.2 is unproven. The simulation-based stationarity check (Delta O < 0.05 across ten initialisations) is a finite-horizon heuristic and does not establish uniqueness of an invariant measure. Moreover, the accessible parameter range is defined by this same heuristic and then used to select the configurations for the F1 evaluation, so the reported scores are conditional on the conjecture. Since Appendix E shows that performance degrades outside this range, the paper should either provide a proof or at least a substantially stronger argument for Conjecture 4.2, or explicitly re-frame the contribution as a heuristic boundary with empirical validation in the finite-horizon stationary regime.","section":"§4, Conjecture 4.2 and §5.2"}],"minor_comments":[{"comment":"The set in Eq. (4.4) is written as {tilde pi^i = BR(pi) for some pi in Pi_epsilon}; this should specify that BR is the best response to the opponent's strategy, not to an arbitrary strategy in Pi_epsilon. As written it is ambiguous.","section":"§4, Theorem 4.1"},{"comment":"The text states that the discount-factor grid is Theta_delta = {0.65, 0.70, ..., 0.85}, but Figure 3 shows panels for delta = 0.55 and Figure 4 also uses delta = 0.55. The grid definition should be corrected.","section":"§5.2 and Figure 3"},{"comment":"The normalisation in Eq. (5.4) divides by the total occupation of the five focal profiles, but Table 1 shows this total is only 0.77-0.90 depending on the hyperparameters. The interpretation of 'dominance' should explicitly note that the classifier is relative to the focal set, not to the full strategy-profile space.","section":"§5.3, Eq. (5.4) and Table 1"},{"comment":"The phrase 'derive a boundary' overstates the status of the result, since the derivation is a heuristic resting on an unproven conjecture. I suggest 'propose and empirically validate a heuristic boundary'.","section":"Abstract and §4"},{"comment":"The perturbation plot along the delta dimension uses the x-axis 'delta - delta_c' with truncation to [-1,1]. Since delta_c can be outside [0,1], the truncation and the interpretation of the resulting scores should be explained.","section":"Figure 6"}],"recommendation":"major_revision","confidential_remarks":"The paper contains an interesting and potentially useful heuristic with a very thorough simulation study. However, the main theoretical derivation currently has an internal inconsistency between the defined barrier height and the Appendix B formulas, and the paper applies the boundary in parameter regions where WSLS is not an equilibrium. The fluctuation-size issue further weakens the theoretical interpretation. These are fixable in a revision if the authors restrict the scope of the theoretical claim and add the requested robustness checks, but they are load-bearing for the paper's central claim as stated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Janusz and Max have written a useful paper. The genuinely new thing is the time-averaged occupation-time criterion under constant exploration, plus the closed-form δ_c boundary that separates WSLS-stability from AD-stability. That boundary is derived from Bellman equations, not fitted, and the simulations back it: macro F1 0.85–0.95 in the accessible range, with perturbation analysis showing the boundary is locally optimal. The paper is also candid about what it does not do: Conjecture 4.2 (unique stationary distribution) is unproven, and the stationarity filter is post hoc. That honesty matters.\n\nThe weak point is the one the authors flag in footnote 6: the stability heuristic uses only the height of the smallest Q-value gap, min_s ΔQ, and ignores the size of fluctuations in each equilibrium. The stress-test note puts this squarely: if σ_AD differs from σ_WSLS, the equal-stability condition should be ΔQ/σ, not ΔQ. On reading Appendix B, the derivation indeed equates raw gaps. The simulations validate the raw-gap boundary within the tested grid, but the grid is not designed to discriminate raw-gap from variance-corrected conditions, so the theory's load-bearing simplification is not directly tested. This is not a demonstrated error — the paper calls it a heuristic and it performs well — but it is the place a reviewer should push.\n\nOther soft spots in proportion: no code or data is released, which slows independent verification; the focal strategy set is partly chosen from preliminary simulations, so the F1 metric has a small circular component; and the finite-horizon stationarity check substitutes for Conjecture 4.2. None of these are fatal. The central empirical claim — that the boundary tracks the transition away from defection-dominated play — holds up in the accessible region.\n\nWho benefits: anyone working on algorithmic collusion or multi-agent Q-learning who cares about persistent exploration. The paper deserves a serious referee; the main demands on revision should be a sharper treatment of the fluctuation-scale issue and, ideally, release of the simulation code, not a redo of the core argument.","headline":"A sharp, honestly-labeled heuristic boundary for time-averaged cooperation in Q-learning; solid simulations, but the stability argument ignores fluctuation variance and the stationarity conjecture is unproven.","tokens_in":23752,"tokens_out":1560,"would_cite":true,"duration_ms":15291,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T05","60J20","91A05"],"pacs":[],"model":"deepseek-v4-flash","headline":"Cooperation can dominate the long-run behavior of Q-learning agents even when exploration never fades, and the paper derives the payoff threshold at which it does.","keywords":["algorithmic collusion","Q-learning","iterated Prisoner's Dilemma","constant exploration","occupation time","equilibrium stability","win-stay lose-shift","epsilon-greedy"],"falsifier":"Simulate epsilon-greedy Q-learning in the same repeated Prisoner's Dilemma and sweep the discount factor across δ_c while measuring the average occupation time of cooperative profiles; if the empirical transition occurs at a markedly different δ, or fails to shift in the predicted direction when ε changes, the boundary is falsified. A complementary check is to measure the Q-value distributions around the AD and WSLS equilibria: if the difference in their fluctuation widths changes which equilibrium is more likely to be exited, the barrier-only heuristic will misclassify.","tokens_in":22905,"feed_emoji":"🤖","tokens_out":5472,"duration_ms":60418,"temperature":0.7,"pith_summary":"The paper studies two epsilon-greedy Q-learning agents playing the iterated Prisoner's Dilemma with one-period memory, keeping both exploration and learning rates constant. In this regime the agents never settle permanently into one strategy, so the paper shifts the question from convergence to time-averaged behavior: what fraction of time do the agents spend cooperating? The authors claim that cooperative play, centered on the win-stay, lose-shift equilibrium, can dominate in this time-averaged sense, and they derive a closed-form boundary on the discount factor that separates cooperative from defection-dominated regimes. The boundary is obtained by comparing the stability of the All-Defect and WSLS equilibria, measured by the smallest gap between Q-values in each. Large-scale simulations show that this boundary predicts the observed transition away from defection-dominated behavior with high classification accuracy.","feed_headline":"Past one threshold, learning algorithms cooperate most of the time","feed_subtitle":"The threshold comes from comparing the stability of two equilibria in a constantly exploring Q-learning Prisoner's Dilemma.","key_machinery":"The machinery is a stability comparison between the All-Defect and Win-Stay, Lose-Shift equilibria of the expected Q-learning dynamics. For each equilibrium, the minimum over the four states of the optimal Q-value difference ΔQ(s) = Q(s,argmax_a Q(s,a)) - Q(s,argmin_a Q(s,a)) is computed from the Bellman optimality equation; that minimum represents the height of the smallest Q-value barrier. Setting the two minima equal and solving for δ gives the critical discount factor δ_c, which is the paper's predictor. The derivation is heuristic—it deliberately ignores differences in fluctuation sizes across equilibria—but the resulting boundary is the object that organizes the simulations and the cla","core_discovery":"The central claim is that the relative stability of two Nash equilibria of the memory-one repeated Prisoner's Dilemma—All-Defect (AD) and Win-Stay, Lose-Shift (WSLS)—controls which equilibrium the persistent fluctuations of constant-exploration Q-learning favor. Stability is measured by the smallest optimal Q-value gap min_s ΔQ(s), interpreted as the barrier random exploration must overcome to reverse a player's action preference in some state. Equating this barrier for WSLS and AD and solving for the discount factor yields δ_c = 2(T+P-(R+S)) / ((1-ε)[2(R-P)+ε(P+S-(R+T))]): for δ > δ_c, WSLS is the more stable equilibrium, so the long-run stationary distribution should place more occupation","pith_inferences":["If the barrier-only stability heuristic is correct, the same Q-value-gap comparison could be applied to any pair of equilibria in other Q-learning games with a finite state space, yielding a general time-averaged dominance predictor.","A direct test beyond the paper's grid would be to measure the stationary fluctuation width of Q-values around AD and WSLS; if those widths differ enough to reverse the ordering of exit probabilities, the boundary should misclassify in a detectable way.","Because the boundary itself is independent of α but the accessible stationary regime depends strongly on α, a possible extension is a finite-horizon mixing-time correction that extends the prediction to non-stationary parameter regions.","For regulators, the result suggests that auditing algorithms for collusion should include the fraction of time spent pricing supra-competitively, since deployed algorithms with constant exploration may never converge to a fixed policy."],"forward_implications":["Cooperation can be sustained among reinforcement-learning pricing algorithms without vanishing exploration; the relevant measure of collusive harm becomes the long-run fraction of time spent in cooperative or supra-competitive play.","The boundary δ_c depends on payoffs and exploration rate but not on the learning rate, so within the accessible stationary regime the prediction should transfer across choices of α.","The boundary is best interpreted as locating the transition away from defection-dominated behavior rather than a sharp prediction of full mutual cooperation, since asymmetric states occupy a non-negligible share of time near the boundary.","The time-averaged perspective implies that social-welfare effects of algorithmic collusion depend on how long supra-competitive prices are sustained by fluctuating learning dynamics, not only on whether such prices can be learned.","The prediction presupposes convergence to a unique stationary occupation-time distribution; in practice the theory applies only where the simulation horizon is long enough for stationarity to be reached."],"fun_headline_variants":["Stability threshold flips Q-learners toward cooperation","Equilibrium stability drives Q-learning cooperation","Why exploring Q-learners cooperate: a stability tipping point","Stability barrier decides when Q-learners cooperate","A stability condition for cooperative Q-learning"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The central prediction collapses if an equilibrium's stability is not captured by its smallest Q-value gap, or if the occupation-time distribution does not converge to a unique stationary law; the paper leaves both as assumptions, and the fluctuation-size part is explicitly conceded not to hold literally.","fun_headline_variants_meta":{"raw":{"variants":["Stability threshold flips Q-learners toward cooperation","Equilibrium stability drives Q-learning cooperation","Why exploring Q-learners cooperate: a stability tipping point","Stability barrier decides when Q-learners cooperate","A stability condition for cooperative Q-learning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000914,"raw_usage":{"total_tokens":3756,"prompt_tokens":733,"completion_tokens":3023,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":477,"completion_tokens_details":{"reasoning_tokens":2967}},"tokens_in":477,"tokens_out":3023,"duration_ms":21289,"temperature":1.0,"reasoning_tokens":2967,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T04:40:31.651582+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Simulate epsilon-greedy Q-learning in the same repeated Prisoner's Dilemma and sweep the discount factor across δ_c while measuring the average occupation time of cooperative profiles; if the empirical transition occurs at a markedly different δ, or fails to shift in the predicted direction when ε changes, the boundary is falsified. A complementary check is to measure the Q-value distributions around the AD and WSLS equilibria: if the difference in their fluctuation widths changes which equilibrium is more likely to be exited, the barrier-only heuristic will misclassify.","supporting_citations":[],"review_version":1}