{"id":"576e4c60-bb14-4a03-b631-a5368da58e26","arxiv_id":"2506.23276","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"In a public goods game with costly sanctioning, reasoning-focused LLMs (o1, o3-mini) cooperated less and free-rode more than traditional LLMs such as Llama-3.3 and GPT-4o.","lead":"Large language models played a public goods game where players could spend tokens to punish or reward each other. Reasoning-focused models such as OpenAI's o1 series free-rode more often than traditional models, suggesting stronger reasoning does not automatically make AI agents more cooperative.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reasoning-vs-traditional gap is confounded by model family, provider, and temperature; within-family o3-mini results even point the opposite way, so the causal claim needs matched controls.","rationale":"The reader's weakest_assumption correctly identifies the core confound: reasoning models are all OpenAI o-series, while traditional models come from different families and API configurations. My stress-test confirms this is the most load-bearing concern and adds two pieces of internal evidence that strengthen it: (1) the o3-mini low/medium/high comparison within a single reasoning model family does not show the predicted monotonic decrease in cooperation—if anything, it shows the opposite ordering of means; and (2) o3-mini-medium, a reasoning model, achieves 100% SI participation and 0% free-riders, which directly contradicts the category-level narrative. These observations do not disprove the descriptive finding that some OpenAI reasoning models free-ride more in this game; they do show that the causal attribution to 'reasoning capability' is unsupported by the current design. The robustness checks in Appendix F are helpful but are run on one traditional model and one reasoning model, so they inherit the same confound and cannot disambiguate the causal claim. The RQ4 classifier is a secondary concern because the headline behavioral gap does not depend on it. Given this, the reader's CONDITIONAL verdict is appropriate; no verdict change is needed, but the requested matched-control experiments are necessary before the causal claim can be accepted.","tokens_in":29464,"tokens_out":3342,"duration_ms":34677,"concrete_test":"Run a matched within-family experiment: (1) o3-mini at low/medium/high reasoning effort with at least 10 runs per setting, and (2) the same underlying model family with vs. without an explicit 'reason step-by-step' instruction, holding temperature and API defaults fixed. If the effect of reasoning manipulation on mean contribution is not monotonic or its bootstrapped 95% CI includes zero, the causal 'reasoning causes free-riding' claim in §4.1 and Figure 1 fails, and the paper should be reframed as reporting descriptive model-family differences.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline causal claim—\"reasoning capabilities lead to less cooperative behavior\" (Fig. 1 caption; §4.1)—is not identified by the experimental design. The \"reasoning\" group consists exclusively of OpenAI o1/o3 models, while the \"traditional\" group is GPT-4o/4o-mini, DeepSeek-V3, and Llama-3.3, with temperature 1.0 for traditional models and internal reasoning effort for reasoning models (Table 6). Family, provider, scale, RLHF, temperature, and API defaults are all entangled with the reasoning label. No matched control isolates reasoning. Two key models (o1-preview, o3-mini-high) have n=1 runs (Table 1), and no significance tests accompany the means. The within-family evidence is actually inconsistent with the causal story: o3-mini-low/medium/high average contributions are 9.28/11.07/12.57 (Table 1), i.e., higher reasoning effort is associated with higher, not lower, mean contribution in the only family that varies reasoning effort; o3-mini-medium also reaches 100% SI and 0% free-riders. Thus the causal interpretation is unsupported even if the descriptive category-level gap is real.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper adapts the Gürerk et al. (2006) public goods game with endogenous institutional choice to LLM agents, running 15-round simulations with seven model families and using GPT-4o to classify agents' stated reasoning. It reports four behavioral archetypes and claims that reasoning-focused LLMs (o1/o3 series) free-ride substantially more than traditional LLMs (GPT-4o, DeepSeek-V3, Llama-3.3), despite similar or higher stated reasoning effort. The authors interpret this as evidence that reasoning capability does not necessarily promote cooperation and may impede it.","tokens_in":29640,"tokens_out":2496,"duration_ms":28043,"significance":"The paper builds a clean, reproducible experimental environment for studying costly sanctioning in LLM societies, and the code, prompts, taxonomy, and classification pipeline are fully released. The empirical observation of large descriptive differences in contribution behavior across models is potentially useful for the multi-agent LLM community, and the qualitative reasoning traces in Appendix G give concrete examples of free-riding justifications. However, the headline causal claim that 'reasoning capabilities lead to less cooperative behavior' is not identified by the experimental design, and several load-bearing comparisons rest on single runs and overlapping archetype assignments. The contribution is best viewed as a descriptive benchmark and a cautionary demonstration of model heterogeneity, not as a controlled test of reasoning versus traditional architectures.","major_comments":[{"comment":"The central claim that reasoning LLMs are less cooperative is confounded. All reasoning models are OpenAI o-series models, while the traditional group comprises GPT-4o, GPT-4o-mini, DeepSeek-V3, and Llama-3.3-70B, which differ in provider, model family, scale, RLHF procedure, and API defaults; Table 6 also shows temperature 1.0 for traditional models versus internal reasoning effort for reasoning models. No matched control or intervention isolates 'reasoning' as a variable. The within-family evidence in Table 1 actually points against the causal reading: o3-mini low/medium/high average contributions are 9.28, 11.07, and 12.57, so higher reasoning effort is associated with higher mean contribution in the only family that varies reasoning effort, and o3-mini-medium reaches 100% SI participation and 0% free-riders. The paper should either reframe the finding as descriptive model-level differences or add matched controls (e.g., the same base model with and without chain-of-thought).","section":"§4.1 and Table 1 / Table 6"},{"comment":"Two key models, o1-preview and o3-mini-high, have only a single run, yet Table 1 reports no significance tests anywhere, and §4.1 states that 'a significant performance gap' exists between traditional and reasoning LLMs. With n=1 for two of the five reasoning-model rows and no inferential statistics, the aggregate means in the reasoning column are not robust. At minimum, the paper should report per-run data, effect sizes, and some test (bootstrap or permutation) that respects the small number of runs, or explicitly restrict the strong claims to models with repeated runs.","section":"Table 1 and §4.1"},{"comment":"The behavioral archetypes are assigned in a way that creates internal contradictions. Figure 3 lists o3-mini-low under both 'Increasingly defecting' and 'No change', o3-mini-med under both 'Increasingly cooperative' and 'No change', and DeepSeek-V3 under both 'Increasingly cooperative' and 'No change'. The same overlap appears in Figure 6. If the archetypes are not mutually exclusive, the subsequent comparisons of 'increasingly cooperative' versus each other archetype in §4.4 and Tables 3–5 become ambiguous, because a single model can contribute to both sides of a comparison. The paper should define the archetype assignment algorithm and either enforce disjoint assignment or treat archetype membership as a soft label with the appropriate statistical treatment.","section":"§4.3 and Figure 3 / Figure 6"},{"comment":"The RQ4 analysis is partly circular. The behavioral archetypes are defined from the contribution and institution-choice trajectories (Section 4.3), and the reasoning strategies are then classified from the same agents' stated rationales for those same decisions and used to 'explain' the archetype differences. A difference in reasoning labels between groups is expected when the groups are already separated by their behavior; the analysis does not establish that reasoning causes the behavior. The paper should clarify that RQ4 is descriptive and correlational, or use a held-out prediction design (e.g., predict future rounds' contributions from earlier reasoning classifications).","section":"§4.4 and Appendix D.2"},{"comment":"The robustness checks are too narrow to support the broad conclusion that 'the core behavioral archetypes remained largely consistent'. They cover only Llama-3.3-70B and o1-mini, use single runs for each ablation, and the narrative-prompt result in Table 8 changes o1-mini's average contribution from 5.39 to 9.95 and free-rider rate from 69.3% to 45.7%, which is a substantial sensitivity to prompt framing. This does not contradict the main finding, but it weakens the claim that the behaviors are stable across design choices, and the single-run ablations should be labeled as exploratory.","section":"Appendix F"}],"minor_comments":[{"comment":"The caption says 'reasoning capabilities lead to less cooperative behavior', which is a causal claim not supported by the design; recommend rewording to 'are associated with' or 'correlate with'.","section":"Figure 1 caption"},{"comment":"The narrative prompt templates contain the literal string 'REW ARD_COST' where 'REWARD_COST' is intended; this is likely a LaTeX typo and should be corrected.","section":"Appendix F.2.1"},{"comment":"The captions refer to 'Llama 3.1 70B' and the legend says 'Llama 3.1', while the main text consistently uses Llama-3.3-70B; the labels should be unified.","section":"Figure 7 and Figure 8 captions"},{"comment":"The human row reports only one value for the punish/reward ratio and free-rider percentage with an asterisk, but the footnote does not state whether the human free-rider value is a period-average or final-period value; please clarify.","section":"Table 1 footnote"},{"comment":"The statistical analysis section describes hierarchical bootstrapping, but the main text (Section 4.1) does not apply any significance test to the headline comparison; a pointer to the appendix would help readers locate the inferential analysis.","section":"Appendix D.2.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is a competent descriptive study of LLM behavior in a sanctioning public goods game, and the code release is a genuine asset. The main concern is that the abstract and Figure 1 state a causal interpretation—'reasoning capabilities lead to less cooperative behavior'—that the experimental design cannot support. The reviewer should ask the authors to either add matched comparisons or substantially soften the causal language and restructure the paper around the descriptive contribution. If the authors can provide a matched-control experiment (e.g., the same base model with and without chain-of-thought), the paper could become a strong contribution; without it, the central claim should be presented as a correlational observation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a useful descriptive paper with a new experimental setup, but the headline causal claim about reasoning causing free-riding is not supported by the design, and the paper's own within-family numbers point against it. The reader's verdict is about right.\n\nWhat's actually new: the institutional-choice public goods game with costly sanctioning has not been run on LLM agents before, and the four behavioral archetypes — increasingly cooperative, increasingly defecting, no change, unstable — are a handy descriptive taxonomy. The authors ship code, prompts, and the classification taxonomy, and they compare against human behavior from Gürerk et al. (2006), which is a nice touch. The qualitative reasoning traces in Appendix G are genuinely instructive; the contrast between o1-mini's explicit free-riding justification and Llama-3.3's cooperative framing tells a real story about how different models frame the game.\n\nThe soft spots are proportional to how strongly the paper states the conclusion. The abstract and Figure 1 say reasoning capabilities lead to less cooperative behavior, but the reasoning group is entirely OpenAI o1/o3 models, tested with different temperature defaults, while the traditional group is GPT-4o, DeepSeek-V3, and Llama. No matched control isolates reasoning. More importantly, the within-family evidence goes the wrong way: o3-mini-low/med/high contribute 9.28/11.07/12.57 tokens on average, so more reasoning effort is associated with higher, not lower, contributions. o3-mini-medium also reaches 100% SI participation and zero free-riders. Whatever is going on, it is not a clean dose-response of reasoning depth. Two key models (o1-preview, o3-mini-high) have single runs, and no significance tests accompany the means, so the descriptive gap in Table 1 is suggestive but not established at the level of the claim.\n\nThe RQ4 analysis is mildly circular — archetypes are defined from the same trajectories that the reasoning classifications are then mapped back onto — but that is a minor issue for this kind of qualitative study. The payoff description also has small inconsistencies among Section 3.2, Figure 2, and Section 3.3, though nothing load-bearing.\n\nWho gets value: people deploying LLM agents in repeated mixed-motive settings, and researchers studying emergent cooperation. The descriptive phenomenon — that some well-known models free-ride heavily in this environment — is worth knowing and replicating. The authors are transparent about budget constraints and single runs in the appendix, which I credit.\n\nRecommendation: if this came to me as a desk editor, I would send it to review, not reject on sight. The right outcome would be a revision that tempers the causal language, adds matched controls or within-family reasoning-effort comparisons, and reports significance or effect sizes. My own verdict on the strong claim is skeptical, but the paper deserves referee time.","headline":"Useful descriptive benchmark with a new game setup, but the causal 'reasoning causes free-riding' claim is confounded and undercut by the paper's own o3-mini results; deserves review with mandatory revisions.","tokens_in":30251,"tokens_out":5796,"would_cite":true,"duration_ms":54248,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"In repeated public-goods games with costly sanctioning, reasoning-focused LLMs such as o1-mini become free-riders while traditional LLMs like GPT-4o and Llama-3.3-70B sustain near-universal cooperation, challenging the assumption that…","keywords":["public goods game","LLM agents","cooperation","free-riding","social dilemmas","costly sanctioning","institutional choice","reasoning LLMs"],"falsifier":"Run the same 15-round game on a single model family with reasoning effort as the only variable, for instance o3-mini at minimum, medium, and maximum effort with all prompts, temperature, and history identical, or GPT-4o with and without a mandated chain-of-thought reasoning block, and compare free-rider rates; if cooperation does not drop as reasoning effort rises, the paper's central claim that reasoning drives free-riding is falsified.","tokens_in":2022,"feed_emoji":"🤝","tokens_out":2281,"duration_ms":97160,"temperature":0.7,"pith_summary":"This paper asks whether LLM agents will spend their own resources to enforce cooperation, not just follow norms, when placed in a social dilemma. The authors adapt a public goods game with institutional choice from behavioral economics: seven agents play fifteen rounds, each round choosing between a sanctioning institution where rewards and punishments are available at personal cost and a sanction-free institution, then deciding how much to contribute to a shared pool. The central finding is a behavioral divide: traditional LLMs (GPT-4o, GPT-4o-mini, DeepSeek-V3, Llama-3.3-70B) converge to sustained high cooperation with near-zero free-riding, while reasoning-optimized models (o1-mini, o1-preview, o3-mini at all effort levels) defect, abandon the sanctioning institution, or lock into rigid suboptimal strategies. The paper argues that stronger reasoning pushes models toward individually rational but collectively harmful strategies, so cooperation is not a natural byproduct of capability gains.","feed_headline":"Reasoning LLMs free-ride where traditional LLMs cooperate","feed_subtitle":"In a public-goods game, o1-class models free-ride while GPT-4o and Llama-3.3 sustain high cooperation.","key_machinery":"The carrying object is the institutional-choice public goods game, adapted from the human experiment of Gurerk et al.: each round an agent picks a Sanctioning Institution (SI) or a Sanction-Free Institution (SFI), contributes 0-20 tokens to a common pool multiplied by 1.6 and split equally among the institution's members, and, if in the SI, spends from a separate 20-token endowment to reward (+1 to recipient, cost 1) or punish (-3 to recipient, cost 1) others. With N=7 agents and a multiplier below group size, the payoff structure $\\pi_i = (e - c_i) + \\alpha \\sum_j c_j / N$ makes zero contribution the Nash equilibrium while full contribution is the social optimum (52 vs. 40 tokens), a 12-token tension that institutional choice and sanctioning sit on top of. The second load-bearing mechanism is the reasoning-analysis pipeline: every decision is accompanied by a verbal justification, which the authors classify into a 15-category taxonomy (payoff maximization, cooperative argument, complexity aversion, free-riding, and so on, grouped into economic, social, risk, and control families) using GPT-4o as annotator. The taxonomy is what lets the paper map behavioral archetypes onto decision-making styles: cooperative agents reason about collective welfare, defecting agents reason about equilibrium and exploitation.","core_discovery":"On its own terms, the paper's discovery is that reasoning ability and cooperative behavior come apart in LLM agents. In the same public goods game with costly sanctioning, reasoning-focused models free-ride: o1-mini contributes on average 5.39 of 20 tokens with 69.3% of rounds classified as free-riding, o1-preview and o3-mini-high oscillate between cooperation and defection, and o3-mini-low/medium hold fixed suboptimal contributions. Traditional models instead match or exceed the human benchmark of 18.3 tokens (Llama-3.3-70B reaches 18.71, with 0% free-riding) and nearly all migrate to the sanctioning institution, just as human participants did. The reasoning traces explain the mechanism the paper attributes: o1-mini explicitly justifies zero contributions by invoking the Nash equilibrium and the absence of enforcement, while Llama-3.3-70B justifies maximum contributions by appeal to collective earnings and influence on others. A secondary discovery is that every LLM enforces norms with rewards, not punishment, inverting the human pattern (humans punished 1.66 times more than they rewarded), which the authors read as evidence that the models optimize positive interactions rather than understanding deterrence.","pith_inferences":["The causal reading that 'reasoning itself' corrupts cooperation is not isolated by the experiment: every reasoning model is an OpenAI o-series model, while the cooperative group is a different family, scale, and training regime, and two of the reasoning results come from single runs; a same-family comparison with reasoning effort toggled on and off would be needed to confirm the mechanism.","The reward-over-punishment preference is plausibly an alignment artifact, since models trained to avoid negative or confrontational outputs may systematically under-use punishment even when it is the equilibrium-stabilizing tool; this is testable by comparing sanctioning behavior across models with different alignment objectives.","A quick falsification probe: test a reasoning model from a different vendor on the same game; if it cooperates at traditional-LLM levels, the observed gap tracks the o-series family rather than reasoning capability.","Because the o3-mini effort settings already show internally inconsistent results (low effort: 0% high contributors but only 7.2% free riders; high effort: 65.7% high contributors in a single run), the archetype taxonomy may be sensitive to run count and reasoning-effort configuration."],"forward_implications":["Model choice becomes a cooperation decision: in multi-agent LLM systems that share resources or produce joint goods, deploying reasoning-optimized models risks free-riding and collapse of the shared pool.","Improving reasoning ability does not automatically improve social behavior; the paper concludes that cooperation may require targeted pro-social alignment rather than general capability gains.","LLM-based institutions will likely enforce norms through rewards, not punishment, which the authors argue may undermine the long-term stability of cooperation because deterrence is not being modeled.","The four behavioral archetypes, increasingly cooperative, increasingly defecting, no change, and unstable, provide a diagnostic taxonomy for predicting which agent societies will sustain cooperation."],"supporting_citations":[{"why":"Supplies the human public-goods experiment with institutional choice that this paper adapts, and the human baseline it compares against.","marker":"Gurerk et al., 2006"},{"why":"Establishes costly punishment as a mechanism that sustains cooperation, motivating the sanctioning-institution design.","marker":"Fehr & Gächter, 2000"},{"why":"The prior LLM social-dilemma study (GovSim) this work extends from resource extraction to contribution and norm enforcement.","marker":"Piatti et al., 2024"},{"why":"Defines the o1 reasoning-focused models whose free-riding behavior is the paper's central contrast.","marker":"Jaech et al., 2024"},{"why":"Supports the paper's choice of GPT-4o as the classifier for annotating agent reasoning strategies.","marker":"Gilardi et al., 2023"},{"why":"Gives the game-theoretic result that voluntary-contribution equilibria collapse toward zero when the multiplier is below group size.","marker":"Isaac & Walker, 1988"},{"why":"Defines GPT-4o, one of the traditional models that sustain high cooperation in the experiments.","marker":"Hurst et al., 2024"}],"fun_headline_variants":["Reasoning LLMs free-ride in public goods games","o1-mini free-rides, Llama-3.3 cooperates in costly sanctions","Corrupted by reasoning: LLM free-riders in cooperation","Smarter reasoning, stingier cooperation in LLM agents","Public goods game: reasoning models choose self over group"],"cache_read_input_tokens":32384,"weakest_assumption_plain":"The comparison assumes the cooperation gap is caused by reasoning capability, but every reasoning model tested comes from a single model family (OpenAI o-series) that differs from the traditional models in training, scale, and API settings, so the design never isolates reasoning as the cause.","fun_headline_variants_meta":{"raw":{"variants":["Reasoning LLMs free-ride in public goods games","o1-mini free-rides, Llama-3.3 cooperates in costly sanctions","Corrupted by reasoning: LLM free-riders in cooperation","Smarter reasoning, stingier cooperation in LLM agents","Public goods game: reasoning models choose self over group"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000404,"raw_usage":{"total_tokens":2157,"prompt_tokens":1053,"completion_tokens":1104,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":669,"completion_tokens_details":{"reasoning_tokens":1014}},"tokens_in":669,"tokens_out":1104,"duration_ms":12172,"temperature":1.0,"reasoning_tokens":1014,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:48:29.665271+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same 15-round game on a single model family with reasoning effort as the only variable, for instance o3-mini at minimum, medium, and maximum effort with all prompts, temperature, and history identical, or GPT-4o with and without a mandated chain-of-thought reasoning block, and compare free-rider rates; if cooperation does not drop as reasoning effort rises, the paper's central claim that reasoning drives free-riding is falsified.","supporting_citations":[],"review_version":1}