{"id":"bbc78393-e0c4-4d5f-87a0-756a74be917a","arxiv_id":"2503.15516","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"In a 241-participant Hanabi study, AI behavioral metrics like action diversity and strategic dominance predict human preference ratings more strongly than the final game score, though all correlations are weak to moderate.","lead":"Human players in a 241-person study of the cooperative card game Hanabi preferred AI teammates that behaved diversely and avoided irrational moves, and their ratings were predicted more by those behavioral measures than by the team's final score. The finding challenges the common reinforcement-learning assumption that high task performance is the main driver of human preference toward AI teammates.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported correlations use 241 participants but only 8 agents; per-agent constant predictors make the p-values invalid and the headline comparison of predictor strength is confounded by level of analysis.","rationale":"To support the abstract's claim, it must be possible to estimate a stable association between an agent-level objective metric and human teamwork ratings with valid uncertainty. The current evidence cannot do this because the independent variable is an agent constant for only 8 agents. The reader's stated weakest assumption (transfer from AI-AI to human-AI games) is real and untested, but I see the unit-of-analysis problem as the more direct reason to reject: even if transfer held perfectly, Table 5 would still not establish predictive power. The paper's own sensitivity columns show that one outlier can move or reverse the correlations, and Appendix E concedes that the low p-values are driven by large N without addressing agent-level clustering. A corrected agent-level or multilevel analysis is necessary before the comparative claims can be evaluated. I therefore agree with the reader's REJECT verdict; no adjustment is needed.","tokens_in":23153,"tokens_out":11980,"duration_ms":129278,"concrete_test":"Re-run the Table 5 analysis at the agent level: compute the mean teamwork rating for each of the 8 agents and correlate each objective metric against these 8 means (Pearson and Spearman), with a permutation test over agent labels and a leave-one-agent-out analysis; equivalently, fit a mixed-effects model with random intercepts for agent and participant and report cluster-robust confidence intervals. If no objective metric's agent-level effect excludes zero, or if the rank ordering changes so that human-AI game score is not the weakest predictor, the paper's central claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step is the cross-level correlation in Section 3.2 / Table 5. Each AI-only metric is a per-agent constant, and only 8 agents were used. The regression then treats the 482 block-level ratings (two per participant) as independent observations, so the p-values reported for the AI-only metrics are far too small. Appendix D's Bonferroni correction addresses multiple comparisons, not clustering or the tiny number of agents. The instability is visible in Table 5 itself: AD-entropy flips from r=-0.086 with RandomBot to r=0.301 without it, inter-XP falls from 0.370 to 0.047, and G2 changes sign. The headline comparison is also confounded: human-AI game score is a participant-level variable measured in the same games as the rating, while every other metric is an agent-level mean, so comparing r values across levels conflates aggregation with predictive utility. With at most 8 independent observations for the AI-only side, Table 5 cannot support the claim that any AI-only metric is a better predictor than human-AI game score.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper asks whether objective, human-free measurements of an AI agent's behavior can predict human subjective preferences toward that agent as a teammate. The authors evaluate a pool of Hanabi AI agents on task-performance metrics (self-play, intra-XP, inter-XP), information-theoretic metrics (action entropy, action-response entropy, instantaneous coordination, context independence), and game-theoretic metrics (frequencies of dominated and dominant moves) using AI-only games. They then collect human teamwork ratings (N=241 participants, each playing with two of eight agents) and correlate the agent-level AI-only metric means with the participant-level teamwork ratings. The headline result is that human-AI game score is a weaker predictor of subjective preference than several AI-only metrics, including self-play score, ARD-entropy, and G1-dominated move frequency, with the stated long-term goal of using such metrics to shape RL reward functions. The paper also reports that the AI-only metrics correlate only weakly to moderately overall and that results are sensitive to inclusion of RandomBot.","tokens_in":23389,"tokens_out":4145,"duration_ms":42415,"significance":"If the central claim were statistically valid, the paper would make a genuinely useful contribution: it would offer a practical, human-free proxy signal for training human-collaborative AI and would empirically challenge the common assumption that self-play or cross-play performance predicts human preference. The study has notable strengths: a large human-subjects sample, a diverse set of publicly available agents, deliberate reporting of results with and without an outlier agent, and careful documentation of metric definitions and experiment procedures in the appendices. However, the statistical analysis at the core of the paper has a load-bearing flaw: the AI-only metrics are per-agent constants measured for only eight agents, while the regression treats hundreds of participant-level ratings as independent observations. This invalidates the reported p-values and makes the headline comparison between human-AI game score and AI-only metrics confounded by level of analysis. The instability of the correlations to the exclusion of a single agent further undermines the claim that these metrics are predictive.","major_comments":[{"comment":"The correlation analysis in Section 3.2 regresses each participant's block-level teamwork rating (two ratings per participant, 482 total) on per-agent constant predictors that take only eight distinct values. This treats ratings as independent despite being nested within agents, and the effective sample size for each AI-only metric is at most eight agents, not 482. The Bonferroni correction in Appendix D addresses multiple comparisons but does not address clustering or the tiny number of independent units on the predictor side. Consequently, the p-values in Table 5 are artificially small, and the relative ordering of correlation strengths across metrics is not a valid basis for the paper's central claim. A mixed-effects model or cluster-robust inference with agents as the clustering unit would be required, and with eight agents the confidence intervals around the agent-level correlations would be very wide.","section":"Section 3.2, Table 5; Appendix D"},{"comment":"The headline comparison is confounded by level of analysis: the human-AI game score is a participant-level variable measured in the same games as the rating, whereas self-play, inter-XP, entropy, CI, and game-theoretic frequencies are agent-level means computed from a different set of games. Comparing Pearson correlations across these two levels conflates the unit of analysis with predictive utility. Even if the within-level correlations were computed correctly, the statement that AI-only metrics are 'more predictive' than human-AI game score cannot be supported unless both are evaluated on the same observational units or with a comparable aggregation scheme.","section":"Section 3.2, Table 5; Section E.2"},{"comment":"The results are not robust to the inclusion of a single agent. Excluding RandomBot flips AD-entropy from r=-0.086 to r=0.301, reduces inter-XP from r=0.370 (p<0.001) to r=0.047 (p=0.344), changes the sign of G2-dominated from 0.332 to 0.058, and drops self-play from 0.447 to 0.168. Since the analysis has only eight agents, these large swings show that the correlations are driven substantially by one outlier rather than by a stable relationship. The paper does report both analyses, which is transparent, but the instability undercuts the abstract's claim that the AI-only metrics are generally predictive of human preference.","section":"Section 3.2, Table 5; Section E.1, E.2"},{"comment":"The transfer assumption that AI-only metrics computed from AI-AI games are representative of behavior with human partners is load-bearing and not justified. Hanabi behavior is strongly partner-dependent, as the paper itself demonstrates: Table 3 and Section E.2 explain RandomBot's low ARD-entropy and CI as artifacts of short AI-AI games, and the paper notes that OP+Aux and AP+Aux have high inter-XP scores yet low human ratings. Without validation that the AI-only metric distributions match those in human-AI games, the correlations in Table 5 may reflect game-length or partner-policy artifacts rather than stable agent traits. The paper should either compute the metrics from the human-AI game traces (where feasible) or provide evidence that the metric values are consistent across partner types.","section":"Section 2.2, 3.1, Table 3; Appendix E.2"},{"comment":"The interpretation of Context Independence is reversed after the fact: the paper asserts that lower CI is better in Hanabi because the number of concepts exceeds the number of actions, but RandomBot has the lowest CI and is the least preferred agent. The paper explains this by game-length effects, but that explanation relies on the same transfer assumption criticized above. Additionally, the Instantaneous Coordination analysis uses a parabolic fit with three free parameters (a,b,c) selected after viewing the data, so the reported r values (0.515 and 0.332) are not confirmatory correlations; they overstate the strength of the relationship. These issues should be addressed with a pre-specified model or a correction for model selection, and the CI reversal should be validated independently.","section":"Appendix C.2, Section E.2; Table 5"}],"minor_comments":[{"comment":"The Shannon entropy citation appears as a malformed placeholder '(?)Chp 2]cover2006elements' rather than a proper citation; the reference should be corrected.","section":"Appendix C.2"},{"comment":"The table note says IC 'had no m and p values,' but the text in Section E.2 reports r values for the nonlinear fit; it would be helpful to state explicitly that no p-value is computed because the fit is nonlinear and post-hoc.","section":"Section 3.2, Table 5 note"},{"comment":"The discussion of game length (Figure 8) is very useful for interpreting the entropy and CI results, but it would be improved by reporting the mean and standard deviation of game length for each agent in the main text rather than only in the appendix.","section":"Figure 8 and Section E.2"},{"comment":"There are several typographical and formatting issues, such as 'Jacques et al.' in Section C.2 (should be 'Jaques et al.') and inconsistent spacing in Table 3; a careful proofread is needed.","section":"General"}],"recommendation":"reject","confidential_remarks":"The statistical problems are not local: the effective sample size for the AI-only predictors is eight agents, and the comparison between participant-level and agent-level correlations is confounded by construction. Even a reanalysis with mixed models or cluster-robust standard errors would leave the central claim unsupported, and the transfer assumption from AI-AI to human-AI behavior would require new validation data. In my view this goes beyond a revision that could be completed within the manuscript's current scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know. First, this is a real empirical effort: 241 people played Hanabi with eight AI agents, rated them on a teamwork scale, and the authors correlate those ratings with a broad set of AI-only metrics. The data collection is serious, and the paper is honest about its weak-to-moderate effect sizes. Second, the central correlational claim is not supported by the analysis as run. The predictor for every AI-only metric is a single mean per agent, with eight agents total, while the outcome is 482 participant-block ratings. Those observations are not independent, so the p-values in Table 5 are far too small. The instability is visible in their own table: AD-entropy flips from negative to positive without RandomBot, inter-XP drops from r=0.370 to r=0.047, and G2 changes sign.\n\nWhat is new and good: the agent pool is broader than prior Hanabi work, the information-theoretic and game-theoretic metrics go beyond simple score, and the \"questionable move\" button gives a behavioral handle on what humans notice. The negative result on inter-XP is useful: OP+Aux and AP+Aux score high in AI-AI games but are not preferred by humans, which is a concrete counterpoint to assumptions in the MARL literature. The paper also acknowledges that N=241 drives low p-values and that effect sizes are r=0.2-0.5, which is more candour than many papers manage.\n\nThe soft spots are real. The unit-of-analysis problem is load-bearing, not cosmetic. Comparing \"human-AI game score\"—a participant-level variable measured in the same games as the rating—with agent-level means of other metrics conflates aggregation with predictive utility. The transfer assumption that AI-AI behavior represents AI-human behavior is acknowledged for RandomBot's game length but not tested, and the IC parabola is fit post-hoc. The CI interpretation reversal is argued after the fact, though the argument is at least plausible.\n\nWho is this for? People working on human-AI teaming evaluation and reward design. The descriptive data and some secondary findings are worth having. But as is, Table 5 cannot support the claim that esoteric metrics outperform final game score. A revision that does agent-level inference or properly clustered analysis, with pre-registered outlier rules, could make a modest version of the conclusion work.\n\nI would send it to a serious referee—the data and question matter—but the referee should be told to focus on the correlational analysis. I would not cite the headline result until the analysis is fixed.","headline":"Large, honest Hanabi study with a load-bearing statistics problem: the headline correlations treat eight agents as hundreds of independent observations.","tokens_in":23906,"tokens_out":3375,"would_cite":false,"duration_ms":32552,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Human preferences toward AI teammates can be predicted from human-free measurements of agent behavior, and final team score predicts those preferences more weakly than action diversity, strategic dominance, and coordination metrics.","keywords":["human-AI teaming","Hanabi","reinforcement learning","human preferences","teamwork rating","objective metrics","information theory","game theory"],"falsifier":"A held-out prediction test would settle the claim: use the fitted regressions from the eight evaluated agents to predict teamwork ratings for new agents, run the same human protocol, and check whether the predicted preference ordering matches the observed ratings. A cleaner experiment would train two agents matched on self-play score but differing in G1-dominated frequency, since the claim predicts the low-G1 agent earns higher teamwork ratings despite equal game scores.","tokens_in":22946,"feed_emoji":"🎴","tokens_out":10105,"duration_ms":83621,"temperature":0.7,"pith_summary":"The paper asks whether an AI agent's behavior can be scored without any human involvement in a way that predicts how much human teammates will like working with it. Using the cooperative card game Hanabi, the authors measured a pool of AI agents on task-performance, information-theoretic, and game-theoretic metrics through AI-only games, then had 241 human participants play with a subset of those agents and rate each as a teammate. The central finding is that the final game score a human-AI team achieves is a weaker predictor of human preference than several AI-only metrics, including the diversity of the agent's actions, its instantaneous coordination with a partner, and how often it takes strategically dominated moves. If the correlations hold, objective metrics computable without humans in the loop could serve as reward signals for training collaborative AI, replacing expensive human feedback during training.","feed_headline":"Final score fails to predict how humans rate AI teammates","feed_subtitle":"In Hanabi, action diversity and rare 'irrational' moves predict human ratings better than winning does.","key_machinery":"The load-bearing machinery is a three-stage pipeline. First, AI-only game traces are compressed into three families of objective metrics: task performance (self-play, intra-algorithm cross-play, and inter-algorithm cross-play scores), information-theoretic measures (action-distribution entropy, action-response distribution entropy, instantaneous coordination, and context independence), and game-theoretic move frequencies (discarding a known-playable card, playing a known-unplayable card, and playing a known-playable card). Second, a subset of agents spanning this metric space is played by 241 human participants who rate each teammate on a summed Likert teamwork scale. Third, single-variable regressions with a Bonferroni-corrected significance threshold link each objective metric to the subjective rating, and it is the correlation table produced by this step that carries the argument's weight.","core_discovery":"The paper's central claim is that subjective human preferences toward AI teammates are partially predictable from objective metrics that require no human interaction, and that task performance is not the most informative metric for this purpose. In Hanabi, the human-AI game score correlates with teamwork ratings at $r = 0.350$ including the random agent and $r = 0.203$ without it, while action-response distribution entropy ($r = 0.466$) and instantaneous coordination (nonlinear fit, $r = 0.517$) correlate more strongly. The game-theoretic measures show the sharpest practical signal: a 1-percent increase in the frequency of discarding a card known to be playable corresponds to a 9-point drop on the 35-point teamwork scale, whereas self-play and intra-algorithm cross-play scores are weak-to-moderate predictors and inter-algorithm cross-play score shows no significant correlation once the random agent is excluded. The authors read these results as refuting the common assumption that better self-play or cross-play performance implies a more preferred teammate, and they propose that human-collaborative agents should be trained with rewards that blend task score with information-theoretic and game-theoretic terms.","pith_inferences":["The asymmetry between the steep negative slope for dominated moves and the gentler positive slope for dominant moves suggests preference is driven more by avoiding visible mistakes than by maximizing visible competence; a study that varies only G1-move frequency while holding game score constant could test this loss-aversion pattern directly.","The paper's transfer assumption, that AI-AI statistics represent human-AI behavior, could be checked by recomputing the objective metrics with a scripted human-modeled partner; if agent rankings shift, the correlations would need re-interpretation.","An engineering test would train a new agent with a reward penalty on G1-dominated moves and run a fresh human study to see whether it outranks a score-matched agent without the penalty.","Because the strongest predictors carry domain-specific interpretations, the same correlation pipeline may need to be rebuilt for each new teaming task, which argues against expecting a single portable human-preference metric."],"forward_implications":["Reward functions for training collaborative agents should blend task score with information-theoretic and game-theoretic terms rather than maximizing raw performance alone.","Self-play and cross-play scores, widely used as coordination benchmarks, would be demoted from proxies for human preference to weak indicators at best.","Designers could screen candidate AI teammates with cheap AI-AI evaluations before committing to expensive human-subject experiments.","Trainers should suppress single-step dominated moves even at very low frequencies, since a 1-percent increase in such moves predicts a 9-point teamwork-rating drop.","Because no metric reached strong predictive power even in Hanabi's stripped-down setting, a universal human-preference reward function is unlikely to exist across less-constrained teaming tasks."],"supporting_citations":[{"why":"Establishes Hanabi as the AI-teaming benchmark and defines the 0-25 scoring that all task-performance metrics use.","marker":"(Bard et al., 2020)"},{"why":"Supplies the rule-based agents (SimpleBot through SmartBot) that anchor the low-complexity end of the agent pool.","marker":"(O'Dwyer, 2019)"},{"why":"Provides the RL agents IQL, VDN, and SAD(+Aux) that populate the learning-based portion of the pool.","marker":"(Hu & Foerster, 2019)"},{"why":"Introduces Other-Play and the cross-play evaluation paradigm, and supplies the OP+Aux agent with high inter-XP but low human ratings.","marker":"(Hu et al., 2020)"},{"why":"Defines Off-Belief Learning and supplies the OBL1/OBL2/OBL4 agents that receive the highest teamwork ratings.","marker":"(Hu et al., 2021)"},{"why":"Defines instantaneous coordination, the objective metric with the strongest correlation to human teamwork rating.","marker":"(Jaques et al., 2019)"},{"why":"Defines context independence, which the paper adapts to Hanabi with a reversed interpretation.","marker":"(Bogin et al., 2018)"},{"why":"Defines self-play, intra-XP, and inter-XP scoring and the Any-Play agent; the paper directly refutes its assumption that high inter-XP implies human preference.","marker":"(Lucas & Allen, 2022)"},{"why":"Prior human-AI Hanabi experiment whose open question, whether learned agents can beat rule-based agents in human preference, this study resolves.","marker":"(Siu et al., 2021)"}],"fun_headline_variants":["Hanabi: AI action diversity beats score for teammate ratings","Game score less predictive than AI 'quirks' for human ratings","Action diversity predicts human-AI teammate preference over score","In Hanabi, rare moves predict teammate quality better than winning","For AI teammates, diversity and 'irrational' moves matter more"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The correlations assume that how an agent behaves next to another AI, including the actions it favors and the game lengths it produces, is representative of how it behaves next to a human, even though Hanabi strategies are strongly partner-dependent.","fun_headline_variants_meta":{"raw":{"variants":["Hanabi: AI action diversity beats score for teammate ratings","Game score less predictive than AI 'quirks' for human ratings","Action diversity predicts human-AI teammate preference over score","In Hanabi, rare moves predict teammate quality better than winning","For AI teammates, diversity and 'irrational' moves matter more"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000147,"raw_usage":{"total_tokens":1187,"prompt_tokens":947,"completion_tokens":240,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":563,"completion_tokens_details":{"reasoning_tokens":155}},"tokens_in":563,"tokens_out":240,"duration_ms":3366,"temperature":1.0,"reasoning_tokens":155,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T19:52:05.337713+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A held-out prediction test would settle the claim: use the fitted regressions from the eight evaluated agents to predict teamwork ratings for new agents, run the same human protocol, and check whether the predicted preference ordering matches the observed ratings. A cleaner experiment would train two agents matched on self-play score but differing in G1-dominated frequency, since the claim predicts the low-G1 agent earns higher teamwork ratings despite equal game scores.","supporting_citations":[{"cited_title":"N., Chandar, S., Burch, N., Lanctot, M., Song, H","cited_arxiv_id":null,"evidence_quote":"Establishes Hanabi as the AI-teaming benchmark and defines the 0-25 scoring that all task-performance metrics use."},{"cited_title":"quuxplusone/hanabi: framework for writing bots that play hanabi","cited_arxiv_id":null,"evidence_quote":"Supplies the rule-based agents (SimpleBot through SmartBot) that anchor the low-complexity end of the agent pool."},{"cited_title":"Z., and De Freitas, N","cited_arxiv_id":null,"evidence_quote":"Defines instantaneous coordination, the objective metric with the strongest correlation to human teamwork rating."},{"cited_title":"Emergence of Communication in an Interactive World with Consistent Speakers","cited_arxiv_id":"1809.00549","evidence_quote":"Defines context independence, which the paper adapts to Hanabi with a reversed interpretation."},{"cited_title":"and Allen, R","cited_arxiv_id":null,"evidence_quote":"Defines self-play, intra-XP, and inter-XP scoring and the Any-Play agent; the paper directly refutes its assumption that high inter-XP implies human preference."},{"cited_title":"C., Pe \\ n a, J., Chen, E., Zhou, Y., Lopez, V., Palko, K., Chang, K., and Allen, R","cited_arxiv_id":null,"evidence_quote":"Prior human-AI Hanabi experiment whose open question, whether learned agents can beat rule-based agents in human preference, this study resolves."}],"review_version":1}