REVIEW 3 major objections 6 minor 87 references
Ad-Hoc Human-AI Coordination Challenge
T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Human proxies make Hanabi AI benchmarking cheap and reproducible
desk verdict Useful benchmark infrastructure, but the proxies' human-likeness is not yet established; treat leaderboard scores as coordination with these specific policies, not with humans. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the human proxy agent trained with Human-Data-Regularised IPPO (HDR-IPPO). The procedure first trains an LSTM-based behavioural cloning policy on a large-scale human dataset (101,096 two-player and 46,525 three-player games from hanab.live following H-group conventions), then initializes from the best BC checkpoint and continues with IPPO while adding a KL regularisation term anchoring the policy to the BC action distribution, weighing the IPPO objective against KL divergence with a weight lambda.
What would settle it
A direct human-AI play experiment in which human participants play Hanabi with both the human proxies and with other humans, comparing scores and subjective coordination quality: if proxy-AI teams score systematically lower than human-AI teams (or proxy-human teams differ from human-human teams), the proxies do not stand in for real human partners.
Extended reading notes
Core claim
The central claim is that human proxy agents trained by behavioural cloning on a large-scale human dataset followed by KL-regularized independent PPO serve as robust, cheap, and reproducible human-like evaluation partners for ad-hoc human-AI coordination in Hanabi. The paper demonstrates that these proxies substantially improve self-play scores over pure BC policies—especially in three-player games, where zero-score games drop from over 70% to well under 1%—while retaining human-like behaviour, evidenced by cross-play compatibility with BC policies, comparable action-prediction accuracy on held-out human games, and nearly identical information-per-play and communicativeness metrics. On the resulting leaderboard, no baseline matches the human proxies themselves, and OBL, which uses no human data, outperforms methods that use the provided limited human data, highlighting the difficulty of efficient human-data use.
Load-bearing premise
The central assumption is that human proxy agents, trained on and validated against H-group hanab.live players, faithfully represent the humans an AI agent will actually coordinate with.
Editorial extensions
If this is right
- AI coordination research gains a standardized, low-cost evaluation protocol for human-AI coordination in Hanabi, with 1,000-game evaluation runs through a controlled API instead of costly human studies.
- The benchmark can track progress in data-efficient human-AI coordination, since participants receive only 3,079 open-source games while the proxies are trained on the full closed dataset.
- The baseline results set a reference: OBL's strong zero-shot performance without human data suggests that existing methods underutilize limited human data, and the lagging performance of BC, BR-BC, FCP, OP, IPPO, and DeepSeek-R1 defines the current gap.
- The action-prediction sub-challenge provides a complementary, replicable measure of human-likeness via teacher-forced cross-entropy loss on held-out human games.
- If the proxies are accepted as human stand-ins, the leaderboard provides a meaningful ranking of human-AI coordination ability that can guide future algorithmic development.
Reading between the lines
- The proxies' validation is internal to one community (H-group on hanab.live); the paper itself defers direct human-play validation. If the goal is coordination with general humans, the proxies may over-represent a specific convention system, and leaderboard scores might not transfer to casual or differently conventioned players.
- A direct testable extension would be a human study comparing human scores and subjective experience when paired with the proxies versus with real humans; if scores diverge, the proxies' role as a surrogate would need recalibration.
- The KL-regularisation mechanism is quite general: the same BC-plus-regularised-RL recipe could create human proxies for other cooperative games or real-world human-AI collaboration settings where human demonstrations are available but live evaluation is expensive.
- The near-identical behavioural metrics (IPP and communicativeness) between proxies and the human dataset are coarse summaries; finer-grained behavioural distributions (e.g., convention-following rates per game position) could reveal whether the proxies reproduce human strategy spread or only its average.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces AH2AC2, a benchmark for ad-hoc human-AI coordination in Hanabi, and proposes human proxy agents as cheap, reproducible stand-ins for human partners. The proxies are trained by behavioral cloning on 147,621 games from the hanab.live H-group population, then refined with IPPO regularized by a KL term toward the BC policy. The authors open-source a limited dataset of 3,079 games, provide a controlled evaluation API and leaderboard, and report baselines including IPPO, BC, BR-BC, OBL, OP, FCP, and a prompted DeepSeek-R1 agent. The paper also defines an optional action-prediction challenge. Section 5 validates the proxies via self-play scores, cross-play with BC policies, action-prediction metrics on held-out games, and two behavioral statistics (IPP and Communicativeness); Section 7 explicitly defers direct human-AI play validation to future work.
Significance. If the central claim holds, AH2AC2 would be a valuable community resource: it is the first open-source human Hanabi dataset, it provides a controlled evaluation protocol, and it assembles a broad set of baselines that clarify the difficulty of human-AI coordination. The HDR-IPPO recipe, the hosted API, and the open baselines are concrete and reproducible contributions, and the paper gives credit where prior datasets were closed. However, the paper's headline claim is that the proxies are 'human-like evaluation partners', and the evidence for this is indirect: all validation strands are measured against the same H-group population used for training. The leaderboard therefore measures coordination with a learned proxy, not yet with humans. The benchmark artifact is useful even under that caveat, but the claims and the 'golden standard' label in Table 5 outrun the current evidence.
major comments (3)
- [Section 5.2, Table 3, Section 7] The load-bearing property of the central claim—that scores achieved with the proxies measure human-AI coordination—is not demonstrated. The three validation strands all measure agreement with the same H-group population used for training: action-prediction loss/accuracy on held-out games from that population, cross-play with BC policies trained on that population, and two aggregate behavioral statistics. None of these establishes that a candidate's score with a proxy predicts its score with a human, especially a human outside H-group conventions. Moreover, Table 3 shows that the HDR-IPPO refinement systematically degrades the direct human-likeness metrics relative to BC (accuracy −0.03 to −0.08; loss +0.05 to +0.08). The paper itself states in Section 7 that 'The ultimate validation of our human proxy agents requires direct human-AI play' and defers it. The Abstract and Section 4.2 claims that the proxies 'serve as robust, cheap, and reproducible human-like evaluation partners' therefore outrun the current evidence.
- [Table 5, Section 4.3] The leaderboard footnote labels the human proxies as a 'golden standard' because they are not constrained by game limits. This is circular relative to the paper's own caveat: the proxies are the proposed substitute for human evaluation, so their scores cannot serve as the criterion against which candidate agents are judged until the proxy-human correspondence is established. I recommend either adding a human-subject validation study or relabeling these scores as 'proxy-coordination scores' and removing 'golden standard' until such validation exists.
- [Section 4.3, Appendix A.7] The evaluation protocol is described inconsistently: the main text says registered participants receive access to 1,000 evaluation games, while Appendix A.7 states that the evaluation is 2,000 games total (1,000 in each of two- and three-player settings). Please disambiguate. Additionally, the text calls pre-registration 'the gold standard for empirical science'; pre-registration alone does not prevent iterative fitting to the hosted proxies across repeated evaluation requests, so the integrity guarantee needs a concrete statement about how many evaluation runs a participant may request and what controls exist.
minor comments (6)
- [Abstract and Section 4.1] The abstract says the proxies are trained on 'a large-scale human dataset', but the full dataset is closed and only 3,079 games are open-sourced; please make this asymmetry explicit in the abstract.
- [Section 4.1] There is a typo: 'gives them access to to 1,000 evaluation games' should read 'access to 1,000 evaluation games'.
- [Section 2] In the Dec-POMDP definition, the sentence 'i.e. sH is always a terminal state' lacks a period and is missing the preceding comma; please fix the punctuation.
- [Section 5.2] The phrase 'highlight the brittleness of BC policies in unfamiliar scenarios' and 'BC policies' poor single-player performance' use 'single-player' where 'self-play' is the intended term; please align the terminology with the rest of the paper.
- [Appendix A.6] The qualitative analysis of the proxy's convention-following is based on a single game; please state explicitly that this is illustrative rather than a systematic behavioral evaluation.
- [Appendix A.10] The LLM prompt templates contain several typos that may affect reproducibility, including 'your shop' for 'your chop', 'rnak clue' for 'rank clue', 'follwing' for 'following', 'Teamamte' for 'Teammate', and 'has not been discarded yet, where discarding'.
Circularity Check
No significant circularity: the proxy pipeline and benchmark are self-contained, and held-out human-action prediction provides independent evidence; direct human validation is explicitly deferred.
full rationale
The paper is an empirical benchmark contribution, not a derivation whose conclusion is algebraically forced by its inputs. The proxy agents are trained by behavioral cloning on a large human dataset followed by KL-regularized IPPO (Eq. 2); the KL term uses the BC policy as a reference, but this is a training objective, and the paper does not define 'human-like' as 'close to this BC policy.' Human-likeness is then assessed on held-out games excluded from training (Table 3), which is a genuine out-of-sample prediction, together with cross-play against BC policies and aggregate behavioral metrics (Table 4). The cross-play check is partly internal—BC policies are themselves derived from the same H-group data—so it is not independent evidence of human equivalence, but it does not reduce a prediction to a fitted parameter. The paper explicitly acknowledges that 'The ultimate validation of our human proxy agents requires direct human-AI play' (Section 7), i.e., the human-likeness claim is an assumption awaiting external validation, which is a limitation rather than circularity. Self-citations to Hu et al. and Bakhtin et al. support the regularized-RL methodology, but the paper provides its own experiments (Tables 2-4, Appendix A.8), so the citations are not load-bearing. No uniqueness theorem, ansatz smuggled via citation, or renaming of a known result is present. Therefore no circular step is substantiated under the quoted-evidence standard.
Assumptions & free parameters
free parameters (3)
- KL regularization weight lambda (human proxies) =
0.3, 0.2, 0.1, 0.15 for theta1..theta4
- Entropy coefficient (IPPO) =
1e-5, 1e-4, 1e-4, 1e-4
- Open dataset size per setting =
1,000 training games plus validation (858 two-player, 221 three-player)
assumptions (3)
- domain assumption H-group conventions on hanab.live constitute a meaningful distribution of human Hanabi play.
- domain assumption Behavioral cloning supervised on human actions, then refined by KL-regularized self-play, yields human-compatible policies.
- standard math Standard PPO/IPPO convergence and the environment implementation are correct.
invented entities (1)
-
Human proxy agents as evaluation partners
Cite this review
Pith. "Pith review of Ad-Hoc Human-AI Coordination Challenge." pith.science (2026). https://pith.science/paper/PJQDE7WY
@misc{pith2026250621490,
author = {Pith},
title = {Pith review of: Ad-Hoc Human-AI Coordination Challenge},
year = {2026},
howpublished = {\url{https://pith.science/paper/PJQDE7WY}},
note = {Machine review of arXiv:2506.21490}
}
read the original abstract
Achieving seamless coordination between AI agents and humans is crucial for real-world applications, yet it remains a significant open challenge. Hanabi is a cooperative card game featuring imperfect information, constrained communication, theory of mind requirements, and coordinated action -- making it an ideal testbed for human-AI coordination. However, its use for human-AI interaction has been limited by the challenges of human evaluation. In this work, we introduce the Ad-Hoc Human-AI Coordination Challenge (AH2AC2) to overcome the constraints of costly and difficult-to-reproduce human evaluations. We develop \textit{human proxy agents} on a large-scale human dataset that serve as robust, cheap, and reproducible human-like evaluation partners in AH2AC2. To encourage the development of data-efficient methods, we open-source a dataset of 3,079 games, deliberately limiting the amount of available human gameplay data. We present baseline results for both two- and three- player Hanabi scenarios. To ensure fair evaluation, we host the proxy agents through a controlled evaluation system rather than releasing them publicly. The code is available at \href{https://github.com/FLAIROx/ah2ac2}{https://github.com/FLAIROx/ah2ac2}.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
Turn 0: Success (Hint blue, play clue on the blue 1)
-
[2]
Turn 1: Success (Play clue on both the 1’s)
-
[3]
Turn 2: Failure (Looks like a 2 save, but doubles the yellow 2’s, violating good touch principle) 18 Ad-Hoc Human-AI Coordination Challenge
-
[4]
Turn 3: Success (Actor 1 plays card slot 2, knowing it’s a B1, which follows the ”play clue” convention)
-
[5]
Turn 4: Success (Actor 0 gives a 5 save when Red 5 is on the chop)
-
[6]
Turn 5: Success (Actor 1 plays a known playable card)
-
[7]
Turn 6: Failure (Actor 0 should have 2-saved the Red 2 on chop)
-
[8]
Turn 7: Success (Actor 1 discards Red 2, the chop card)
Show all 87 references
-
[9]
Turn 8: Success (Actor 0 plays slot 0, a known playable 1)
-
[10]
Turn 9: Success (Actor 1 discards chop on slot 3)
-
[11]
Turn 10: Success (Fix clued the duplicate cards)
-
[12]
Turn 11: Success (Plays known playable card)
-
[13]
Turn 12: Success (Gives play clue to the Red 1)
-
[14]
Turn 13: Success (Plays the Red 1)
-
[15]
Turn 14: Success (Discards chop)
-
[16]
Turn 15: Success (Gives play clue to the Yellow 3)
-
[17]
Turn 16: Success (Plays Yellow 3)
-
[18]
Turn 17: Success (Discards known trash)
-
[19]
Turn 18: Success (Discards chop)
-
[20]
Turn 19: Failure (Doesn’t understand chop-focus, giving play clue to wrong card, and violates good touch principle by clueing Red 1 and Red 3 twice)
-
[21]
Turn 20: Failure (The focus of the last clue was the chop, so it should have played the Red 3, but played Red 2 instead)
-
[22]
Turn 21: Success (Fix clue on the duplicated Red 3’s)
-
[23]
Turn 22: Success (Plays known playable Red 3)
-
[24]
Turn 23: Success (Discards chop, position 1)
-
[25]
Turn 24: Success (Gives play clue to White 2)
-
[26]
Turn 25: Success (Plays White 2)
-
[27]
Turn 26: Success (Play clue on the Red 4)
-
[28]
Turn 27: Success (Plays the Red 4)
-
[29]
Turn 28: Success (Play clue on Blue 4, and filling in White 3)
-
[30]
Turn 29: Success (Plays White 3)
-
[31]
Turn 30: Success (Discards known trash)
-
[32]
Turn 31: Success (Play clue on White 4)
-
[33]
Turn 32: Success (Plays White 4)
-
[34]
Turn 33: Success (Plays known playable Red 5)
-
[35]
Turn 34: Success (Discards known trash Red 1) 19 Ad-Hoc Human-AI Coordination Challenge
-
[36]
Turn 35: Failure (Should have played its Blue 3 because of the play clue, instead gave a 5 hint off chop, which is illegal after the late game)
-
[37]
Turn 36: Success (Discards chop)
-
[38]
Turn 37: Failure (Should have played its Blue 3, instead gave a 2 hint which is illegal)
-
[39]
Turn 38: Success (Discards chop)
-
[40]
Turn 39: Failure (Should have played its Blue 3, instead discarded chop)
-
[41]
Turn 40: Success (Play clue on Blue 4, filling in Blue 3)
-
[42]
Turn 41: Success (Plays Blue 3)
-
[43]
Turn 42: Success (Discards chop)
-
[44]
Turn 43: Success (Plays Blue 4)
-
[45]
Turn 44: Success (Discards chop)
-
[46]
Turn 45: Success (Hint Blue, filling in Blue 5)
-
[47]
Turn 46: Success (Plays Blue 5)
-
[48]
Turn 47: Success (Play clue on Green 1)
-
[49]
Turn 48: Success (Plays Green 1)
-
[50]
Turn 49: Success (Discards chop)
-
[51]
Turn 50: Success (Play clue on Yellow 4)
-
[52]
Turn 51: Success (Plays Yellow 4)
-
[53]
Turn 52: Success (Plays Green 2)
-
[54]
Turn 53: Success (Discards chop)
-
[55]
Turn 54: Success (Reveals Green 4 identity)
-
[56]
Turn 55: Success (Discards chop)
-
[57]
Turn 56: Success (Plays Yellow 5)
-
[58]
Turn 57: Success (Discards chop)
-
[59]
Turn 58: Success (5 save on Green 5)
-
[60]
Turn 59: Success (Stalling, hinting 1s)
-
[61]
Turn 60: Failure (Hinting Green is seen as a play clue on Green 1, which is illegal)
-
[62]
Turn 61: Success (Plays Green 1, which it thought was Green 3 because of convention)
-
[63]
Turn 62: Success (Play clue on Green 3)
-
[64]
Turn 63: Success (Plays Green 3)
-
[65]
Turn 64: Success (Hints White 5)
-
[66]
Turn 65: Success (Plays Green 4) In summary, in this game, the human proxy followed H-group conventions for 88% of the moves and used various strategies while playing the game. 20 Ad-Hoc Human-AI Coordination Challenge •Successful H-Group conventions played: –Giving play clue ...
2010
-
[73]
We train a new BC policy that serves as both a starting point for subsequent HDR-IPPO training and a baseline for comparison in this ablation study
-
[74]
These agents share identical architectures, hyperparameters, and training procedures, with the sole exception of the human data regularisation weight, λ
From the baseline BC policy, we train multiple HDR-IPPO agents. These agents share identical architectures, hyperparameters, and training procedures, with the sole exception of the human data regularisation weight, λ. We vary the weight of regularisation term across a range of...
-
[75]
lost-in-the-middle
We evaluate, analyse and compare the trained HDR-IPPO agents and the baseline BC policy. Precisely, we: (a) Assess the SP performance of each agent to understand how the strength of KL regularisation influences its ability to play Hanabi effectively on its own. (b) Evaluate th...
2014
-
[76]
One New Card: If exactly one card is newly clued (had no prior clues), that card is the focus of the clue
-
[77]
Multiple New Cards (chop focus): If more than one card is newly clued and the chop is included in the clue, the chop card is the focus
-
[78]
Multiple New Cards (not including chop): If more than one card is newly clued and the chop is not included in the clue, the leftmost new card is the focus
-
[79]
- Clue Type: - If the chop is included in the clue, the chop card is the focus and it is either a playable card (if it’s a Play Clue) or a critical card (if it’s a Save Clue)
No New Cards: If the clue only touches cards that already had clues, the leftmost re-clued card is the focus. - Clue Type: - If the chop is included in the clue, the chop card is the focus and it is either a playable card (if it’s a Play Clue) or a critical card (if it’s a Sav...
-
[80]
34 Ad-Hoc Human-AI Coordination Challenge
Good Touch Principle: Only give clues to cards that are have not been played yet; avoid clueing cards that have already been played. 34 Ad-Hoc Human-AI Coordination Challenge
-
[81]
protected
Save Principle: Including cards that are saved with Save Clues, do not allow other players to discard playable cards. All cards that are playable need to be "protected" by giving them a Play Clue. The following cards must not be discarded: All rank 5 cards, Unique rank 2 cards...
-
[82]
If a clue does not make a card playable or prevent the discard of a critical card, you should discard instead of wasting a clue
Minimum Clue Value Principle: Every clue must either make one or more cards safely playable or prevent the discard of a critical card. If a clue does not make a card playable or prevent the discard of a critical card, you should discard instead of wasting a clue. ## The Early ...
-
[83]
Identify your current Finesse Position (leftmost unclued slot)
-
[84]
Valid Actions
Blind-play that card at once, assuming it is the connecting card 1 rank below the clued card. - Instructions (for the clue-giver): - You can only give a finesse clue to Teammate 1, which would trigger Teammate 0 to blind-play their card in their Finesse Position before Teamamt...
-
[85]
Generate ONLY a single JSON object matching the schema
-
[86]
Do NOT include any text outside the JSON object (e.g., explanations, markdown formatting like```json, apologies, or status messages)
-
[87]
Ensure all required fields from the schema are present in the JSON. 38
-
[503]
Springer Berlin Heidelberg, Berlin, Heidelberg,
-
[1997]
doi: 10.1162/neco
ISSN 0899-7667, 1530-888X. doi: 10.1162/neco. 1997.9.8.1735. URL https://direct.mit.edu/ neco/article/9/8/1735-1780/6109. Hu, H., Lerer, A., Peysakhovich, A., and Foerster, J. ”Other-Play” for Zero-Shot Coordination. 2020. doi: 10.48550/ARXIV .2003.02979. URLhttps://arxiv. org...
-
[2012]
doi: 10.1007/ 978-3-642-27645-3 15
ISBN 978-3-642-27645-3. doi: 10.1007/ 978-3-642-27645-3 15. URL https://doi.org/ 10.1007/978-3-642-27645-3_15. Rawas, S. AI: the future of humanity.Discover Artificial Intelligence, 4(1):25, March 2024. ISSN 2731-0809. doi: 10.1007/s44163-024-00118-3. URL https://doi. org/10.1...
-
[2017]
URLhttps: //arxiv.org/abs/1707.06347
doi: 10.48550/ARXIV .1707.06347. URLhttps: //arxiv.org/abs/1707.06347. Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V ., Lanctot, M., Dieleman, S., Grewe, D., Nham, J., Kalchbrenner, N., ...
- [2018]
-
[2022]
Kingma, D
URLhttps://proceedings.mlr.press/ v162/jacob22a.html. Kingma, D. P. and Ba, J. Adam: A Method for Stochastic Optimization. 2014. doi: 10.48550/ARXIV .1412.6980. URLhttps://arxiv.org/abs/1412.6980. Kullback, S. and Leibler, R. A. On Information and Suf- ficiency.The Annals of M...
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.