Pith. sign in

REVIEW 4 major objections 6 minor 47 references

Codenames as a Benchmark for Large Language Models

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read LLMs handle a wider range of Codenames teammates than word-vector agents, though they don't yet beat those agents when paired with an identical partner.

desk verdict Useful benchmark artifact with a headline claim that runs ahead of the evidence: the cross-family generalization result is asserted but never actually tested. read the letter →

arxiv 2412.11373 v2 pith:HTSMXJO3 submitted 2024-12-16 cs.AI cs.CL

Matthew Stephenson , Matthew Sidji , Benoît Ronval This is my paper · ORCID
classification cs.AIcs.CL
keywords CodenamesLargeLanguageModelsAIbenchmarktheoryofmindcooperativegameplayingemergentplaystylewordembeddingsepistemicreasoning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes the word-association board game Codenames as a benchmark for evaluating large language models, arguing that the game's need for language understanding, theory of mind, and epistemic reasoning makes it a better test than static knowledge quizzes. The authors compare nine LLMs (including GPT-4o, Claude 3.5 Sonnet, Gemini 1.5, and Llama 3.1) against traditional word-vector agents in both a single-team scoring version and a full two-team competitive version of the game. Their central result is that current LLMs do not beat the word-vector agents when paired with an identical technique, but they lose far less when paired with a different agent, which the authors read as greater generalisability to unknown teammates. Each LLM also displayed a distinct, stable risk profile, and OpenAI's o1-preview achieved the best single-team score despite a risky style. The paper offers this as evidence that Codenames can serve as a behaviourally rich LLM benchmark.

What carries the argument

The central object is the Codenames game itself, used as a two-role benchmark in which one agent (codemaster) gives a single-word clue plus a number, and a partner (guesser) selects words from a 25-word board. The paper's key mechanism is the measurement of cross-agent generalisation: it pairs every codemaster with every guesser and records mean score, win-rate, loss rate, average clue number, and stop-early/stop-late rates. The framework updates a prior single-team version to full two-team rules, which changes the optimal risk strategy and is itself part of the claim about benchmark design.

What would settle it

Run the single-team and two-team experiments with a morphological 'no derived words' checker that accepts Education as valid (instead of rejecting it for containing CAT) and without the retry/random-fallback mechanism; if the model rankings and win-rates change materially, then the paper's central claims depend on the framework's rule approximations rather than on LLM reasoning ability.

Watch

Extended reading notes

Core claim

The paper's central claim is that LLM agents are more generalisable to a wider range of teammates than prior word-vector techniques, and that each LLM exhibits a unique emergent style of play. The quantitative evidence shows that mixed-model LLM pairs (e.g., GPT-4o with Gemini-1.5, or Llama-3.1 with Sonnet-3.5) perform roughly as well as same-model pairs in the single-team version, whereas word-vector pairs such as Word2Vec with GloVe see their mean score jump from roughly 5-6 to 12 and their loss rate from 0% to over 20%. In the two-team version, GloVe beats Word2Vec by a 72% win-rate when they face off, while LLM pairings mostly stay within a 40-60 win-rate band. The single-team results put o1-preview ahead of all other LLMs (mean score 8.41, with the next-best at 9.70 and most other LLM pairs above 10), and qualitative observations report LLMs drawing on cultural references like 'Hogwarts' to connect three board words at once, something no word-vector agent could do. The authors conclude that the full rules of Codenames expose cooperation and reasoning traits that simpler benchmarks miss.

Load-bearing premise

The load-bearing premise is that the framework's approximations of official Codenames rules—the substring-only check for the 'no derived words' rule, the lenient retry policy for invalid responses, and the random fallback after ten failures—do not distort which models rank higher; if these approximations favour models that produce short, substring-safe, format-compliant clues rather than models that reason about meaning, the comparative conclusions about generalisability and playstyle would not hold.

Editorial extensions

If this is right

  • If LLMs generalise across teammates, Codenames becomes a reproducible way to test theory of mind in models: a good guesser must infer what clue the codemaster intended, and a good codemaster must anticipate how a different teammate will interpret a clue.
  • Because each model has a stable playstyle, single-model leaderboard scores are only meaningful relative to a fixed teammate; future benchmark reporting should include the full teammate matrix rather than just self-play results.
  • The reversal between game versions—cautious play wins the single-team scoring game, risky play wins the two-team win-rate game—shows that benchmark design decisions (scoring vs win-rate, presence of an opponent) determine which model appears best, so conclusions about 'better' LLMs must be tied to the rule set.
  • The o1-preview result suggests that reasoning-specialised models may belong on a separate evaluation track from chat models for cooperative language games.
  • Word-vector agents remain the best same-model baseline (0% loss, scores near 5-7), so they should stay in the benchmark as the point of comparison for LLM generalisation rather than being removed.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension of this paper would replace the substring-only 'no derived words' check with a morphological or dictionary-based checker; if models like Gemini-1.5 and Llama-3.1, which tend to give longer meaning-rich clues, improve their ranks relative to models that favour short safe clues, then the reported playstyle ordering is partly an artifact of the current checker.
  • The two-team result suggests a general design principle for LLM game benchmarks: the payoff function (mean score vs win-rate) selects the optimal risk attitude, so benchmark authors should report both scoring and win-rate to avoid conflating strategic caution with inability.
  • The paper's observation that LLMs produce culturally-grounded clues such as 'Hogwarts' points to a concrete way to test shared-knowledge theory of mind: measure how well human guessers, not just LLM guessers, solve clues from each model, and compare against word-vector clues.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Codenames as a benchmark for evaluating large language models, extending an existing Codenames AI framework to support the full rules and two-team play. It evaluates nine LLMs (o1-preview, o1-mini, o3-mini, GPT-4o, Gemini-1.5, Sonnet-3.5, DeepSeek-R1, DeepSeek-V3, Llama-3.1) and three word-vector agents (Word2Vec, GloVe, Combined) over 100 trials of single-team and two-team games, reporting scores, win-rates, and playstyle metrics. The authors claim that LLM agents are more generalisable to a wider range of teammates than prior word-vector techniques, that each LLM exhibits a distinct emergent playstyle, and that o1-preview performs best in the single-team setting. Code, prompts, outputs, and results are made publicly available.

Significance. Assuming the central claims survive scrutiny, Codenames is a well-motivated and behaviorally rich benchmark for LLM language understanding, theory of mind, and cooperation. The paper's strengths include a publicly available framework with full rules, 100 trials per condition, detailed per-pair statistics, and transparent reporting of API costs. The playstyle metrics (clue number, stop-early/stop-late rates) are a useful contribution and could support future work on emergent LLM behaviour. However, the headline generalization claim is broader than the evidence, and several methodological choices need tightening before the benchmark results can be taken at face value.

major comments (4)
  1. [Abstract; Section I; Section IV.A.2; Section VII] The central claim that 'LLM agents are more generalisable to a wider range of teammates than prior techniques' is not directly tested. In the experiments, all LLM teams are paired only with other LLMs and all word-vector teams only with other word-vector agents; Section IV.A.2 states that the authors 'were unable to produce reliable performance results for games with both LLM and Word-Vector agents on the same team' and therefore evaluate each agent group separately. Section VII then notes that word-vector agents 'can still operate as Codemaster alongside an LLM guesser,' acknowledging that a cross-type pairing was feasible but not run. As a result, the observed robustness of LLM cross-pairs could reflect within-class compatibility (shared natural-language semantics and prompt format) rather than a genuinely wider range of teammates. Please either run the missing cross-type conditions (e.g., word-vector codemaster with LLM guesser) or narrow the claim to generalization across LLM teammates.
  2. [Section IV.A.1; Section IV.D.1] The model-selection procedure weakens the breadth of the claims. Six of the fifteen initially selected LLMs are excluded after 'preliminary testing' because they could not consistently follow the rules, but no results, criteria, or error rates are reported for them; this makes the comparison susceptible to selection bias and makes it difficult to interpret 'LLMs' as a class. In addition, five of the nine retained models (o1-preview, o1-mini, o3-mini, DeepSeek-R1, DeepSeek-V3) are evaluated only in self-pair configurations (Section IV.D.1), so the evidence for cross-teammate generalization actually rests on four models. Please report the excluded models' performance in an appendix, define a principled inclusion rule, or explicitly frame the results as applying only to high-performing, format-compliant LLMs.
  3. [Table II; Section VI.A.2; Table I] Win-rate and mean-score differences are reported without confidence intervals or significance tests. With 100 trials per condition, a 52% vs. 48% win-rate is within binomial sampling error, so statements such as Sonnet-3.5 having 'the highest average win-rate for the red team of 56.75%' and GPT-4o having 'the highest overall win-rate for the blue team of 58.75%' are not supported without uncertainty quantification. The same applies to the mean-score rankings in Table I. Please add confidence intervals or significance tests (e.g., bootstrap or exact binomial tests) for the key comparisons, especially those underlying the playstyle and generalization claims.
  4. [Section II.C; Section VI.B.2] The rule approximations can systematically bias the comparisons. The invalid-clue checker is a substring match, and Section VI.B.2 gives a concrete example where the valid clue Education is rejected because it contains the board word CAT; combined with the lenient retry policy and the random fallback after ten failures, this can favor models that produce short, substring-safe, format-compliant clues over models that reason about meaning. Because the paper does not report per-model invalid-response frequencies or re-run results with a more accurate adjudicator, it is unclear how much of the observed playstyle and generalization differences is an artifact of the checker. Please quantify the frequency of invalid responses per model and test sensitivity with an improved checker or human adjudication.
minor comments (6)
  1. [Table II] The 'GloVe vs. Combined' row reports win-rates of 63% and 35%, which sum to 98%; please correct the typo or explain any draws, since Codenames normally has no draw outcome.
  2. [Section IV.C.2] There is a typo in the guesser prompting description: 'invliad response' should be 'invalid response'.
  3. [Table I] The column header 'Civillian avg' should be spelled 'Civilian avg'.
  4. [Section VI.A.1] The Spearman rank correlation of 0.821 between average clue number and loss percentage is reported without a p-value or a statement of the sample size; please clarify what is being correlated and how many model pairs are included.
  5. [Figure 2] The text says the figure confirms 'our previous ordering of models from most risky to most cautious' but then lists only Sonnet-3.5, Gemini-1.5, GPT-4o, and Llama-3.1; please make explicit that this is a subset of the full ordering given in Section VI.A.1.
  6. [References [28] and [41]] References [28] and [41] appear to cite the same Sidji, Smith, and Rogerson paper on human-AI collaboration in Codenames; please merge or disambiguate them.

Circularity Check

0 steps flagged · score 1.0 of 10

No circularity: the generalisability claim is broader than the tested pairing design, but it is not derived from its own inputs or from a fitted parameter.

full rationale

This paper makes no mathematical derivation and fits no parameters: agent performance is measured directly from 100 gameplay trials per condition (Tables I and II). The central claim that LLM agents are 'more generalisable to a wider range of teammates than prior techniques' is inferred from comparing LLM-LLM pairs against word-vector/word-vector pairs. Section IV.A.2 states that the authors 'were unable to produce reliable performance results for games with both LLM and Word-Vector agents on the same team, and have instead chosen to evaluate each agent group separately,' and Section VII acknowledges that word-vector agents 'can still operate as Codemaster alongside an LLM guesser' even though that cross-type condition was not run. That is a scope/validity limitation, not a circular reduction: 'wider range of teammates' is not defined in terms of the LLM group's own outputs, and the reported LLM cross-pair results are genuine measurements rather than predictions forced by construction. The only self-citations (Sidji et al. 2024 on prompt engineering and on human-LLM Codenames collaboration) appear as related-work and future-work context, and the paper explicitly distinguishes its goal from the earlier prompt-engineering study ('Rather than exploring the effectiveness of prompt engineering techniques, our presented research instead aims to explore the inherent abilities of different state-of-the-art LLMs'). No equation, fitted parameter, uniqueness theorem, or ansatz is imported through those citations. Accordingly, no specific circular step can be exhibited, and the paper is self-contained as an empirical benchmark study.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No parameters are fitted to data; the central claims rest on the framework's rule approximations, the adequacy of 100 seeded trials, and the representativeness of the nine selected LLMs.

assumptions (4)
  • domain assumption The Codenames framework, including the relaxed invalid-clue retry rule and substring-based derivability check, is a faithful enough approximation of the official game rules to support comparative agent evaluation.
    Invoked in Section II.C; false positives like Education/CAT can change which clues are accepted.
  • domain assumption The 100 seeded trials provide stable estimates of agent performance, despite no significance testing or confidence intervals.
    Tables I and II report means and percentages over 100 trials; no inferential statistics are given; exact replication is not possible.
  • ad hoc to paper Models that could not consistently follow output formatting (six of fifteen) can be excluded without biasing the comparison of state-of-the-art LLMs.
    Section IV.A.1 excludes smaller models because their scores were significantly worse due to rule violations; this selection affects the scope of conclusions.
  • domain assumption The game of Codenames as implemented measures the intended constructs: language understanding, theory of mind, and epistemic/strategic reasoning.
    Section III.B argues for construct validity; no external validation against human reasoning tasks is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Codenames as a Benchmark for Large Language Models." pith.science (2026). https://pith.science/paper/HTSMXJO3

@misc{pith2026241211373,
  author       = {Pith},
  title        = {Pith review of: Codenames as a Benchmark for Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HTSMXJO3}},
  note         = {Machine review of arXiv:2412.11373}
}
read the original abstract

In this paper, we propose the use of the popular word-based board game Codenames as a suitable benchmark for evaluating the reasoning capabilities of Large Language Models (LLMs). Codenames presents a highly interesting challenge for achieving successful AI performance, requiring both a sophisticated understanding of language, theory of mind, and epistemic reasoning capabilities. Prior attempts to develop agents for Codenames have largely relied on word embedding techniques, which have a limited vocabulary range and perform poorly when paired with differing approaches. LLMs have demonstrated enhanced reasoning and comprehension capabilities for language-based tasks, but can still suffer in lateral thinking challenges. We evaluate the capabilities of several state-of-the-art LLMs, including GPT-4o, Gemini 1.5, Claude 3.5 Sonnet, and Llama 3.1, across a variety of board setups. Our results indicate that while certain LLMs perform better than others overall, different models exhibit varying emergent behaviours during gameplay and excel at specific roles. We also evaluate the performance of different combinations of LLMs when playing cooperatively together, demonstrating that LLM agents are more generalisable to a wider range of teammates than prior techniques.

Figures

Figures reproduced from arXiv: 2412.11373 by the authors.

Figure 1
Figure 1. Codenames example board setup (seed = 0). Words associated with [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Average clue number provided by each codemaster model as the turn [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

47 extracted references · 30 canonical work pages

  1. [1]

    A survey of large language models,

    W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y . Hou, Y . Min, B. Zhang, J. Zhang, Z. Dong, Y . Du, C. Yang, Y . Chen, Z. Chen, J. Jiang, R. Ren, Y . Li, X. Tang, Z. Liu, P. Liu, J.-Y . Nie, and J.-R. Wen, “A survey of large language models,” 2024. [Online]. Available: https://arxiv.org/abs/2303.18223

    arXiv 2024

  2. [2]

    Gpt for games: A scoping review (2020-2023),

    D. Yang, E. Kleinman, and C. Harteveld, “Gpt for games: A scoping review (2020-2023),” in 2024 IEEE Conference on Games (CoG) . IEEE, Aug. 2024, p. 1–8. [Online]. Available: http://dx.doi.org/10.1109/CoG60054.2024.10645548

    arXiv 2020

  3. [3]

    Level generation through large language models,

    G. Todd, S. Earle, M. U. Nasir, M. C. Green, and J. Togelius, “Level generation through large language models,” in Proceedings of the 18th International Conference on the Foundations of Digital Games . Association for Computing Machinery, 2023

    work page 2023

  4. [4]

    Langbirds: An agent for angry birds using a large language model,

    S. Oh, I. Chung, and K.-J. Kim, “Langbirds: An agent for angry birds using a large language model,” in 2024 IEEE Conference on Games (CoG), 2024, pp. 1–8

    work page 2024

  5. [5]

    Playing nethack with llms: Potential & limitations as zero-shot agents,

    D. Jeurissen, D. Perez-Liebana, J. Gow, D. Cakmak, and J. Kwan, “Playing nethack with llms: Potential & limitations as zero-shot agents,” 2024. [Online]. Available: https://arxiv.org/abs/2403.00690

    arXiv 2024

  6. [6]

    The Go Transformer: Natural Language Modeling for Game Play

    M. Ciolino, D. Noever, and J. Kalin, “The go transformer: Natural language modeling for game play,” 2020. [Online]. Available: https://arxiv.org/abs/2007.03500

    work page Pith review arXiv 2020

  7. [7]

    Generative ai in mafia-like game simulation,

    M. Kim and S. Kim, “Generative ai in mafia-like game simulation,”

    work page

  8. [8]

    Language-driven play: Large language models as game-playing agents in slay the spire,

    B. Bateni and J. Whitehead, “Language-driven play: Large language models as game-playing agents in slay the spire,” in Proceedings of the 19th International Conference on the Foundations of Digital Games, ser. FDG ’24. New York, NY , USA: Association for Computing Machinery,

    work page

Show all 47 references
  1. [9]

    Gamebench: Evaluating strategic reasoning abilities of llm agents,

    A. Costarelli, M. Allen, R. Hauksson, G. Sodunke, S. Hariharan, C. Cheng, W. Li, J. Clymer, and A. Yadav, “Gamebench: Evaluating strategic reasoning abilities of llm agents,” 2024. [Online]. Available: https://arxiv.org/abs/2406.06613

    2024 arXiv

  2. [10]

    A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play,

    D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L. Sifre, D. Kumaran, T. Graepel, T. Lillicrap, K. Simonyan, and D. Hassabis, “A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play,” Science, vol. 362...

    2018

  3. [11]

    Mastering the game of Go without human knowledge,

    D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, Y . Chen, T. Lillicrap, F. Hui, L. Sifre, G. van den Driessche, T. Graepel, and D. Hassabis, “Mastering the game of Go without human knowledge,” Nature, vol. 550...

    2017

  4. [12]

    TAG: Pandemic Competition,

    R. D. Gaina and M. Balla, “TAG: Pandemic Competition,” in 2022 IEEE Conference on Games (CoG) , 2022, pp. 552–559

    2022

  5. [13]

    Chv ´atil, Codenames

    V . Chv ´atil, Codenames. Czech Games Edition, 2015

    2015

  6. [14]

    The codenames ai competition,

    A. Summerville, A. Kim, M. Ruzmaykin, and A. Truong, “The codenames ai competition,” https://sites.google.com/view/ the-codenames-ai-competition, accessed: 2024-11-25. 12

    2024

  7. [15]

    Cooperation and Codenames: Understanding Natural Language Processing via Code- names,

    A. Kim, M. Ruzmaykin, A. Truong, and A. Summerville, “Cooperation and Codenames: Understanding Natural Language Processing via Code- names,” Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, vol. 15, no. 1, pp. 160–166, 2019

    2019

  8. [16]

    Theory of mind may have spontaneously emerged in large language models,

    M. Kosinski, “Theory of mind may have spontaneously emerged in large language models,” arXiv preprint arXiv:2302.02083 , 2023

    2023 arXiv

  9. [17]

    Measuring massive multitask language understanding,

    D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt, “Measuring massive multitask language understanding,” arXiv preprint arXiv:2009.03300 , 2020

    2009 arXiv

  10. [18]

    Hel- laswag: Can a machine really finish your sentence?

    R. Zellers, A. Holtzman, Y . Bisk, A. Farhadi, and Y . Choi, “Hel- laswag: Can a machine really finish your sentence?” arXiv preprint arXiv:1905.07830, 2019

    1905 arXiv

  11. [19]

    Challenging big-bench tasks and whether chain-of-thought can solve them,

    M. Suzgun, N. Scales, N. Sch ¨arli, S. Gehrmann, Y . Tay, H. W. Chung, A. Chowdhery, Q. V . Le, E. H. Chi, D. Zhou et al. , “Challenging big-bench tasks and whether chain-of-thought can solve them,” arXiv preprint arXiv:2210.09261, 2022

    arXiv

  12. [20]

    Towards reasoning in large language models: A survey,

    J. Huang and K. C.-C. Chang, “Towards reasoning in large language models: A survey,” arXiv preprint arXiv:2212.10403 , 2022

    2022 arXiv

  13. [21]

    Llm as a mastermind: A survey of strategic reasoning with large language models,

    Y . Zhang, S. Mao, T. Ge, X. Wang, A. de Wynter, Y . Xia, W. Wu, T. Song, M. Lan, and F. Wei, “Llm as a mastermind: A survey of strategic reasoning with large language models,” arXiv preprint arXiv:2404.01230, 2024

    2024 arXiv

  14. [22]

    Tydi qa: A benchmark for information-seeking question answering in typologically diverse languages,

    J. H. Clark, E. Choi, M. Collins, D. Garrette, T. Kwiatkowski, V . Niko- laev, and J. Palomaki, “Tydi qa: A benchmark for information-seeking question answering in typologically diverse languages,” Transactions of the Association for Computational Linguistics , vol. 8, pp. 454...

    2020

  15. [23]

    Measuring mathematical problem solving with the math dataset,

    D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt, “Measuring mathematical problem solving with the math dataset,” arXiv preprint arXiv:2103.03874 , 2021

    2021 arXiv

  16. [24]

    Neural theory-of-mind? on the limits of social intelligence in large lms,

    M. Sap, R. LeBras, D. Fried, and Y . Choi, “Neural theory-of-mind? on the limits of social intelligence in large lms,” arXiv preprint arXiv:2210.13312, 2022

    2022 arXiv

  17. [25]

    Large Language Models Fail on Trivial Alterations to Theory-of-Mind Tasks,

    T. Ullman, “Large Language Models Fail on Trivial Alterations to Theory-of-Mind Tasks,” Mar. 2023

    2023

  18. [26]

    Prompt Engineering ChatGPT for Co- denames,

    M. Sidji and M. Stephenson, “Prompt Engineering ChatGPT for Co- denames,” in 2024 IEEE Conference on Games (CoG) , Aug. 2024, pp. 1–4

    2024

  19. [27]

    Strategic reasoning with language models,

    K. Gandhi, D. Sadigh, and N. D. Goodman, “Strategic reasoning with language models,” ArXiv, vol. abs/2305.19165, 2023

    2023 arXiv

  20. [28]

    Human-AI Collaboration in Cooperative Games: A Study of Playing Codenames with an LLM Assistant,

    M. Sidji, W. Smith, and M. J. Rogerson, “Human-AI Collaboration in Cooperative Games: A Study of Playing Codenames with an LLM Assistant,” Proceedings of the ACM on Human-Computer Interaction , vol. 8, no. CHI PLAY , pp. 1–25, Oct. 2024

    2024

  21. [29]

    LLMs achieve adult human performance on higher-order theory of mind tasks,

    W. Street, J. O. Siy, G. Keeling, A. Baranes, B. Barnett, M. McKibben, T. Kanyere, A. Lentz, B. A. y Arcas, and R. I. M. Dunbar, “LLMs achieve adult human performance on higher-order theory of mind tasks,” May 2024

    2024

  22. [30]

    Word autobots: Using transformers for word association in the game codenames,

    C. Jaramillo, M. Charity, R. Canaan, and J. Togelius, “Word autobots: Using transformers for word association in the game codenames,” Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, vol. 16, no. 1, pp. 231–237, Oct. 2020

    2020

  23. [31]

    Playing Codenames with Language Graphs and Word Embeddings,

    D. Koyyalagunta, A. Sun, R. L. Draelos, and C. Rudin, “Playing Codenames with Language Graphs and Word Embeddings,” Journal of Artificial Intelligence Research , vol. 71, pp. 319–346, 2021

    2021

  24. [32]

    Adapting to teammates in a cooperative language game,

    C. Archibald and S. Brosnahan, “Adapting to teammates in a cooperative language game,” arXiv, 2024

    2024

  25. [33]

    Noisy communication modeling for improved cooperation in codenames,

    C. Archibald and D. Blaylock, “Noisy communication modeling for improved cooperation in codenames,” in 2024 IEEE Conference on Games (CoG), 2024, pp. 1–8

    2024

  26. [34]

    ThinkSum: Probabilis- tic reasoning over sets using large language models,

    B. Ozturkler, N. Malkin, Z. Wang, and N. Jojic, “ThinkSum: Probabilis- tic reasoning over sets using large language models,” in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . Association for Computational Lingu...

    2023

  27. [35]

    Large Language Models are Zero-Shot Reasoners,

    T. Kojima, S. S. Gu, M. Reid, Y . Matsuo, and Y . Iwasawa, “Large Language Models are Zero-Shot Reasoners,” Advances in Neural Infor- mation Processing Systems , vol. 35, pp. 22 199–22 213, 2022-12-06

    2022

  28. [36]

    Self-Refine: Iterative Refinement with Self-Feedback,

    A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y . Yang, S. Gupta, B. P. Majumder, K. Hermann, S. Welleck, A. Yazdanbakhsh, and P. Clark, “Self-Refine: Iterative Refinement with Self-Feedback,” Advances in Neural Informa- t...

    2023

  29. [37]

    Unleashing the Emergent Cognitive Synergy in Large Language Models: A Task- Solving Agent through Multi-Persona Self-Collaboration,

    Z. Wang, S. Mao, W. Wu, T. Ge, F. Wei, and H. Ji, “Unleashing the Emergent Cognitive Synergy in Large Language Models: A Task- Solving Agent through Multi-Persona Self-Collaboration,” arXiv, 2024

    2024

  30. [38]

    Efficient estimation of word representations in vector space,

    T. Mikolov, K. Chen, G. Corrado, and J. Dean, “Efficient estimation of word representations in vector space,” 2013. [Online]. Available: https://arxiv.org/abs/1301.3781

    2013 arXiv

  31. [39]

    GloVe: Global vectors for word representation,

    J. Pennington, R. Socher, and C. Manning, “GloVe: Global vectors for word representation,” in Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) , A. Moschitti, B. Pang, and W. Daelemans, Eds. Doha, Qatar: Association for Computation...

    2014

  32. [40]

    Concatenated power mean word embeddings as universal cross-lingual sentence representations,

    A. R ¨uckl´e, S. Eger, M. Peyrard, and I. Gurevych, “Concatenated power mean word embeddings as universal cross-lingual sentence representations,” 2018. [Online]. Available: https://arxiv.org/abs/1803. 01400

    2018

  33. [41]

    Human-ai collaboration in cooperative games: A study of playing codenames with an llm assistant,

    M. Sidji, W. Smith, and M. J. Rogerson, “Human-ai collaboration in cooperative games: A study of playing codenames with an llm assistant,” Proc. ACM Hum.-Comput. Interact. , vol. 8, no. CHI PLAY , Oct. 2024. [Online]. Available: https://doi.org/10.1145/3677081

    2024 doi

  34. [42]

    Semantic Priming Effects In Visual Word Recognition: A Selective Review Of Current Findings And Theories,

    J. H. Neely, “Semantic Priming Effects In Visual Word Recognition: A Selective Review Of Current Findings And Theories,” inBasic Processes in Reading. Routledge, 1990

    1990

  35. [43]

    Prototypes Revisited,

    R. E. MacLaury, “Prototypes Revisited,” Annual Review of Anthropol- ogy, vol. 20, pp. 55–74, 1991

    1991

  36. [44]

    Context-independent and context-dependent informa- tion in concepts,

    L. W. Barsalou, “Context-independent and context-dependent informa- tion in concepts,” Memory & Cognition , vol. 10, no. 1, pp. 82–93, Jan. 1982

    1982

  37. [45]

    Human Learning from Artificial Intel- ligence: Evidence from Human Go Players’ Decisions after AlphaGo,

    M. Shin, J. Kim, and M. Kim, “Human Learning from Artificial Intel- ligence: Evidence from Human Go Players’ Decisions after AlphaGo,” Proceedings of the Annual Meeting of the Cognitive Science Society , vol. 43, no. 43, 2021. Matthew Stephenson Dr. Matthew Stephenson is a Lec...

    2021

  38. [2023]

    Available: https://arxiv.org/abs/2309.11672

    [Online]. Available: https://arxiv.org/abs/2309.11672

    arXiv

  39. [2024]

    Available: https://doi.org/10.1145/3649921.3650013

    [Online]. Available: https://doi.org/10.1145/3649921.3650013

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.