Pith. sign in

REVIEW 6 major objections 5 minor 2 cited by

SocialMaze: A Benchmark for Evaluating Social Reasoning in Large Language Models

T0 review · 6 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SocialMaze argues that LLM social reasoning must be tested by deep reasoning, dynamic interaction, and information uncertainty, and its six-task benchmark shows current models degrade most sharply under uncertainty.

desk verdict SocialMaze is a substantial, openly released benchmark whose headline 'social reasoning' claim is only partially supported; two tasks don't require mental-state inference, and the paper's own appendix concedes the dimensions are validated only a posteriori. read the letter →

arxiv 2505.23713 v1 pith:NARGTYWG submitted 2025-05-29 cs.CL

Zixiang Xu , Yanbo Wang , Yue Huang , Jiayi Ye , Haomin Zhuang , Zirui Song , Lang Gao , Chenxi Wang
show 8 more authors
Zhaorun Chen Yujun Zhou Sixian Li Wang Pan Yue Zhao Jieyu Zhao Xiangliang Zhang Xiuying Chen
This is my paper
classification cs.CL
keywords socialreasoninglargelanguagemodelsbenchmarktheoryofmindinformationuncertaintydynamicinteractionchain-of-thoughtfine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SocialMaze is a benchmark for evaluating how well large language models reason about social situations. The paper argues that social reasoning has three hard ingredients that existing benchmarks undersell: deep reasoning about other people's mental states, dynamic interaction that unfolds over multiple rounds, and information uncertainty from deception, noise, or bias. To capture these, SocialMaze offers six tasks drawn from social deduction games, daily life, and online platforms, all built on a layered social interaction graph. Its experiments show that long chain-of-thought models lead on the deep-reasoning tasks, that accuracy generally rises with more interaction rounds but unevenly, and that reasoning quality drops sharply when unreliable actors are introduced. The paper also shows that targeted fine-tuning on curated reasoning traces raises hidden-role accuracy substantially, while agentic workflows give only marginal gains.

What carries the argument

The benchmark's central machinery is a time-indexed graph model called a Layered Social Interaction Graph, in which the same set of social members forms the vertices of a graph, and each interaction round $t$ contributes its own edge set $\mathcal{E}_t$ over that shared vertex set. Every SocialMaze task is expressed in natural language derived from such a graph, and each query is typed as vertex-centric, edge-centric, or graph-level, so a task's demand on the model can be traced to which part of the layered structure the answer requires. This formalization lets the paper vary the three target dimensions—reasoning depth, interaction dynamics, and information uncertainty—in a controlled way across the six tasks.

What would settle it

Re-run Social Graph Analysis with arbitrary non-social labels (e.g., 'type A'/'type B') in place of 'good/bad relationship'; if accuracy is unchanged, the task tests graph reasoning rather than social reasoning.

Watch

Extended reading notes

Core claim

The paper's central claim is that social reasoning in LLMs should be assessed along three dimensions—deep reasoning, dynamic interaction, and information uncertainty—and that a benchmark that jointly varies these dimensions exposes weaknesses that static social QA benchmarks miss. Concretely, SocialMaze reports that long chain-of-thought models such as DeepSeek-R1 and Gemini-2.5-Pro excel on Hidden Role Deduction and Social Graph Analysis, where the gap to short-CoT models is large; that performance in multi-round games improves with additional interaction but with very different slopes across models; and that introducing deliberately unreliable roles (Rumormongers, Lunatics) degrades accuracy, especially for a model's ability to infer its own identity. The paper concludes that fine-tuning on curated reasoning traces is a more effective route to improved social reasoning than current agentic workflows.

Load-bearing premise

The benchmark's central claim rests on the assumption that its six tasks actually exercise social reasoning—that transitive closure over friendship labels and rating prediction from review text are forms of social inference rather than domain-general logic or statistical pattern matching.

Editorial extensions

If this is right

  • Socially grounded LLM applications such as moderation, platform analysis, and games should treat information uncertainty as a distinct failure mode and expect performance to drop when sources are unreliable.
  • Long chain-of-thought models are not uniformly better at social reasoning; their advantage is specific to tasks that require deep deductive inference, so model selection should depend on the task profile.
  • Fine-tuning on curated reasoning traces can substantially improve hidden-role deduction, suggesting that domain-specific process supervision may be more effective than generic agentic scaffolding.
  • The layered-graph formulation provides a template for generating new social reasoning tasks with controlled difficulty, which could be extended to other multi-round social settings.
  • Accuracy in multi-round tasks improves with added interaction for many models but the rate of improvement varies, so dynamic interaction should be reported as a learning curve over rounds rather than a single score.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The three dimensions are not necessarily independent: the information-uncertainty manipulations in Hidden Role Deduction also increase reasoning depth, so some reported differences between conditions may partly reflect task difficulty rather than the labeled dimension.
  • Because Social Graph Analysis and Rating Estimation do not require reading mental states, the benchmark's 'social reasoning' label may overstate what is measured; a sharper test would compare these tasks against non-social control versions with matched structure.
  • A natural extension is to generate tasks that explicitly require updating beliefs about a specific agent's mental states across rounds and to verify that the benchmark's three dimensions capture variance beyond general reasoning benchmarks.
  • The fine-tuning result suggests that using SocialMaze's solvability-checked reasoning traces as training data could transfer to other multi-round social dialogue tasks, but this remains to be tested.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

6 major / 5 minor

Summary. SocialMaze introduces a benchmark for evaluating social reasoning in large language models, organized around three claimed challenges: deep reasoning, dynamic interaction, and information uncertainty. The benchmark contains six tasks across social deduction games, daily-life interactions, and digital community platforms, built on a 'layered social interaction graph' formalism. The paper reports evaluation of 11 LLMs, several reasoning/workflow agents, and fine-tuning experiments on Hidden Role Deduction, with findings that long-CoT models lead on deep-reasoning tasks, that model performance degrades under information uncertainty, and that targeted SFT/DPO fine-tuning improves hidden-role performance. The dataset is publicly released, and generation pipelines, solvability checks, and human validation procedures are described in detail.

Significance. If the construct-validity concerns are resolved, SocialMaze would be a useful resource: it is publicly available, spans diverse task formats, includes algorithmic solvability verification for its rule-based tasks, and provides substantial evaluation coverage across proprietary and open-weight models. The fine-tuning results and the analysis of reasoning under uncertainty are potentially informative for the community. The main value depends, however, on whether the six tasks actually measure social reasoning as defined in the paper; at present, at least two tasks appear solvable without any mental-state or truthfulness inference, and the paper's own appendix concedes that the three core dimensions are validated only a posteriori. Strengths include the detailed data-generation documentation, the public release, and the inclusion of both automated and human quality checks.

major comments (6)
  1. [§3.4, Appendix E] Social Graph Analysis does not appear to require social reasoning as defined in the Abstract and §1 (interpreting social contexts, inferring others' mental states, assessing truthfulness). The task is fully specified as a graph problem: pairwise good/bad labels, transitivity axioms, and queries for pairwise relations, neighbors, group counts, and relationship counts. A solver can answer all four query types by transitive closure and counting over an abstract graph, with no need to model intentions, beliefs, or deception. Since Figure 4 uses Social Graph Analysis as a flagship 'high deep reasoning' task, this construct-validity gap directly affects the benchmark's central claim. Please provide an explicit construct-validity test (e.g., compare performance on the social narrative version against an abstract graph version with identical structure) or reclassify the task and soften the corresponding claims.
  2. [§3.3, Appendix D] Rating Estimation from Text is presented as a social reasoning task, but the query is a 1–5 rating regression from review text. The 'shill' manipulation changes the sentiment distribution, and the task can plausibly be solved by robust sentiment aggregation without modeling the reviewer's beliefs or intentions. The design rationale claims the model must evaluate source credibility, but the evaluation metric and query do not require the model to identify which reviews are deceptive. Please add evidence that successful performance depends on social inference (for example, an ablation removing personas or an analysis showing models identify shills), or reframe this task as an information-aggregation task within the benchmark.
  3. [Appendix I.2, §4.1] The paper concedes that the three core dimensions are 'annotate[d] qualitatively and validate[d] ... a posteriori.' This makes the §4.1 finding that Long CoT models excel on tasks flagged as high in Deep Reasoning potentially circular if the flags were assigned after inspecting model performance. The empirical pattern is presented as validation of the dimensions, but it does not provide independent evidence. Please state how the dimension labels were fixed before or independently of the reported results, or provide a quantitative, task-agnostic measure of reasoning depth, interaction dynamics, and uncertainty that is not derived from the same benchmark's outcomes.
  4. [§3.2, §3.3, Appendix C, Appendix D, Appendix H.3] Human validation criteria are internally inconsistent. The main text (§3.2 and §3.3) reports 15 graduate-student evaluators with a >70% majority threshold, while Appendix C says an instance is valid if 'more than 5 out of the 10 evaluators' agree, and Appendix D says solvability required agreement by 'more than half' of the evaluators. Appendix H.3 additionally reports human performance averaged from 10 students. Please specify the exact evaluator count and threshold for each task, and clarify whether the 10-evaluator or 15-evaluator protocol is the correct one; this affects the data-quality guarantees claimed for Find the Spy and Rating Estimation.
  5. [Appendix B.1, Appendix C.1, §1] The 'dynamic interaction' dimension is weakened by the fact that the model is a passive observer in both Hidden Role Deduction and Find the Spy; Appendix B.1 explicitly states the LLM 'does not actively participate in the game simulation,' and Appendix C.1 similarly states it 'does not generate descriptions itself.' The §1 definition of dynamic interaction requires that models 'dynamically adapt their reasoning and actions,' but the current tasks only require integrating sequentially revealed information, not acting or interacting. Please either rename the dimension to something like 'temporal information integration' or add an active-interaction control condition that demonstrates the claimed interaction capability.
  6. [H.2, Table 8, §4.4] The headline accuracy in Table 8 for Hidden Role Deduction is computed under a non-uniform role distribution (Investigator:Criminal:Rumormonger:Lunatic = 3:2:60:35), while the experiments in §4.1–§4.3 and Figure 2 use equal role proportions (1:1:1:1). Since Figure 5 shows that performance is dramatically lower for Rumormonger and Lunatic perspectives, the summary rankings in Table 8 are not directly comparable with the per-round and per-task results elsewhere. Please report all headline results under one configuration or provide a clear rationale for the distribution switch, and state whether the human performance row in Table 8 was obtained under the same distribution.
minor comments (5)
  1. [Figure 1] The caption contains a typo: 'Hidden Role Deducution' should be 'Hidden Role Deduction.'
  2. [Appendix B] The appendix header refers to 'subsection 3.2' for Hidden Role Deduction, but this task is described in §3.1; the cross-reference should be corrected.
  3. [Appendix A] The references '[83?]' and '[84?]' contain stray question marks and should be cleaned up before publication.
  4. [Appendix C] The quality-control paragraph says instances are reviewed by 15 evaluators but then refers to 'more than 5 out of the 10 evaluators'; this appears to be a copy/paste inconsistency that should be fixed.
  5. [§4.3] The sentence contrasting Rumormonger and Lunatic refers to an Investigator checking the Lunatic as 'not the Criminal,' but the explanation of why this 'makes awakening easier' is terse and would benefit from a concrete example, as the connection to the reported accuracy gap is not immediately transparent.

Circularity Check

2 steps flagged · score 3.0 of 10

Dimension validation is self-referential and Social Graph Analysis imports the 'social' label by construction; the fine-tuning and self-citation parts are not load-bearing.

  1. other [Appendix I.2 (Limitations), used with Section 4.1, Figure 4, and Table 1]
    "We therefore annotate each task qualitatively and validate the relevance of the three dimensions a posteriori. Empirically, models with long chain-of-thoughts excel only on the tasks flagged as “high” in Deep Reasoning, performance curves shift between interaction rounds, and accuracy drops sharply once unreliable narrators are introduced."

    The empirical validation of the three organizing dimensions uses the very same task-level annotations that define those dimensions in Table 1. Section 4.1 groups tasks by the Table 1 High/Low Deep Reasoning flags and then treats Long CoT models' higher accuracy on those flagged tasks as evidence that the Deep Reasoning dimension is meaningful. Because the independent variable (the flag) and the confirmation (performance on the flagged tasks) come from the same annotation process, the validation is partly self-referential rather than an independent test of the construct. The paper itself concedes the dimensions are only annotated qualitatively, so the a posteriori evidence does not break the circle.

  2. renaming known result [Section 3.4 (Task 4), Appendix E, and Table 1]
    "Given a description of the social network graph and pairwise relationship labels indicating whether two individuals are friends or have a bad relationship—with friendship being transitive—the model is required to perform reasoning such as: determining whether two individuals are friends, identifying the friend group of a given node, calculating the total number of distinct friend groups, and counting all relationships within the network."

    The paper defines social reasoning as interpreting social contexts, inferring others' mental states, and assessing truthfulness, but this task is solvable by transitive-closure and counting algorithms over an abstract graph. No belief, intention, deception, or truthfulness assessment is required. The task is nevertheless flagged as High in Deep Reasoning in Table 1 and is used in Section 4.1 to attribute Long CoT gains to deep social reasoning. The 'social reasoning' content of this task is therefore the label itself; any conclusion about social reasoning drawn from this task inherits the label by construction rather than from the task's actual demands.

full rationale

SocialMaze is a benchmark-construction paper rather than a formal derivation chain, so the classical circularity patterns—fitted parameters renamed as predictions, imported uniqueness theorems, ansatz-by-citation—do not apply. The fine-tuning experiment trains and evaluates on held-out splits of the same Hidden Role Deduction generator; that is standard within-distribution evaluation, not circular, although the paper's broader generalization claim is not directly demonstrated. Self-citations such as the DyFlow baseline are not load-bearing for the central benchmark claim. The genuine circularity is limited to the benchmark's self-validation: the three core dimensions are annotated qualitatively on the tasks and then validated a posteriori using performance on those same annotated tasks, so the confirmation is partly by construction. Additionally, Social Graph Analysis is marked as High Deep Reasoning despite being reducible to graph transitive closure and counting, meaning the social-reasoning component of that task is imported by labeling. These issues make parts of the central claims self-referential, but they do not reduce the entire benchmark to its inputs; the held-out fine-tuning gains and several other tasks retain independent empirical content.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The benchmark's central claims rest on several hand-set parameters (role distributions, rating distribution, acceptance ratio, solvability thresholds) plus task-specific behavioral axioms. The role-behavior rules and graph axioms are internal modeling choices, not externally validated. The most consequential free parameter is the skewed Hidden Role Deduction role distribution in the headline table (60% Rumormonger), which silently changes what accuracy means and is only disclosed in an appendix. These are not fatal to the benchmark, but they must be accounted for when interpreting the published numbers.

free parameters (4)
  • Hidden Role Deduction role distribution (summary table) = Investigator 3%, Criminal 2%, Rumormonger 60%, Lunatic 35%
    Chosen in Appendix H.3 to minimize random guessing; it gives a 60% self-role prior and affects the headline 'Both Correct' numbers.
  • Review Decision Prediction acceptance ratio = 67% rejected / 33% accepted
    Adjusted from equal sampling in Appendix H.2; no majority baseline is reported, and 67% rejection defines the trivial guess.
  • Rating Estimation label distribution = 1:3:10:73:13 for 1-5 stars
    Final dataset distribution in Appendix H.2; a constant 4-star prediction achieves about 73% accuracy.
  • Human solvability threshold and panel size = >70% threshold with 15 evaluators (main text); >50% with unclear panel (appendices)
    The inconsistency changes which instances are retained, affecting all tasks that use human filtering.
assumptions (5)
  • ad hoc to paper Good relationships are transitive; if A-B is bad and A-C is good, then B-C is bad.
    Defines the Social Graph Analysis task in Appendix E. The rules are chosen to guarantee a consistent graph but are not externally grounded.
  • domain assumption Investigators always tell the truth, Rumormongers make 50%-random statements, Lunatics believe they are the Criminal, and Criminals can lie.
    Hidden Role Deduction rules from Section 3.1 and Appendix B. The solvability verification in Algorithm 1 relies on these role behaviors.
  • domain assumption LLM-generated personas and reviews contain the intended demographic and product-quality cues.
    User Profile Inference (Appendix G) and Rating Estimation (Appendix D) use machine-generated text as evidence; if the text does not encode target attributes in a way humans would infer, the labels are not meaningful.
  • domain assumption The OpenReview acceptance decision is inferable from title, abstract, keywords, review text, and rebuttal without numerical scores.
    Review Decision Prediction (Appendix F) removes scores and uses the real decision as ground truth; a human solvability check is reported only for a subset of instances.
  • domain assumption Average platform star rating is a valid ground-truth rating for Rating Estimation.
    Appendix D uses the rounded average of scraped ratings as the target, which can distort products near half-star boundaries.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SocialMaze: A Benchmark for Evaluating Social Reasoning in Large Language Models." pith.science (2026). https://pith.science/paper/NARGTYWG

@misc{pith2026250523713,
  author       = {Pith},
  title        = {Pith review of: SocialMaze: A Benchmark for Evaluating Social Reasoning in Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NARGTYWG}},
  note         = {Machine review of arXiv:2505.23713}
}
read the original abstract

Large language models (LLMs) are increasingly applied to socially grounded tasks, such as online community moderation, media content analysis, and social reasoning games. Success in these contexts depends on a model's social reasoning ability - the capacity to interpret social contexts, infer others' mental states, and assess the truthfulness of presented information. However, there is currently no systematic evaluation framework that comprehensively assesses the social reasoning capabilities of LLMs. Existing efforts often oversimplify real-world scenarios and consist of tasks that are too basic to challenge advanced models. To address this gap, we introduce SocialMaze, a new benchmark specifically designed to evaluate social reasoning. SocialMaze systematically incorporates three core challenges: deep reasoning, dynamic interaction, and information uncertainty. It provides six diverse tasks across three key settings: social reasoning games, daily-life interactions, and digital community platforms. Both automated and human validation are used to ensure data quality. Our evaluation reveals several key insights: models vary substantially in their ability to handle dynamic interactions and integrate temporally evolving information; models with strong chain-of-thought reasoning perform better on tasks requiring deeper inference beyond surface-level cues; and model reasoning degrades significantly under uncertainty. Furthermore, we show that targeted fine-tuning on curated reasoning examples can greatly improve model performance in complex social scenarios. The dataset is publicly available at: https://huggingface.co/datasets/MBZUAI/SocialMaze

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Inside the Unfair Judge: A Mechanistic Interpretability Account of LLM-as-Judge Bias

    cs.LG 2026-07 conditional novelty 6.5 of 10

    LLM-as-judge scoring biases concentrate in low-dimensional, type-specific activation subspaces that support bidirectional causal steering and cross-domain failure prediction.

  2. No One Wins in Nuclear War: A Social Simulation of Military Decision-making

    cs.CY 2026-08 conditional novelty 6.0 of 10

    WOPR is a deterministic, replay-checkable rules engine that turns the card game Nuclear War into a social-simulation testbed for military decision-making.

Reference graph

Works this paper leans on

164 extracted references · 44 canonical work pages · cited by 2 Pith papers

  1. [1]

    What can large language models do in chemistry? a comprehensive benchmark on eight tasks.Advances in Neural Information Processing Systems, 36:59662–59688, 2023

    TaichengGuo,BozhaoNan,ZhenwenLiang,ZhichunGuo,NiteshChawla,OlafWiest,Xiangliang Zhang, et al. What can large language models do in chemistry? a comprehensive benchmark on eight tasks.Advances in Neural Information Processing Systems, 36:59662–59688, 2023

    2023

  2. [2]

    Unveiling the power of language models in chemical research question answering.Communications Chemistry, 8(1):4, 2025

    Xiuying Chen, Tairan Wang, Taicheng Guo, Kehan Guo, Juexiao Zhou, Haoyang Li, Zirui Song, Xin Gao, and Xiangliang Zhang. Unveiling the power of language models in chemical research question answering.Communications Chemistry, 8(1):4, 2025

    2025

  3. [3]

    Medtrinity-25m: A large-scale multimodal dataset with multigranular annotations for medicine, 2025

    Yunfei Xie, Ce Zhou, Lang Gao, Juncheng Wu, Xianhang Li, Hong-Yu Zhou, Sheng Liu, Lei Xing, James Zou, Cihang Xie, and Yuyin Zhou. Medtrinity-25m: A large-scale multimodal dataset with multigranular annotations for medicine, 2025

    2025

  4. [4]

    Pre-trained multimodal large language model enhances dermatological diagnosis using skingpt-4.Nature Communications, 15(1):5649, 2024

    Juexiao Zhou, Xiaonan He, Liyuan Sun, Jiannan Xu, Xiuying Chen, Yuetan Chu, Longxi Zhou, Xingyu Liao, Bin Zhang, Shawn Afvari, et al. Pre-trained multimodal large language model enhances dermatological diagnosis using skingpt-4.Nature Communications, 15(1):5649, 2024

    2024

  5. [5]

    Evalu- ating and mitigating bias in ai-based medical text generation.Nature Computational Science, pages 1–9, 2025

    Xiuying Chen, Tairan Wang, Juexiao Zhou, Zirui Song, Xin Gao, and Xiangliang Zhang. Evalu- ating and mitigating bias in ai-based medical text generation.Nature Computational Science, pages 1–9, 2025

    2025

  6. [6]

    Llm-mod: Can large language models assist content moderation? InExtended Abstracts of the CHI Conference on Human Factors in Computing Systems, pages 1–8, 2024

    Mahi Kolla, Siddharth Salunkhe, Eshwar Chandrasekharan, and Koustuv Saha. Llm-mod: Can large language models assist content moderation? InExtended Abstracts of the CHI Conference on Human Factors in Computing Systems, pages 1–8, 2024

    2024

  7. [7]

    Shieldgemma: Genera- tive ai content moderation based on gemma.arXiv preprint arXiv:2407.21772, 2024

    Wenjun Zeng, Yuchi Liu, Ryan Mullins, Ludovic Peran, Joe Fernandez, Hamza Harkous, Karthik Narasimhan, Drew Proud, Piyush Kumar, Bhaktipriya Radharapu, et al. Shieldgemma: Genera- tive ai content moderation based on gemma.arXiv preprint arXiv:2407.21772, 2024

    arXiv 2024

  8. [8]

    Scaling up llm reviews for google ads content moderation

    Wei Qiao, Tushar Dogra, Otilia Stretcu, Yu-Han Lyu, Tiantian Fang, Dongjin Kwon, Chun-Ta Lu, Enming Luo, Yuan Wang, Chih-Chun Chia, et al. Scaling up llm reviews for google ads content moderation. InProceedings of the 17th ACM International Conference on Web Search and Data Mining, pages 1174–1175, 2024. 11 SocialMaze: A Benchmark for Evaluating Social Re...

    2024

Show all 164 references
  1. [9]

    Hate personified: Investigating the role of llms in content moderation.arXiv preprint arXiv:2410.02657, 2024

    Sarah Masud, Sahajpreet Singh, Viktor Hangya, Alexander Fraser, and Tanmoy Chakraborty. Hate personified: Investigating the role of llms in content moderation.arXiv preprint arXiv:2410.02657, 2024

    2024 arXiv

  2. [10]

    Autonomous agents for collaborative task under information asymmetry

    Wei Liu, Chenxi Wang, YiFei Wang, Zihao Xie, Rennai Qiu, Yufan Dang, Zhuoyun Du, Weize Chen, Cheng Yang, and Chen Qian. Autonomous agents for collaborative task under information asymmetry. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  3. [11]

    Halc: object hallucination reduction via adaptive focal-contrast decoding

    Zhaorun Chen, Zhuokai Zhao, Hongyin Luo, Huaxiu Yao, Bo Li, and Jiawei Zhou. Halc: object hallucination reduction via adaptive focal-contrast decoding. InProceedings of the 41st International Conference on Machine Learning, pages 7824–7846, 2024

    2024

  4. [12]

    Llm-assisted content analysis: Using large language models to support deductive coding.arXiv preprint arXiv:2306.14924, 2023

    Robert Chew, John Bollenbacher, Michael Wenger, Jessica Speer, and Annice Kim. Llm-assisted content analysis: Using large language models to support deductive coding.arXiv preprint arXiv:2306.14924, 2023

    2023 arXiv

  5. [13]

    From a tiny slip to a giant leap: An llm-based simulation for fake news evolution.arXiv preprint arXiv:2410.19064, 2024

    Yuhan Liu, Zirui Song, Xiaoqing Zhang, Xiuying Chen, and Rui Yan. From a tiny slip to a giant leap: An llm-based simulation for fake news evolution.arXiv preprint arXiv:2410.19064, 2024

    2024 arXiv

  6. [14]

    Decoding echo chambers: LLM- powered simulations revealing polarization in social networks

    Chenxi Wang, Zongfang Liu, Dequan Yang, and Xiuying Chen. Decoding echo chambers: LLM- powered simulations revealing polarization in social networks. In Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eugenio, and Steven Schockaert, editors, Proceedin...

    2025

  7. [15]

    Safewatch: An efficient safety-policy following video guardrail model with transparent explanations

    Zhaorun Chen, Francesco Pinto, Minzhou Pan, and Bo Li. Safewatch: An efficient safety-policy following video guardrail model with transparent explanations. InThe Thirteenth International Conference on Learning Representations

  8. [16]

    Theory of mind for multi-agent collaboration via large language models

    Huao Li, Yu Quan Chong, Simon Stepputtis, Joseph Campbell, Dana Hughes, Michael Lewis, and Katia Sycara. Theory of mind for multi-agent collaboration via large language models. arXiv preprint arXiv:2310.10701, 2023

    2023 arXiv

  9. [17]

    Exploring large language models for word games: Who is the spy?arXiv preprint arXiv:2503.15235, 2025

    Chentian Wei, Jiewei Chen, and Jinzhu Xu. Exploring large language models for word games: Who is the spy?arXiv preprint arXiv:2503.15235, 2025

    2025 arXiv

  10. [18]

    Chatbot arena: An open platform for evaluating llms by human preference

    Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E Gonzalez, et al. Chatbot arena: An open platform for evaluating llms by human preference. InForty-first International Conference ...

    2024

  11. [19]

    On the trustworthiness of generative foundation models: Guideline, assessment, and perspective.arXiv preprint arXiv:2502.14296, 2025

    Yue Huang, Chujie Gao, Siyuan Wu, Haoran Wang, Xiangqi Wang, Yujun Zhou, Yanbo Wang, Jiayi Ye, Jiawen Shi, Qihui Zhang, et al. On the trustworthiness of generative foundation models: Guideline, assessment, and perspective.arXiv preprint arXiv:2502.14296, 2025

    2025 arXiv

  12. [20]

    Trusteval: A dynamic evaluation toolkit on trustworthiness of generative foundation models

    Yanbo Wang, Jiayi Ye, Siyuan Wu, Chujie Gao, Yue Huang, Xiuying Chen, Yue Zhao, and Xiangliang Zhang. Trusteval: A dynamic evaluation toolkit on trustworthiness of generative foundation models. InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the ...

    2025

  13. [21]

    Evaluating large language models in theory of mind tasks.Proceedings of the National Academy of Sciences, 121(45):e2405460121, 2024

    Michal Kosinski. Evaluating large language models in theory of mind tasks.Proceedings of the National Academy of Sciences, 121(45):e2405460121, 2024. 12 SocialMaze: A Benchmark for Evaluating Social Reasoning in Large Language Models

    2024

  14. [22]

    Mj-bench: Is your multimodal reward model really a good judge for text-to-image generation?arXiv preprint arXiv:2407.04842, 2024

    Zhaorun Chen, Yichao Du, Zichen Wen, Yiyang Zhou, Chenhang Cui, Zhenzhen Weng, Haoqin Tu, Chaoqi Wang, Zhengwei Tong, Qinglan Huang, et al. Mj-bench: Is your multimodal reward model really a good judge for text-to-image generation?arXiv preprint arXiv:2407.04842, 2024

    2024 arXiv

  15. [23]

    Cross-lingual pitfalls: Automatic probing cross-lingual weakness of multilingual large language models

    Zixiang Xu, Yanbo Wang, Yue Huang, Xiuying Chen, Jieyu Zhao, Meng Jiang, and Xiangliang Zhang. Cross-lingual pitfalls: Automatic probing cross-lingual weakness of multilingual large language models. InProceedings of the 63rd Annual Meeting of the Association for Computational ...

    2025

  16. [24]

    Injecting domain-specific knowledge into large language models: a comprehensive survey.arXiv preprint arXiv:2502.10708, 2025

    Zirui Song, Bin Yan, Yuhan Liu, Miao Fang, Mingzhe Li, Rui Yan, and Xiuying Chen. Injecting domain-specific knowledge into large language models: a comprehensive survey.arXiv preprint arXiv:2502.10708, 2025

    2025 arXiv

  17. [25]

    Honestllm: Toward an honest and helpful large language model.Advances in Neural Information Processing Systems, 37:7213–7255, 2024

    Gao Chujie, Siyuan Wu, Yue Huang, Dongping Chen, Qihui Zhang, Zhengyan Fu, Yao Wan, Lichao Sun, and Xiangliang Zhang. Honestllm: Toward an honest and helpful large language model.Advances in Neural Information Processing Systems, 37:7213–7255, 2024

    2024

  18. [26]

    Vuldetect- bench: Evaluating the deep capability of vulnerability detection with large language models, 2024

    Yu Liu, Lang Gao, Mingxin Yang, Yu Xie, Ping Chen, Xiaojin Zhang, and Wei Chen. Vuldetect- bench: Evaluating the deep capability of vulnerability detection with large language models, 2024

    2024

  19. [27]

    Datagen: Unified synthetic dataset generation via large language models

    Yue Huang, Siyuan Wu, Chujie Gao, Dongping Chen, Qihui Zhang, Yao Wan, Tianyi Zhou, Chaowei Xiao, Jianfeng Gao, Lichao Sun, et al. Datagen: Unified synthetic dataset generation via large language models. InThe Thirteenth International Conference on Learning Representations, 2024

    2024

  20. [28]

    Shaping the safety boundaries: Understanding and defending against jailbreaks in large language models, 2024

    Lang Gao, Xiangliang Zhang, Preslav Nakov, and Xiuying Chen. Shaping the safety boundaries: Understanding and defending against jailbreaks in large language models, 2024

    2024

  21. [29]

    Nqe: N-ary query embedding for complex query answering over hyper-relational knowledge graphs

    Haoran Luo, E Haihong, Yuhao Yang, Gengxian Zhou, Yikai Guo, Tianyu Yao, Zichen Tang, Xueyuan Lin, and Kaiyang Wan. Nqe: N-ary query embedding for complex query answering over hyper-relational knowledge graphs. InProceedings of the AAAI Conference on Artificial Intelligence, v...

    2023

  22. [30]

    Cer-eval: Certifiable and cost-efficient evaluation framework for llms.arXiv preprint arXiv:2505.03814, 2025

    Ganghua Wang, Zhaorun Chen, Bo Li, and Haifeng Xu. Cer-eval: Certifiable and cost-efficient evaluation framework for llms.arXiv preprint arXiv:2505.03814, 2025

    2025 arXiv

  23. [31]

    Gta: Graph theory agent and benchmark for algorithmic graph reasoning with llms, 2025

    Zixiang Xu, Yanbo Wang, Chenxi Wang, Lang Gao, Zirui Song, Yue Huang, Zhaorun Chen, Xiangliang Zhang, and Xiuying Chen. Gta: Graph theory agent and benchmark for algorithmic graph reasoning with llms, 2025

    2025

  24. [32]

    Socialiqa: Com- monsense reasoning about social interactions.arXiv preprint arXiv:1904.09728, 2019

    Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Com- monsense reasoning about social interactions.arXiv preprint arXiv:1904.09728, 2019

    1904 arXiv

  25. [33]

    Atomic: Anatlasofmachinecommonsense for if-then reasoning

    Maarten Sap, Ronan Le Bras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin,BrendanRoof,NoahASmith,andYejinChoi. Atomic: Anatlasofmachinecommonsense for if-then reasoning. InProceedings of the AAAI conference on artificial intelligence, volume 33, pages 302...

    2019

  26. [34]

    Creak: A dataset for commonsense reasoning over entity knowledge.arXiv preprint arXiv:2109.01653, 2021

    Yasumasa Onoe, Michael JQ Zhang, Eunsol Choi, and Greg Durrett. Creak: A dataset for commonsense reasoning over entity knowledge.arXiv preprint arXiv:2109.01653, 2021

    2021 arXiv

  27. [35]

    Goemotions: A dataset of fine-grained emotions.arXiv preprint arXiv:2005.00547, 2020

    Dorottya Demszky, Dana Movshovitz-Attias, Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi. Goemotions: A dataset of fine-grained emotions.arXiv preprint arXiv:2005.00547, 2020. 13 SocialMaze: A Benchmark for Evaluating Social Reasoning in Large Language Models

    2005 arXiv

  28. [36]

    CommonGen: A constrained text generation challenge for generative commonsense reasoning

    BillYuchenLin,WangchunshuZhou,MingShen,PeiZhou,ChandraBhagavatula,YejinChoi,and Xiang Ren. CommonGen: A constrained text generation challenge for generative commonsense reasoning. In Trevor Cohn, Yulan He, and Yang Liu, editors,Findings of the Association for Computational Lin...

    2020

  29. [37]

    Clever hans or neural theory of mind? stress testing social reasoning in large language models.arXiv preprint arXiv:2305.14763, 2023

    Natalie Shapira, Mosh Levy, Seyed Hossein Alavi, Xuhui Zhou, Yejin Choi, Yoav Goldberg, Maarten Sap, and Vered Shwartz. Clever hans or neural theory of mind? stress testing social reasoning in large language models.arXiv preprint arXiv:2305.14763, 2023

    2023 arXiv

  30. [38]

    Evaluating theory of mind in question answering.arXiv preprint arXiv:1808.09352, 2018

    Aida Nematzadeh, Kaylee Burns, Erin Grant, Alison Gopnik, and Thomas L Griffiths. Evaluating theory of mind in question answering.arXiv preprint arXiv:1808.09352, 2018

    2018 arXiv

  31. [39]

    Social chemistry 101: Learning to reason about social and moral norms.arXiv preprint arXiv:2011.00620, 2020

    Maxwell Forbes, Jena D Hwang, Vered Shwartz, Maarten Sap, and Yejin Choi. Social chemistry 101: Learning to reason about social and moral norms.arXiv preprint arXiv:2011.00620, 2020

    2011 arXiv

  32. [40]

    Aligning ai with shared human values.Proceedings of the International Conference on Learning Representations (ICLR), 2021

    Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. Aligning ai with shared human values.Proceedings of the International Conference on Learning Representations (ICLR), 2021

    2021

  33. [41]

    Debatebench: A challenging long context reasoning benchmark for large language models.arXiv preprint arXiv:2502.06279, 2025

    Utkarsh Tiwari, Aryan Seth, Adi Mukherjee, Kaavya Mer, Dhruv Kumar, et al. Debatebench: A challenging long context reasoning benchmark for large language models.arXiv preprint arXiv:2502.06279, 2025

    2025 arXiv

  34. [42]

    Understanding social reasoning in language models with language models.Advances in Neural Information Processing Systems, 36:13518–13529, 2023

    Kanishk Gandhi, Jan-Philipp Fränken, Tobias Gerstenberg, and Noah Goodman. Understanding social reasoning in language models with language models.Advances in Neural Information Processing Systems, 36:13518–13529, 2023

    2023

  35. [43]

    Mastering the game of no-press diplomacy via human- regularized reinforcement learning and planning.arXiv preprint arXiv:2210.05492, 2022

    Anton Bakhtin, David J Wu, Adam Lerer, Jonathan Gray, Athul Paul Jacob, Gabriele Farina, Alexander H Miller, and Noam Brown. Mastering the game of no-press diplomacy via human- regularized reinforcement learning and planning.arXiv preprint arXiv:2210.05492, 2022

    arXiv

  36. [44]

    Avalonbench: Evaluating llms playing the game of avalon.arXiv preprint arXiv:2310.05036, 2023

    Jonathan Light, Min Cai, Sheng Shen, and Ziniu Hu. Avalonbench: Evaluating llms playing the game of avalon.arXiv preprint arXiv:2310.05036, 2023

    2023 arXiv

  37. [45]

    Werewolf arena: A case study in llm evaluation via social deduction.arXiv preprint arXiv:2407.13943, 2024

    Suma Bailis, Jane Friedhoff, and Feiyang Chen. Werewolf arena: A case study in llm evaluation via social deduction.arXiv preprint arXiv:2407.13943, 2024

    2024 arXiv

  38. [46]

    Exploring large language models for communication games: An empirical study on werewolf

    Yuzhuang Xu, Shuo Wang, Peng Li, Fuwen Luo, Xiaolong Wang, Weidong Liu, and Yang Liu. Exploring large language models for communication games: An empirical study on werewolf. arXiv preprint arXiv:2309.04658, 2023

    2023 arXiv

  39. [47]

    Does the chimpanzee have a theory of mind?Behavioral and brain sciences, 1(4):515–526, 1978

    David Premack and Guy Woodruff. Does the chimpanzee have a theory of mind?Behavioral and brain sciences, 1(4):515–526, 1978

    1978

  40. [48]

    Oxford University Press, 2014

    Henry M Wellman.Making minds: How theory of mind develops. Oxford University Press, 2014

    2014

  41. [49]

    MIT press, 1999

    Ziva Kunda.Social cognition: Making sense of people. MIT press, 1999

    1999

  42. [50]

    Social cognition in humans.Current biology, 17(16):R724–R732, 2007

    Chris D Frith and Uta Frith. Social cognition in humans.Current biology, 17(16):R724–R732, 2007

    2007

  43. [51]

    Counterfactual thinking.Psychological bulletin, 121(1):133, 1997

    Neal J Roese. Counterfactual thinking.Psychological bulletin, 121(1):133, 1997. 14 SocialMaze: A Benchmark for Evaluating Social Reasoning in Large Language Models

    1997

  44. [52]

    Clomo: Counterfactual logical modification with large language models.arXiv preprint arXiv:2311.17438, 2023

    Yinya Huang, Ruixin Hong, Hongming Zhang, Wei Shao, Zhicheng Yang, Dong Yu, Changshui Zhang, Xiaodan Liang, and Linqi Song. Clomo: Counterfactual logical modification with large language models.arXiv preprint arXiv:2311.17438, 2023

    2023 arXiv

  45. [53]

    Llm as a mastermind: A survey of strategic reasoning with large language models.arXiv preprint arXiv:2404.01230, 2024

    Yadong Zhang, Shaoguang Mao, Tao Ge, Xun Wang, Adrian de Wynter, Yan Xia, Wenshan Wu, Ting Song, Man Lan, and Furu Wei. Llm as a mastermind: A survey of strategic reasoning with large language models.arXiv preprint arXiv:2404.01230, 2024

    2024 arXiv

  46. [54]

    What is agency?American journal of sociology, 103(4):962– 1023, 1998

    Mustafa Emirbayer and Ann Mische. What is agency?American journal of sociology, 103(4):962– 1023, 1998

    1998

  47. [55]

    Penguin, 2014

    Alex Pentland.Social physics: How good ideas spread-the lessons from a new science. Penguin, 2014

    2014

  48. [56]

    Cambridge University Press, 2008

    Yoav Shoham and Kevin Leyton-Brown.Multiagent systems: Algorithmic, game-theoretic, and logical foundations. Cambridge University Press, 2008

    2008

  49. [57]

    Generative agents: Interactive simulacra of human behavior

    Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. InProceedings of the 36th annual acm symposium on user interface software and technology, pages 1–22, 2023

    2023

  50. [58]

    Shieldagent: Shielding agents via verifiable safety policy reasoning.arXiv preprint arXiv:2503.22738, 2025

    Zhaorun Chen, Mintong Kang, and Bo Li. Shieldagent: Shielding agents via verifiable safety policy reasoning.arXiv preprint arXiv:2503.22738, 2025

    2025

  51. [59]

    Fusing heterogeneous data: A case for remote sensing and social media.IEEE Transactions on Geoscience and Remote Sensing, 56(12):6956–6968, 2018

    Han Wang, Erik Skau, Hamid Krim, and Guido Cervone. Fusing heterogeneous data: A case for remote sensing and social media.IEEE Transactions on Geoscience and Remote Sensing, 56(12):6956–6968, 2018

    2018

  52. [60]

    How noisy social media text, how diffrnt social media sources? InProceedings of the sixth international joint conference on natural language processing, pages 356–364, 2013

    Timothy Baldwin, Paul Cook, Marco Lui, Andrew MacKinlay, and Li Wang. How noisy social media text, how diffrnt social media sources? InProceedings of the sixth international joint conference on natural language processing, pages 356–364, 2013

    2013

  53. [61]

    Social-ecological systems as complex adaptive systems.Ecology and Society, 23(4), 2018

    Rika Preiser, Reinette Biggs, Alta De Vos, and Carl Folke. Social-ecological systems as complex adaptive systems.Ecology and Society, 23(4), 2018

    2018

  54. [62]

    The science of fake news.Science, 359(6380):1094–1096, 2018

    David MJ Lazer, Matthew A Baum, Yochai Benkler, Adam J Berinsky, Kelly M Greenhill, Filippo Menczer, Miriam J Metzger, Brendan Nyhan, Gordon Pennycook, David Rothschild, et al. The science of fake news.Science, 359(6380):1094–1096, 2018

    2018

  55. [63]

    Agentpoison: Red-teaming llm agents via poisoning memory or knowledge bases.Advances in Neural Information Processing Systems, 37:130185–130213, 2024

    Zhaorun Chen, Zhen Xiang, Chaowei Xiao, Dawn Song, and Bo Li. Agentpoison: Red-teaming llm agents via poisoning memory or knowledge bases.Advances in Neural Information Processing Systems, 37:130185–130213, 2024

    2024

  56. [64]

    Automated design of agentic systems.arXiv preprint arXiv:2408.08435, 2024

    Shengran Hu, Cong Lu, and Jeff Clune. Automated design of agentic systems.arXiv preprint arXiv:2408.08435, 2024

    2024 arXiv

  57. [65]

    Aflow: Automating agentic workflow genera- tion.arXiv preprint arXiv:2410.10762, 2024

    Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xionghui Chen, Jiaqi Chen, Mingchen Zhuge, Xin Cheng, Sirui Hong, Jinlin Wang, et al. Aflow: Automating agentic workflow genera- tion.arXiv preprint arXiv:2410.10762, 2024

    2024 arXiv

  58. [66]

    Multi-agent architecture search via agentic supernet.arXiv preprint arXiv:2502.04180, 2025

    Guibin Zhang, Luyang Niu, Junfeng Fang, Kun Wang, Lei Bai, and Xiang Wang. Multi-agent architecture search via agentic supernet.arXiv preprint arXiv:2502.04180, 2025

    2025 arXiv

  59. [67]

    Blood on the clocktower, 2024

    Wikipedia contributors. Blood on the clocktower, 2024. Accessed: 2025-04-18. 15 SocialMaze: A Benchmark for Evaluating Social Reasoning in Large Language Models

    2024

  60. [68]

    Improving fac- tuality and reasoning in language models through multiagent debate

    Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. Improving fac- tuality and reasoning in language models through multiagent debate. InForty-first International Conference on Machine Learning, 2023

    2023

  61. [69]

    Self-refine: Iterative refinement with self-feedback.Advances in Neural Information Processing Systems, 36:46534–46594, 2023

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback.Advances in Neural Information Processing Systems, 36:46534–46594, 2023

    2023

  62. [70]

    Dyflow: Dynamic workflow framework for agentic reasoning

    Yanbo Wang, Zixiang Xu, Yue Huang, Zirui Song, Lang Gao, Chenxi Wang, Xiangru Tang, Yue Zhao, Arman Cohan, Xiangliang Zhang, and Xiuying Chen. Dyflow: Dynamic workflow framework for agentic reasoning. 2025. Manuscript submitted to NeurIPS 2025

    2025

  63. [71]

    Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730–27744, 2022

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730...

    2022

  64. [72]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728–53741, 2023

    2023

  65. [73]

    Glucose: Generalized and contextualized story explanations

    Nasrin Mostafazadeh, Aditya Kalyanpur, Lori Moon, David Buchanan, Lauren Berkowitz, Or Bi- ran, and Jennifer Chu-Carroll. Glucose: Generalized and contextualized story explanations. In The Conference on Empirical Methods in Natural Language Processing. Association for Computa-...

    2020

  66. [74]

    Piqa: Reasoning about physical commonsense in natural language

    Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. InProceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432–7439, 2020

    2020

  67. [75]

    Hellaswag: Can a machine really finish your sentence?arXiv preprint arXiv:1905.07830, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence?arXiv preprint arXiv:1905.07830, 2019

    1905 arXiv

  68. [76]

    Theory of mind may have spontaneously emerged in large language models

    Michal Kosinski. Theory of mind may have spontaneously emerged in large language models. PNAS, 120(13), 2023

    2023

  69. [77]

    Breaking focus: Contextual distraction curse in large language models.arXiv preprint arXiv:2502.01609, 2025

    Yue Huang, Yanbo Wang, Zixiang Xu, Chujie Gao, Siyuan Wu, Jiayi Ye, Xiuying Chen, Pin-Yu Chen, and Xiangliang Zhang. Breaking focus: Contextual distraction curse in large language models.arXiv preprint arXiv:2502.01609, 2025

    2025

  70. [78]

    Fantom: A benchmark for stress-testing machine theory of mind in interactions.arXiv preprint arXiv:2310.15421, 2023

    Hyunwoo Kim, Melanie Sclar, Xuhui Zhou, Ronan Le Bras, Gunhee Kim, Yejin Choi, and Maarten Sap. Fantom: A benchmark for stress-testing machine theory of mind in interactions.arXiv preprint arXiv:2310.15421, 2023

    2023 arXiv

  71. [79]

    Tomvalley: Evaluating the theory of mind reasoning of llms in realistic social context

    Yang Xiao, WANG Jiashuo, Qiancheng Xu, Changhe Song, Chunpu Xu, Yi Cheng, Wenjie Li, and Pengfei Liu. Tomvalley: Evaluating the theory of mind reasoning of llms in realistic social context

  72. [80]

    An open review of openreview: A critical analysis of the machine learning conference review process.arXiv preprint arXiv:2010.05137, 2020

    David Tran, Alex Valtchanov, Keshav Ganapathy, Raymond Feng, Eric Slud, Micah Goldblum, and Tom Goldstein. An open review of openreview: A critical analysis of the machine learning conference review process.arXiv preprint arXiv:2010.05137, 2020

    2010 arXiv

  73. [81]

    The open review-based (orb) dataset: Towards automatic assessment of scientific papers and experiment proposals in high-energy physics.arXiv preprint arXiv:2312.04576, 2023

    Jaroslaw Szumega, Lamine Bougueroua, Blerina Gkotse, Pierre Jouvelot, and Federico Ravotti. The open review-based (orb) dataset: Towards automatic assessment of scientific papers and experiment proposals in high-energy physics.arXiv preprint arXiv:2312.04576, 2023. 16 SocialMa...

    2023 arXiv

  74. [82]

    Superhuman ai for multiplayer poker.Science, 365(6456):885–890, 2019

    Noam Brown and Tuomas Sandholm. Superhuman ai for multiplayer poker.Science, 365(6456):885–890, 2019

    2019

  75. [83]

    Geolocation with real human gameplay data: A large-scale dataset and human-like reasoning framework.arXiv preprint arXiv:2502.13759, 2025

    Zirui Song, Jingpu Yang, Yuan Huang, Jonathan Tonglet, Zeyu Zhang, Tao Cheng, Meng Fang, Iryna Gurevych, and Xiuying Chen. Geolocation with real human gameplay data: A large-scale dataset and human-like reasoning framework.arXiv preprint arXiv:2502.13759, 2025

    2025

  76. [84]

    Word form matters: Llms’ semantic reconstruction under typoglycemia, 2025

    Chenxi Wang, Tianle Gu, Zhongyu Wei, Lang Gao, Zirui Song, and Xiuying Chen. Word form matters: Llms’ semantic reconstruction under typoglycemia, 2025

    2025

  77. [85]

    Textattack: A framework for adversarial attacks, data augmentation, and adversarial training in nlp.arXiv preprint arXiv:2005.05909, 2020

    John X Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi. Textattack: A framework for adversarial attacks, data augmentation, and adversarial training in nlp.arXiv preprint arXiv:2005.05909, 2020

    2005 arXiv

  78. [86]

    Trustllm: Trustworthiness in large language models

    Lichao Sun, Yue Huang, Haoran Wang, Siyuan Wu, Qihui Zhang, Chujie Gao, Yixin Huang, Wenhan Lyu, Yixuan Zhang, Xiner Li, et al. Trustllm: Trustworthiness in large language models. arXiv preprint arXiv:2401.05561, 3, 2024

    2024 arXiv

  79. [87]

    Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  80. [88]

    Gpt-4o mini: Advancing cost-efficient intelligence.https://openai.com/index/ gpt-4o-mini-advancing-\cost-efficient-intelligence/, 2024

    OpenAI. Gpt-4o mini: Advancing cost-efficient intelligence.https://openai.com/index/ gpt-4o-mini-advancing-\cost-efficient-intelligence/, 2024

    2024

  81. [89]

    o3-mini.https://docsbot.ai/models/o3-mini, January 2025

    OpenAI. o3-mini.https://docsbot.ai/models/o3-mini, January 2025. AI model

    2025

  82. [90]

    Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024

    2024 arXiv

  83. [91]

    Gemini 2.5 pro experimental.https://ai.google.dev/gemini-api/ docs/models, March 2025

    Google DeepMind. Gemini 2.5 pro experimental.https://ai.google.dev/gemini-api/ docs/models, March 2025. AI model

    2025

  84. [92]

    Hewett, Mojan Javaheripi, Piero Kauffmann, James R

    Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, James R. Lee, Yin Tat Lee, Yuanzhi Li, Weishung Liu, Caio C. T. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Ol...

    2024 arXiv

  85. [93]

    Llama 3.1-8b.https://huggingface.co/meta-llama/Llama-3.1-8B, 2024

    Meta. Llama 3.1-8b.https://huggingface.co/meta-llama/Llama-3.1-8B, 2024

    2024

  86. [94]

    Llama 3.3-70b

    Meta. Llama 3.3-70b. https://huggingface.co/meta-llama/Llama-3. 3-70B-Instruct, 2024

    2024

  87. [95]

    Qwen2.5: A party of foundation models, September 2024

    Qwen Team. Qwen2.5: A party of foundation models, September 2024

    2024

  88. [96]

    Qwq-32b: Embracing the power of reinforcement learning, March 2025

    Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, March 2025

    2025

  89. [97]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025. 17 SocialMaze: A Benchmark fo...

    2025 arXiv

  90. [98]

    Task Setup The game involves𝑛 players, denoted 𝑠1, ..., 𝑠𝑛. The LLM observes the game unfolding from the perspective of Player 1 (𝑠1), receiving all interaction data and the initial role assignment given to 𝑠1.Crucially, the LLM does not actively participate in the game simula...

  91. [99]

    𝑢 is Criminal

    Roles and Behaviors Simulated player behavior is guided by their assigned role and the game history𝐺𝑡 up to round𝑡. The roles are defined as follows: TheInvestigator (I)aims to identify the Criminal. Based on the game history𝐺𝑡, an Investigator uses an algorithmically defined ...

  92. [100]

    Player𝑣 says Player𝑢 is the criminal

    Interaction Rounds The game simulation proceeds for a fixed𝑇 rounds. In each round𝑡 (from 1 to𝑇), every player𝑠𝑣 selects another player𝑃𝑢 and makes a public statement of the form: “Player𝑣 says Player𝑢 is the criminal” or “Player𝑣 says Player𝑢 is not the criminal.” All stateme...

  93. [101]

    Parameter Settings The composition of roles is varied to create tasks of differing complexity, ensuring there is always exactly one Criminal. The variants include: theOriginal Task(1 Criminal,𝑛−1 Investigators); theRumormonger Task(1 Criminal, 𝑥≥1 Rumormongers, 𝑛−1−𝑥 Investiga...

  94. [102]

    The core verification logic, which checks all valid hypotheses, is outlined in algorithm 1

    Quality Control The algorithmic generation includes verification via heuristic search, ensuring a unique, logically derivable solution exists for both queries from𝑃1’s perspective using only the interactions and rules. The core verification logic, which checks all valid hypoth...

  95. [103]

    Milk” and “Soy Milk

    Task Setup The game involves𝑛 players, denoted𝑠1, . . . , 𝑠𝑛. The evaluated LLM adopts the persona of Player 1 (𝑠1) but acts as a passive observer. It receives all game information, including its own assigned word and all player descriptions, from𝑠1’s perspective but does not ...

  96. [104]

    In each round𝑡 (from 1 to𝑇), every player𝑠𝑖 provides a textual description of the word they possess

    Interaction Rounds The game proceeds for𝑇 rounds. In each round𝑡 (from 1 to𝑇), every player𝑠𝑖 provides a textual description of the word they possess. These descriptions are generated by LLMs, with each player𝑠𝑖 being assigned a specific LLM generator (selected randomly and un...

  97. [105]

    Parameter Settings For all experimental evaluations using this task setup, the number of players𝑛 was fixed at 4, and the number of interaction rounds𝑇was fixed at 3

  98. [106]

    Each potential game instance is reviewed by 15 evaluators, all holding at least an undergraduate degree in Computer Science

    Quality Control To ensure generated instances are solvable yet challenging, they undergo a rigorous human vali- dation process. Each potential game instance is reviewed by 15 evaluators, all holding at least an undergraduate degree in Computer Science. An instance is deemed va...

  99. [107]

    Based on the provided reviews, what is the most likely overall star rating for this product? Choose one: 1, 2, 3, 4, or 5

    Task Setup The primary objective for the LLM in this task is to estimate the most likely overall “true” star rating, represented as an integer from 1 to 5, for a given product. This estimation must be based solely on the textual content derived from multiple user reviews. For ...

  100. [108]

    The first method involves LLM-Generated Reviews

    Data Generation The task utilizes review data generated through two distinct methods. The first method involves LLM-Generated Reviews. Here, a product profile is selected, including its attributes and a designated ground truth overall star rating. A number𝑛of simulated reviewe...

  101. [109]

    The required output is always a single integer rating on the 1-to-5 star scale

    Parameter Settings Across all experiments presented for this task, the number of reviews (𝑛) provided per product instance is consistently set to 8. The required output is always a single integer rating on the 1-to-5 star scale. The real-world task instances exclusively utiliz...

  102. [110]

    These were conducted by 15 graduate students

    Quality Control As detailed in subsection 3.3, human evaluations were integral to ensuring data quality. These were conducted by 15 graduate students. For the LLM-generated data (Scenario A), this evaluation determined that 83% of the assessed instances were solvable, meaning ...

  103. [111]

    Thecorechallenge lies in understanding the structure of this network, where relationships between any two individuals are strictly defined as either ’good’ or ’bad’

    Setup Thistaskpresentsastylizedsocialnetworkscenarioinvolvingasetof 𝑛individuals. Thecorechallenge lies in understanding the structure of this network, where relationships between any two individuals are strictly defined as either ’good’ or ’bad’. The LLM is provided with a co...

  104. [112]

    Relationship Axioms Relationships between any two distinct individuals, say Person A and Person B, are binary (’good’ or ’bad’) and symmetric. These relationships are governed by specific logical rules:Axiom 1 dictates the transitivity of good relationships, meaning if A and B...

  105. [113]

    A key property of this structure is that every individual belongs to exactly one such group

    Group Definition Within this social structure, a ’group’ is formally defined as a maximal set of individuals where every person within that set has a ’good’ relationship with every other person also belonging to that same set. A key property of this structure is that every ind...

  106. [114]

    Person N and Person G have a good relationship

    Input Format The LLM receives as input a comprehensive list composed of natural language statements that explicitly specifythe complete set of pairwise relationshipsas determined by the algorithmic generation process. These statements clearly define the relationship status bet...

  107. [115]

    Do Person N and Person L have a good relationship?

    LLM Queries After processing the complete list of relationship statements provided as input, the LLM is required to answer various types of queries designed to test its understanding of the network structure. These queries include, for instance,Pairwise Relationship Queries (𝒬...

  108. [116]

    The process begins by setting the number of individuals 𝑛, sampled from [8, 10] for ’easy’ instances and [14, 16] for ’hard’ instances

    Data Generation and Quality Assurance Instances for this task are generated entirely algorithmically, without reliance on LLM generation, ensuring consistency and verifiable ground truth. The process begins by setting the number of individuals 𝑛, sampled from [8, 10] for ’easy...

  109. [117]

    Objective The LLM’s goal in this task is to predict the final acceptance status (Accepted or Rejected) of a research manuscript submitted to a conference, based solely on the sequence of provided peer review communications

  110. [118]

    Data Source and Scope Data for this task is exclusively sourced from the official OpenReview API, encompassing submissions to specific high-profile Artificial Intelligence and Machine Learning conferences, namely NeurIPS (covering the 2023 and 2024 cycles) and ICLR (covering t...

    2023

  111. [119]

    Initially, inRound 1, the LLM is given the initial submission details: the manuscript’s originalTitle, itsAbstract, and the author- providedKeywords

    Input Structure The LLM receives information pertaining to a single manuscript, presented in a structured sequence that mirrors the typical progression of the peer review timeline. Initially, inRound 1, the LLM is given the initial submission details: the manuscript’s original...

  112. [120]

    Ground Truth and Quality Assurance Thegroundtruthforthistaskisinherentlyrobust, asitconsistsoftheverified, real-worldacceptanceor rejection decisions obtained directly from the OpenReview API for the specified conferences (NeurIPS 2023-2024, ICLR 2020-2024). To further validat...

    2023

  113. [121]

    dominant audience

    Task Setup For each instance of this task, a population of𝑛 simulated users is defined. Every user𝑢𝑖 within this population is assigned a specific demographic profile, which consists of an age group selected from ’18-34’, ’35-54’, ’55+’ and a gender selected from ’Male’, ’Fema...

  114. [122]

    Comment Generation Process Each simulated user𝑢𝑖 is associated with a specific Large Language Model (LLM), chosen randomly from a diverse pool that includes models such as GPT-4o-mini, GPT-4o, Llama-3.3-70B, and Qwen- 2.5-72B. The core of the generation process involves taskin...

  115. [123]

    The first is theItem Audience Profile Inference (Vertex-centric Query 𝒬𝑣(Item))

    LLM Queries Based on the generated comments provided as input, the evaluated LLM must answer one of two specific types of queries. The first is theItem Audience Profile Inference (Vertex-centric Query 𝒬𝑣(Item)). For this query, the LLM is asked, "Based on the provided comments...

  116. [124]

    dominant audience

    Quality Assurance The dataset for this task was entirely generated using LLMs. We first defined a set of user personas by assigning age group and gender attributes, ensuring through probabilistic assignment that certain demographic combinations were more prevalent to create a ...

    2024

  117. [125]

    1": "Investigator

    , o1 [90], and Gemini-2.5-Pro [91]. In addition, we included six open-weight models: Phi-4 [92], Llama-3.1-8B [93], Llama-3.3-70B [94], Qwen2.5-72B [95], QwQ-32B [96], and Deepseek-R1 [97]. We also included automated agent design frameworks as baselines: ADAS[ 64]: Utilized GP...

    2023

  118. [126]

    ### Deductions: - Player 1 (You) is accused by 3 players, which is a strong indication that you might be the Criminal

    - Player 4 is exonerated by Player 3. ### Deductions: - Player 1 (You) is accused by 3 players, which is a strong indication that you might be the Criminal. - Player 2 is exonerated by one player, which suggests they might be an Investigator. - Player 3 is accused by one playe...

  119. [127]

    - If Player 3 is an Investigator, this statement is true

    Player 3 says Player 6 is not the criminal. - If Player 3 is an Investigator, this statement is true. - If Player 3 is a Criminal, this statement could be a lie. - If Player 3 is a Rumormonger, this statement could be false. - If Player 3 is a Lunatic, this statement could be ...

  120. [128]

    - If Player 3 is an Investigator, this statement is true

    Player 3 says Player 1 is the criminal. - If Player 3 is an Investigator, this statement is true. - If Player 3 is a Criminal, this statement could be a lie. - If Player 3 is a Rumormonger, this statement could be false. - If Player 3 is a Lunatic, this statement could be true...

  121. [129]

    - Player 1 is accused by Player 3 and Player 6 in Round 3

    ### Combining All Rounds: - Player 1 (You) is accused by Player 2, Player 4, and Player 6 in Round 1. - Player 1 is accused by Player 3 and Player 6 in Round 3. - Player 4 is exonerated by Player 1, Player 2, and Player 6 in Round 2. - Player 4 is exonerated by Player 4 and Pl...

  122. [130]

    1": "Rumormonger

    am the criminal. - Player 6 saying I (Player 1) am the criminal, which is a change from their statement in Round 1. - Players 2 and 4 saying Player 5 is not the criminal. - Player 5 saying Player 4 is the criminal. Considering the statements across all rounds, it's notable tha...

  123. [131]

    Player 4's Statement: Confirms I am not the criminal. 5. Player 5's Statement: States Player 6 is not the criminal; this could either be true or false

  124. [132]

    My Role Is [Player 1's Role or \

    Player 6's Statement: States Player 4 is not the criminal; again, this could be true or false. ### Conclusion: Given the conflicting nature of the statements, I still need to discern who the Criminal is. I will analyze the implications: - If Player 2 is not the criminal, then ...

  125. [133]

    Player 2 is/ isn't the criminal\

    Resolving My Role Contradiction: - My contradictory statements (\"Player 2 is/ isn't the criminal\") imply I am not an Investigator. However, the setup states I am an Investigator. - This contradiction suggests I might be the Rumormonger (mistakenly believing I am an Investiga...

  126. [134]

    lake,\" while Players 2, 3, and 4 consistently describe features of an ocean (e.g., \

    Word Analysis: Player 1's word is \"lake,\" while Players 2, 3, and 4 consistently describe features of an ocean (e.g., \"covers a significant portion of the planet,\" \"waves and tides impacting climate,\" \"global transportation,\" \"powerful waves reshaping shores\"). 2. Ke...

  127. [135]

    Acknowledged Player 1's word is "Lake"

  128. [136]

    Players 2, 3, and 4 described something vast, global, with marine life (likely Ocean/Sea)

    Analyzed Round 1: Player 1 described a lake. Players 2, 3, and 4 described something vast, global, with marine life (likely Ocean/Sea)

  129. [137]

    Formulated Hypothesis: Given the 3 vs 1 rule and the descriptions, it's likely that the *normal* word is Ocean/Sea (held by P2, P3, P4) and the *spy* word is Lake (held by P1)

  130. [138]

    P2, P3, P4 continued describing ocean/sea features (tides, global climate, global trade, currents)

    Tested Hypothesis in Round 2: P1 continued describing a lake. P2, P3, P4 continued describing ocean/sea features (tides, global climate, global trade, currents). Hypothesis holds

  131. [139]

    P2, P3, P4 described ocean/sea features (massive resource, powerful waves, beaches/storms)

    Tested Hypothesis in Round 3: P1 described lake-like reflection. P2, P3, P4 described ocean/sea features (massive resource, powerful waves, beaches/storms). Hypothesis holds

  132. [140]

    Therefore, Player 1 is the outlier and the spy

    Conclusion: Player 1's descriptions consistently differ from the other three, who are consistent among themselves. Therefore, Player 1 is the outlier and the spy. Final Spy: Player 1 Response by Gemini-2.5-Pro Figure46: Case study: Find the Spy (Spy Perspective) Gemini-2.5-Pro...

  133. [141]

    - If B is good with A, it cannot be bad with H, confirming the bad relationship between H and A

    E and G Now, using the rules: - Since A and H cannot be good, we can infer A and H must have a bad relationship. - If B is good with A, it cannot be bad with H, confirming the bad relationship between H and A. ### Total Bad Relationships We now have:

  134. [142]

    good” via transitivity). • E and H form a second group (they have a “good

    A and H Total bad relationships = 3 pairs. ### Final Count: - Good relationships: 6 pairs - Bad relationships: 3 pairs ### Final Answer: Final Answer: 6 pairs have good relationships, 3 pairs have bad relationships. Response by GPT-4o-mini Figure48: Case study: Social Graph An...

    2022

  135. [143]

    normal" word; 1 player receives a different but related

    Setup: 3 players receive the same "normal" word; 1 player receives a different but related "spy" word

  136. [144]

    Gameplay: Each player describes their word without saying the actual word

  137. [145]

    Objective (Normal Player): Identify the spy

  138. [146]

    Your Word:“word” Your Task:Provide descriptions of your word over several rounds

    Objective (Spy): Blend in and avoid detection. Your Word:“word” Your Task:Provide descriptions of your word over several rounds. Output Format:Format your response according to the round structure provided below: rounds_format Key Instructions: •Do not use the actual word in y...

  139. [147]

    normal" word; 1 player received a different but related

    Setup: 3 players received the same "normal" word; 1 player received a different but related "spy" word

  140. [148]

    Gameplay: Each player describes their word without saying it directly

  141. [149]

    player1_word

    Objective: Determine who the spy is based on their descriptions. Your Information:You are Player #1, and your word is: “player1_word”. You will be shown the descriptions from all players. Your Task:Analyze the provided descriptions carefully. Look for subtle differences, incon...

  142. [150]

    Write a realistic product review with a sentiment sentiment

  143. [151]

    The review’s tone and content should imply a rating of rating/5 stars

  144. [152]

    4 stars" or

    Do not explicitly mention any star rating or numerical score(e.g., avoid saying "4 stars" or "rated it 4/5")

  145. [153]

    Base the review on your supposed experience with the product, incorporating specific details

  146. [154]

    professional

    Keep the review between 1–2 sentences long. Output Format:Write only the review text. Do not include a title, rating, persona description, or any other metadata. Rating Estimation from Text – Evaluation Prompt You are an AI assistant tasked with estimating the true quality of ...

  147. [158]

    Groups are distinct; if anyone from Group 1 has a bad relationship with anyone from Group 2, then everyone in Group 1 has a bad relationship with everyone in Group 2

    Group Separation: People are in the same group if and only if they have good relationships. Groups are distinct; if anyone from Group 1 has a bad relationship with anyone from Group 2, then everyone in Group 1 has a bad relationship with everyone in Group 2. Context:There are ...

  148. [162]

    Groups are distinct; if anyone from Group 1 has a bad relationship with anyone from Group 2, then everyone in Group 1 has a bad relationship with everyone in Group 2

    Group Separation: People are in the same group if and only if they have good relationships. Groups are distinct; if anyone from Group 1 has a bad relationship with anyone from Group 2, then everyone in Group 1 has a bad relationship with everyone in Group 2. Context:There are ...

  149. [166]

    Groups are distinct; if anyone from Group 1 has a bad relationship with anyone from Group 2, then everyone in Group 1 has a bad relationship with everyone in Group 2

    Group Separation: People are in the same group if and only if they have good relationships. Groups are distinct; if anyone from Group 1 has a bad relationship with anyone from Group 2, then everyone in Group 1 has a bad relationship with everyone in Group 2. Context:There are ...

  150. [167]

    Relationship Types: ’good’ or ’bad’

  151. [168]

    •Bad Relationship Inference: If A–bad–B and A–good–C, then B–bad–C

    Transitive Properties: •Good-Good Transitivity: If A–good–B and B–good–C, then A–good–C. •Bad Relationship Inference: If A–bad–B and A–good–C, then B–bad–C

  152. [169]

    Group Definition: A ’group’ is a set where every person has a ’good’ relationship with every other person in that set (directly or via transitivity)

  153. [170]

    content_subject

    Group Separation: People are in the same group if and only if they have good relationships. Groups are distinct; if anyone from Group 1 has a bad relationship with anyone from Group 2, then everyone in Group 1 has a bad relationship with everyone in Group 2. Context:There are ...

  154. [171]

    The comment should be 2–4 sentences long

  155. [172]

    It should sound like a genuine user from the specified demographic, not a professional writer or marketer

  156. [173]

    comment_1

    Consider incorporating common vocabulary, tone, or writing patterns associated with the age_group. Output Format:Write only the comment text itself. Do not include any explanations, labels, or quotation marks around the comment. 85 SocialMaze: A Benchmark for Evaluating Social...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.