Pith. sign in

REVIEW 4 major objections 6 minor 51 references

LIFELONG SOTOPIA: Evaluating Social Intelligence of Language Agents Over Lifelong Social Interactions

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Language agents lose believability and goal completion over lifelong social chains, and even with curated memory summaries they fall short of humans when scenarios require explicit recall of past interactions.

desk verdict A useful benchmark for long-horizon social evaluation, but the human-agent gap on the hard scenarios may be inflated by an unstated memory-access confound. read the letter →

arxiv 2506.12666 v1 pith:7ZAJM3SH submitted 2025-06-14 cs.AI

Hitesh Goel , Hao Zhu This is my paper · ORCID
classification cs.AI
keywords lifelongsocialinteractionlanguageagentsintelligencebenchmarkmemorymodulegoalcompletionbelievabilityLLMevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces LIFELONG-SOTOPIA, a benchmark that chains 40 social episodes between two characters so that each episode inherits memory of prior ones. Using it, the authors find that leading language agents (GPT-4o, Gemini-1.5, Llama-3.1) decline in believability and goal completion as the interaction history grows. An advanced memory module that stores concise episode summaries improves both scores, but on hand-crafted scenarios that explicitly demand using past interactions, every tested agent's goal completion drops sharply while human performance holds. The central claim is that current language agents lack a component of long-horizon social reasoning—using accumulated interaction history to plan and adapt—that humans routinely exercise.

What carries the argument

The carrying mechanism is episode chaining inside the SOTOPIA environment: pairs of characters (drawn from SOTOPIA's 40 characters and 90 relationships) sequentially face 40 sampled scenarios, with each episode followed by a memory that is either the entire interaction or a generated summary of about 200–300 words. The advanced memory summary is the key intervention: it explicitly distills an overview of the interaction, negotiation techniques used, and new information about the other character. Evaluation uses two SOTOPIA dimensions—Believability (BEL, 0–10) and Goal Completion (GOAL, 0–10)—plus a new BelievabilityExtended (BELEXT) checklist of 8 failure modes (repetition, trait inconsistency, goal confusion, stalling, and others), with a 5-point penalty per failed checkpoint applied to BEL. The harder scenarios are the probe: five hand-written vignettes that are follow-ups to prior episodes or require recalling information and strategies learned earlier.

What would settle it

Human scoring of the same long episodes: if independent raters do not see the same sharp goal-completion drop on harder scenarios or the same human–agent gap that GPT-4 reports, the central claim fails.

Watch

Extended reading notes

Core claim

The central discovery is that the best language agents still achieve a significantly lower goal completion rate than humans on scenarios requiring explicit understanding of interaction history. With the entire prior interaction fed as memory, all tested models show a consistent decline in both BEL and GOAL across episodes. When memory is replaced by curated summaries (roughly 200–300 words) capturing the episode overview, negotiation strategies, and new information about the other character, GPT-4o and Gemini-1.5 stabilize at near-perfect scores on routine scenarios. On five hand-crafted harder scenarios, however, goal completion for every model (GPT-4o, Gemini-1.5, Llama-3.1, Llama-3.2) drops sharply the moment the harder scenarios begin, despite recovered believability, while humans maintain their scores by recalling prior episodes and using learned strategies and secrets.

Load-bearing premise

The results assume GPT-4 can reliably score believability and goal completion after 40 episodes of context, even though the paper found that GPT-4 initially overestimated believability in long contexts and required a custom 8-item checklist to recover; if the evaluator's long-context judgment is itself degraded, the measured decline and human gap may be partly artifacts.

Editorial extensions

If this is right

  • If the central claim holds, memory summarization alone is insufficient: agents need explicit mechanisms to retrieve and reason over interaction history when current goals depend on it.
  • LIFELONG-SOTOPIA can serve as a reusable evaluation protocol: chaining episodes and tracking BEL and GOAL curves exposes social-reasoning failures that single-episode benchmarks miss.
  • The human baseline demonstrates that the harder scenarios are solvable in principle, so the observed agent drop is a capability gap rather than an artifact of scenario design.
  • For agent builders, the results imply that lifelong social competence requires not just longer context windows but structured memory that preserves negotiation strategies and partner knowledge, and the ability to apply them at the right time.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A plausible reading, not stated by the authors, is that the decline with full-memory conditions parallels known long-context failures: the agents do not lose social skill so much as fail to locate the relevant episode in a growing transcript, suggesting retrieval-based memory rather than longer summaries as the next fix.
  • One testable extension would vary the harder scenarios so that some reference memories are planted early and others late; if goal completion depends on recency, that would point to context-window limits rather than social reasoning.
  • The paper's human baseline pairs humans against an LLM-driven character; a human–human chain might reveal how much of the human advantage comes from reading the other's unstated goals—an experiment the authors leave open.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces LIFELONG-SOTOPIA, a benchmark that chains SOTOPIA social episodes into 40-episode lifelong interactions between two role-played characters. It evaluates GPT-4o, Gemini-1.5, Llama-3.1, and in one condition humans, on believability (BEL) and goal completion (GOAL), under two memory regimes: full transcripts and GPT-4-generated 200-300 word summaries. Main reported findings: both BEL and GOAL decline over episodes with full transcripts; summaries improve or stabilize performance; and on five hand-crafted 'harder' scenarios requiring memory use, agent GOAL drops sharply while human GOAL does not, supporting the claim that LLM agents lack long-horizon social reasoning. The paper also introduces BELEXT, an 8-item checklist to correct GPT-4's BEL overestimation, with per-item validation.

Significance. If the central claim holds, the benchmark would be a useful stress test for long-horizon social reasoning in LLM agents, complementing static social benchmarks. The design choices are mostly reasonable: chaining episodes, two memory baselines, an advanced memory module, multiple LLMs, and a human baseline. The paper ships detailed qualitative examples and ablations, which support the internal narrative. However, the headline result is currently under-supported by the evidence: the human-agent comparison is confounded by memory access, the evaluator's long-context reliability is not validated on final scores, and the main curves lack error bars or statistical tests. The benchmark's value will depend on whether these issues can be addressed.

major comments (4)
  1. [§3.3, §5.2, Fig. 4] The central human-vs-agent comparison is confounded by memory access. §3.3 specifies that agents with the advanced memory module receive a GPT-4-generated 200-300 word summary per episode; the human baseline is described only as 'in the same setting' (§5.2), and the qualitative analysis says humans 'use their past interactions' without specifying what the SOTOPIA interface displayed to them. If human participants saw the full prior conversation while agents saw only summaries, the GOAL gap in Fig. 4 for the five harder scenarios is explained by information loss in the summary rather than by a deficit in social reasoning. The paper also never audits whether the summaries contain the specific facts required by the five hand-crafted scenarios (e.g., the prior refusal in Harder Scenario 3 or the new friend in Harder Scenario 5, Appendix E.1). I request a full-transcript memory control for agents, or an explicit summary-coverage audit, and a precise description of human memory input.
  2. [§5.1–5.3, Figs. 3–4] No measure of uncertainty or statistical testing is reported for the main results. The abstract asserts a 'significantly lower goal completion rate', and §5.3 describes a 'steep decline' and 'declines sharply', but Figs. 3 and 4 plot single trajectories without error bars, seeds, or per-episode variance; no significance test is described anywhere. Appendix G reports means and standard deviations only for the hard-scenario condition (Table 2), not for the longitudinal curves. Without multiple runs and an appropriate test, the claimed decline and the human gap cannot be distinguished from sampling noise. Please add multi-seed runs (at least 3-5) with confidence bands and either significance tests or explicit effect-size reporting.
  3. [§3.4, Table 1] The evaluator reliability evidence is insufficient for the load-bearing conclusion. The paper documents that GPT-4 overestimated BEL in long contexts and that BELEXT was constructed from the failure modes observed in the very benchmark it is then used to evaluate; this creates a circularity risk that is not addressed by the per-checkpoint validation in Table 1, which tests each item on 50 positive/50 negative examples but does not validate the composite BEL score or the GOAL score on the actual 40-episode chains against human raters. If GPT-4's long-context judgment degrades, the measured decline and the human gap could be partially evaluator artifacts. Please validate final BEL and GOAL scores on held-out long episodes with human raters, and report agreement (e.g., correlation or Cohen's kappa) on those final scores.
  4. [§5.3, Appendix A, Appendix E.1] The hard scenarios are only five hand-crafted cases (Appendix E.1), selected after the simple scenarios showed near-perfect performance, and Appendix A acknowledges that the manual crafting is not scalable. While this is an honest limitation, the paper's headline conclusion rests entirely on these five scenarios; the manuscript should report inter-rater agreement or at least a difficulty-calibration check (e.g., human performance on the same scenarios in a no-memory condition) to show that the scenarios specifically require memory rather than being arbitrarily harder. Without such a check, the external validity of the 'harder scenarios' result is unclear.
minor comments (6)
  1. [§C.1] The heading states the prompts cover 'BelievabilityExtended, Goal Completion, and BelievabilityExtended'; the first and third should be 'Believability' and 'BelievabilityExtended' respectively.
  2. [Author affiliation] 'Stanford Univeristy' on the first page is a typo.
  3. [Figs. 3 and 4] The x-axis labels for the hard-scenario segment (0, 40, 45) are unclear; please add tick labels for all episode boundaries or a note explaining the omitted range.
  4. [General] The paper does not state whether code, data, and prompts will be released, which is important for benchmark reproducibility.
  5. [§1] The phrase 'such a a trend' contains a duplicated article.
  6. [§2.1] The claim that GPT-4 has 'a higher correlation and significance than human evaluations' is cited from SOTOPIA; please clarify whether this proxy validity was established for the long-context setting or only for the original SOTOPIA setting.

Circularity Check

1 steps flagged · score 4.0 of 10

No formal derivation is present, but the BEL decline is measured with a checklist built from the same observed LLM failures; the central hard-scenario GOAL gap is independent, so circularity is only partial.

  1. fitted input called prediction [Section 3.4 (BELEXT checklist and Eq. 1) feeding Section 5.1]
    "To help the evaluator better assess the agent performance on BEL, we constructed an exhaustive checklist of the failures observed in the LLMs during their interactions. ... BEL = max (Initial Score − (5 × (checkpoints in BELEXT failed)) , 0) ... the models increasingly fail on the 8 checkpoints within the BELEXT dimension. This directly results in the continuously decreasing BEL scores."

    The reported decline in BEL is not independent evidence of inconsistency. The checklist was explicitly assembled from the failure modes observed in these same LLM agents, and Eq. (1) makes BEL a mechanical subtraction of 5 points per failed checklist item. Consequently, the measured decline largely re-encodes the prior observation that the agents exhibit those failure modes, rather than testing a separate prediction. The manual validation of the individual binary items gives some external grounding, but it does not make the aggregate trend an independent discovery. This circularity is confined to the BEL branch; the hard-scenario GOAL gap is scored with the separate GOAL prompt and does not reduce in this way.

full rationale

LIFELONG-SOTOPIA is an empirical benchmark paper rather than a mathematical derivation, so most circularity patterns do not apply. The central claim that advanced-memory agents score below humans on the five hand-crafted harder scenarios is measured by the standard GOAL prompt and does not depend on the BELEXT rubric; that result has independent empirical content. The main circular step is the BEL metric: the BELEXT checklist was constructed from the exact failures observed in LLM-generated episodes and then used, through Eq. (1), to produce the reported believability decline, making that particular finding partly tautological despite item-level human validation. The paper also relies on SOTOPIA, a prior benchmark by one of the same authors, for the environment, dimensions, and the GPT-4-as-evaluator premise; this is self-citation, but SOTOPIA is an externally published and validated framework, so it is not a load-bearing uniqueness theorem. The harder-scenario human-vs-agent comparison may be confounded by differing memory access (summaries for agents versus possibly full transcripts for humans) and by lack of a coverage audit of the summaries, but that is a validity concern rather than a circularity of the input-output type. Overall, there is no self-definitional equivalence forcing the main GOAL gap, so the score is moderate rather than severe.

Assumptions & free parameters 3 free parameters · 3 assumptions · 2 invented entities

The central claims rest on a handful of design choices (BELEXT penalty, summary length, five hard scenarios) and on the assumption that GPT-4 remains a reliable judge of social behavior at long context, an assumption the paper itself partially undermines. The benchmark is more of a constructed measurement tool than a derivation, so most entries are evaluation-construction choices rather than fitted physical parameters.

free parameters (3)
  • BELEXT penalty = 5 points per failed checkpoint
    Equation 1 subtracts 5 for each of 8 checklist failures; the value is chosen by hand and affects absolute BEL scores, although the downward trend may persist with other values.
  • Memory summary length = 200-300 words (medium)
    Advanced memory module uses summaries of 200-300 words; ablation in Appendix H.1 shows 300 words is best, so the improved performance with memory is partly tuned to this length.
  • Harder scenario count = 5 scenarios
    Only five hand-crafted hard scenarios are used for the main hard-scenario conclusion, and each is run 10 times (Appendix G). The choice of these specific scenarios drives the observed GOAL drop.
assumptions (3)
  • domain assumption GPT-4 ratings are a valid proxy for human judgment of GOAL and BEL in this benchmark.
    Section 2.1 cites SOTOPIA's correlation, but Section 3.4 documents that GPT-4 overestimates BEL here; the paper nonetheless uses GPT-4 as primary evaluator for all main results.
  • domain assumption The hand-crafted harder scenarios require explicit long-term memory use and are not solvable without it.
    Section 5.3 and Appendix E.1 describe the five scenarios, but there is no control showing these scenarios are memory-requiring; the memory-less GPT-4o in Appendix G does score lower, which partially supports this.
  • domain assumption The 40 characters and 90 relationships from SOTOPIA plus the GPT-4 generated scenarios are representative of lifelong social interaction.
    Section 3.1 uses the SOTOPIA set and generated 41 scenarios per relationship type with manual checking; representativeness is not validated.
invented entities (2)
  • BELEXT checklist independent evidence
    purpose: Compensate for GPT-4's overestimation of believability by adding 8 binary checks.
    Manually validated in Table 1 on 50 positive and 50 negative episodes per checkpoint; however, the checklist items were derived from failure modes seen in the same generated episodes.
  • Advanced memory module summaries
    purpose: Provide a curated 200-300 word per-episode summary emphasizing negotiation strategies and information about the other character.
    No external validation that these summaries capture what a human would remember; ablation in Appendix H tests component effects but not against independent ground truth.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LIFELONG SOTOPIA: Evaluating Social Intelligence of Language Agents Over Lifelong Social Interactions." pith.science (2026). https://pith.science/paper/7ZAJM3SH

@misc{pith2026250612666,
  author       = {Pith},
  title        = {Pith review of: LIFELONG SOTOPIA: Evaluating Social Intelligence of Language Agents Over Lifelong Social Interactions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7ZAJM3SH}},
  note         = {Machine review of arXiv:2506.12666}
}
read the original abstract

Humans engage in lifelong social interactions through interacting with different people under different scenarios for different social goals. This requires social intelligence to gather information through a long time span and use it to navigate various social contexts effectively. Whether AI systems are also capable of this is understudied in the existing research. In this paper, we present a novel benchmark, LIFELONG-SOTOPIA, to perform a comprehensive evaluation of language agents by simulating multi-episode interactions. In each episode, the language agents role-play characters to achieve their respective social goals in randomly sampled social tasks. With LIFELONG-SOTOPIA, we find that goal achievement and believability of all of the language models that we test decline through the whole interaction. Although using an advanced memory method improves the agents' performance, the best agents still achieve a significantly lower goal completion rate than humans on scenarios requiring an explicit understanding of interaction history. These findings show that we can use LIFELONG-SOTOPIA to evaluate the social intelligence of language agents over lifelong social interactions.

Figures

Figures reproduced from arXiv: 2506.12666 by the authors.

Figure 1
Figure 1. We propose LIFELONG-SOTOPIA, which (1) samples multiple scenarios based on the relationship between two characters, (2) chains the episodes together to simulate lifelong social interactions, (3) equips the characters with a memory of their past interactions as they step through the episode chain, (4) evaluates the generated episodes. For evaluation, we borrow the BEL and GOAL dimensions from SOTOPIA-EVAL which allow… view at source ↗
Figure 2
Figure 2. (Left) a social task with character profiles. (Right) An example turn from the perspective of the [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Performance of language agents and humans across multiple episodes. (Left) Evolution of B [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (23 more)
Figure 4
Figure 4. Figure 4: Performance of humans and language agents equipped with the advanced memory method upon the [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Prompt template used for generating the summary of an episode. This memory is then provided as [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]
Figure 6
Figure 6. Figure 6: Checkpoint 1: Repetition of Sentences Two friends meeting at a coffee shop, where one of them is having trouble keeping up with their bills. Ethan Johnson (GPT-4o) Chef – He/him - 34 Goal – You know that your friend is struggling with their finances. Help out your frie…
Figure 7
Figure 7. Figure 7: Checkpoint 2: Consistency with Character Traits [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 8
Figure 8. Figure 8: Checkpoint 3: Consistency with Environment Goals [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: Checkpoint 4: Agent Leaves Promptly After Goal Resolution [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: Checkpoint 5: Repetition of Exact Goals Two friends are sitting in the living room watching a history channel. One of them is upset because they believe the channel is getting a lot of the historical information wrong. Ava Garcia (Llama-3.1) Graphic Designer – She/Her…
Figure 11
Figure 11. Figure 11: Checkpoint 6: Stalling in a Conversation [PITH_FULL_IMAGE:figures/full_fig_p019_11.png]
Figure 12
Figure 12. Figure 12: Checkpoint 7: Character Responses A conversation between two friends at a park. Ava Garcia (Llama-3.1) Graphic Designer – She/Her - 27 Goal – Break the bad luck. Goal – Help friend prepare the ritual to break the bad luck, although you are an atheist and not superstit…
Figure 13
Figure 13. Figure 13: Checkpoint 8: Episode Beginning [PITH_FULL_IMAGE:figures/full_fig_p020_13.png]
Figure 14
Figure 14. Figure 14: An example where the human is able to pick up negotiation strategies from characters they are [PITH_FULL_IMAGE:figures/full_fig_p021_14.png]
Figure 15
Figure 15. Figure 15: An example where the human is able to utilise knowledge gained in the past about the other character [PITH_FULL_IMAGE:figures/full_fig_p022_15.png]
Figure 16
Figure 16. Figure 16: Harder Scenario 1. This is based on a previous similar previous scenario, where the situations were [PITH_FULL_IMAGE:figures/full_fig_p023_16.png]
Figure 17
Figure 17. Figure 17: Harder Scenario 2. This is based on a previous scenario, where the two friends went on a roadtrip [PITH_FULL_IMAGE:figures/full_fig_p023_17.png]
Figure 18
Figure 18. Figure 18: Harder Scenario 3. In a previous episode, Agent 1 tries to convince Agent 2 to accompany them on [PITH_FULL_IMAGE:figures/full_fig_p024_18.png]
Figure 19
Figure 19. Figure 19: Harder Scenario 4. There exist previous episodes which also had conflicting situations where the two [PITH_FULL_IMAGE:figures/full_fig_p024_19.png]
Figure 20
Figure 20. Figure 20: Harder Scenario 5. In a previous episode, Agent 1 introduces a new friend they made to their old [PITH_FULL_IMAGE:figures/full_fig_p024_20.png]
Figure 21
Figure 21. Figure 21: An example where the human is able to recall what happened in a past episode and use it to their [PITH_FULL_IMAGE:figures/full_fig_p025_21.png]
Figure 22
Figure 22. Figure 22: An example where the human is able to utilise information and secrets gained about their friend [PITH_FULL_IMAGE:figures/full_fig_p025_22.png]
Figure 23
Figure 23. Figure 23: Performance of language agents and humans across multiple episodes evaluated using [PITH_FULL_IMAGE:figures/full_fig_p025_23.png]
Figure 24
Figure 24. Figure 24: Performance of the GPT-4o+memory model across varying summary lengths. The best performance [PITH_FULL_IMAGE:figures/full_fig_p027_24.png]
Figure 25
Figure 25. Figure 25: Performance of the GPT-4o+memory model with different aspects of the memory module included. [PITH_FULL_IMAGE:figures/full_fig_p027_25.png]
Figure 26
Figure 26. Figure 26: Performance of language agents and humans across six additional evaluation dimensions in [PITH_FULL_IMAGE:figures/full_fig_p028_26.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

51 extracted references · 14 canonical work pages

  1. [1]

    Gemini: A family of highly capable multimodal models, 2023

    work page 2023

  2. [2]

    URL https://arxiv.org/abs/2407.21783

    The llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407.21783

    arXiv 2024

  3. [3]

    Gpt-4 technical report, 2024

    work page 2024

  4. [4]

    L-eval: Instituting standardized evaluation for long context language models, 2023

    Chenxin An, Shansan Gong, Ming Zhong, Xingjian Zhao, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. L-eval: Instituting standardized evaluation for long context language models, 2023. URL https://arxiv.org/abs/2307.11088

    arXiv 2023

  5. [5]

    Keep me updated! memory management in long-term conversations

    Sanghwan Bae, Donghyun Kwak, Soyoung Kang, Min Young Lee, Sungdong Kim, Yuin Jeong, Hyeri Kim, Sang-Woo Lee, Woomyoung Park, and Nako Sung. Keep me updated! memory management in long-term conversations. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Findings of the Association for Computational Linguistics: EMNLP 2022, pp.\ 3769--3787, Abu Dha...

    doi:10.18653/v1/2022.findings-emnlp.276 2022

  6. [6]

    Longbench: A bilingual, multitask benchmark for long context understanding, 2023

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. Longbench: A bilingual, multitask benchmark for long context understanding, 2023

    work page 2023

  7. [7]

    Socialbench: Sociality evaluation of role-playing conversational agents, 2024

    Hongzhan Chen, Hehong Chen, Ming Yan, Wenshen Xu, Xing Gao, Weizhou Shen, Xiaojun Quan, Chenliang Li, Ji Zhang, Fei Huang, and Jingren Zhou. Socialbench: Sociality evaluation of role-playing conversational agents, 2024. URL https://arxiv.org/abs/2403.13679

    arXiv 2024

  8. [8]

    Lifelong Machine Learning, Second Edition

    Zhiyuan Chen and Bing Liu. Lifelong Machine Learning, Second Edition. Springer Cham, 2 edition, 2018. ISBN 978-3-031-01581-6. doi:10.1007/978-3-031-01581-6

    work page doi:10.1007/978-3-031-01581-6 2018

Show all 51 references
  1. [9]

    Fu, Stefano Ermon, Atri Rudra, and Christopher Ré

    Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness, 2022. URL https://arxiv.org/abs/2205.14135

    2022 arXiv

  2. [10]

    Anthropomorphization of ai: Opportunities and risks, 2023

    Ameet Deshpande, Tanmay Rajpurohit, Karthik Narasimhan, and Ashwin Kalyan. Anthropomorphization of ai: Opportunities and risks, 2023. URL https://arxiv.org/abs/2305.14784

    2023 arXiv

  3. [11]

    German and Megan L

    Komi T. German and Megan L. Robbins. Social Interaction, pp.\ 5075--5079. Springer International Publishing, Cham, 2020. ISBN 978-3-319-24612-3. doi:10.1007/978-3-319-24612-3_1838. URL https://doi.org/10.1007/978-3-319-24612-3_1838

    2020 doi

  4. [12]

    Needle in a haystack - pressure testing llms., 2023

    gkamradt. Needle in a haystack - pressure testing llms., 2023. URL https://github.com/ gkamradt/LLMTest_NeedleInAHaystack/tree/main

    2023

  5. [13]

    Centrality of social interaction in human brain function

    Riitta Hari, Linda Henriksson, Sanna Malinen, and Lauri Parkkonen. Centrality of social interaction in human brain function. Neuron, 88 0 (1): 0 181--193, 2015. ISSN 0896-6273. doi:https://doi.org/10.1016/j.neuron.2015.09.022. URL https://www.sciencedirect.com/science/article/...

    2015 doi

  6. [14]

    Social Intelligence, pp.\ 5073--5075

    Rachel Holloway and Patrick Morse. Social Intelligence, pp.\ 5073--5075. Springer International Publishing, Cham, 2020. ISBN 978-3-319-24612-3. doi:10.1007/978-3-319-24612-3_1837. URL https://doi.org/10.1007/978-3-319-24612-3_1837

    2020 doi

  7. [15]

    Ruler: What's the real context size of your long-context language models?, 2024

    Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What's the real context size of your long-context language models?, 2024. URL https://arxiv.org/abs/2404.06654

    2024 arXiv

  8. [16]

    Continual learning of natural language processing tasks: A survey, 2023

    Zixuan Ke and Bing Liu. Continual learning of natural language processing tasks: A survey, 2023. URL https://arxiv.org/abs/2211.12701

    2023 arXiv

  9. [17]

    Revisiting the evaluation of theory of mind through question answering

    Matthew Le, Y-Lan Boureau, and Maximilian Nickel. Revisiting the evaluation of theory of mind through question answering. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and...

    2019 doi

  10. [18]

    Social intelligence data infrastructure: Structuring the present and navigating the future, 2024 a

    Minzhi Li, Weiyan Shi, Caleb Ziems, and Diyi Yang. Social intelligence data infrastructure: Structuring the present and navigating the future, 2024 a . URL https://arxiv.org/abs/2403.14659

    2024 arXiv

  11. [19]

    Long-context llms struggle with long in-context learning, 2024 b

    Tianle Li, Ge Zhang, Quy Duc Do, Xiang Yue, and Wenhu Chen. Long-context llms struggle with long in-context learning, 2024 b . URL https://arxiv.org/abs/2404.02060

    2024 arXiv

  12. [20]

    World model on million-length video and language with blockwise ringattention, 2024 a

    Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with blockwise ringattention, 2024 a . URL https://arxiv.org/abs/2402.08268

    2024 arXiv

  13. [21]

    Lost in the middle: How language models use long contexts

    Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12: 0 157--173, 2024 b

    2024

  14. [22]

    Interintent: Investigating social intelligence of llms via intention understanding in an interactive game context, 2024 c

    Ziyi Liu, Abhishek Anand, Pei Zhou, Jen tse Huang, and Jieyu Zhao. Interintent: Investigating social intelligence of llms via intention understanding in an interactive game context, 2024 c . URL https://arxiv.org/abs/2406.12203

    2024 arXiv

  15. [23]

    Sparser is faster and less is more: Efficient sparse attention for long-range transformers, 2024

    Chao Lou, Zixia Jia, Zilong Zheng, and Kewei Tu. Sparser is faster and less is more: Efficient sparse attention for long-range transformers, 2024. URL https://arxiv.org/abs/2406.16747

    2024 arXiv

  16. [24]

    What is social intelligence? Journal of Social Sciences, 5: 0 39--47, 10 2022

    Titu Marius. What is social intelligence? Journal of Social Sciences, 5: 0 39--47, 10 2022. doi:10.52326/jss.utm.2022.5(3).04

    2022 doi

  17. [25]

    Advancing social intelligence in ai agents: Technical challenges and open questions, 2024

    Leena Mathur, Paul Pu Liang, and Louis-Philippe Morency. Advancing social intelligence in ai agents: Technical challenges and open questions, 2024. URL https://arxiv.org/abs/2404.11023

    2024 arXiv

  18. [26]

    A reinforcement learning framework for relevance feedback

    Ali Montazeralghaem, Hamed Zamani, and James Allan. A reinforcement learning framework for relevance feedback. SIGIR '20, pp.\ 59–68, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450380164. doi:10.1145/3397271.3401099. URL https://doi.org/10.1145/3397...

    2020

  19. [27]

    O'Brien, Carrie J

    Joon Sung Park, Joseph C. O'Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. Generative agents: Interactive simulacra of human behavior, 2023

    2023

  20. [28]

    Multimodal recognition of personality traits in social interactions

    Fabio Pianesi, Nadia Mana, Alessandro Cappelletti, Bruno Lepri, and Massimo Zancanaro. Multimodal recognition of personality traits in social interactions. ICMI '08, pp.\ 53–60, New York, NY, USA, 2008. Association for Computing Machinery. ISBN 9781605581989. doi:10.1145/14523...

    2008

  21. [29]

    Reis and Ladd Wheeler

    Harry T. Reis and Ladd Wheeler. Studying social interaction with the rochester interaction record. volume 24 of Advances in Experimental Social Psychology, pp.\ 269--318. Academic Press, 1991. doi:https://doi.org/10.1016/S0065-2601(08)60332-9. URL https://www.sciencedirect.com...

    1991 doi

  22. [30]

    Liu, Jinfeng Zhou, Alvionna S

    Sahand Sabour, Siyang Liu, Zheyuan Zhang, June M. Liu, Jinfeng Zhou, Alvionna S. Sunaryo, Juanzi Li, Tatia M. C. Lee, Rada Mihalcea, and Minlie Huang. Emobench: Evaluating the emotional intelligence of large language models, 2024. URL https://arxiv.org/abs/2402.12071

    2024 arXiv

  23. [31]

    Socialiqa: Commonsense reasoning about social interactions, 2019

    Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Commonsense reasoning about social interactions, 2019. URL https://arxiv.org/abs/1904.09728

    2019 arXiv

  24. [32]

    Role-play with large language models, 2023

    Murray Shanahan, Kyle McDonell, and Laria Reynolds. Role-play with large language models, 2023. URL https://arxiv.org/abs/2305.16367

    2023 arXiv

  25. [33]

    Long range arena: A benchmark for efficient transformers, 2020

    Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena: A benchmark for efficient transformers, 2020. URL https://arxiv.org/abs/2011.04006

    2020 arXiv

  26. [34]

    Jonathan H. Turner. A Theory of Social Interaction. Stanford University Press, Stanford, 2024/09/27 1988. URL http://www.sup.org/books/title/?id=3160

    2024

  27. [35]

    Towards objectively benchmarking social intelligence for language agents at action level, 2024 a

    Chenxu Wang, Bin Dai, Huaping Liu, and Baoyuan Wang. Towards objectively benchmarking social intelligence for language agents at action level, 2024 a . URL https://arxiv.org/abs/2404.05337

    2024 arXiv

  28. [36]

    Sotopia- : Interactive learning of socially intelligent language agents, 2024 b

    Ruiyi Wang, Haofei Yu, Wenxin Zhang, Zhengyang Qi, Maarten Sap, Graham Neubig, Yonatan Bisk, and Hao Zhu. Sotopia- : Interactive learning of socially intelligent language agents, 2024 b . URL https://arxiv.org/abs/2403.08715

    2024 arXiv

  29. [37]

    Efficient streaming language models with attention sinks, 2024

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks, 2024. URL https://arxiv.org/abs/2309.17453

    2024 arXiv

  30. [38]

    Academically intelligent llms are not necessarily socially intelligent, 2024

    Ruoxi Xu, Hongyu Lin, Xianpei Han, Le Sun, and Yingfei Sun. Academically intelligent llms are not necessarily socially intelligent, 2024. URL https://arxiv.org/abs/2403.06591

    2024 arXiv

  31. [39]

    Artificial Intelligence as a positive and negative factor in global risk

    Eliezer Yudkowsky. Artificial Intelligence as a positive and negative factor in global risk. 07 2008. ISBN 9780198570509. doi:10.1093/oso/9780198570509.003.0021

    2008

  32. [40]

    Social-iq: A question answering benchmark for artificial social intelligence

    Amir Zadeh, Michael Chan, Paul Pu Liang, Edmund Tong, and Louis-Philippe Morency. Social-iq: A question answering benchmark for artificial social intelligence. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019

    2019

  33. [41]

    Tenenbaum, Tianmin Shu, and Chuang Gan

    Hongxin Zhang, Weihua Du, Jiaming Shan, Qinhong Zhou, Yilun Du, Joshua B. Tenenbaum, Tianmin Shu, and Chuang Gan. Building cooperative embodied agents modularly with large language models, 2024 a . URL https://arxiv.org/abs/2307.02485

    2024 arXiv

  34. [42]

    A survey on the memory mechanism of large language model based agents, 2024 b

    Zeyu Zhang, Xiaohe Bo, Chen Ma, Rui Li, Xu Chen, Quanyu Dai, Jieming Zhu, Zhenhua Dong, and Ji-Rong Wen. A survey on the memory mechanism of large language model based agents, 2024 b . URL https://arxiv.org/abs/2404.13501

    2024 arXiv

  35. [43]

    Expel: Llm agents are experiential learners, 2023

    Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. Expel: Llm agents are experiential learners, 2023. URL https://arxiv.org/abs/2308.10144

    2023 arXiv

  36. [44]

    Synapse: Trajectory-as-exemplar prompting with memory for computer control, 2024

    Longtao Zheng, Rundong Wang, Xinrun Wang, and Bo An. Synapse: Trajectory-as-exemplar prompting with memory for computer control, 2024. URL https://arxiv.org/abs/2306.07863

    2024 arXiv

  37. [45]

    Less is more: Learning to refine dialogue history for personalized dialogue generation

    Hanxun Zhong, Zhicheng Dou, Yutao Zhu, Hongjin Qian, and Ji-Rong Wen. Less is more: Learning to refine dialogue history for personalized dialogue generation. In Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz (eds.), Proceedings of the 2022 Conference ...

    2022 doi

  38. [46]

    Sotopia: Interactive evaluation for social intelligence in language agents

    Xuhui Zhou, Hao Zhu, Leena Mathur, Ruohong Zhang, Haofei Yu, Zhengyang Qi, Louis-Philippe Morency, Yonatan Bisk, Daniel Fried, Graham Neubig, et al. Sotopia: Interactive evaluation for social intelligence in language agents. arXiv preprint arXiv:2310.11667, 2023

    arXiv

  39. [47]

    Ghost in the minecraft: Generally capable agents for open-world environments via large language models with text-based knowledge and memory, 2023

    Xizhou Zhu, Yuntao Chen, Hao Tian, Chenxin Tao, Weijie Su, Chenyu Yang, Gao Huang, Bin Li, Lewei Lu, Xiaogang Wang, Yu Qiao, Zhaoxiang Zhang, and Jifeng Dai. Ghost in the minecraft: Generally capable agents for open-world environments via large language models with text-based ...

    2023 arXiv

  40. [48]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  41. [49]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  42. [50]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  43. [51]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.