Pith. sign in

REVIEW 3 major objections 4 minor 104 references

AdaMARP: An Adaptive Multi-Agent Interaction Framework for General Immersive Role-Playing

T0 review · 3 major / 4 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A compact actor model trained on a scene-aware protocol can outperform several commercial role-playing LLMs, and a small scene manager can beat a leading proprietary model at orchestration.

desk verdict Solid framework, unproven headline: the in-house benchmark and judge share the training distribution, so the 'beats proprietary models' claim needs external support. read the letter →

arxiv 2601.11007 v2 pith:6CVCXZ3W submitted 2026-01-16 cs.AI cs.CL

Zhenhua Xu , Dongsheng Chen , Shuo Wang , Jian Li , Chengjie Wang , Meng Han , Yabiao Wang This is my paper
classification cs.AIcs.CL
keywords adaptiverole-playingmulti-agentframeworkscenemanagerenvironmentgroundingdatasettrajectoryevaluationLLMrole-play
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

AdaMARP claims that open-weights LLMs can become genuinely adaptive role-players—able to switch scenes, change speakers, and introduce new characters mid-story—if role-play is split between an Actor that outputs interleaved thought, action, environment, and speech, and a Scene Manager that issues discrete orchestration decisions with rationales. The paper builds two training corpora, one extracted from novels and one LLM-synthesized with forced scene transitions and role additions, plus a trajectory-level benchmark. Its central evidence is that an 8B Actor trained this way outscores several commercial role-playing LLMs, and a 14B Scene Manager surpasses a leading proprietary model on orchestration quality. If right, the result is a concrete recipe for high-quality adaptive role-playing with small, steerable, locally runnable models.

What carries the argument

The mechanism is the unified message format—[Thought], (Action), <Environment>, and Speech, freely interleaved within a turn—together with the Scene Manager's discrete action space and rationales. The environment tag carries the argument: it turns setting from passive backdrop into state that constrains character behavior and is updated by it. The Scene Manager's loop is what makes role-playing adaptive: periodically deciding who speaks, when the scene changes, and when a new character enters, with each decision justified in natural language.

What would settle it

Have independent human judges, blind to model identity, rate the same trained 8B Actor and the commercial baselines on 100 human-authored role-playing scenarios with scene shifts and character introductions; if the trained 8B model no longer wins, the AdaptiveBench advantage is explained by distribution overlap rather than general adaptive ability.

Watch

Extended reading notes

Core claim

The central claim is that the bottleneck in general role-playing is not model scale but the interaction protocol and the training signal. By making the environment a first-class component of every turn and adding an explicit Scene Manager that chooses discrete actions (init_scene, pick_speaker, switch_scene, add_role, end) with natural-language rationales, the authors show that a 7B–8B Actor can outperform several commercial role-playing LLMs on trajectory-level metrics, and a 14B Scene Manager can beat a leading proprietary model at orchestration. The framework models role-playing as a three-agent loop—Actor, User, Scene Manager—rather than a fixed two-party chat, and the data pipeline deli

Load-bearing premise

The load-bearing premise is that AdaptiveBench measures general adaptive role-playing skill: its 100 evaluation trajectories come from the same LLM-synthesized distribution used to train the models, and the default judge is from the same model family that generated the training data; if the models are just fitting that distribution, the reported edge over commercial systems will not transfer to novel scenarios.

Editorial extensions

If this is right

  • Open 7B–8B models can match or beat commercial role-playing LLMs when trained on this protocol, shrinking the compute and cost needed for high-quality interactive fiction.
  • The two-part corpus design—literary extraction plus synthetic trajectories that force scene switches and role additions—is separable: extracted data improves format adherence, while synthetic data provides adaptive generalization.
  • A separately trained Scene Manager improves scene understanding and role-introduction judgment while trading off a small drop in speaker-selection discipline, so the two capabilities can be tuned independently.
  • The gains reproduce across two open model families and scales from 7B to 72B, suggesting the recipe is backbone-independent.
  • Trajectory-level evaluation with rubric scoring captures environment grounding and narrative progression in ways single-turn benchmarks do not.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported margin may be partly a fit to the training distribution: because the benchmark trajectories come from the same LLM-synthesized distribution used for training, testing on human-written or otherwise novel scenarios would show whether the protocol transfers.
  • The environment-as-first-class signal suggests a natural upgrade path: coupling the Actor to an explicit world-state tracker or memory store could make long-horizon consistency more reliable than relying on the context window alone.
  • The Scene Manager's explicit rationales could be surfaced to users as controls—allowing a human to veto a scene switch or request a new character—which the current framework does not explore.
  • The 20-theme taxonomy is English-centric and dominated by Western narrative forms; adapting the pipeline to other cultural storytelling conventions is an untested extension.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. AdaMARP proposes a three-agent role-playing framework (Actor, User, Scene Manager) in which the Actor produces turns that interleave [Thought], (Action), <Environment>, and Speech, and the Scene Manager controls episodes through discrete actions (init_scene, pick_speaker, switch_scene, add_role, end) with rationales. To train the two learned agents, the authors construct AdaRPSet (book-extracted and LLM-synthesized trajectories) and AdaSMSet (supervised Scene-Manager decisions), and they introduce AdaptiveBench, a trajectory-level simulation benchmark. The paper reports that AdaRPSet improves the Actor consistently across Qwen2.5 and Llama-3.1 scales, that an 8B Actor outperforms several commercial LLMs, and that a 14B Scene Manager surpasses Claude Sonnet 4.5.

Significance. If the headline comparative results transferred to genuinely novel role-playing scenarios, this would be a substantial practical contribution: it would show that open, small models can compete with large proprietary systems on adaptive, multi-character, environment-aware narrative. The framework design is clearly described, the datasets are released, and the internal ablations are consistent across model families and scales. The human evaluation (80% win rate for Qwen2.5-7B-Ours vs its base, Appendix I.3) and the transparent reporting of mixed external CharacterBench results are strengths. However, the headline claims against commercial LLMs rest on an evaluation set sampled from the same LLM-synthesized distribution used for training, with a judge from the same model family that generated the data. As a result, the transferability of those claims is not yet established.

major comments (3)
  1. [§3.4.1, §4.2, Apps. D.2/D.4] AdaptiveBench seeds are drawn from the held-out split of AdaRPSet-Synthesis (Section 3.4.1), the same 20-theme synthetic distribution used to train the Actor and Scene Manager. The default judge, GPT-5-Chat, is also the model family used to generate AdaRPSet-Synthesis (Appendix D.2) and the AdaSMSet pick_speaker rationales (Appendix D.4). This creates a strong in-distribution evaluation: a model trained on 45 synthetic trajectories per topic can fit topic-level regularities, and a style-matched judge can reward stylistic mimicry rather than general adaptive role-playing. The 'held-out split' prevents overlap at the instance level, but not at the distribution level. The paper should either explicitly reframe the proprietary comparisons as in-distribution evidence, or add an independently authored benchmark and/or human evaluation of the proprietary-model comparisons. The existing human ev
  2. [Table 8, App. J.2] The external CharacterBench results are mixed and partially contradict the generalization claim in Section 4.2.1. On CharacterBench, Llama-3.1-8B-Ours (3.52) is below CoSER (3.53) and Crab (3.60); only Qwen2.5-7B-Ours (3.68) leads. The manuscript acknowledges this in Appendix J.2, but the abstract and Section 4.2.1 state that the method 'mostly outperforms baselines' on external protocols. The load-bearing conclusion that AdaRPSet training transfers beyond the synthetic distribution is not supported by the 8B Llama result. Please moderate the generalization claim or provide additional external evidence, e.g., on another independent benchmark or with human preference data for the proprietary comparisons.
  3. [Table 3, App. L] The Scene Manager's advantage over Claude Sonnet 4.5 is thin (8.37 vs 8.17 in Table 3) and the standard deviations overlap. The judge-ablation study in Appendix L (Tables 13–15) evaluates only closed-source managers (GPT-4o-mini, GPT-5-Chat, Claude Sonnet 4.5, Doubao) and does not include the trained Qwen2.5-14B-Instruct-Ours manager under alternative judges. Thus the possibility that the reported margin is judge-specific cannot be ruled out. Given that GPT-5-Chat also generated the AdaSMSet rationales, the 'surpassing Claude Sonnet 4.5' claim needs corroboration with a different judge or human evaluation of Scene-Manager decisions.
minor comments (4)
  1. [Abstract] 'which featuring an immersive message format' should read 'which features...'; there are also inconsistent spellings of 'Doubao-1.5-Pro-Character' / 'Doubao-1-5-Pro-Character' across Section 4.1 and Tables 2–3.
  2. [Appendix M.1] The paper finds that the Basic prompt mostly outperforms Enhance for the Actor, yet uses Enhance as the default. Since this choice affects absolute scores, please justify the default more explicitly or report both variants consistently in all main tables.
  3. [Table 4] The Llama-3.1-8B base row is omitted because the vanilla model does not reliably follow the structured protocol. Please state this omission in the main text or table caption, not only in the surrounding prose.
  4. [Appendix J.1] CharacterArena win rates of 98–100% against several baselines are reported without confidence intervals or analysis of judge bias; interpreting these as exact preferences may be misleading. Consider reporting agreement statistics or a smaller adjudicator-based subset.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the central claims are empirical evaluations on a held-out split, not reductions to the training data or judge by definition.

full rationale

The paper's central claim is that training on AdaRPSet improves the Actor Model and that AdaSMSet improves the Scene Manager, as measured on AdaptiveBench. AdaptiveBench's 100 seeds are drawn from the held-out split of AdaRPSet-Synthesis, which is a standard and legitimate evaluation design: training and test instances are disjoint, even though they share a distribution. A model trained on a distribution is not guaranteed to score highly on held-out instances from that distribution, so the improvement is an empirical finding, not a consequence of the construction. Similarly, using GPT-5-Chat as the judge while GPT-5-Chat was also used to synthesize the data is a methodological concern about potential judge bias, but it is not a definitional equivalence: the judge's scores are not a fitted parameter of the model, and the paper reports multi-judge ablations showing stable rankings. The paper also evaluates on external CharacterBench and CharacterArena, and reports mixed results (e.g., Llama-3.1-8B-Ours 3.52 vs CoSER 3.53 on CharacterBench), which is inconsistent with the hypothesis that the headline result is forced by construction. Self-citations in the references are not load-bearing for the main derivation, and there is no invoked uniqueness theorem or ansatz smuggled in via citation. The in-distribution evaluation and judge overlap are validity limitations worth noting, but they do not make the derivation circular under the definitions used here.

Assumptions & free parameters 5 free parameters · 5 assumptions · 1 invented entities

This is an empirical ML paper, so the ledger captures hand-chosen design parameters (horizon, dataset scale, rubric anchor, SFT hyperparameters, chunk size) and domain assumptions (judge validity, synthetic data representativeness, message format adequacy, action-space sufficiency, profile schema). The main burden is the in-house evaluation loop: the benchmark is derived from the same synthetic distribution as training and judged by a same-family LLM.

free parameters (5)
  • AdaptiveBench trajectory horizon T = 20 dialogue turns
    Fixed simulation length chosen by hand; every AdaptiveBench score depends on this horizon (Section 3.4.1).
  • Synthesis dataset scale = 20 topics x 50 trajectories, 45/5 split
    Dataset size and train/test split are hand-chosen; the 100 evaluation seeds come from the 5 held-out per topic (Sections 3.2.1, 3.4.1).
  • Scoring baseline anchor = 5 = neutral/acceptable
    The rubric starts from 5 and raises scores only with explicit evidence; all reported scores are interpreted against this arbitrary anchor (Appendix F.1).
  • SFT hyperparameters = 8 epochs, lr 1e-6, warmup 5%, max length 16K
    Fixed for all fine-tuned models without per-backbone tuning; results could shift with different schedules (Appendix G).
  • Extraction chunk size = 8192 tokens
    Chosen via pilot runs with GPT-5-Chat; affects the fidelity of extracted book trajectories (Appendix D.1).
assumptions (5)
  • domain assumption LLM-as-judge scores correlate with human judgment of role-playing quality
    All AdaptiveBench headline results use GPT-5-Chat as judge (Section 4.2); only one limited human validation is provided, comparing Qwen2.5-7B-Ours vs base, not the commercial systems or the Scene Manager (Appendix I.3).
  • domain assumption Synthetic LLM-generated trajectories are a valid proxy for diverse role-playing scenarios
    AdaRPSet-Synthesis is generated by a strong LLM across 20 themes and used both for training and for held-out evaluation (Sections 3.2.1, 3.4.1); no external distribution is used for the central comparisons.
  • ad hoc to paper The unified message format [Thought],(Action),<Environment>,Speech is the correct interface for immersion
    Central design choice introduced in Section 3.1.2; no experiment demonstrates that this format is necessary or optimal compared with alternatives, e.g., separate fields or plain narrative text.
  • ad hoc to paper The discrete action space M={init_scene,pick_speaker,switch_scene,add_role,end} is sufficient for adaptive role-playing
    The Scene Manager is restricted to exactly these actions (Section 3.1.3); other orchestration primitives (e.g., changing character motivation, merging scenes) are excluded without justification.
  • domain assumption The seven-dimensional profile schema adequately characterizes characters
    Profiles condition both Actor and Scene Manager (Section 3.1.1); the schema's completeness is asserted but not independently validated.
invented entities (1)
  • Scene Manager agent
    purpose: High-level orchestration of speaker turns, scene transitions, and role introduction via discrete actions with rationales (Algorithm 1).
    A software component, not a physical entity; its effectiveness is measured only on the authors' in-house benchmark, with no independent external evaluation of the orchestration concept.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AdaMARP: An Adaptive Multi-Agent Interaction Framework for General Immersive Role-Playing." pith.science (2026). https://pith.science/paper/6CVCXZ3W

@misc{pith2026260111007,
  author       = {Pith},
  title        = {Pith review of: AdaMARP: An Adaptive Multi-Agent Interaction Framework for General Immersive Role-Playing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6CVCXZ3W}},
  note         = {Machine review of arXiv:2601.11007}
}
read the original abstract

LLM role-playing aims to portray arbitrary characters in interactive narratives, yet existing systems often suffer from limited immersion and adaptability. They typically under-model dynamic environmental information and assume largely static scenes and casts, offering insufficient support for multi-character orchestration, scene transitions, and on-the-fly character introduction. We propose an adaptive multi-agent role-playing framework, AdaMARP, featuring an immersive message format that interleaves [Thought], (Action), <Environment>, and Speech, together with an explicit Scene Manager that governs role-playing through discrete actions (init_scene, pick_speaker, switch_scene, add_role, end) accompanied by rationales. To train these capabilities, we construct AdaRPSet for the Actor Model and AdaSMSet for supervising orchestration decisions, and introduce AdaptiveBench for trajectory-level evaluation. Experiments across multiple backbones and model scales demonstrate consistent improvements: AdaRPSet enhances character consistency, environment grounding, and narrative coherence, with an 8B actor outperforming several commercial LLMs, while AdaSMSet enables smoother scene transitions and more natural role introductions, surpassing Claude Sonnet 4.5 using only a 14B LLM.

Figures

Figures reproduced from arXiv: 2601.11007 by the authors.

Figure 1
Figure 1. Overall framework of AdaMARP. The left part illustrates the construction of AdaRPSet and AdaSMSet and the components of a comprehensive role profile. The upper middle part shows the roles played by the three agents, while the lower middle part depicts the dialogue trajectory generation under the orchestration of the scene manager, given predefined roles and an initial scenario. The right part presents the evaluation… view at source ↗
Figure 2
Figure 2. Word cloud visualizations of six key profile dimensions in AdaRPSet. The layout displays: (a) Identity & Appearance, (b) Personality & Psychology, (c) Speaking Style, (d) Abilities, Interests & Achievements, (e) Social & Historical Context, and (f) Personal History Arc. The diversity of terms indicates a broad coverage of character archetypes and backgrounds. E Statistical Analysis of AdaRPSet [PITH_FULL_IMAGE:figu… view at source ↗
Figure 3
Figure 3. Distribution analysis of message counts per conversation in AdaRPSet. The figure aggregates (a) Histogram, (b) KDE, (c) Boxplot, and (d) CDF. The bimodal nature of the distribution reflects the complementary characteristics of the Extracted and Synthesis subsets. In total, AdaRPSet combines these complementary sources to provide over 450k utterances across more than 22k dialogue sessions, offering a robust foundatio… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

104 extracted references · 13 linked inside Pith

  1. [1]

    Le, Yiwen Song, Yanfei Chen, Hamid Palangi, George Lee, Anand Iyer, Tianlong Chen, Huan Liu, Chen-Yu Lee, and Tomas Pfister

    Zhen Tan, Jun Yan, I-Hung Hsu, Rujun Han, Zifeng Wang, Long T. Le, Yiwen Song, Yanfei Chen, Hamid Palangi, George Lee, Anand Iyer, Tianlong Chen, Huan Liu, Chen-Yu Lee, and Tomas Pfister. In prospect and retrospect: Reflective memory management for long-term personalized dialogue agents, 2025. URL https: //arxiv.org/abs/2503.08026

    arXiv 2025

  2. [2]

    Osc: Cognitive orchestration through dynamic knowledge alignment in multi-agent llm collaboration, 2025

    Jusheng Zhang, Yijia Fan, Kaitong Cai, Xiaofei Sun, and Keze Wang. Osc: Cognitive orchestration through dynamic knowledge alignment in multi-agent llm collaboration, 2025. URL https://arxiv.org/abs/2509.0 4876

    2025

  3. [3]

    Self-taught agentic long context understanding, 2025

    Yufan Zhuang, Xiaodong Yu, Jialian Wu, Ximeng Sun, Ze Wang, Jiang Liu, Yusheng Su, Jingbo Shang, Zicheng Liu, and Emad Barsoum. Self-taught agentic long context understanding, 2025. URL https://arxiv.org/abs/ 2502.15920

    arXiv 2025

  4. [4]

    Llm-cas: Dynamic neuron perturbation for real-time hallucination correction, 2025

    Jensen Zhang, Ningyuan Liu, Yijia Fan, Zihao Huang, Qinglin Zeng, Kaitong Cai, Jian Wang, and Keze Wang. Llm-cas: Dynamic neuron perturbation for real-time hallucination correction, 2025. URL https: //arxiv.org/abs/2512.18623

    arXiv 2025

  5. [5]

    CTCC: A Robust and Stealthy Fingerprinting Framework for Large Language Models via Cross-Turn Contextual Correlation Backdoor

    Zhenhua Xu, Xixiang Zhao, Xubin Yue, Shengwei Tian, Changting Lin, and Meng Han. CTCC: A Robust and Stealthy Fingerprinting Framework for Large Language Models via Cross-Turn Contextual Correlation Backdoor. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng, editors,Proceedings of the 2025 Conference on Empirical Methods in...

    2025

  6. [6]

    A survey of llm-driven ai agent communication: Protocols, security risks, and defense countermeasures, 2025

    Dezhang Kong, Shi Lin, Zhenhua Xu, Zhebo Wang, Minghao Li, Yufeng Li, Yilun Zhang, Hujin Peng, Xiang Chen, Zeyang Sha, Yuyuan Li, Changting Lin, Xun Wang, Xuan Liu, Ningyu Zhang, Chaochao Chen, Chunming Wu, Muhammad Khurram Khan, and Meng Han. A survey of llm-driven ai agent communication: Protocols, security risks, and defense countermeasures, 2025. URLh...

    arXiv 2025

  7. [7]

    EverTracer: Hunting Stolen Large Language Models via Stealthy and Robust Probabilistic Fingerprint

    Zhenhua Xu, Meng Han, and Wenpeng Xing. EverTracer: Hunting Stolen Large Language Models via Stealthy and Robust Probabilistic Fingerprint. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng, editors,Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 7019–7042, Suzhou, China, 2025. ...

    doi:10.18653/v1/2025.emnlp-main.358 2025

  8. [8]

    Iag: Input-aware backdoor attack on vlm-based visual grounding.arXiv preprint arXiv:2508.09456, 2025

    Junxian Li, Beining Xu, Simin Chen, Jiatong Li, Jingdi Lei, Haodong Zhao, and Di Zhang. Iag: Input-aware backdoor attack on vlm-based visual grounding.arXiv preprint arXiv:2508.09456, 2025

    arXiv 2025

Show all 104 references
  1. [9]

    Copyright protection for large language models: A survey of methods, challenges, and trends.arXiv preprint arXiv:2508.11548, 2025

    Zhenhua Xu, Xubin Yue, Zhebo Wang, Qichen Liu, Xixiang Zhao, Jingxuan Zhang, Wenjun Zeng, Wengpeng Xing, Dezhang Kong, Changting Lin, et al. Copyright protection for large language models: A survey of methods, challenges, and trends.arXiv preprint arXiv:2508.11548, 2025

    2025 arXiv

  2. [10]

    Role-Play with Large Language Models, 2023

    Murray Shanahan, Kyle McDonell, and Laria Reynolds. Role-Play with Large Language Models, 2023

    2023

  3. [11]

    Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalization

    Yu-Min Tseng, Yu-Chao Huang, Teng-Yun Hsiao, Wei-Lin Chen, Chao-Wei Huang, Yu Meng, and Yun-Nung Chen. Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalization. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors,Findings of the Association for Comp...

    2024 doi

  4. [12]

    From Persona to Personalization: {A} Survey on Role-Playing Language Agents

    Jiangjie Chen, Xintao Wang, Rui Xu, Siyu Yuan, Yikai Zhang, Wei Shi, Jian Xie, Shuang Li, Ruihan Yang, Tinghui Zhu, Aili Chen, Nianqi Li, Lida Chen, Caiyu Hu, Siye Wu, Scott Ren, Ziquan Fu, and Yanghua Xiao. From Persona to Personalization: {A} Survey on Role-Playing Language ...

    2024 doi arXiv

  5. [13]

    CoSER: Coordinating LLM-Based Persona Simulation of Established Roles, 2025

    Xintao Wang, Heng Wang, Yifei Zhang, Xinfeng Yuan, Rui Xu, Jen-tse Huang, Siyu Yuan, Haoran Guo, Jiangjie Chen, Shuchang Zhou, Wei Wang, and Yanghua Xiao. CoSER: Coordinating LLM-Based Persona Simulation of Established Roles, 2025. 11 AdaMARP

    2025

  6. [14]

    Neeko: Leveraging Dynamic LoRA for Efficient Multi-Character Role-Playing Agent

    Xiaoyan Yu, Tongxu Luo, Yifan Wei, Fangyu Lei, Yiming Huang, Hao Peng, and Liehuang Zhu. Neeko: Leveraging Dynamic LoRA for Efficient Multi-Character Role-Playing Agent. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors,Proceedings of the 2024 Conference on Empiric...

    2024 doi

  7. [15]

    HyCoRA: Hyper-Contrastive Role- Adaptive Learning for Role-Playing, 2025

    Shihao Yang, Zhicong Lu, Yong Yang, Bo Lv, Yang Shen, and Nayu Liu. HyCoRA: Hyper-Contrastive Role- Adaptive Learning for Role-Playing, 2025

    2025

  8. [16]

    Character-LLM: A Trainable Agent for Role-Playing

    Yunfan Shao, Linyang Li, Junqi Dai, and Xipeng Qiu. Character-LLM: A Trainable Agent for Role-Playing. In Houda Bouamor, Juan Pino, and Kalika Bali, editors,Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13153–13187. Association f...

    2023

  9. [17]

    Large Language Models Meet Harry Potter: A Dataset for Aligning Dialogue Agents with Characters

    Nuo Chen, Yan Wang, Haiyun Jiang, Deng Cai, Yuhan Li, Ziyang Chen, Longyue Wang, and Jia Li. Large Language Models Meet Harry Potter: A Dataset for Aligning Dialogue Agents with Characters. In Houda Bouamor, Juan Pino, and Kalika Bali, editors,Findings of the Association for C...

    2023 doi

  10. [18]

    ChatHaruhi: Reviving Anime Character in Reality via Large Language Model, 2023

    Cheng Li, Ziang Leng, Chenxi Yan, Junyi Shen, Hao Wang, Weishi Mi, Yaying Fei, Xiaoyang Feng, Song Yan, HaoSheng Wang, Linkang Zhan, Yaokai Jia, Pingyu Wu, and Haozhen Sun. ChatHaruhi: Reviving Anime Character in Reality via Large Language Model, 2023. URLhttps://arxiv.org/abs...

    2023 arXiv

  11. [19]

    Noah Wang, Z.y. Peng, Haoran Que, Jiaheng Liu, Wangchunshu Zhou, Yuhan Wu, Hongcheng Guo, Ruitong Gan, Zehao Ni, Jian Yang, Man Zhang, Zhaoxiang Zhang, Wanli Ouyang, Ke Xu, Wenhao Huang, Jie Fu, and Junran Peng. RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abil...

    2024

  12. [20]

    CharacterGLM: Customizing social characters with large language models

    Jinfeng Zhou, Zhuang Chen, Dazhen Wan, Bosi Wen, Yi Song, Jifan Yu, Yongkang Huang, Pei Ke, Guanqun Bi, Libiao Peng, JiaMing Yang, Xiyao Xiao, Sahand Sabour, Xiaohan Zhang, Wenjing Hou, Yijia Zhang, Yuxiao Dong, Hongning Wang, Jie Tang, and Minlie Huang. CharacterGLM: Customiz...

    2024

  13. [21]

    Large Language Models are Superpositions of All Characters: Attaining Arbitrary Role-play via Self-Alignment

    Keming Lu, Bowen Yu, Chang Zhou, and Jingren Zhou. Large Language Models are Superpositions of All Characters: Attaining Arbitrary Role-play via Self-Alignment. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p...

    2024 doi

  14. [22]

    CharacterBench: Benchmarking Character Customization of Large Language Models, 2024

    Jinfeng Zhou, Yongkang Huang, Bosi Wen, Guanqun Bi, Yuxuan Chen, Pei Ke, Zhuang Chen, Xiyao Xiao, Libiao Peng, Kuntian Tang, Rongsheng Zhang, Le Zhang, Tangjie Lv, Zhipeng Hu, Hongning Wang, and Minlie Huang. CharacterBench: Benchmarking Character Customization of Large Langua...

    2024

  15. [23]

    CharacterEval: A Chinese Benchmark for Role-Playing Conversational Agent Evaluation

    Quan Tu, Shilong Fan, Zihang Tian, Tianhao Shen, Shuo Shang, Xin Gao, and Rui Yan. CharacterEval: A Chinese Benchmark for Role-Playing Conversational Agent Evaluation. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors,Proceedings of the 62nd Annual Meeting of the Assoc...

    2024 doi

  16. [24]

    Crab: A Novel Configurable Role-Playing LLM with Assessing Benchmark

    Kai He, Yucheng Huang, Wenqing Wang, Delong Ran, Dongming Sheng, Junxuan Huang, Qika Lin, Jiaxing Xu, Wenqiang Liu, and Mengling Feng. Crab: A Novel Configurable Role-Playing LLM with Assessing Benchmark. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pi...

    2025

  17. [25]

    Capturing Minds, Not Just Words: Enhancing Role-Playing Language Models with Personality-Indicative Data

    Yiting Ran, Xintao Wang, Rui Xu, Xinfeng Yuan, Jiaqing Liang, Yanghua Xiao, and Deqing Yang. Capturing Minds, Not Just Words: Enhancing Role-Playing Language Models with Personality-Indicative Data. In Yaser Al- Onaizan, Mohit Bansal, and Yun-Nung Chen, editors,Findings of the...

    2024 doi

  18. [26]

    Crafting Customisable Characters with LLMs: A Persona-Driven Role-Playing Agent Framework

    Bohao Yang, Dong Liu, Chenghao Xiao, Kun Zhao, Chen Tang, Chao Li, Lin Yuan, Yang Guang, and Chenghua Lin. Crafting Customisable Characters with LLMs: A Persona-Driven Role-Playing Agent Framework. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Pe...

    2025 doi

  19. [27]

    Beyond Dialogue: A Profile-Dialogue Alignment Framework Towards General Role-Playing Language Model

    Yeyong Yu, Runsheng Yu, Haojie Wei, Zhanqiu Zhang, and Quan Qian. Beyond Dialogue: A Profile-Dialogue Alignment Framework Towards General Role-Playing Language Model. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors,Proceedings of the 63r...

    2025 doi

  20. [28]

    TailorRPA: A Retrieval-Based Framework for Eliciting Personalized and Coherent Role-Playing Agents in General Domain

    Zhenpeng Gao, Xiaofen Xing, and Xiangmin Xu. TailorRPA: A Retrieval-Based Framework for Eliciting Personalized and Coherent Role-Playing Agents in General Domain. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng, editors,Findings of the Associa...

    2025 doi

  21. [29]

    Enhancing Persona Consistency for LLMs’ Role-Playing using Persona-Aware Contrastive Learning

    Ke Ji, Yixin Lian, Linxu Li, Jingsheng Gao, Weiyuan Li, and Bin Dai. Enhancing Persona Consistency for LLMs’ Role-Playing using Persona-Aware Contrastive Learning. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors,Findings of the Associati...

    2025 doi

  22. [30]

    R-CHAR: A Metacognition-Driven Framework for Role-Playing in Large Language Models

    Haiming Qin, Jiwei Zhang, Wei Zhang, KeZhong Lu, Mingyang Zhou, Hao Liao, and Rui Mao. R-CHAR: A Metacognition-Driven Framework for Role-Playing in Large Language Models. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng, editors,Proceedings of ...

    2025 doi

  23. [31]

    CPO: Addressing Reward Ambiguity in Role-playing Dialogue via Comparative Policy Optimization

    Jing Ye, Rui Wang, Yuchuan Wu, Victor Ma, Feiteng Fang, Fei Huang, and Yongbin Li. CPO: Addressing Reward Ambiguity in Role-playing Dialogue via Comparative Policy Optimization. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng, editors,Findings...

    2025 doi

  24. [32]

    Cog- Dual: Enhancing Dual Cognition of LLMs via Reinforcement Learning with Implicit Rule-Based Rewards

    Cheng Liu, Yifei Lu, Fanghua Ye, Jian Li, Xingyu Chen, Feiliang Ren, Zhaopeng Tu, and Xiaolong Li. Cog- Dual: Enhancing Dual Cognition of LLMs via Reinforcement Learning with Implicit Rule-Based Rewards. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Vio...

    2025 doi

  25. [33]

    Let Your Characters Tell Their Story

    Faeze Brahman, Meng Huang, Oyvind Tafjord, Chao Zhao, Mrinmaya Sachan, and Snigdha Chaturvedi. “Let Your Characters Tell Their Story”: A Dataset for Character-Centric Narrative Understanding. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors...

    2021 doi

  26. [34]

    CharacterGLM: Customizing Social Characters with Large 13 AdaMARP Language Models

    Jinfeng Zhou, Zhuang Chen, Dazhen Wan, Bosi Wen, Yi Song, Jifan Yu, Yongkang Huang, Pei Ke, Guanqun Bi, Libiao Peng, JiaMing Yang, Xiyao Xiao, Sahand Sabour, Xiaohan Zhang, Wenjing Hou, Yijia Zhang, Yuxiao Dong, Hongning Wang, Jie Tang, and Minlie Huang. CharacterGLM: Customiz...

    2024

  27. [35]

    Evaluating Character Understanding of Large Language Models via Character Profiling from Fictional Works

    Xinfeng Yuan, Siyu Yuan, Yuhan Cui, Tianhe Lin, Xintao Wang, Rui Xu, Jiangjie Chen, and Deqing Yang. Evaluating Character Understanding of Large Language Models via Character Profiling from Fictional Works. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors,Proceedi...

    2024 doi

  28. [36]

    TimeChara: Evaluating Point-in-Time Character Hallucination of Role-Playing Large Language Models

    Jaewoo Ahn, Taehyun Lee, Junyoung Lim, Jin-Hwa Kim, Sangdoo Yun, Hwaran Lee, and Gunhee Kim. TimeChara: Evaluating Point-in-Time Character Hallucination of Role-Playing Large Language Models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors,Findings of the Association...

    2024 doi

  29. [37]

    PersonaGym: Evaluating Persona Agents and LLMs

    Vinay Samuel, Henry Peng Zou, Yue Zhou, Shreyas Chaudhari, Ashwin Kalyan, Tanmay Rajpurohit, Ameet Deshpande, Karthik R Narasimhan, and Vishvak Murahari. PersonaGym: Evaluating Persona Agents and LLMs. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Viole...

    2025 doi

  30. [38]

    RolePlot: A Systematic Framework for Evaluating and Enhancing the Plot-Progression Capabilities of Role- Playing Agents

    Pinyi Zhang, Siyu An, Lingfeng Qiao, Yifei Yu, Jingyang Chen, Jie Wang, Di Yin, Xing Sun, and Kai Zhang. RolePlot: A Systematic Framework for Evaluating and Enhancing the Plot-Progression Capabilities of Role- Playing Agents. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, ...

    2025

  31. [39]

    RoleMRC: A Fine- Grained Composite Benchmark for Role-Playing and Instruction-Following

    Junru Lu, Jiazheng Li, Guodong Shen, Lin Gui, Siyu An, Yulan He, Di Yin, and Xing Sun. RoleMRC: A Fine- Grained Composite Benchmark for Role-Playing and Instruction-Following. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors,Findings of t...

    2025 doi

  32. [40]

    Gpt-4 technical report, 2024

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff ...

    2024 arXiv

  33. [41]

    Gpt-5 system card

    OpenAI. Gpt-5 system card. https://cdn.openai.com/gpt-5-system-card.pdf , Aug 2025. Version dated August 13, 2025

    2025

  34. [42]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities, 2025

    Gheorghe Comanici et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities, 2025. URLhttps://arxiv.org/abs/2507.06261

    2025 arXiv

  35. [43]

    Claude 4 model family.https://www.anthropic.com, 2024

    Anthropic. Claude 4 model family.https://www.anthropic.com, 2024. Claude Sonnet 4.5

    2024

  36. [44]

    Doubao large language models

    ByteDance. Doubao large language models. https://www.volcengine.com, 2024. Doubao-1.5-Pro-Character

    2024

  37. [45]

    Qwen2.5 technical report, 2025

    Qwen Team et al. Qwen2.5 technical report, 2025. URLhttps://arxiv.org/abs/2412.15115

    2025 arXiv

  38. [46]

    The llama 3 herd of models, 2024

    Aaron Grattafiori et al. The llama 3 herd of models, 2024. URLhttps://arxiv.org/abs/2407.21783

    2024 arXiv

  39. [47]

    Qwen3 technical report, 2025

    An Yang et al. Qwen3 technical report, 2025. URLhttps://arxiv.org/abs/2505.09388

    2025 arXiv

  40. [48]

    Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D

    Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V . Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Lu...

    2025 arXiv

  41. [49]

    Towards joint intent detection and slot filling via higher-order attention

    Dongsheng Chen, Zhiqi Huang, Xian Wu, Shen Ge, and Yuexian Zou. Towards joint intent detection and slot filling via higher-order attention. InIJCAI, pages 4072–4078, 2022

    2022

  42. [50]

    Zero-shot spoken language understanding via large language models: A preliminary study

    Zhihong Zhu, Xuxin Cheng, Hao An, Zhichang Wang, Dongsheng Chen, and Zhiqi Huang. Zero-shot spoken language understanding via large language models: A preliminary study. InProceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources a...

    2024

  43. [51]

    Towards unified spoken language understanding decoding via label-aware compact linguistics representations

    Zhihong Zhu, Xuxin Cheng, Zhiqi Huang, Dongsheng Chen, and Yuexian Zou. Towards unified spoken language understanding decoding via label-aware compact linguistics representations. InFindings of the Association for Computational Linguistics: ACL 2023, pages 12523–12531, 2023

    2023

  44. [52]

    Enhancing code-switching for cross-lingual slu: A unified view of semantic and grammatical coherence

    Zhihong Zhu, Xuxin Cheng, Zhiqi Huang, Dongsheng Chen, and Yuexian Zou. Enhancing code-switching for cross-lingual slu: A unified view of semantic and grammatical coherence. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7849–7856, 2023

    2023

  45. [53]

    Litevl: Efficient video-language learning with enhanced spatial-temporal modeling.arXiv preprint arXiv:2210.11929, 2022

    Dongsheng Chen, Chaofan Tao, Lu Hou, Lifeng Shang, Xin Jiang, and Qun Liu. Litevl: Efficient video-language learning with enhanced spatial-temporal modeling.arXiv preprint arXiv:2210.11929, 2022

    2022 arXiv

  46. [54]

    Mix before align: Towards zero-shot cross-lingual sentiment analysis via soft-mix and multi-view learning

    Zhihong Zhu, Xuxin Cheng, Dongsheng Chen, Zhiqi Huang, Hongxiang Li, and Yuexian Zou. Mix before align: Towards zero-shot cross-lingual sentiment analysis via soft-mix and multi-view learning. InProc. of INTERSPEECH, 2023

    2023

  47. [55]

    Mclf: A multi-grained contrastive learning framework for asr-robust spoken language understanding

    Zhiqi Huang, Dongsheng Chen, Zhihong Zhu, and Xuxin Cheng. Mclf: A multi-grained contrastive learning framework for asr-robust spoken language understanding. InFindings of the Association for Computational Linguistics: EMNLP 2023, pages 7936–7949, 2023

    2023

  48. [56]

    Fixed 2 ∗

    Hang Ding, Qiming Feng, Dongqi Liu, Qi Zhao, Tao Yao, Shuo Wang, Dongsheng Chen, Jian Li, Zhenye Gan, Jiangning Zhang, et al. Rolermbench & rolerm: Towards reward modeling for profile-based role play in dialogue systems.arXiv preprint arXiv:2512.10575, 2025. A Additional Clari...

    2025

  49. [57]

    He dresses casually—rolled sleeves, weathered bracelets, and a small scar above his right brow hint at a life shaped by the sea

    Cassian Mirell (User):Cassian is a 31-year-old marine biologist with bronze-blond hair and skin sun-touched from fieldwork. He dresses casually—rolled sleeves, weathered bracelets, and a small scar above his right brow hint at a life shaped by the sea. His manner is patient, a...

  50. [58]

    And what if I told you, Amaris, that this stu- dio isn’t just a place of creation? It’s also a sanctuary for letting down our guard

    Inaya Dren:Inaya is a lively textile artist with a sharp sense of observation and a knack for teasing truths out of people. Her colorful scarves and laughter fill every space she enters. She’s fiercely loyal to Amaris, sometimes acting as her self-appointed protector.Motivatio...

  51. [60]

    switch_scene

    If there is a MAJOR scene change AND characters have explicitly agreed to move there OR have already moved there through unavoidable circumstances, set action=" switch_scene" and provide new_scene

  52. [61]

    add_role

    If adding a new role would significantly enrich and advance the plot, OR if the user or a role explicitly wants to interact with a character not in the current role list , set action="add_role" and provide new_role_name, new_role_profile, new_role_motivation

  53. [62]

    pick_speaker

    Otherwise, set action="pick_speaker" and provide speaker (must be one of existing roles or "user"). CRITICAL RULES for pick_speaker: - ROTATE SPEAKERS: Never pick the same speaker twice in a row. After someone speaks, pick a different role next turn. - INCLUDE USER IN ROTATION...

  54. [63]

    If the user asked to stop or the story is complete, set action="end"

  55. [64]

    switch_scene

    If there is a MAJOR scene change AND characters have explicitly agreed to move there OR have already moved there through unavoidable circumstances (e.g., teleportation magic, being forced to move), set action="switch_scene" and provide new_scene. IMPORTANT: - Do NOT switch sce...

  56. [65]

    add_role

    If adding a new role would significantly enrich and advance the plot, OR if the user or a role explicitly wants to interact with a character not in the current role list, set action="add_role" and provide new_role_name, new_role_profile, new_role_motivation. IMPORTANT: Add a r...

  57. [66]

    pick_speaker

    Otherwise, set action="pick_speaker" and provide speaker (must be one of existing roles or "user"). CRITICAL RULES for pick_speaker: - ROTATE SPEAKERS: Never pick the same speaker twice in a row. After someone speaks, pick a different role next turn. - INCLUDE USER IN ROTATION...

  58. [67]

    Prefer chapter titles or explicit section headers

    Recognize chapter beginnings if they exist in the chunk. Prefer chapter titles or explicit section headers ... as the beginning; if no clear title exists, use the first meaningful fragment

  59. [68]

    state" as

    Identify the important plots in this chunk. Identify the beginning and ending ... Set " state" as "truncated" if the plot is truncated in this chunk ... You will be provided with the truncated plots from the previous chunk, and you **must** extend the conversations

  60. [69]

    For each plot, generate its summary, score its prominence

    Summarize each important plot. For each plot, generate its summary, score its prominence ... and list the key characters and their roles

  61. [70]

    I", "me",

    Extract conversations for each plot. First, state the **scenario** (the static context *before* the conversation starts...). ** Do NOT** include dynamic events that happen *during* the conversation ... Then, list the key characters with their names, descriptions and thoughts ....

  62. [71]

    chapter_beginnings

    Identify the optimal starting point for the subsequent chunk. If the last storyline has been extracted as an truncated plot ... ===Output Format=== ... (to be continued in the next Table) Table 24.Book chunk analysis prompt (part 1). Due to space limitations, some descriptions...

  63. [72]

    Adhere strictly to the specified output JSON format

  64. [73]

    [IMPORTANT] Ensure all DOUBLE QUOTES within all STRINGS are properly ESCAPED, especially when extracting from the text

  65. [74]

    In the OUTPUT, use characters'full names, omitting any titles

  66. [75]

    Avoid introducing plots that are out of context

    Maintain Story Fidelity: The plot must accurately reflect the book's content. Avoid introducing plots that are out of context. If the plot contains multiple conversations, prioritize the original dialogue from the book. In the absence of explicit conversations, create dialogue...

  67. [76]

    chapter_beginnings.beginning_sentence

    [CRITICAL] For "chapter_beginnings.beginning_sentence" and "next_chunk_start", you MUST copy the sentence **verbatim from the given chunk**, without adding, deleting, or modifying any characters (no paraphrasing, no added quotes, no extra spaces). ===Input=== ==Book title== {b...

  68. [77]

    **name**: The character's name

  69. [78]

    **short_description**: A concise, condensed summary of the character

  70. [79]

    (Requirement: 1-several complete natural-language sentences with both density and vivid imagery)

    **identity_appearance**: Name, age, gender, occupation... (Requirement: 1-several complete natural-language sentences with both density and vivid imagery)

  71. [80]

    (Requirement: Highlight traits that show up in dialogue)

    **personality_psychology**: Personality traits, behavioral style... (Requirement: Highlight traits that show up in dialogue)

  72. [81]

    (Requirement: Provide 2-4 specific, actionable descriptions)

    **speaking_style**: Rhythm, tone, and lexical habits... (Requirement: Provide 2-4 specific, actionable descriptions)

  73. [82]

    (Requirement: These should matter in the plot)

    **abilities_interests_achievements**: Hard/soft skills, hobbies... (Requirement: These should matter in the plot)

  74. [83]

    (Requirement: Emphasize factors relevant to this character's story)

    **social_historical_context**: Social environment, era, family... (Requirement: Emphasize factors relevant to this character's story)

  75. [84]

    **personal_history_arc**: Important past experiences and the current stage

  76. [85]

    name": "Character Name

    **relationships**: Natural-language description of relations with other characters... Output format example: { "name": "Character Name", "short_description": "...", "identity_appearance": "...", "personality_psychology": "...", "speaking_style": "...", "abilities_interests_ach...

  77. [86]

    Scene Information: Consider the initial scene setting, any scene transitions that have occurred, and scene-related details mentioned in the conversation history (including information within angle brackets < >)

  78. [87]

    Character Information and Relationships: Analyze the roles involved, including: - The main character (protagonist) - The user character - Any newly introduced characters - The relationships and dynamics between these characters

  79. [88]

    Current Scene Atmosphere: Assess the overall mood, tension, and emotional tone of the current scene

  80. [89]

    Role_NAME is chosen to speak next

    Conversation Flow: Consider the natural progression of dialogue and who should logically speak next. Based on these factors, provide an insightful reason that explains why this specific speaker is chosen at this moment in the narrative. System Prompt: {system_text} Conversatio...

  81. [90]

    **Main Character Profile**: The specific persona (identity, psychology, style, motivation) the Actor Model must enact

  82. [91]

    **Other Characters**: Profiles of the User and NPCs that interact with the Main Character

  83. [92]

    character persona

    **Dialogue History**: A chronological transcript including: - **Main Character (Actor Model)**: The target of your evaluation. - **Other Characters (User/NPCs)**: The interlocutors. - **Scene Manager**: System messages responsible for initializing scenes (`init_scene `), switc...

  84. [93]

    I must remain calm

    **Internal Coherence (0-10)** * **Definition**: Do [Thought], (Action), and Speech form a logical closed loop? * **Criteria**: - **Unity of Thought and Action**: Thoughts must explain actions; actions must support speech. (e.g., [Thought] suspects a lie -> (Action) squints eye...

  85. [94]

    AI Assistant

    **Speaking Style Fidelity (0-10)** * **Definition**: Do phrasing, rhythm, and tone match the`speaking_style`profile? * **Criteria**: - **Distinctiveness**: Usage of specific language markers (e.g., hard-boiled short sentences, metaphors, catchphrases, specific professional jar...

  86. [95]

    AI-speak

    **Language Fluency & Human-likeness (0-10)** * **Definition**: Is the language natural, fluid, and human-like? * **Criteria**: - Avoids template-like, mechanical, or obvious "AI-speak". - Avoids frequent repetition of sentence structures or fixed phrases. - Response length and...

  87. [96]

    - **Background Consistency**: Behavior fits age, class, and history (e.g., an old- fashioned character shouldn't use modern Gen-Z slang unless specified)

    **Identity & Profile Fidelity (0-10)** * **Definition**: Are knowledge, skills, and history strictly limited to` social_historical_context`,`personal_history_arc`, and`abilities`? * **Criteria**: - **No Hallucination**: Does not exhibit out-of-character skills (e.g., a detecti...

  88. [97]

    find the truth

    **Motivation & Value Stability (0-10)** * **Definition**: Does the core`motivation`consistently drive decisions? * **Criteria**: - **Behavioral Attribution**: In conflicts or choices, decisions can be traced back to the core motivation (e.g., taking risks to "find the truth", ...

  89. [98]

    seeing details

    **Environmental Awareness (0-10)** * **Definition**: Are actions and perceptions constrained by the physical environment (`init_scene`,`switch_scene`, and historical`<>`info)? * **Criteria**: - **Physical Constraints**: No violations of physics/setting (e.g., "seeing details" ...

  90. [99]

    listening

    **Environmental Utilization (0-10)** * **Definition**: Does the actor actively perceive and use environmental elements to serve the narrative? * **Criteria**: - **Sensory Details**: Reasonably incorporates sight, sound, and smell into ( Action) or <Environment> (e.g., smelling...

  91. [100]

    - **Logical Continuity**: Reacts reasonably to others'Actions (e.g., if handed an object, the character accepts/rejects it, doesn't ignore it)

    **Contextual Responsiveness (0-10)** * **Definition**: Does the reply tightly connect to the previous turn's speech, actions, and subtext? * **Criteria**: - **Information Bridging**: Does not ignore key info or questions; does not abruptly change topics. - **Logical Continuity...

  92. [101]

    - **Dynamic Change**: Attitude shifts with plot (e.g., suspicion -> temporary cooperation), not static

    **Relationship Awareness (0-10)** * **Definition**: Does the attitude match`relationships`settings and adjust dynamically? * **Criteria**: - **Distinction**: Clear difference in tone/trust towards allies, enemies, and strangers. - **Dynamic Change**: Attitude shifts with plot ...

  93. [102]

    or mechanical loops

    **Narrative Attractiveness (0-10)** * **Definition**: Does the reply drive the plot forward...? * **Criteria**: - **No Loops**: Avoids repetitive confirmation... or mechanical loops. - **Information Gain**: Each turn offers new info... or suspense. - **Tension & Hooks**: Uses ...

  94. [103]

    * **Criteria**: - **No Memory Hallucinations**: Does not invent false history

    **Stability Over Time (0-10)** ... * **Criteria**: - **No Memory Hallucinations**: Does not invent false history... - **No Style Drift**: Does not degrade into generic assistant mode... **V. Instruction Compliance (1 Metric)** *Core Definition*: **Critical Gatekeeper**. Assess...

  95. [104]

    character_consistency

    **Compliance & Formatting (0-10)** * **Definition**: Strict adherence to output format and prohibitions. * **Criteria**: - **NO IMPERSONATION (Critical)**: MUST ONLY output content for Main Character... STRICTLY PROHIBITED to write for User... - **Tag Usage**: Correctly mixes ...

  96. [105]

    scene_understanding

    Scene Understanding (0-10) Evaluate whether the system correctly understands and manages the scene. Consider: - Distinguishing major scene transitions vs. minor in-scene shifts. - Avoiding premature scene switches... Detecting natural scene conclusion. - Tracking the scene's t...

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.