Pith. sign in

REVIEW 4 major objections 4 minor 2 cited by

OmniCharacter: Towards Immersive Role-Playing Agents with Seamless Speech-Language Personality Interaction

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read OmniCharacter claims that a role-playing agent can sustain a character's personality and voice traits through a conversation that freely mixes spoken and written replies, with speech response latency as low as 289ms.

desk verdict Solid dataset and plausible system, but the paper's central claim about dynamic vocal traits is not backed by the evidence. read the letter →

arxiv 2505.20277 v2 pith:DBVBIM3J submitted 2025-05-26 cs.CL cs.CV

Haonan Zhang , Run Luo , Xiong Liu , Yuchuan Wu , Ting-En Lin , Pengpeng Zeng , Qiang Qu , Feiteng Fang
show 5 more authors
Min Yang Lianli Gao Jingkuan Song Fei Huang Yongbin Li
This is my paper
classification cs.CLcs.CV
keywords role-playingagentsspeech-languageinteractionvoicetraitsspeechtokenpredictionconditionalflowmatchinglow-latencyresponseOmniCharacter-10Kmulti-turndialogue
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a role-playing agent can carry a conversation partly in text and partly in spoken audio while keeping the character's personality and voice consistent throughout, at a latency low enough for real-time use. Existing role-playing agents simulate personas in text and ignore voice traits such as tone, emotion, and pauses, which the paper argues are central to immersion. To make the claim concrete, the authors build OmniCharacter, a model that reads text and speech inputs and generates either text or speech replies, and they release a dataset of 20 characters, 10K multi-turn dialogues, and 135K audio responses. Their experiments report better content and style than existing role-playing agents and general speech-language models, with a speech response latency as low as 289ms.

What carries the argument

The central object is the Role Speech Decoder, which wraps the base speech-language model and turns its outputs into speech. It has two load-bearing parts. Role-context Guided Speech Token Prediction takes the LLM's hidden representations $H$, projects them into the space of a small SpeechLLM, and has that SpeechLLM auto-regressively emit discrete speech tokens $O^S$; this is what keeps the audio semantically aligned with the text while avoiding repeated or hallucinated tokens. Role-aware Speech Synthesis then feeds the context $H$, the speech tokens $O^S$, and a speaker embedding $v$ extracted from reference clips of the character into an optimal-transport conditional flow matching model that produces a Mel spectrogram, and a HiFi-GAN vocoder renders the waveform. The speaker embedding $v$ is the designated carrier of the character's vocal identity.

What would settle it

Run a listening test in which raters are given a role profile and a dialogue turn and asked whether the synthesized audio conveys the emotion specified for that turn, and compare the full model against the same pipeline with the character's speaker embedding replaced by a different speaker's embedding. If emotional appropriateness scores do not change when the embedding is swapped, the fixed speaker embedding is not carrying dynamic voice traits and the paper's core claim would be falsified.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that role-specific voice traits can be generated alongside textual responses by splitting the work into two stages: a speech-language collaborative model interprets the conversation and produces text, a lightweight speech LLM predicts discrete speech tokens under the guidance of that text context, and a role-aware vocoder conditioned on a speaker embedding renders the waveform. The paper claims this two-stage design avoids the duplicate tokens and hallucinations that arise when a single LLM tries to predict text and speech tokens together, and that the resulting audio matches each character's voice while remaining aligned in content with the text response. It further claims that adding audio input improves character consistency and conversational ability over the text-only version, and that the synthesized voice is more similar to the original character than the voice cloning of general TTS systems.

Load-bearing premise

The load-bearing premise is that a fixed voice fingerprint captured from a few reference clips of a character, combined with the conversation context, is enough to render that character's changing emotions, pauses, and speaking style in every synthesized reply; if that fingerprint carries only static timbre, the system reduces to ordinary voice cloning and the central 'seamless speech-language personality interaction' claim loses its force.

Editorial extensions

If this is right

  • If the central claim is correct, a role-playing agent can switch between typed and spoken replies within one conversation without losing its character's persona or voice, which is what separates immersion-focused NPCs and virtual assistants from text-only chat.
  • If correct, the reported 289ms speech response latency puts character-consistent spoken dialogue within reach of real-time interactive use rather than offline generation.
  • If correct, the OmniCharacter-10K dataset provides a reusable testbed with profile-, dialogue-, and audio-level annotations for comparing future speech-language role-playing agents.
  • If correct, the finding that audio input improves text-side character consistency implies that hearing the user speak contributes to understanding who they are and what they want, not only to producing audio.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: a natural extension is to condition speech synthesis on turn-level emotion annotations rather than only on a fixed speaker embedding; if dynamic emotional coloring matters, such conditioning should improve expressiveness scores in listening tests.
  • Editorial inference: the two-stage design invites reuse for other multimodal generation tasks where a lightweight decoder needs to stay aligned with a frozen LLM's hidden representations.
  • Editorial inference: because the dataset covers 10 Chinese and 10 English video-game characters, a testable follow-up is whether the model can clone a new character's voice from a short reference clip and profile without retraining.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The manuscript introduces OmniCharacter, a role-playing agent that accepts text and speech inputs and outputs text or speech responses in a character's voice. It combines a speech-language collaborative model built on Qwen2.5-7B-Instruct with a role speech decoder: a SpeechLLM predicts speech tokens conditioned on LLM hidden states, and an OT-CFM module synthesizes a Mel spectrogram conditioned on hidden states, a speaker embedding extracted from reference audio, and predicted speech tokens; a HiFi-GAN vocoder produces the waveform. The authors also release OmniCharacter-10K, a dataset of 20 game characters, 10,072 multi-turn dialogues, and 135K audio responses synthesized with TTS. Experiments compare against text RPAs on CharacterEval and SocialBench, against speech-language models on the OmniCharacter-10K test split (GPT-4o scoring and five-expert human scoring), and against ASR/TTS systems on LibriSpeech, AISHELL-2, and LibriTTS. The paper claims better content and style than baselines and a 289 ms response latency.

Significance. The strongest parts are the scale and breadth of the effort: a reproducible dataset, a concrete two-stage training recipe, and evaluations on external benchmarks (CharacterEval) and a held-out split of the new dataset. If the dynamic-expressiveness claim were established, the system would be a meaningful step toward speech-language role-playing agents. The paper also makes a useful attempt to ablate audio input and to measure voice similarity. The current evidence, however, does not yet support the central claim that context-dependent vocal traits (emotions, pauses, speaking style) are generated rather than a static cloned timbre. The absence of significance tests and agreement metrics further weakens the quantitative claims.

major comments (4)
  1. [Section 4.4 and Eq. (7)] The central claim is that the model conveys 'voice traits such as voice style and emotions' turn by turn. The conditioning in Eq. (7) includes the speaker embedding v, LLM context H, and predicted speech tokens O^S, so dynamic expressiveness is architecturally possible. However, the experiments in Section 4.4 and Figure 5 measure only static timbre identity (cosine similarity of speaker embeddings), and Section 4.5 shows only that speaker embeddings are discriminable across characters. Neither experiment tests whether H and O^S modulate emotional delivery, pauses, or speaking style. The human evaluation in Table 5 includes an 'Emotional Expression' dimension, but experts score unpaired generated samples and the rubric does not ask whether the emotion matches the current dialogue context or whether the same character varies emotion across turns. No objective prosody/emotion classifier or matched-context control is reported. As written, the results are consistent with ordinary voice cloning plus an LLM, and the 'seamless speech-language personality interaction' claim is therefore not yet established. A matched-context emotion-rating study or a prosody analysis that isolates the contribution of H and O^S would address this gap.
  2. [Table 7] The text states that removing audio 'significantly improves character consistency and conversational ability,' but the reported averages show Conversational Ability actually decreases when audio is included (3.522 without audio vs. 3.516 with audio), while Character Consistency changes by only 0.013 (2.928 vs. 2.941). No variance, confidence intervals, or significance tests are reported, so 'significantly' is unsupported and the conversational-ability claim is contradicted by the table. This is load-bearing because the paper uses this ablation to argue that the audio modality matters for language understanding.
  3. [Tables 5, 9, and 10] The headline human-evaluation results are averaged over five experts, but no inter-annotator agreement (e.g., Fleiss' kappa or Krippendorff's alpha) and no per-item variance or statistical test are reported. With five annotators and 1-10 integer scales, the large margins in Table 5 could still be consistent with high annotator noise, especially on subjective dimensions such as Immersion and Emotional Expression. The claim that OmniCharacter 'reaches the best performance across all dimensions' needs at least agreement metrics and ideally per-item paired comparisons.
  4. [Section 4.4 / Figure 5] The voice-similarity evaluation is at least partially circular. The speaker embedding v used as a conditioning input is extracted from the reference speech, and the WavLLM cosine similarity used for scoring is computed in a speaker-embedding space of the same type. This can inflate identity scores even if the generated speech is otherwise not faithful to the character. The threshold of 0.9 for 'same character' is also unvalidated. An independent speaker-verification model or human same/different listening test would make the voice-cloning result more convincing.
minor comments (4)
  1. [Table 6] The claim of 'comparable performance' on general benchmarks is overstated: the model's LibriTTS WER (7.23) is more than twice CosyVoice's (3.17), and its LibriSpeech test-clean WER (3.26) is higher than Whisper-large-v3 (2.50). Please qualify the claim or report matched-condition comparisons.
  2. [Table 4] The GPT-4o scoring for S2TIF and S2SIF lacks a prompt template, scoring rubric, and correlation with human judgments; a few example scored responses would help.
  3. [Section 3.1] The dataset section describes a 50% ratio for user voice gender assignment, but the paper does not report whether the training/test split preserves this ratio or whether results are sensitive to it.
  4. [References and Appendix A.1] There are citation inconsistencies: the appendix cites 'SpeechGPT (Zhang et al., 2024a)' while the reference list assigns SpeechGPT to (Zhang et al., 2023a) and SpeechGPT-Gen to (Zhang et al., 2024a); also, multiple entries are cited as 'Wang et al., 2024b' with different author groups. Please check and disambiguate these references.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity found; the dynamic-expression claim is under-evidenced but the derivation is not circular.

full rationale

Walking the derivation chain: the model is specified by Eqs. (4)-(7) as an auto-regressive text/speech-token predictor plus a CFM vocoder conditioned on context H, speaker embedding v, and speech tokens O^S. No predicted quantity is defined as a fitted parameter, and no equation reduces to its own input by construction. The main empirical claims are checked on external benchmarks (CharacterEval, SocialBench, LibriSpeech, AISHELL-2, LibriTTS) and on a held-out split of OmniCharacter-10K, with prior RPAs and speech-language models tested under the same protocol. Two minor caveats do not rise to circularity: (1) the speech-language collaborative model and SpeechLLM are initialized from the authors' own OpenOmni, but this initialization is not load-bearing for the role-playing or voice-trait conclusions because the model is fine-tuned and evaluated on external and held-out data; (2) WavLLM is used both to filter synthesized training audio (Sec. 3.1, Step 4: similarity less than 0.8 is rejected) and to measure test-time voice similarity (Sec. 4.4), which is an evaluation-overlap concern, but the model is not trained to optimize WavLLM similarity and the comparison against zero-shot voice-cloning TTS baselines remains meaningful. The paper's real weakness is evidential rather than circular: the role-aware synthesis equation is never ablated to show that H and O^S change emotional delivery or speaking style beyond static timbre carried by v, so the central 'vocal traits' claim is not fully established. That is a validation gap, not a self-referential derivation.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

No new physical entities, forces, or conserved quantities are introduced. The central claim rests on pre-trained components (OpenOmni, GLM-4-Voice, Whisper, Qwen2.5) and on the assumption that the synthetic OmniCharacter-10K pipeline is faithful enough to real role-play. The manually chosen dataset filters and training hyperparameters are the main free choices; their sensitivity is unexamined.

free parameters (5)
  • Quality-filter thresholds in dataset construction = WER <= 10; WavLLM similarity >= 0.8
    Hand-set in Step-4 of dataset construction; determine which synthesized audios are kept. No sensitivity analysis on these thresholds.
  • Dialogue curation criteria = ABAB pattern; length > 3 turns
    Hand-chosen filters that select 9,672 training dialogues; their effect on model performance is not measured.
  • User-voice gender assignment ratio = 50% male / 50% female
    Chosen to balance user speech data; arbitrary.
  • Stage-wise learning rates and epochs = lr 5e-4 then 5e-5; 3 epochs; batch 32
    Training hyperparameters chosen by hand; no search or sensitivity analysis.
  • Speech adaptor frame grouping factor k = not reported
    Equation 2 groups k consecutive frames; k is never stated, yet it controls the speech sequence length fed to the LLM.
assumptions (5)
  • domain assumption Role-specific voice traits are representable in the GLM-4-Voice discrete speech token space plus a static speaker embedding
    Section 2.3 generates speech tokens O^S with SpeechLLM and conditions OT-CFM on speaker embedding v; if the tokenizer discards prosody and emotion, the voice traits cannot be recovered. No analysis of token-level voice information is given.
  • domain assumption Static speaker embedding v is sufficient to control the character's full vocal persona
    Eq. 7 uses v as the only explicit role-identity condition; dynamic emotion and style must come from H and O^S. The paper does not ablate expressiveness, only timbre similarity in Figure 5.
  • domain assumption OpenOmni and GLM-4-Voice pretrained components transfer to role-play with fine-tuning
    Section 4.1 initializes from OpenOmni and GLM-4-Voice; the method inherits their alignment and synthesis limitations.
  • domain assumption Synthetic LLM-generated dialogues and TTS audio are representative of real role-playing interactions
    Section 3.1 pipelines LLM-generated dialogues and synthesized speech; the main speech-language evaluation is on the same pipeline's test split, so real-world transfer is not measured.
  • domain assumption Whisper-large-v3 speech encoding followed by k-frame downsampling preserves enough semantic content
    Section 2.2 compresses speech encoder outputs; if semantic information is lost, dialogue understanding degrades. No analysis of compression loss is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of OmniCharacter: Towards Immersive Role-Playing Agents with Seamless Speech-Language Personality Interaction." pith.science (2026). https://pith.science/paper/DBVBIM3J

@misc{pith2026250520277,
  author       = {Pith},
  title        = {Pith review of: OmniCharacter: Towards Immersive Role-Playing Agents with Seamless Speech-Language Personality Interaction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DBVBIM3J}},
  note         = {Machine review of arXiv:2505.20277}
}
read the original abstract

Role-Playing Agents (RPAs), benefiting from large language models, is an emerging interactive AI system that simulates roles or characters with diverse personalities. However, existing methods primarily focus on mimicking dialogues among roles in textual form, neglecting the role's voice traits (e.g., voice style and emotions) as playing a crucial effect in interaction, which tends to be more immersive experiences in realistic scenarios. Towards this goal, we propose OmniCharacter, a first seamless speech-language personality interaction model to achieve immersive RPAs with low latency. Specifically, OmniCharacter enables agents to consistently exhibit role-specific personality traits and vocal traits throughout the interaction, enabling a mixture of speech and language responses. To align the model with speech-language scenarios, we construct a dataset named OmniCharacter-10K, which involves more distinctive characters (20), richly contextualized multi-round dialogue (10K), and dynamic speech response (135K). Experimental results showcase that our method yields better responses in terms of both content and style compared to existing RPAs and mainstream speech-language models, with a response latency as low as 289ms. Code and dataset are available at https://github.com/AlibabaResearch/DAMO-ConvAI/tree/main/OmniCharacter.

Figures

Figures reproduced from arXiv: 2505.20277 by the authors.

Figure 1
Figure 1. (a) Existing role-playing agents (RPAs) con [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The overview of our OmniCharacter framework. We first build a speech-language collaborative model, a large language model that receives both speech and language inputs for unified modeling. Furthermore, we propose a role speech decoder to synthesize speech responses containing vocal traits of different characters by devising two innovative modules: i) Role-context Guided Speech Token Prediction, which aims to enhanc… view at source ↗
Figure 3
Figure 3. Illustration of OmniCharacter-10K. (a) Dataset Construction Pipeline, which consists of four steps: (1) Characters Profile Creation, (2) Dialogue Generation, (3) Speech Synthesis, and (4) Quality Verification. (b) Dataset Properties, which has three appealing properties of our dataset: (1) Large Vocabulary, the dataset includes 20 characters, 10K multi-turn dialogues, and 135K audio responses, (2) Rich Annotations, … view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Left: Distribution of dialogue turns across samples in the OmniCharacter-10K dataset. Right: Dis￾tribution of audio duration for user speech and character speech respectively. (TTS) model, i.e., VITS (Kim et al., 2021) for the audio synthesis of open-text in dialogue. …
Figure 6
Figure 6. Figure 6 [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: The prompt template used to organize our [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: The prompt template used for human evaluation on OmniCharacter-10K test split. [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. X$^3$-OPD: Distilling Reasoning into Large Audio-Language Models via On-Policy Alignment

    cs.LG 2026-07 conditional novelty 5.0 of 10

    X3-OPD improves audio-grounded reasoning by training the audio student on its own rollouts with token-level teacher feedback, using a three-tier paired text-audio corpus.

  2. VoxRole: A Comprehensive Benchmark for Evaluating Speech-Based Role-Playing Agents

    cs.CL 2025-09 reject novelty 5.0 of 10

    A 65.6-hour movie-dialogue benchmark for spoken role-playing agents, with an evaluation framework whose main judge is also an evaluated model.

Reference graph

Works this paper leans on

42 extracted references · 13 canonical work pages · cited by 2 Pith papers

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Sharifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, and Neil Zeghidour. 2023. https://arxiv.org/abs/2209.03143 Audiolm: a language modeling approach to audio generation . Preprint, arXiv:2209.03143

    arXiv 2023

  4. [4]

    Hongzhan Chen, Hehong Chen, Ming Yan, Wenshen Xu, Xing Gao, Weizhou Shen, Xiaojun Quan, Chenliang Li, Ji Zhang, Fei Huang, et al. 2024. Socialbench: Sociality evaluation of role-playing conversational agents. arXiv preprint arXiv:2403.13679

    arXiv 2024

  5. [5]

    Alexandre D \'e fossez, Laurent Mazar \'e , Manu Orsini, Am \'e lie Royer, Patrick P \'e rez, Herv \'e J \'e gou, Edouard Grave, and Neil Zeghidour. 2024. Moshi: a speech-text foundation model for real-time dialogue. arXiv preprint arXiv:2410.00037

    arXiv 2024

  6. [6]

    Jiayu Du, Xingyu Na, Xuechen Liu, and Hui Bu. 2018. Aishell-2: Transforming mandarin asr research into industrial scale. arXiv preprint arXiv:1808.10583

    arXiv 2018

  7. [7]

    Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Heng Lu, Yexin Yang, Hangrui Hu, Siqi Zheng, Yue Gu, Ziyang Ma, et al. 2024. Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens. arXiv preprint arXiv:2407.05407

    arXiv 2024

  8. [8]

    Qingkai Fang, Shoutao Guo, Yan Zhou, Zhengrui Ma, Shaolei Zhang, and Yang Feng. 2024. Llama-omni: Seamless speech interaction with large language models. arXiv preprint arXiv:2409.06666

    arXiv 2024

Show all 42 references
  1. [9]

    Shujie Hu, Long Zhou, Shujie Liu, Sanyuan Chen, Lingwei Meng, Hongkun Hao, Jing Pan, Xunying Liu, Jinyu Li, Sunit Sivasankaran, et al. 2024. Wavllm: Towards robust and adaptive speech large language model. arXiv preprint arXiv:2404.00656

    2024 arXiv

  2. [10]

    Jaehyeon Kim, Jungil Kong, and Juhee Son. 2021. Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech. In ICML, pages 5530--5540. PMLR

    2021

  3. [11]

    Jinlong Li, Zequn Jie, Xu Wang, Xiaolin Wei, and Lin Ma. 2022. Expansion and shrinkage of localization for weakly-supervised semantic segmentation. NeurIPS, 35:16037--16051

    2022

  4. [12]

    Jinlong Li, Cristiano Saltori, Fabio Poiesi, and Nicu Sebe. 2025. Cross-modal and uncertainty-aware agglomeration for open-vocabulary 3d scene understanding. arXiv preprint arXiv:2503.16707

    2025 arXiv

  5. [13]

    Run Luo, Yunshui Li, Longze Chen, Wanwei He, Ting-En Lin, Ziqiang Liu, Lei Zhang, Zikai Song, Xiaobo Xia, Tongliang Liu, et al. 2024 a . Deem: Diffusion models serve as the eyes of large language models for image perception. arXiv preprint arXiv:2405.15232

    2024 arXiv

  6. [14]

    Run Luo, Ting-En Lin, Haonan Zhang, Yuchuan Wu, Xiong Liu, Min Yang, Yongbin Li, Longze Chen, Jiaming Li, Lei Zhang, et al. 2025. Openomni: Large language models pivot zero-shot omnimodal alignment across language with real-time self-aware emotional speech synthesis. arXiv pre...

    2025

  7. [15]

    Run Luo, Haonan Zhang, Longze Chen, Ting-En Lin, Xiong Liu, Yuchuan Wu, Min Yang, Minzheng Wang, Pengpeng Zeng, Lianli Gao, et al. 2024 b . Mmevol: Empowering multimodal large language models with evol-instruct. arXiv preprint arXiv:2409.05840

    2024 arXiv

  8. [16]

    Openai OpenAI. 2022. Openai: Introducing chatgpt. URL https://openai. com/blog/chatgpt

    2022

  9. [17]

    Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: an asr corpus based on public domain audio books. In ICASSP, pages 5206--5210. IEEE

    2015

  10. [18]

    Joon Sung Park, Joseph O'Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. 2023. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th annual acm symposium on user interface software and technology, pages 1--22

    2023

  11. [19]

    Letian Peng and Jingbo Shang. 2024. Quantifying and optimizing global faithfulness in persona-driven role-playing. arXiv preprint arXiv:2405.07726

    2024 arXiv

  12. [20]

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In ICML, pages 28492--28518. PMLR

    2023

  13. [21]

    Paul K Rubenstein, Chulayuth Asawaroengchai, Duc Dung Nguyen, Ankur Bapna, Zal \'a n Borsos, F \'e lix de Chaumont Quitry, Peter Chen, Dalia El Badawy, Wei Han, Eugene Kharitonov, et al. 2023. Audiopalm: A large language model that can speak and listen. arXiv preprint arXiv:2306.12925

    2023 arXiv

  14. [22]

    Nafis Sadeq, Zhouhang Xie, Byungkyu Kang, Prarit Lamba, Xiang Gao, and Julian McAuley. 2024. Mitigating hallucination in fictional character role-play. arXiv preprint arXiv:2406.17260

    2024 arXiv

  15. [23]

    Yunfan Shao, Linyang Li, Junqi Dai, and Xipeng Qiu. 2023. Character-llm: A trainable agent for role-playing. In EMNLP, pages 13153--13187

    2023

  16. [24]

    Qwen Team. 2024. https://qwenlm.github.io/blog/qwen2.5/ Qwen2.5: A party of foundation models

    2024

  17. [25]

    Quan Tu, Shilong Fan, Zihang Tian, and Rui Yan. 2024. Charactereval: A chinese benchmark for role-playing conversational agent evaluation. arXiv preprint arXiv:2401.01275

    2024 arXiv

  18. [26]

    Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al. 2023 a . Neural codec language models are zero-shot text to speech synthesizers. arXiv preprint arXiv:2301.02111

    2023 arXiv

  19. [27]

    Hui Wang, Siqi Zheng, Yafeng Chen, Luyao Cheng, and Qian Chen. 2023 b . Cam++: A fast and efficient network for speaker verification using context-aware masking

    2023

  20. [28]

    Lei Wang, Jianxun Lian, Yi Huang, Yanqi Dai, Haoxuan Li, Xu Chen, Xing Xie, and Ji-Rong Wen. 2024 a . Characterbox: Evaluating the role-playing capabilities of llms in text-based virtual worlds. arXiv preprint arXiv:2412.05631

    2024 arXiv

  21. [29]

    Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. 2024 b . A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6):186345

    2024

  22. [30]

    Tianrui Wang, Long Zhou, Ziqiang Zhang, Yu Wu, Shujie Liu, Yashesh Gaur, Zhuo Chen, Jinyu Li, and Furu Wei. 2024 c . Viola: Conditional language models for speech recognition, synthesis, and translation. IEEE/ACM Transactions on Audio, Speech, and Language Processing

    2024

  23. [31]

    Xintao Wang, Yunze Xiao, Jen-tse Huang, Siyu Yuan, Rui Xu, Haoran Guo, Quan Tu, Yaying Fei, Ziang Leng, Wei Wang, et al. 2024 d . Incharacter: Evaluating personality fidelity in role-playing agents through psychological interviews. In ACL, pages 1840--1873

    2024

  24. [32]

    Zekun Moore Wang, Zhongyuan Peng, Haoran Que, Jiaheng Liu, Wangchunshu Zhou, Yuhan Wu, Hongcheng Guo, Ruitong Gan, Zehao Ni, Jian Yang, et al. 2023 c . Rolellm: Benchmarking, eliciting, and enhancing role-playing abilities of large language models. arXiv preprint arXiv:2310.00746

    2023 arXiv

  25. [33]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115

    2024 arXiv

  26. [34]

    Tao Yang, Yuhua Zhu, Xiaojun Quan, Cong Liu, and Qifan Wang. 2025. Psyplay: Personality-infused role-playing conversational agents. arXiv preprint arXiv:2502.03821

    2025 arXiv

  27. [35]

    Yeyong Yu, Runsheng Yu, Haojie Wei, Zhanqiu Zhang, and Quan Qian. 2024. Beyond dialogue: A profile-dialogue alignment framework towards general role-playing language model. arXiv preprint arXiv:2408.10903

    2024 arXiv

  28. [36]

    Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron J Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu. 2019. Libritts: A corpus derived from librispeech for text-to-speech. arXiv preprint arXiv:1904.02882

    2019 arXiv

  29. [37]

    Aohan Zeng, Zhengxiao Du, Mingdao Liu, Kedong Wang, Shengmin Jiang, Lei Zhao, Yuxiao Dong, and Jie Tang. 2024. Glm-4-voice: Towards intelligent and human-like end-to-end spoken chatbot. arXiv preprint arXiv:2412.02612

    2024 arXiv

  30. [38]

    Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. 2023 a . Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities. arXiv preprint arXiv:2305.11000

    2023 arXiv

  31. [39]

    Dong Zhang, Xin Zhang, Jun Zhan, Shimin Li, Yaqian Zhou, and Xipeng Qiu. 2024 a . Speechgpt-gen: Scaling chain-of-information speech generation. arXiv preprint arXiv:2401.13527

    2024 arXiv

  32. [40]

    Haonan Zhang, Pengpeng Zeng, Lianli Gao, Xinyu Lyu, Jingkuan Song, and Heng Tao Shen. 2023 b . Spt: Spatial pyramid transformer for image captioning. TCSVT, 34(6):4829--4842

    2023

  33. [41]

    Qinglin Zhang, Luyao Cheng, Chong Deng, Qian Chen, Wen Wang, Siqi Zheng, Jiaqing Liu, Hai Yu, Chaohong Tan, Zhihao Du, et al. 2024 b . Omniflatten: An end-to-end gpt model for seamless voice conversation. arXiv preprint arXiv:2410.17799

    2024 arXiv

  34. [42]

    Jinfeng Zhou, Zhuang Chen, Dazhen Wan, Bosi Wen, Yi Song, Jifan Yu, Yongkang Huang, Libiao Peng, Jiaming Yang, Xiyao Xiao, et al. 2023. Characterglm: Customizing chinese conversational ai characters with large language models. arXiv preprint arXiv:2311.16832

    2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.