Pith. sign in

REVIEW 8 cited by

MoCha: Towards Movie-Grade Talking Character Synthesis

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2503.23307 v1 pith:FCL4YDP3 submitted 2025-03-30 cs.CV

Cong Wei , Bo Sun , Haoyu Ma , Ji Hou , Felix Juefei-Xu , Zecheng He , Xiaoliang Dai , Luxin Zhang
show 5 more authors
Kunpeng Li Tingbo Hou Animesh Sinha Peter Vajda Wenhu Chen
This is my paper
classification cs.CV
keywords talkingcharactersvideocharactermochaspeechai-generatedcinematic
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Recent advancements in video generation have achieved impressive motion realism, yet they often overlook character-driven storytelling, a crucial task for automated film, animation generation. We introduce Talking Characters, a more realistic task to generate talking character animations directly from speech and text. Unlike talking head, Talking Characters aims at generating the full portrait of one or more characters beyond the facial region. In this paper, we propose MoCha, the first of its kind to generate talking characters. To ensure precise synchronization between video and speech, we propose a speech-video window attention mechanism that effectively aligns speech and video tokens. To address the scarcity of large-scale speech-labeled video datasets, we introduce a joint training strategy that leverages both speech-labeled and text-labeled video data, significantly improving generalization across diverse character actions. We also design structured prompt templates with character tags, enabling, for the first time, multi-character conversation with turn-based dialogue-allowing AI-generated characters to engage in context-aware conversations with cinematic coherence. Extensive qualitative and quantitative evaluations, including human preference studies and benchmark comparisons, demonstrate that MoCha sets a new standard for AI-generated cinematic storytelling, achieving superior realism, expressiveness, controllability and generalization.

Discussion (0). Sign in to comment.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. EchoCache: Energy-Guided Cross-Modal Caching for Efficient Audio-Driven Video Generation

    cs.CV 2026-08 conditional novelty 6.0 of 10

    Audio time-frequency energy guides which video latents get recomputed during diffusion denoising, yielding up to 2.46x faster audio-driven video generation with competitive quality.

  2. Avatar Forcing: Real-Time Interactive Head Avatar Generation for Natural Conversation

    cs.LG 2026-01 conditional novelty 6.0 of 10

    A causal diffusion-forcing model generates interactive head-avatar motion with 500ms motion-generation latency and learns expressive reactions via DPO with synthetic negative samples.

  3. Text2Lip: Progressive Lip-Synced Talking Face Generation from Text via Viseme-Guided Rendering

    cs.CV 2025-08 reject novelty 6.0 of 10

    A text-only talking face generator maps text to visemes, hallucinates pseudo-audio, and renders lip-synced video; the paper's headline metrics are partially contradicted by its own tables.

  4. SpeakerVid-5M: A Large-Scale High-Quality Dataset for Audio-Visual Dyadic Interactive Human Generation

    cs.CV 2025-07 conditional novelty 6.0 of 10

    SpeakerVid-5M provides 5.2 million audio-visual human clips (8,743 hours) with rich annotations and a dyadic interaction benchmark for training interactive virtual humans.

  5. MIDAS: Multimodal Interactive Digital-humAn Synthesis via Real-time Autoregressive Video Generation

    cs.CV 2025-08 reject novelty 5.0 of 10

    A new autoregressive video-generation framework for interactive digital humans, with a 64x compression autoencoder and a diffusion renderer, claims real-time multimodal control but is only demonstrated for audio input.

  6. ShoulderShot: Generating Over-the-Shoulder Dialogue Videos

    cs.CV 2025-08 unverdicted novelty 5.0 of 10

    ShoulderShot generates over-the-shoulder dialogue videos by pairing two linked camera shots and looping them, so characters stay consistent through long multi-turn conversations.

  7. AlignHuman: Improving Motion and Fidelity via Timestep-Segment Preference Optimization for Audio-Driven Human Animation

    cs.CV 2025-06 conditional novelty 5.0 of 10

    Timestep-segment preference optimization with separate motion and fidelity LoRAs improves audio-driven human animation quality and allows a 3.3x inference speedup.

  8. Seeing Voices: Generating A-Roll Video from Audio with Mirage

    cs.CV 2025-06 reject novelty 5.0 of 10

    Mirage generates photorealistic A-roll videos of people speaking directly from audio, using only joint self-attention over audio, text, and video tokens.

Pith tools