Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

ARIG: Autoregressive Interactive Head Generation for Real-time Conversations

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read ARIG proposes a frame-by-frame autoregressive model that generates each interactive head motion in real time by sampling continuous motion values with a diffusion procedure, guided by conversational context and predicted interaction states.

desk verdict A real-time frame-wise AR diffusion system for interactive head generation with solid gains over DIM, but the central comparison to INFP is missing, so the headline claim of better interaction realism needs a matched experiment. read the letter →

arxiv 2507.00472 v1 pith:EKEZZ6OG submitted 2025-07-01 cs.CV

Ying Guo , Xi Liu , Cheng Zhen , Pengfei Yan , Xiaoming Wei This is my paper
classification cs.CV
keywords interactiveheadgenerationautoregressivemotiondiffusionprobabilitymodelingconversationalstateunderstandingdual-trackdual-modalinteractionreal-timeavataranimationdyadicconversationfacialsynthesis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes ARIG, a system that generates a virtual agent's head motions one frame at a time during a live two-person conversation, so the agent can react without waiting for a future audio clip. It argues that earlier interactive-head-generation designs are limited in two ways: clip-wise systems that generate a whole short clip at once are delayed and cannot see enough context, and systems that switch between explicit listener and speaker generators cannot handle interruptions, overlap, and feedback smoothly. ARIG instead treats the next motion as a continuous autoregressive prediction, using a diffusion procedure to sample from the motion distribution rather than choosing from a discrete codebook. Two conditioning modules carry the interaction understanding: one fuses audio and visual motion from both participants over short and long ranges, and one predicts conversational states such as interruption, feedback, and pause to guide the motion. Experiments report that this design improves motion synchronization, lip-sync, realism, and diversity over the compared baselines while running at 31 frames per second.

What carries the argument

The load-bearing mechanism is the continuous autoregressive motion predictor: instead of a softmax over a finite codebook, the next frame's motion is sampled from a diffusion-modeled distribution, with a lightweight MLP denoiser using adaptive normalization to inject conditions. Around this predictor sit three modules. The interactive behavior understanding module splits time into short chunks, learns each participant's own audio-visual behavior, exchanges information between participants through attention, integrates the two tracks, and compresses each chunk into a summary that a causally masked decoder reads over a long-range cache. The conversational state understanding module embeds voice-activity detection signals and the context token, then uses cross-attention to produce a state feature supervised by seven state labels, which is passed on as guidance. The progressive motion prediction module first predicts a coarse motion outline from audio, then refines it with context, state features, and recent motion frames before the diffusion sampler produces the final continuous motion.

What would settle it

Retrain ARIG with the seven state labels randomly permuted while keeping all other inputs identical: if SyncScore, FID, and the visual comparisons barely change, the state module is not doing the claimed work. Separately, measure live per-frame latency from audio arrival to motion output at 25 frames per second; if the average exceeds one frame interval, the real-time claim fails.

Watch

Extended reading notes

Core claim

The central claim is that interactive head generation should be a frame-wise autoregressive problem in continuous space, not a clip-wise generation problem and not an explicit role-switching problem. At each time step, only the previous frames and the agent's current audio are needed to produce the next motion frame; because the motion distribution is represented by a diffusion process, the model samples continuous optical-flow keypoint coordinates instead of quantized codebook indices. To make the sampled motion interactive, an interactive behavior understanding module builds a combined audio-visual summary of both participants' recent behavior and reads longer-range context from cached summaries, while a conversational state understanding module uses voice activity signals plus that context to predict one of seven states—interruption, feedback, pause, and similar—that condition the final progressive motion prediction. The paper's claim is that this combination delivers real-time generation at 31 frames per second with interaction realism—natural handling of interruptions, feedback, and pauses—that clip-wise or explicit-role methods cannot match.

Load-bearing premise

The claimed improvement from conversational-state guidance rests on the seven manually annotated state labels being both accurate and complete; the paper reports the annotation but not how it was done or how well annotators agreed, so noisy or incomplete labels could weaken the effect.

Editorial extensions

If this is right

  • A system built this way can begin generating a response as soon as the current audio frame arrives, so interactive avatars in live calls, games, and virtual meetings no longer wait one clip behind.
  • Because motion values are sampled in continuous space rather than picked from a codebook, subtle expressions are not limited by codebook size.
  • Conversational states such as interruption, feedback, and pause become explicit conditions, so the same spoken words can drive different facial motions depending on the interaction context.
  • The same model handles talking, listening, and interacting without a separate role switcher, because the role is expressed implicitly through inputs and predicted state.
  • Long-range context is kept in compressed chunk summaries, allowing the model to track conversational intent beyond a single short window without exploding memory cost.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to test the seven-state taxonomy itself: conversations also contain laughter, sarcasm, and side reactions that may not fit the voice-activity-derived categories, so a wider or self-supervised state space could push realism further.
  • The manual state annotations are the least-specified link in the pipeline, since no annotation protocol or agreement measure is reported; an easy check is to retrain with shuffled labels and see whether the SyncScore and FID gains from the state module survive.
  • The same non-quantized autoregressive sampler and chunked context compression could apply to full-body avatars, since the method is tied to the motion representation rather than to the head specifically.
  • A deployment caveat is that the 31 fps figure was measured for motion generation in isolation; end-to-end latency including audio capture and rendering could differ, and the real-time argument assumes both participants' audio arrives promptly.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes ARIG, a frame-wise autoregressive framework for interactive head generation. ARIG generates the current frame of the agent's head motion from user and agent audio and motion history, modeling the conditional motion distribution with a continuous diffusion-based sampler instead of discrete codebook indices. It introduces an Interactive Behavior Understanding (IBU) module that summarizes short-range dual-track, dual-modal audio-visual behavior and performs long-range contextual understanding via a cached cis-token, and a Conversation States Understanding (CSU) module that predicts one of seven conversational states to guide motion generation. The authors report quantitative improvements over DIM on RealTalk, inherited comparisons on ViCo, qualitative comparisons with INFP, and talking-head results on HDTF, along with ablations and a small user study.

Significance. If the claims are substantiated, the paper would make a useful contribution by demonstrating a real-time, frame-wise alternative to clip-wise interactive head generation, with continuous autoregressive modeling via diffusion as an interesting technical idea. The inference speed of 31 fps and the modular design are concrete strengths. The ablations for long-range context, visual modality, continuous AR, and bidirectional-integrated learning are informative and broadly support the usefulness of these components internally. The main significance is currently limited by the lack of a controlled quantitative comparison with INFP, which is the most relevant prior method for the paper's motivating argument; the available evidence is qualitative or inherited, so the headline claim of 'better interaction realism' over clip-wise methods is not yet established.

major comments (3)
  1. [§4.2.2] The central claim that ARIG achieves better interaction realism than the clip-wise paradigm, specifically INFP, is not quantitatively supported on the primary dataset. The paper states in §4.2.2 that 'Since INFP[39] is not open-source, we only retrained DIM[27] on our dataset and present the quantitative comparisons between our method and DIM[27]' (Table 1). The only INFP comparisons are qualitative in §4.2.1 (two videos from INFP's website) and the ViCo numbers in Table 2, where INFP's results are 'inherited' from [27] and [39] rather than recomputed under the same protocol, and INFP's pose RPCC is missing. Because the motivation for ARIG is specifically to outperform clip-wise generation, the absence of a matched quantitative comparison with INFP leaves the headline claim untested. Please provide a quantitative comparison on RealTalk (e.g., using the INFP authors' released code/checkpoints if available, or a faithful reimplementation), or explicitly revise the claim to say the comparison is qualitative and that DIM is the only quantitative baseline on the interactive task.
  2. [§4.1] The CSU module is one of the paper's two core technical contributions, and the ablation in Table 1 shows that removing conversational state guidance lowers SyncScore from 7.218 to 6.973 and FID from 21.64 to 22.96. However, the paper does not describe how the 'additional state annotations on the training dataset' (§4.1) were collected. No information is given about the number of annotators, the annotation interface, the label distribution of the seven states, or inter-annotator agreement. If the labels are noisy or the taxonomy is incomplete, the state guidance could be learning annotation artifacts rather than genuine conversational states. Add a description of the annotation protocol and agreement statistics, or provide a robustness analysis (e.g., training on only high-agreement labels).
  3. [§4.4 and Appendix B.2] Several ablations reported in Table 1 and Table 4 are based on single runs, and some differences are small relative to typical metric variability. For example, the conversational-state ablation changes CSIM only from 0.901 to 0.892, and RPCC from 0.125 to 0.127, while the long-range-context ablation changes SyncScore from 7.218 to 7.128. Without multiple seeds, standard deviations, or significance tests, the paper does not demonstrate that these differences are stable. The user study in Appendix B.2 similarly reports mean scores for DIM and ARIG but gives no significance testing, despite the modest sample size (25 participants, 20 videos). Report results over at least three seeds with variance, or perform statistical tests on the key metrics.
minor comments (5)
  1. [Abstract and §3.4] The terms 'non-vector-quantized' and 'non-quantized' are used interchangeably; pick one and define it at first use.
  2. [§3.1, Eq. (2)] The notation is inconsistent with Eq. (1). Eq. (1) includes A^a_0~T as an input, but Eq. (2) only shows A^a_T, A^u_{T-1}, M^a_{T-1}, M^u_{T-1}. Clarify how the history of A^a is represented in the 'historical cache' and why Eq. (2) omits it.
  3. [Figure 2] The caption and diagram use A^u_{T-1} but the equation uses A^a_T; check whether the variable naming reflects the flow correctly.
  4. [§4.2.2, Table 4 caption] There is a duplicated article in 'the the cosine similarity' and a typo 'Comparision' should be 'Comparison'.
  5. [§3.4] The phrase 'we preform the fine-grained prediction' should be 'we perform the fine-grained prediction'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: ARIG's claims are empirical and self-contained; the central result does not reduce to its inputs by construction.

full rationale

The paper is an empirical system paper and contains no derivation chain whose conclusions are equivalent to its inputs. The autoregressive formulation (Eqs. 1-2), the diffusion denoising loss (Eq. 3), and the progressive motion prediction are standard architectural choices trained on external data. The CSU module predicts conversational states from voice activity signals and IBU context using seven manually annotated state labels on the training data; the state feature is a learned intermediate condition, not a re-use of the target motion as an input, so it does not constitute self-definitional circularity. The only self-citation is to the authors' prior CustomListener work in the related-work survey, and it is not load-bearing for any claimed result. The absence of a direct quantitative comparison with INFP on RealTalk, and the reliance on inherited ViCo numbers for INFP, is a substantive evidence and correctness concern about whether the headline advantage over INFP is established, but it is not circular reasoning: those inherited numbers are external reported results, not outputs of the present model, and no fitted parameter is being relabeled as a prediction. The ablation studies compare the full model against explicitly removed modules and report independent metrics, which further supports that the central claims stand on measurement rather than on definitional identity with the inputs.

Assumptions & free parameters 4 free parameters · 4 assumptions · 2 invented entities

The framework's central claims rest on several hand-chosen hyperparameters, domain assumptions about the motion and audio representations, and an ad hoc taxonomy of conversational states. The state taxonomy and annotations are particularly load-bearing because the CSU module's guidance is claimed to improve realism, yet the validity of the labels is not established.

free parameters (4)
  • chunk size c = 6
    Hand-chosen window size for short-range behavioral summarization.
  • context cache capacity w = 512
    Hand-chosen number of chunk summaries retained for long-range context.
  • number of attention heads h = 6
    Hand-chosen architecture hyperparameter.
  • diffusion inference steps = 15
    Hand-chosen number of denoising steps at inference.
assumptions (4)
  • standard math The diffusion denoising procedure (Eq. 4) correctly models the conditional motion distribution.
    The paper relies on established diffusion model theory from [4,13] without re-deriving it.
  • domain assumption The motion representation extracted by LivePortrait [8] captures sufficient facial and head information for conversational realism.
    The entire pipeline uses these coefficients as the motion space; if they are lossy, the generated videos may not reflect true interaction quality.
  • domain assumption Voice activity detection (VAD) signals are a reliable basis for classifying conversational states.
    The CSU module uses VAD outputs to help classify the seven states; if VAD is error-prone, state predictions degrade.
  • ad hoc to paper The seven conversational states form a valid and complete taxonomy for the experiments.
    This taxonomy is introduced by the authors without external validation or comparison to alternative state definitions.
invented entities (2)
  • Conversational state categories (seven classes)
    purpose: Used as a conditioning signal to guide motion generation based on the current conversational situation.
    These categories are defined by the authors and are not validated against any external standard or inter-annotator agreement.
  • cis-token (contextual interaction summary)
    purpose: A learned compressed representation of long-range interactive context used as a condition in motion prediction.
    This is a latent feature internal to the model; there is no external validation that it captures the intended semantics.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ARIG: Autoregressive Interactive Head Generation for Real-time Conversations." pith.science (2026). https://pith.science/paper/EKEZZ6OG

@misc{pith2026250700472,
  author       = {Pith},
  title        = {Pith review of: ARIG: Autoregressive Interactive Head Generation for Real-time Conversations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EKEZZ6OG}},
  note         = {Machine review of arXiv:2507.00472}
}
read the original abstract

Face-to-face communication, as a common human activity, motivates the research on interactive head generation. A virtual agent can generate motion responses with both listening and speaking capabilities based on the audio or motion signals of the other user and itself. However, previous clip-wise generation paradigm or explicit listener/speaker generator-switching methods have limitations in future signal acquisition, contextual behavioral understanding, and switching smoothness, making it challenging to be real-time and realistic. In this paper, we propose an autoregressive (AR) based frame-wise framework called ARIG to realize the real-time generation with better interaction realism. To achieve real-time generation, we model motion prediction as a non-vector-quantized AR process. Unlike discrete codebook-index prediction, we represent motion distribution using diffusion procedure, achieving more accurate predictions in continuous space. To improve interaction realism, we emphasize interactive behavior understanding (IBU) and detailed conversational state understanding (CSU). In IBU, based on dual-track dual-modal signals, we summarize short-range behaviors through bidirectional-integrated learning and perform contextual understanding over long ranges. In CSU, we use voice activity signals and context features of IBU to understand the various states (interruption, feedback, pause, etc.) that exist in actual conversations. These serve as conditions for the final progressive motion prediction. Extensive experiments have verified the effectiveness of our model.

Figures

Figures reproduced from arXiv: 2507.00472 by the authors.

Figure 1
Figure 1. Development of interactive head motion generation. a) [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overall framework of ARIG. Given the previous frame’s audio and motion [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Illustration of agent’s conversational state. In addition to [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Visual results of our method on RealTalk[ [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Qualitative comparisons with DIM[27] and INFP[39]. The two sample videos are from the DyConv dataset proposed by INFP[39], which is not open-sourced, thus lacking ground truth. comparisons with DIM are in Appendix C.1. The full gen￾erated videos can be found in our sup…
Figure 6
Figure 6. Figure 6: Ablation study. (a) Ablation of long-range context. (b) [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: The structure of the Bidirectional-learning. [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: The structure of the Integrated-learning. [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]
Figure 9
Figure 9. Figure 9: The structure of the DiffusionMLP. B. More Experiments B.1. More Ablation Continuous AR Modeling To validate the effectiveness of continuous autoregressive (AR) modeling, we utilize dis- [PITH_FULL_IMAGE:figures/full_fig_p011_9.png]
Figure 10
Figure 10. Figure 10: Ablation Study. Methods RPCC ↓ CSIM ↑ SyncScore ↑ PSNR ↑ SSIM ↑ FID ↓ SID↑ Var↑ w/o continuous AR modeling 0.129 0.887 7.036 27.62 0.813 23.78 2.261 2.154 w/o bidirectional-integrated learning 0.173 0.841 6.813 24.17 0.749 25.36 2.138 2.016 Ours 0.125 0.901 7.218 29.6…
Figure 11
Figure 11. Figure 11: Qualitative comparisons with DIM[27] on RealTalk[7] dataset. natural and photorealistic. C.3. Listening Head Generation We compare our method with the SOTA listening head gen￾eration methods (e.g., RLHG[36], L2L[17], DIM[27] and INFP[39]) on ViCo[36] based on the same…
Figure 12
Figure 12. Figure 12: Qualitative comparisons with state-of-the-art talking head generation methods on HDTF[ [PITH_FULL_IMAGE:figures/full_fig_p014_12.png]
Figure 13
Figure 13. Figure 13: Qualitative comparisons with state-of-the-art listenning head generation methods on ViCo[ [PITH_FULL_IMAGE:figures/full_fig_p015_13.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Towards Flexible, Natural, Efficient Interaction for Conversational Talking Face Generation

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    InterTalk is a motion-based real-time framework for flexible multi-round multi-person conversational talking face generation using motion feedback, iterative strategies, and facial component disentanglement, supported...

  2. Live Avatar: Streaming Real-time Audio-Driven Avatar Generation with Infinite Length

    cs.CV 2025-12 conditional novelty 6.0 of 10

    Live Avatar reports real-time streamable generation from a 14B audio-driven diffusion model at ~20 FPS on 5 H800s with stable identity over 10,000 seconds.

Reference graph

Works this paper leans on

40 extracted references · 22 canonical work pages · cited by 2 Pith papers

  1. [39]

    Infp: Audio-driven interactive head generation in dyadic conversations

    Yongming Zhu, Longhao Zhang, Zhengkun Rong, Tianshu Hu, Shuang Liang, and Zhipeng Ge. Infp: Audio-driven interactive head generation in dyadic conversations. arXiv preprint arXiv:2412.04037, 2024. 1, 2, 3, 6, 7, 8, 13 A. Implementation Details A.1. Network Details Due to the page limitation, we show network details of mod- ules in this Appendix. Bidirecti...

    arXiv 2024

  2. [27]

    Dim: Dyadic interaction modeling for social be- havior generation

    Minh Tran, Di Chang, Maksim Siniukov, and Mohammad Soleymani. Dim: Dyadic interaction modeling for social be- havior generation. In European Conference on Computer Vision, pages 484–503. Springer, 2024. 1, 3, 6, 7, 8, 12, 13

    work page 2024

  3. [1]

    wav2vec 2.0: A framework for self-supervised learning of speech representations

    Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural infor- mation processing systems, 33:12449–12460, 2020. 5

    2020

  4. [2]

    Echomimic: Lifelike audio-driven por- trait animations through editable landmark conditions

    Zhiyuan Chen, Jiajiong Cao, Zhiquan Chen, Yuming Li, and Chenguang Ma. Echomimic: Lifelike audio-driven por- trait animations through editable landmark conditions. arXiv preprint arXiv:2407.08136, 2024. 2, 7, 8, 12

    arXiv 2024

  5. [3]

    Scaling vision transformers to 22 billion pa- rameters

    Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdul- mohsin, et al. Scaling vision transformers to 22 billion pa- rameters. In International Conference on Machine Learning, pages 7480–7512. PMLR, 2023. 4

    work page 2023

  6. [4]

    Autoregressive video generation with- out vector quantization

    Haoge Deng, Ting Pan, Haiwen Diao, Zhengxiong Luo, Yufeng Cui, Huchuan Lu, Shiguang Shan, Yonggang Qi, and Xinlong Wang. Autoregressive video generation with- out vector quantization. arXiv preprint arXiv:2412.14169 ,

    arXiv

  7. [5]

    Accurate 3d face reconstruction with weakly-supervised learning: From single image to image set

    Yu Deng, Jiaolong Yang, Sicheng Xu, Dong Chen, Yunde Jia, and Xin Tong. Accurate 3d face reconstruction with weakly-supervised learning: From single image to image set. In Proceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition workshops , pages 0–0, 2019. 7

    work page 2019

  8. [6]

    Scaling recti- fied flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M ¨uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling recti- fied flow transformers for high-resolution image synthesis. In Forty-first international conference on machine learning,

Show all 40 references
  1. [7]

    Affective faces for goal-driven dyadic communication

    Scott Geng, Revant Teotia, Purva Tendulkar, Sachit Menon, and Carl V ondrick. Affective faces for goal-driven dyadic communication. arXiv preprint arXiv:2301.10939, 2023. 5, 6, 7, 12, 13

    2023 arXiv

  2. [8]

    Livepor- trait: Efficient portrait animation with stitching and retarget- ing control

    Jianzhu Guo, Dingyun Zhang, Xiaoqiang Liu, Zhizhou Zhong, Yuan Zhang, Pengfei Wan, and Di Zhang. Livepor- trait: Efficient portrait animation with stitching and retarget- ing control. arXiv preprint arXiv:2407.03168 , 2024. 3, 5, 6

    2024 arXiv

  3. [9]

    Percep- tual conversational head generation with regularized driver and enhanced renderer

    Ailin Huang, Zhewei Huang, and Shuchang Zhou. Percep- tual conversational head generation with regularized driver and enhanced renderer. In Proceedings of the 30th ACM In- ternational Conference on Multimedia (MM’22), 2022. 2

    2022

  4. [10]

    Interact: Capture and modelling of realistic, ex- pressive and interactive activities between two persons in daily scenarios

    Yinghao Huang, Leo Ho, Dafei Qin, Mingyi Shi, and Taku Komura. Interact: Capture and modelling of realistic, ex- pressive and interactive activities between two persons in daily scenarios. arXiv preprint arXiv:2405.11690, 2024. 3

    2024 arXiv

  5. [11]

    Loopy: Taming audio-driven portrait avatar with long-term motion dependency

    Jianwen Jiang, Chao Liang, Jiaqi Yang, Gaojie Lin, Tianyun Zhong, and Yanbo Zheng. Loopy: Taming audio-driven portrait avatar with long-term motion dependency. arXiv preprint arXiv:2409.02634, 2024. 2

    2024 arXiv

  6. [12]

    Uvim: A unified modeling approach for vision with learned guiding codes

    Alexander Kolesnikov, Andr ´e Susano Pinto, Lucas Beyer, Xiaohua Zhai, Jeremiah Harmsen, and Neil Houlsby. Uvim: A unified modeling approach for vision with learned guiding codes. Advances in Neural Information Processing Systems, 35:26295–26308, 2022. 3

    2022

  7. [13]

    Autoregressive image generation without vec- tor quantization

    Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vec- tor quantization. Advances in Neural Information Processing Systems, 37:56424–56445, 2025. 3, 5

    2025

  8. [14]

    Mfr-net: Multi-faceted responsive listening head generation via denoising diffusion model

    Jin Liu, Xi Wang, Xiaomeng Fu, Yesheng Chai, Cai Yu, Jiao Dai, and Jizhong Han. Mfr-net: Multi-faceted responsive listening head generation via denoising diffusion model. In Proceedings of the 31th ACM International Conference on Multimedia (MM’23), 2023. 2

    2023

  9. [15]

    Customlistener: Text-guided responsive inter- action for user-friendly listening head generation

    Xi Liu, Ying Guo, Cheng Zhen, Tong Li, Yingying Ao, and Pengfei Yan. Customlistener: Text-guided responsive inter- action for user-friendly listening head generation. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2415–2424, 2024. 2

    2024

  10. [16]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. 6

    2017 arXiv

  11. [17]

    Learning to lis- ten: Modeling non-deterministic dyadic facial motion

    Evonne Ng, Hanbyul Joo, Liwen Hu, Hao Li, Trevor Dar- rell, Angjoo Kanazawa, and Shiry Ginosar. Learning to lis- ten: Modeling non-deterministic dyadic facial motion. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR), pages 20395–20405,

  12. [18]

    From audio to photoreal embodiment: Synthesizing humans in conversations

    Evonne Ng, Javier Romero, Timur Bagautdinov, Shaojie Bai, Trevor Darrell, Angjoo Kanazawa, and Alexander Richard. From audio to photoreal embodiment: Synthesizing humans in conversations. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition , ...

    2024

  13. [19]

    Let’s go real talk: Spoken dialogue model for face-to-face conversa- tion

    Se Jin Park, Chae Won Kim, Hyeongseop Rha, Minsu Kim, Joanna Hong, Jeong Hun Yeo, and Yong Man Ro. Let’s go real talk: Spoken dialogue model for face-to-face conversa- tion. arXiv preprint arXiv:2406.07867, 2024. 1, 3, 5

    2024 arXiv

  14. [20]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF inter- national conference on computer vision , pages 4195–4205,

  15. [21]

    A lip sync expert is all you need for speech to lip generation in the wild

    KR Prajwal, Rudrabha Mukhopadhyay, Vinay P Nambood- iri, and CV Jawahar. A lip sync expert is all you need for speech to lip generation in the wild. In Proceedings of the 28th ACM international conference on multimedia , pages 484–492, 2020. 7, 8

    2020

  16. [22]

    Zero-shot text-to-image generation

    Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea V oss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International confer- ence on machine learning, pages 8821–8831. Pmlr, 2021. 3

    2021

  17. [23]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 8

    2022

  18. [24]

    Emotional listener portrait: Neural lis- tener head generation with emotion

    Luchuan Song, Guojun Yin, Zhenchao Jin, Xiaoyi Dong, and Chenliang Xu. Emotional listener portrait: Neural lis- tener head generation with emotion. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 20839–20849, 2023. 2

    2023

  19. [25]

    Beyond talking–generating holistic 3d hu- man dyadic motion for communication

    Mingze Sun, Chao Xu, Xinyu Jiang, Yang Liu, Baigui Sun, and Ruqi Huang. Beyond talking–generating holistic 3d hu- man dyadic motion for communication. International Jour- nal of Computer Vision, pages 1–17, 2024. 3

    2024

  20. [26]

    Emo: Emote portrait alive generating expressive portrait videos with audio2video diffusion model under weak conditions

    Linrui Tian, Qi Wang, Bang Zhang, and Liefeng Bo. Emo: Emote portrait alive generating expressive portrait videos with audio2video diffusion model under weak conditions. In European Conference on Computer Vision, pages 244–260. Springer, 2024. 2

    2024

  21. [28]

    Neural discrete representation learning

    Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information pro- cessing systems, 30, 2017. 3

    2017

  22. [29]

    Agentavatar: Disentangling planning, driving and ren- dering for photorealistic avatar agents

    Duomin Wang, Bin Dai, Yu Deng, and Baoyuan Wang. Agentavatar: Disentangling planning, driving and ren- dering for photorealistic avatar agents. arXiv preprint arXiv:2311.17465, 2023. 1, 3, 8

    2023 arXiv

  23. [30]

    Mrecgen: Multimodal appropriate reaction generator

    Jiaqi Xu, Cheng Luo, Weicheng Xie, Linlin Shen, Xiaofeng Liu, Lu Liu, Hatice Gunes, and Siyang Song. Mrecgen: Multimodal appropriate reaction generator. arXiv preprint arXiv:2307.02609, 2023. 1, 3

    2023 arXiv

  24. [31]

    Hallo: Hierarchical audio-driven visual synthesis for portrait image animation

    Mingwang Xu, Hui Li, Qingkun Su, Hanlin Shang, Liwei Zhang, Ce Liu, Jingdong Wang, Yao Yao, and Siyu Zhu. Hallo: Hierarchical audio-driven visual synthesis for portrait image animation. arXiv preprint arXiv:2406.08801, 2024. 2, 7, 8, 12

    2024 arXiv

  25. [32]

    Dialoguenerf: Towards realistic avatar face- to-face conversation video generation

    Yichao Yan, Zanwei Zhou, Zi Wang, Jingnan Gao, and Xi- aokang Yang. Dialoguenerf: Towards realistic avatar face- to-face conversation video generation. Visual Intelligence, 2 (1):24, 2024. 1, 3

    2024

  26. [33]

    Sadtalker: Learning realistic 3d motion coefficients for stylized audio- driven single image talking face animation

    Wenxuan Zhang, Xiaodong Cun, Xuan Wang, Yong Zhang, Xi Shen, Yu Guo, Ying Shan, and Fei Wang. Sadtalker: Learning realistic 3d motion coefficients for stylized audio- driven single image talking face animation. In Proceedings of the IEEE/CVF conference on computer vision and p...

    2023

  27. [34]

    Flow-guided one-shot talking face generation with a high-resolution audio-visual dataset

    Zhimeng Zhang, Lincheng Li, Yu Ding, and Changjie Fan. Flow-guided one-shot talking face generation with a high-resolution audio-visual dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3661–3670, 2021. 7, 8, 12, 14

    2021

  28. [35]

    Memo: Memory-guided diffusion for expressive talking video generation

    Longtao Zheng, Yifan Zhang, Hanzhong Guo, Jiachun Pan, Zhenxiong Tan, Jiahao Lu, Chuanxin Tang, Bo An, and Shuicheng Yan. Memo: Memory-guided diffusion for expressive talking video generation. arXiv preprint arXiv:2412.04448, 2024. 2

    2024 arXiv

  29. [36]

    Responsive listening head generation: A benchmark dataset and baseline

    Mohan Zhou, Yalong Bai, Wei Zhang, Ting Yao, Tiejun Zhao, and Tao Mei. Responsive listening head generation: A benchmark dataset and baseline. In Proceedings of the European conference on computer vision (ECCV), 2022. 2, 5, 7, 8, 13, 15

    2022

  30. [37]

    Interactive conversational head generation

    Mohan Zhou, Yalong Bai, Wei Zhang, Ting Yao, and Tiejun Zhao. Interactive conversational head generation. arXiv preprint arXiv:2307.02090, 2023. 1, 3

    2023 arXiv

  31. [38]

    A unified frame- work for multimodal, multi-part human motion synthesis

    Zixiang Zhou, Yu Wan, and Baoyuan Wang. A unified frame- work for multimodal, multi-part human motion synthesis. arXiv preprint arXiv:2311.16471, 2023. 3

    2023 arXiv

  32. [262]

    In the initial stage, we re- peat the motion vector of the agent’s reference image and the audio corresponding to the first frame to initialize each cache

    We first encode them into 512 dimensions and put them into the historical input. In the initial stage, we re- peat the motion vector of the agent’s reference image and the audio corresponding to the first frame to initialize each cache. The embedding dimension in Bidirectional...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.