Real-time Generation of Various Types of Nodding for Avatar Attentive Listening System

Divesh Lala; Kazushi Kato; Keiko Ochi; Koji Inoue; Tatsuya Kawahara

REVIEW 3 major objections 5 minor 23 references

Real-time Generation of Various Types of Nodding for Avatar Attentive Listening System

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that an audio-only extension of VAP can predict both the timing and type of listener nodding in real time, making an avatar feel more natural, attentive, and encouraging than conventional backchannel-coupled nodding.

desk verdict Novel VAP extension for real-time nod-type prediction, but the pretraining set may include the test dialogues—that ambiguity needs to be settled before the numbers can be trusted. read the letter →

arxiv 2507.23298 v2 pith:XESKRSXS submitted 2025-07-31 cs.HC cs.SDeess.AS

Kazushi Kato , Koji Inoue , Divesh Lala , Keiko Ochi , Tatsuya Kawahara This is my paper · ORCID

classification cs.HCcs.SDeess.AS

keywords NoddingPredictionVoiceActivityProjectionBackchannelMulti-taskLearningAttentiveListeningSystemAvatarNonverbalBehaviorReal-timeGenerationSpokenDialogueSystems

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

An attentive-listening avatar only feels attentive if it reacts while the user is speaking, and a single generic nod repeated on every backchannel quickly feels mechanical. This paper tries to establish that an audio-only model can do better: it can predict, about half a second before a nod would be transcribed, not just that a listener will nod but whether the nod will be short, long, or a long nod with an upward swing, and can run in real time on a laptop CPU. The authors build this by extending the voice activity projection (VAP) model, which already anticipates turn-taking from both speakers' audio, with an extra head that classifies nodding frames, and by co-training that head with verbal backchannel prediction and pretraining on general dialogue data. In crowdsourced ratings, an avatar driven by the model is judged significantly more natural, attentive, and encouraging than the conventional system that always performs a single nod type together with each backchannel. If the result holds, spoken dialogue agents can produce human-like listening behavior using only microphones, with no camera or body-tracking input.

What carries the argument

The carrying mechanism is the voice activity projection (VAP) architecture: each participant's audio is encoded with contrastive predictive coding into a representation of the current and immediately upcoming speech activity, then passed through self-attention and cross-attention transformers so the model can relate the two channels. The paper adds a single linear layer that classifies each processed frame into one of four classes (no nod, short, long, long_p), and optionally a second linear layer that predicts verbal backchannel timing; the total loss combines the nodding cross-entropy with the original VAD/VAP losses and a weighted backchannel loss. Two design choices make the head usable for an interactive avatar: the ground-truth nod intervals are shifted 500 ms earlier during training so the model predicts a nod before it is physically displayed, and the processing rate can be dropped to 10 Hz, which keeps the real-time factor below 1.0 on a CPU. The three types are defined by movement range and upward swing: short nods have small range, long nods have large range without swinging up, and long_p nods combine large range with an upward swing.

What would settle it

Record real dyadic attentive-listening interactions with motion capture of the listener's head, train the same audio-only VAP-based predictor on those naturally occurring nods, and run the avatar against the conventional 'nod with every backchannel' system; the central claim fails if the model no longer beats that baseline on attentiveness or if its timing F1 drops to near chance.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that listener nodding, which is nonverbal and often assumed to need visual or kinematic input, can be projected from speech alone in a continuous, frame-wise manner. Extending VAP with a further linear layer yields an F1 of 55.93 percent on the binary timing task and per-type F1 scores of 28.86, 39.17, and 22.09 percent for short, long, and long_p nods in the four-way timing-and-type task. Multi-task learning with backchannel prediction gives a statistically significant gain in the timing-and-type task (p = .049 and .012 for the two MT-versus-ST comparisons), and VAP pretraining on general dialogue data adds further improvement. Reducing the processing frame rate from 50 Hz to 10 Hz keeps the real-time factor below 1.0 on a CPU while causing no substantial drop in F1, so the model can be embedded in a deployed avatar. In the subjective evaluation, separating backchannel and nod generation is rated significantly higher than always nodding together with backchannels on naturalness (p = .019), attentiveness (p < .001), and facilitation (p = .012).

Load-bearing premise

The training labels come from a single operator re-enacting listener gestures while watching recorded dialogues after the fact, not from the original live interaction, so the model's success depends on that re-enactment faithfully matching the timing and form of natural attentive-listening nods; if the re-enactment is stylized or biased, the learned mapping may not transfer to real users.

Editorial extensions

If this is right

An avatar can convey attentive listening with audio-only input, removing the need for a camera or body-tracker in deployment.
Because the model predicts roughly 500 ms ahead, the avatar's mechanical latency can be absorbed and the nod can land at the moment a human listener would nod.
Joint prediction of backchannel timing and nod type is more effective for large-movement nods (long, long_p) than for small ones, pointing to a tighter coupling between lexical responses and large gestures.
A 10 Hz processing rate is sufficient for real-time use on a laptop-class CPU without a substantial accuracy cost, so the method is deployable outside GPU-equipped servers.
Letting backchannel and nodding occur independently, rather than always together, changes users' perception of naturalness, attentiveness, and encouragement.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the audio-only projection generalizes, the same VAP backbone could be extended to eye gaze, head motion, and facial-expression timing, since the pretrained speech-activity representation is not tied to nodding.
The 500 ms look-ahead makes the nod prediction a candidate turn-taking feature: an early long_p nod may mark an upcoming floor shift, which could let the system decide whether to yield or hold the turn.
Because the multi-task gains concentrate on long and long_p nods, a finer-grained model that conditions nod type on the exact lexical form of the co-occurring backchannel ('mm', 'uh-huh', 'right') might further improve type accuracy.
The recorded-audio evaluation tests the avatar's behavior on fixed user speech; a fully interactive study, in which the user reacts to the avatar's nods while speaking, could reveal whether the attentiveness effect grows or shrinks in a live loop.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague, here's my read on 2507.23298.

The genuinely new thing here is using VAP-style future voice-activity prediction to predict not just timing but three explicit nodding types (short/long/long_p) in real time, with multi-task learning on backchannel prediction. That's a real extension over prior prosody-based or backchannel-conditioned nod models. The architecture is straightforward, and they ship code and trained models, which I appreciate. The real-time CPU analysis is also useful and honestly presented.

The experiments are extensive: timing and type F1, statistical tests on the comparisons, and a user study with 45 workers. The user study is interesting: separating backchannel and nod generation into independent predictions actually beat always-coupling them on naturalness, attentiveness, and facilitation. That's a nice result, if it holds.

But I have two concerns, and one is potentially decisive.

The first is the VAP pretraining set. Section 4 says they pretrain on 203 WOZ dialogues 'including 72 dialogues used as data for training described in Section 2.' The attentive-listening corpus has 90 dialogues; 72 are train, 9 validation, 9 test. As written, the 203 could be a superset of all 90, meaning the test audio was seen during pretraining. That would make the F1 gains and the significance tests transductive, not clean generalization. The paper never says the 9 validation/test dialogues are excluded from the 203. This needs to be stated explicitly, and if they are included, the experiments need rerunning with pretraining restricted to the training split. This is more worrying than the re-enactment issue below.

Second, the nodding ground truth is a single operator re-enacting listener gestures after the fact. That is a real limitation, especially because the same operator is the one who controlled ERICA and knows the dialogue. It's not fatal if the re-enactment is faithful, but it's a major validity caveat.

Smaller issues: absolute F1 scores are modest (55.9 timing, 22-39 types), there are no confidence intervals and multiple t-tests without correction. The subjective evaluation also uses audio that was originally recorded while the user was interacting with the proposed method, which could favor it. The user study itself is otherwise okay.

Bottom line: this is a plausible, competent application that deserves serious referee time, but the pretraining overlap question must be resolved before the reported numbers are citable. If the authors clarify and rerun if needed, this becomes a solid contribution. I would not cite the numerical results as-is, but I'd keep an eye on the revision.

Referee Report

3 major / 5 minor

Summary. The paper proposes a VAP-based model for real-time prediction of listener nodding, predicting both the timing and the type of nod (short, long, long_p) from the audio of both dialogue participants. The model extends the VAP architecture with an extra linear head for nodding, adds multi-task learning with verbal backchannel prediction, and uses pretraining on a larger Wizard-of-Oz dialogue corpus. Frame-level F1 scores are reported for timing (best 55.93) and for the three types (roughly 22–39), with bootstrap significance tests; the authors show that reducing the processing rate to 10 Hz achieves real-time operation on CPU with little F1 loss. The model is integrated into an avatar attentive listening system, and a crowdsourced subjective study (n=45) finds that separately predicting backchannels and multiple nodding types yields significantly higher ratings in naturalness, attentiveness, and facilitation than a conventional system that always couples nods with predicted verbal backchannels.

Significance. If the results are clean, this is a useful and reasonably novel contribution: it extends the VAP framework from turn-taking and backchannel prediction to continuous, real-time, multi-class nodding prediction, and it provides an end-to-end avatar integration with a subjective evaluation. The public release of code and trained models is a concrete reproducibility strength. However, the quantitative claims are tempered by two concerns: the pretraining corpus may overlap the test dialogues (which would invalidate the reported generalization numbers), and the ground-truth nods were re-enacted after the fact by a single operator. The absolute F1 scores are also modest, and most comparisons lack confidence intervals or multiple-comparison control, so the strength of the evidence is weaker than the text sometimes suggests.

major comments (3)

[Section 4 (Experimental Evaluation)] The composition of the 203-dialogue VAP pretraining set is not stated precisely enough. The paper says it 'includes 72 dialogues used as data for training described in Section 2,' but Section 2 divides the attentive-listening corpus into 72 training, 9 validation, and 9 test dialogues; the text does not state that the 9 validation and 9 test dialogues are excluded from pretraining. Under a strict reading, the pretraining set is a superset of the entire attentive-listening corpus, so the self-attention and cross-attention layers would be trained on the exact test recordings before the nodding head is fine-tuned. In that case the F1 scores in Tables 2, 4–7 and the bootstrap significance results in Tables 3 and 8 would reflect transductive leakage rather than clean generalization. The authors must state the exact overlap between the 203 pretraining dialogues and the 90 attentive-listening dialogues, and if any held-out dialogue appears in pretraining, they must rerun the experiments with pretraining restricted to the 72 training dialogues.
[Section 2.1 (Recording of Listener Gestures)] The ground-truth nodding labels are produced by a single operator re-enacting listener gestures after the fact while reviewing recorded dialogues. No validation is provided that the re-enacted timing, duration, and movement-range categories (short, long, long_p) match naturally occurring attentive-listening nods in live interaction. Because every objective F1 number, the model's training signal, and the nodding motions used in the subjective evaluation are derived from these re-enacted labels, the central quantitative claims are contingent on the fidelity of this re-enactment. The paper should provide at least an inter-operator or repeated-recording consistency check, or explicitly scope the claims to the operational setting of this re-enactment procedure.
[Section 4.2 and Table 8] The text states that VAP pretraining was 'confirmed' to improve performance in the timing-and-type task, but the bootstrap t-tests for pretrained versus non-pretrained models are not statistically significant (ST w/ PT vs. ST, p=0.138; MT w/ BC, PT vs. MT w/ BC, p=0.111). The same is true in the timing task (Table 3: p=0.130 and p=0.454). The claims in Sections 4.1 and 4.2 should be revised to separate descriptive F1 gains from statistically supported ones, and the conclusion that 'VAP pretraining with general dialogue data improves performance' should be softened accordingly.

minor comments (5)

[Section 3.2] The 'self-feedback' of predicted backchannel results to the listener's speech signal input is mentioned but never specified; please clarify how this feedback is implemented (e.g., as an additional input feature, a gating mechanism, or a loss-level interaction).
[Section 4.3, Tables 9 and 10] The claim that reducing the frame rate to 10 Hz causes 'no substantial decline' in F1 is based on point estimates only; no confidence intervals or statistical tests are provided for the differences across input lengths and frame rates.
[Section 4, Tables 2 and 4-7] The Random baseline is described only as 'always predicting nodding in all frames' for the timing task. Please clarify what the Random baseline does in the type-prediction task (e.g., always predicts all four classes with uniform probability, or cycles through classes).
[Section 5.1] Please clarify how the nine audio recordings were assigned to the three worker sets and whether each recording came from a distinct user or session; this affects the interpretation of the per-method averaging across the 45 participants.
[Throughout] There are several small typographical issues, including 'lonp_p nodding' and 'predicton' in Section 4.2, and inconsistent spacing in Table 2; these should be corrected in the camera-ready version.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: nodding timing/type predictions are learned empirically from audio and evaluated against baselines; self-citations are not load-bearing in the claimed derivation.

full rationale

The paper's central claim is that an extended VAP model predicts future nodding timing and type from speaker/listener audio. The derivation is an empirical pipeline: the model is trained with cross-entropy losses on shifted ground-truth nod labels (Eqs. 1-3) and evaluated on held-out test dialogues (Section 4) and by subjective comparison (Section 5). No equation reduces a prediction to its input by construction: the 500 ms label offset is an explicit modeling choice that defines what is predicted, not a hidden reintroduction of the answer. The VAP architecture is taken from external work [19], and the self-cited backchannel predictor [20] is used as a system component and comparative basis, not as the justification for the nodding results; the real-time claim is independently measured in Section 4.3. I therefore find no circular step. I do flag two non-circular validity concerns from the manuscript text: Section 4's description of the 203-dialogue pretraining set mentions only that it includes the 72 training dialogues, leaving held-out composition ambiguous, and the Safe and Responsible Innovation Statement claims that recording gestures from the same performer ensures unbiased gesture data, which overstates what same-operator re-enactment can guarantee. These are data-leakage and data-quality risks, not definitional circularity.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claim rests on several hand-set constants (loss weights, class weights, 500 ms offset) and on a dataset built from a single operator's re-enactment of listener gestures. No new physical or conceptual entities are introduced.

free parameters (5)

w_vad, w_vap (loss weights) = 0.2, 0.2
Weights for VAD and VAP losses in the combined loss (Eq. 1, 3); set via a preliminary experiment (Section 4).
w_bc (backchannel loss weight) = 0.5
Weight for the backchannel prediction loss (Eq. 3), tuned on validation.
Positive-class loss weight (timing task) = 3
Positive samples weighted 3x in cross-entropy loss to handle class imbalance (Section 4.1).
Positive-class loss weight (type task) = 5
Positive samples weighted 5x in cross-entropy loss (Section 4.2).
Nod-label offset = 500 ms
Ground-truth nod intervals shifted 500 ms earlier to account for processing delay; this defines the prediction target and is evaluated against itself.

assumptions (5)

domain assumption The three nod types (short, long, long_p) defined by movement range and swing-up are meaningful, distinct categories that can be predicted from audio.
Motivated by refs [18] and [21], but the actual mapping to acoustic cues is not independently established.
domain assumption Re-enacted listener gestures by one operator approximate natural attentive-listening nodding.
The training data are performed post hoc by the same operator (Section 2.1); if this re-enactment is unnatural, the model learns an artificial distribution.
domain assumption VAP pretraining on general dialogue data transfers to nodding prediction.
The authors test this empirically; the improvement is small and not always significant, so the transfer assumption is weakly supported.
ad hoc to paper Frame-level one-hot labels after a 500 ms offset are the correct supervision for real-time nodding.
The offset is a design choice to compensate for system delay; it makes the model predict a shifted version of the label rather than the actual instantaneous behavior.
standard math CPC representations pretrained on Librispeech capture speaker-independent features useful for nodding.
The CPC encoder is frozen after large-scale pretraining; this is an unverified background assumption about representation quality.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Real-time Generation of Various Types of Nodding for Avatar Attentive Listening System." pith.science (2026). https://pith.science/paper/XESKRSXS

@misc{pith2026250723298,
  author       = {Pith},
  title        = {Pith review of: Real-time Generation of Various Types of Nodding for Avatar Attentive Listening System},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XESKRSXS}},
  note         = {Machine review of arXiv:2507.23298}
}

read the original abstract

In human dialogue, nonverbal information such as nodding and facial expressions is as crucial as verbal information, and spoken dialogue systems are also expected to express such nonverbal behaviors. We focus on nodding, which is critical in an attentive listening system, and propose a model that predicts both its timing and type in real time. The proposed model builds on the voice activity projection (VAP) model, which predicts voice activity from both listener and speaker audio. We extend it to prediction of various types of nodding in a continuous and real-time manner unlike conventional models. In addition, the proposed model incorporates multi-task learning with verbal backchannel prediction and pretraining on general dialogue data. In the timing and type prediction task, the effectiveness of multi-task learning was significantly demonstrated. We confirmed that reducing the processing rate enables real-time operation without a substantial drop in accuracy, and integrated the model into an avatar attentive listening system. Subjective evaluations showed that it outperformed the conventional method, which always does nodding in sync with verbal backchannel. The code and trained models are available at https://github.com/MaAI-Kyoto/MaAI.

Figures

Figures reproduced from arXiv: 2507.23298 by the authors.

**Figure 3.** Examples of nodding annotation (red: short (S), yellow: long (L), green: long_p (LP)) [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

**Figure 4.** Proposed model: multi-task learning with VAP and [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗

Figures from the paper (4 more)

**Figure 5.** Figure 5: Proposed model: multi-task learning with VAP and [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]

**Figure 7.** Figure 7: Output samples of timing prediction model (The panels, from the top to bottom, represent speaker speech signal, [PITH_FULL_IMAGE:figures/full_fig_p005_7.png]

**Figure 8.** Figure 8: Output samples of timing and type prediction model (The panels, from the top to bottom, represent speaker speech [PITH_FULL_IMAGE:figures/full_fig_p006_8.png]

**Figure 9.** Figure 9: Avatar attentive listening system integrated with the proposed model [PITH_FULL_IMAGE:figures/full_fig_p007_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

23 extracted references · 22 canonical work pages

[1]

An attentive listening system with android ERICA: Comparison of autonomous and WOZ interactions

Koji Inoue, Divesh Lala, Kenta Yamamoto, Shizuka Nakamura, Katsuya Takanashi, and Tatsuya Kawahara. An attentive listening system with android ERICA: Comparison of autonomous and WOZ interactions. InProceedings of the 21th annual meeting of the special interest group on discourse and dialogue, pages 118–127, 2020

work page 2020
[2]

Genea workshop 2024: The 5th workshop on generation and evaluation of non-verbal behaviour for embodied agents

Youngwoo Yoon, Taras Kucherenko, Alice Delbosc, Rajmund Nagy, Teodor Nikolov, and Gustav Eje Henter. Genea workshop 2024: The 5th workshop on generation and evaluation of non-verbal behaviour for embodied agents. In Proceedings of the 26th International Conference on Multimodal Interaction (ICMI), pages 694–695, 2024

work page 2024
[3]

Diffsheg: A diffusion-based approach for real-time speech-driven holistic 3d expression and gesture generation

Junming Chen, Yunfei Liu, Jianan Wang, Ailing Zeng, Yu Li, and Qifeng Chen. Diffsheg: A diffusion-based approach for real-time speech-driven holistic 3d expression and gesture generation. InConference on Computer Vision and Pattern Recognition (CVPR), pages 7352–7361, 2024

work page 2024
[4]

Am I listening?

Pieter Wolfert, Gustav Eje Henter, and Tony Belpaeme. “Am I listening?”, Evaluat- ing the quality of generated data-driven listening motion.International Conference on Multimodal Interaction (ICMI), page 6–10, 2023

work page 2023
[5]

Viktor Schmuck, Nguyen Tan Viet Tuyen, and Oya Celiktutan. The kcl-sair team’s entry to the genea challenge 2023 exploring role-based gesture generation in Real-time Generation of Various Types of Nodding for Avatar Attentive Listening System ICMI ’25, October 13–17, 2025, Canberra, ACT, Australia dyadic interactions: Listener vs. speaker.International Co...

work page 2023
[6]

Dim: Dyadic interaction modeling for social behavior generation

Minh Tran, Di Chang, Maksim Siniukov, and Mohammad Soleymani. Dim: Dyadic interaction modeling for social behavior generation. InEuropean Conference on Computer Vision (ECCV), pages 484–503, 2024

work page 2024
[7]

INFP: Audio-driven interactive head generation in dyadic conversations.arXiv preprint arXiv:2412.04037, 2024

Yongming Zhu, Longhao Zhang, Zhengkun Rong, Tianshu Hu, Shuang Liang, and Zhipeng Ge. INFP: Audio-driven interactive head generation in dyadic conversations.arXiv preprint arXiv:2412.04037, 2024

arXiv 2024
[8]

Difflistener: Discrete diffusion model for listener generation

Siyeol Jung and Taehwan Kim. Difflistener: Discrete diffusion model for listener generation. In2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5, 2025

work page 2025

Show all 23 references

[9]

Active listener: Continuous gener- ation of listener’s head motion response in dyadic interactions

Bishal Ghosh, Emma Li, and Tanaya Guha. Active listener: Continuous gener- ation of listener’s head motion response in dyadic interactions. In2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5, 2025

2025
[10]

Responsive listening head generation: a benchmark dataset and baseline

Mohan Zhou, Yalong Bai, Wei Zhang, Ting Yao, Tiejun Zhao, and Tao Mei. Responsive listening head generation: a benchmark dataset and baseline. In European conference on computer vision (ECCV), pages 124–142, 2022

2022
[11]

Listenformer: Responsive listening head generation with non-autoregressive transformers

Miao Liu, Jing Wang, Xinyuan Qian, and Haizhou Li. Listenformer: Responsive listening head generation with non-autoregressive transformers. InProceedings of the 32nd ACM International Conference on Multimedia (ACM Multimedia), pages 7094–7103, 2024

2024
[12]

Listening head motion generation for multi- modal dialog system

Tamon Mikawa, Yasuhisa Fujii, Yukoh Wakabayashi, Kengo Ohta, Ryota Nishimura, and Norihide Kitaoka. Listening head motion generation for multi- modal dialog system. In11th International Conference on Advanced Informatics: Concept, Theory and Application (ICAICTA), pages 1–6, 2024

2024
[13]

PhD thesis, Shinshu University Library, 2005

Tomihide Kondo.A Study on the Function and Role of “Nodding” as a Nonverbal Behavior. PhD thesis, Shinshu University Library, 2005. (in Japanese)

2005
[14]

Kurosio Publishers, 1993

Senko Kumiya Maynard.Conversation Analysis. Kurosio Publishers, 1993. (in Japanese)

1993
[15]

An embodied interaction robots system based speech

Tomio Watanabe, Masashi Okubo, and Hiroki Ogawa. An embodied interaction robots system based speech. InIEEE International Workshop on Robot and Human Interaction, pages 225–230, 1999

1999
[16]

Generation model for head nods consistent with features of verbal response tokens.Transactions of the Japanese Society for Artificial Intelligence, 37(3):1–12, 2022

Taiga Mori and Yasuharu Den. Generation model for head nods consistent with features of verbal response tokens.Transactions of the Japanese Society for Artificial Intelligence, 37(3):1–12, 2022. (in Japanese)

2022
[17]

Multimodal listener response prediction for multi-party conversation.Special Interest Group on Spoken Lan- guage Understanding and Dialogue Processing (SIG-SLUD), 96:7–12, 2022

Taiga Mori, Yasuharu Den, and Kristiina Jokinen. Multimodal listener response prediction for multi-party conversation.Special Interest Group on Spoken Lan- guage Understanding and Dialogue Processing (SIG-SLUD), 96:7–12, 2022. (in Japanese)

2022
[18]

Taiga Mori and Yasuharu Den. Relations between forms of response tokens and physical features of nods.Special Interest Group on Spoken Language Understand- ing and Dialogue Processing (SIG-SLUD), 91:62–67, 2021. (in Japanese)

2021
[19]

Voice activity projection: Self-supervised learning of turn-taking events

Erik Ekstedt and Gabriel Skantze. Voice activity projection: Self-supervised learning of turn-taking events. InAnnual Conference of the International Speech Communication Association (INTERSPEECH), pages 5190–5194, 2022

2022
[20]

Yeah, un, oh: Continuous and real-time backchannel prediction with fine-tuning of voice activ- ity projection

Koji Inoue, Divesh Lala, Gabriel Skantze, and Tatsuya Kawahara. Yeah, un, oh: Continuous and real-time backchannel prediction with fine-tuning of voice activ- ity projection. InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Com...

2025
[21]

Co-occurrence relations between forms of re- sponse tokens and nods.Special Interest Group on Spoken Language Understanding and Dialogue Processing (SIG-SLUD), 90:140–145, 2020

Taiga Mori and Yasuharu Den. Co-occurrence relations between forms of re- sponse tokens and nods.Special Interest Group on Spoken Language Understanding and Dialogue Processing (SIG-SLUD), 90:140–145, 2020. (in Japanese)

2020
[22]

Real-time and continuous turn-taking prediction using voice activity projec- tion

Koji Inoue, Bing’er Jiang, Erik Ekstedt, Tatsuya Kawahara, and Gabriel Skantze. Real-time and continuous turn-taking prediction using voice activity projec- tion. InThe 14th International Workshop on Spoken Dialogue Systems Technology (IWSDS), 2024

2024
[23]

Spoken dialogue system for a human-like conversational robot ERICA

Tatsuya Kawahara. Spoken dialogue system for a human-like conversational robot ERICA. In9th International Workshop on Spoken Dialogue System Technology (IWSDS), pages 65–75, 2019

2019

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

An attentive listening system with android ERICA: Comparison of autonomous and WOZ interactions

Koji Inoue, Divesh Lala, Kenta Yamamoto, Shizuka Nakamura, Katsuya Takanashi, and Tatsuya Kawahara. An attentive listening system with android ERICA: Comparison of autonomous and WOZ interactions. InProceedings of the 21th annual meeting of the special interest group on discourse and dialogue, pages 118–127, 2020

work page 2020

[2] [2]

Genea workshop 2024: The 5th workshop on generation and evaluation of non-verbal behaviour for embodied agents

Youngwoo Yoon, Taras Kucherenko, Alice Delbosc, Rajmund Nagy, Teodor Nikolov, and Gustav Eje Henter. Genea workshop 2024: The 5th workshop on generation and evaluation of non-verbal behaviour for embodied agents. In Proceedings of the 26th International Conference on Multimodal Interaction (ICMI), pages 694–695, 2024

work page 2024

[3] [3]

Diffsheg: A diffusion-based approach for real-time speech-driven holistic 3d expression and gesture generation

Junming Chen, Yunfei Liu, Jianan Wang, Ailing Zeng, Yu Li, and Qifeng Chen. Diffsheg: A diffusion-based approach for real-time speech-driven holistic 3d expression and gesture generation. InConference on Computer Vision and Pattern Recognition (CVPR), pages 7352–7361, 2024

work page 2024

[4] [4]

Am I listening?

Pieter Wolfert, Gustav Eje Henter, and Tony Belpaeme. “Am I listening?”, Evaluat- ing the quality of generated data-driven listening motion.International Conference on Multimodal Interaction (ICMI), page 6–10, 2023

work page 2023

[5] [5]

Viktor Schmuck, Nguyen Tan Viet Tuyen, and Oya Celiktutan. The kcl-sair team’s entry to the genea challenge 2023 exploring role-based gesture generation in Real-time Generation of Various Types of Nodding for Avatar Attentive Listening System ICMI ’25, October 13–17, 2025, Canberra, ACT, Australia dyadic interactions: Listener vs. speaker.International Co...

work page 2023

[6] [6]

Dim: Dyadic interaction modeling for social behavior generation

Minh Tran, Di Chang, Maksim Siniukov, and Mohammad Soleymani. Dim: Dyadic interaction modeling for social behavior generation. InEuropean Conference on Computer Vision (ECCV), pages 484–503, 2024

work page 2024

[7] [7]

INFP: Audio-driven interactive head generation in dyadic conversations.arXiv preprint arXiv:2412.04037, 2024

Yongming Zhu, Longhao Zhang, Zhengkun Rong, Tianshu Hu, Shuang Liang, and Zhipeng Ge. INFP: Audio-driven interactive head generation in dyadic conversations.arXiv preprint arXiv:2412.04037, 2024

arXiv 2024

[8] [8]

Difflistener: Discrete diffusion model for listener generation

Siyeol Jung and Taehwan Kim. Difflistener: Discrete diffusion model for listener generation. In2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5, 2025

work page 2025