Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Voice Activity Projection Model with Multimodal Encoders

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Adding a pretrained facial-image encoder to a voice activity projection model improves turn-taking prediction, beating prior multimodal baselines on shift and backchannel accuracy.

desk verdict Solid incremental extension of VAP with a pretrained face encoder, but the central attribution to the face encoder is confounded by a simultaneous fusion-order change. read the letter →

arxiv 2506.03980 v1 pith:ODCFYRSA submitted 2025-06-04 cs.CL

Takeshi Saga , Catherine Pelachaud This is my paper · ORCID
classification cs.CL
keywords voiceactivityprojectionturn-takingpredictionmultimodalencodersfacialexpressionrecognitionpretrainedfaceencoderbackchannelhuman-agentinteraction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that replacing manually coded facial action units with a pretrained facial-image encoder improves turn-taking prediction in a voice activity projection (VAP) model. The authors test three variants that add a face-image encoder to audio and, in two cases, head, gaze, and body signals. On the NoXi French subset, their Proposed1 and Proposed3 variants exceed the prior multimodal baseline on shift prediction and backchannel prediction, for example reaching 0.794 balanced accuracy for shift prediction and 0.503 for backchannels. If the claim holds, pretrained encoders on raw nonverbal signals offer a simpler integration path for conversational agents than hand-crafted feature extractors.

What carries the argument

The load-bearing object is the pretrained face-image encoder, Former-DFER, a Dynamic Facial Expression Recognition Transformer trained on the DFEW dataset. It maps a clipped and resized face image to a facial embedding that is fed, along with audio features from a Contrastive Predictive Coding model, into per-person multimodal fusion followed by cross-user fusion. The architecture's novelty is merging modalities per participant first and then across participants, which differs from the earlier baseline's modality-first merging; this ordering is what lets facial and audio signals interact within each speaker before turn-taking inference.

What would settle it

Run the previous baseline architecture (modality-first fusion, audio plus action units) and add only the pretrained face-image embedding to it; if shift-prediction balanced accuracy does not rise above 0.768, the paper's attribution of the gain to the face encoder is unsupported. Alternatively, run Proposed3's person-first fusion with the face-image encoder removed; if the 0.794 shift accuracy persists, the fusion change is doing the work.

Watch

Extended reading notes

Core claim

The central claim is that a VAP model whose face input is a pretrained facial-expression encoder, rather than hand-coded facial action units, predicts turn shifts and backchannels at least as well as, and sometimes better than, the existing multimodal VAP baseline. On the French subset of NoXi, the model with audio, face-image encoder, face AU, head angle, gaze, and body positions (Proposed3) reaches 0.794 balanced accuracy on shift prediction and 0.503 on backchannel prediction, compared with 0.768 and 0.428 for the same input set without the face-image encoder. The authors interpret this as evidence that richer facial representations capture subtle expression dynamics that action units miss. They also argue that replacing OpenFace and OpenPose feature extractors with a pretrained image encoder reduces the implementation burden for deployed systems.

Load-bearing premise

The paper's conclusion rests on assuming that the comparison between Proposed3 and the baseline isolates the face-image encoder, but the two models also differ in how audio, face, and body signals are merged, so the improved scores could come from the fusion change rather than the pretrained facial encoder.

Editorial extensions

If this is right

  • If face-image embeddings help, adding pretrained encoders for body and head pose could push shift and backchannel accuracy further.
  • VAP models can move toward raw-signal inputs, reducing reliance on external C++ feature extractors like OpenFace and OpenPose.
  • The approach could transfer to other social-signal prediction tasks such as interruption timing or engagement estimation.
  • Training multimodal VAP models on multilingual datasets, such as the Japanese NoXi subset, is a natural next step to test robustness.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported gains for Proposed3 over Baseline2_2 conflate the face-image encoder with a change in fusion order, so an ablation that keeps fusion fixed is needed to attribute the improvement to the encoder alone.
  • The backchannel metric used here defines backchannels purely by silence and speech duration; a model evaluated on functionally typed backchannels (continuers vs assessments) might show larger differences between encoders.
  • The claim that facial expression encoders beat action units could be tested by swapping only the face representation inside Onishi's original architecture; if gains vanish, fusion, not the encoder, is the active ingredient.
  • Because the face encoder was pretrained for emotion recognition, it likely carries affect-related features; using a pose- or identity-focused pretrained encoder could tell whether the benefit comes from expression or from generic visual features.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper extends Ekstedt and Skantze's voice activity projection (VAP) model with a pretrained facial-expression transformer (Former-DFER) and the existing CPC audio encoder, and evaluates three proposed variants against an audio-only baseline and Onishi et al.'s multimodal baseline on the French subset of NoXi. The main reported results are balanced accuracies for shift-hold, short-long, shift prediction, and backchannel prediction; the best proposed model (Proposed3) reaches SP=0.794 and BC=0.503. The authors conclude that pretrained facial encoders improve turn-taking prediction and that they offer a more practical alternative to hand-crafted action units. Source code and pretrained models are released.

Significance. If the central comparison were controlled, the paper would provide a useful indication that pretrained face-image encoders can substitute for or complement engineered facial action units in VAP, with practical integration benefits. The manuscript has concrete strengths: it uses established evaluation metrics, builds on prior public implementations, and releases code and pretrained models, which supports reproducibility. I also find no circularity problem: the pretrained encoders are external and no constants are fitted on the test set. However, the experimental design currently does not isolate the effect of the face encoder, and the absence of uncertainty quantification weakens every pairwise claim. The contribution is therefore plausible but not yet demonstrated.

major comments (3)
  1. [Section 3 and Section 5, Table 1] The attribution of the SP/BC gains to the face encoder is confounded. Section 3 states that, unlike Onishi's modality-first fusion, 'We first merged multimodal signals for each person separately, followed by the fusion across user embeddings.' Thus Proposed3 differs from Baseline2_2 by both the added face-image encoder and the fusion order, so the sentence 'the difference lies in the use of facial embeddings from the pre-trained encoder' is not established by the reported comparisons. The same problem affects the related claim in Section 5 that 'the difference in score is not from the face AU but from the face image encoder.' Please add an ablation that keeps the proposed fusion architecture fixed and toggles the face-image encoder on and off, and report those numbers.
  2. [Section 5, Table 1] No repeated-run or statistical evidence is reported; each accuracy appears to be from a single training run. Differences such as SP 0.794 (Proposed3) versus 0.768 (Baseline2_2) or S/H 0.737 versus 0.735 may be within run-to-run variability. Please report means and standard deviations over at least several seeds and, if feasible, a paired significance test over the same evaluation folds.
  3. [Section 5, paragraph on SP/BC] The claim that 'our models (Proposed1 and Proposed3) outperformed the baselines' is not accurate as stated, since Baseline2_1 achieves SP 0.788, which is higher than Proposed1's SP 0.709. The subsequent dismissal of S/H and S/L as 'less critical' is asserted rather than argued. Please specify exactly which pairwise comparisons support each claim and justify the relative weighting of the metrics.
minor comments (5)
  1. [Abstract and Section 1] The abstract says 'enhanced with pre-trained audio and face encoders,' but the audio encoder (CPC) is the same one already used in the prior baselines; the novelty is the face encoder. Please phrase the contribution more precisely.
  2. [Section 3] The processing of the face-image sequence is underspecified: it is unclear how the video transformer receives the resized (3,112,112) face crops over time, what temporal sampling or padding is used, and whether the pretrained encoder is fine-tuned or frozen. Please add these details.
  3. [Section 4.2] The sentence 'We used the same hyperparameters for these metrics' is ambiguous because metrics do not have hyperparameters; presumably the same model hyperparameters were used across evaluation metrics. Please rephrase.
  4. [References] References [27] and [28] are incomplete (no author/title) and [31] appears to be a preprint without a venue; please complete them.
  5. [Table 1 caption and text] The baseline names 'Baseline2 1' and 'Baseline2 2' render awkwardly; use underscores or subscripts consistently, and consider adding a model-condition column to make the input configurations easier to compare at a glance.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the derivation is empirical and self-contained against external baselines.

full rationale

The paper's central claim is that a pretrained facial-image encoder improves turn-taking prediction. The model is trained on the NoXi corpus and evaluated with balanced-accuracy metrics inherited from Ekstedt and Skantze's VAP work; no parameter is fitted to the evaluation set, no evaluation quantity is defined in terms of the model's own output, and no claimed prediction is equivalent to a training input by construction. The pretrained encoders (Former-DFER and CPC) are external artifacts with their own training data and are not derived from this paper's outputs. The only self-citations are dataset and background references (e.g., Cafaro et al. for NoXi, and Cafaro et al. for interruption perception), and they are not load-bearing for the core result. The manuscript's discussion does contain a causal attribution that is not fully controlled - Section 5 says the SP/BC difference 'lies in the use of facial embeddings from the pre-trained encoder,' while Section 3 changes the fusion architecture simultaneously - but that is a confound in experimental design, not a circular reduction: the model's predictions are not defined in terms of the face encoder's outputs, nor is the performance metric constructed from the proposed model's own fitted values. Accordingly, the circularity score is 0.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new entities. The central claim rests on a handful of hand-chosen hyperparameters and on domain assumptions about the VAP target representation and the transferability of pretrained encoders.

free parameters (5)
  • learning rate = 3.63e-4
    Hand-chosen hyperparameter for training the proposed models; the specific value suggests tuning, but no search is reported.
  • weight decay = 0.001
    AdamW hyperparameter, likely chosen based on prior work.
  • batch size = 256
    Used with gradient accumulation 16; chosen for the two Titan XP GPUs.
  • input sequence length = 20 seconds
    Inherited from Ekstedt and Onishi; a modeling choice.
  • early stopping patience = 5 epochs
    Hand-chosen to stabilize training.
assumptions (4)
  • domain assumption VAP state representation with 2x4 binary bins where bin durations increase with distance
    Assumed from Ekstedt's model (Section 2) as the prediction target; the paper does not justify this representation.
  • domain assumption The NoXi French subset is representative enough for evaluating turn-taking models
    The paper uses only the French subset (7 hours, 21 interactions) to simplify the setting (Section 4.1), without cross-corpus validation.
  • domain assumption Pretrained encoders (Former-DFER and CPC) provide useful representations for turn-taking cues
    The paper relies on these encoders but does not analyze what they capture or whether their pretraining domain (emotion recognition, speech) transfers to turn-taking.
  • domain assumption Balanced accuracy is a sufficient metric without statistical significance testing
    The paper reports single accuracies without standard deviations or significance tests (Section 5).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Voice Activity Projection Model with Multimodal Encoders." pith.science (2026). https://pith.science/paper/ODCFYRSA

@misc{pith2026250603980,
  author       = {Pith},
  title        = {Pith review of: Voice Activity Projection Model with Multimodal Encoders},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ODCFYRSA}},
  note         = {Machine review of arXiv:2506.03980}
}
read the original abstract

Turn-taking management is crucial for any social interaction. Still, it is challenging to model human-machine interaction due to the complexity of the social context and its multimodal nature. Unlike conventional systems based on silence duration, previous existing voice activity projection (VAP) models successfully utilized a unified representation of turn-taking behaviors as prediction targets, which improved turn-taking prediction performance. Recently, a multimodal VAP model outperformed the previous state-of-the-art model by a significant margin. In this paper, we propose a multimodal model enhanced with pre-trained audio and face encoders to improve performance by capturing subtle expressions. Our model performed competitively, and in some cases, even better than state-of-the-art models on turn-taking metrics. All the source codes and pretrained models are available at https://github.com/sagatake/VAPwithAudioFaceEncoders.

Figures

Figures reproduced from arXiv: 2506.03980 by the authors.

Figure 1
Figure 1. VAP window turn-taking cues, proposing to enhance Ekstedt’s voice activity projection model with pre-trained encoders for audio and fa￾cial signals [18]. We hypothesize that combining the encoders will contribute to capturing the coordinated, multimodal inten￾tions of the speakers, conveyed by both facial and audio signals. Specifically, by adding a facial image encoder, the subtle social context of facial expressio… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Multimodal Voice Activity Projection for Turn-Taking in Social Robots with Voice-Activity-Related Pretrained Encoders

    cs.RO 2026-07 conditional novelty 5.0 of 10

    Pretrained audio-visual speech encoders adapted with LoRA improve multimodal voice activity projection for turn-taking prediction across multiple languages and a robot mediation corpus.

Reference graph

Works this paper leans on

52 extracted references · 51 canonical work pages · cited by 1 Pith paper

  1. [1]

    Smooth turn-taking enables coherent dialogue, preventing everyone from speaking simultaneously

    Introduction Turn-taking is a fundamental and universal function in social in- teractions, enabling the coordinated exchange of speaking turns between participants in a conversation, which is essential for successful social communication [1, 2]. Smooth turn-taking enables coherent dialogue, preventing everyone from speaking simultaneously. We can divide t...

    arXiv 2025

  2. [2]

    V oice activity projection One of the successful models in turn-taking prediction is the voice activity projection (V AP) by Ekstedt et al. [18]. Un- like conventional voice activity detection (V AD) models, which consider only the voice activity of a single speaker, the V AP model predicts both users’ speaking states (e.g., speaking or not speaking) and ...

    work page

  3. [3]

    Unlike Onishi’s multimodal model, we in- serted a pretrained encoder for facial images instead of action units

    Proposed model Figure 2 shows the network architecture of our proposed mul- timodal model. Unlike Onishi’s multimodal model, we in- serted a pretrained encoder for facial images instead of action units. Since Onishi and colleagues reported that face infor- mation contributed the most, we hypothesized that richer rep- resentations extracted with a pre-trai...

    work page

  4. [4]

    Proposed1: audio signal, face image sequence

    work page

  5. [5]

    Proposed2: audio signal, face image sequence, head angles, gaze angles, and normalized 2D body positions

    work page

  6. [6]

    Onishi’s approach involved merging user signals for each modality sepa- rately and then fusing across different modalities

    Proposed3: audio signal, face image sequence, face AU se- quence, head angles, gaze angles, and normalized 2D body positions Furthermore, we also changed our fusion methods. Onishi’s approach involved merging user signals for each modality sepa- rately and then fusing across different modalities. We modified the architecture to capture interactions betwee...

    work page

  7. [7]

    Dataset: NoXi We used the NoXi multimodal dataset for our experiments [37]

    Method 4.1. Dataset: NoXi We used the NoXi multimodal dataset for our experiments [37]. NoXi is the dataset of screen-mediated multimodal face-to-face interactions. This dataset includes dyadic interaction videos from Germany, the United Kingdom, and France, and was re- cently extended with a Japanese subset [22]. We used the French subset to simplify the...

    work page

  8. [8]

    For SP and BC, our models (Proposed1 and Proposed3) outperformed the baselines, where the differ- ence lies in the use of facial embeddings from the pre-trained encoder

    Result and discussion Table 1 shows class-balanced accuracies of the trained models, where S/H indicates shift-hold prediction, S/L indicates short- long prediction, SP indicates shift prediction, and BC indicates backchannel prediction. For SP and BC, our models (Proposed1 and Proposed3) outperformed the baselines, where the differ- ence lies in the use ...

    work page

Show all 52 references
  1. [9]

    By incorporating a face image encoder, the proposed models demonstrated competitive or even superior performance compared to the multimodal state-of-the-art (SoTA) models

    Conclusion This paper investigated the performance of the voice activity projection model enhanced by a pretrained facial image en- coder. By incorporating a face image encoder, the proposed models demonstrated competitive or even superior performance compared to the multimoda...

  2. [10]

    Acknowledgements This work was funded by the French ANR (project EN- HANCER ANR22-CE17-0036-02)

  3. [11]

    A simplest systemat- ics for the organization of turn-taking for conversation,

    H. Sacks, E. A. Schegloff, and G. Jefferson, “A simplest systemat- ics for the organization of turn-taking for conversation,”Language (Baltim.), vol. 50, no. 4, pp. 696–735, 1974

    1974

  4. [12]

    Turn-taking: A critical analysis of the research tradition,

    D. C. O’Connell, S. Kowal, and E. Kaltenbacher, “Turn-taking: A critical analysis of the research tradition,” J. Psycholinguist. Res., vol. 19, no. 6, pp. 345–373, Nov. 1990

    1990

  5. [13]

    Some signals and rules for taking speaking turns in conversations,

    S. Duncan, “Some signals and rules for taking speaking turns in conversations,” Journal of Personality and Social Psychology , vol. 23, no. 2, pp. 283–292, Aug. 1972

    1972

  6. [14]

    Interactional units in conver- sation: Syntactic, intonational, and pragmatic resources for the management of turns,

    C. E. Ford and S. A. Thompson, “Interactional units in conver- sation: Syntactic, intonational, and pragmatic resources for the management of turns,” Studies in interactional sociolinguistics , vol. 13, pp. 134–184, 1996

    1996

  7. [15]

    N. G. Ward, Prosodic patterns in English conversation . Cam- bridge University Press, 2019

    2019

  8. [16]

    Using uh and um in spontaneous speaking,

    H. H. Clark and J. E. Fox Tree, “Using uh and um in spontaneous speaking,” Cognition, vol. 84, no. 1, pp. 73–111, May 2002

    2002

  9. [17]

    Using prosodic clues to decide when to produce back- channel utterances,

    N. Ward, “Using prosodic clues to decide when to produce back- channel utterances,” in Proceeding of F ourth International Con- ference on Spoken Language Processing. ICSLP ’96 , vol. 3. IEEE, 1996, pp. 1728–1731 vol.3

    1996

  10. [18]

    Nonverbal behaviours improving a simulation of small group discussion,

    E. Padilha and J. Carletta, “Nonverbal behaviours improving a simulation of small group discussion,” in The 1st Nordic Sympo- sium on Multimodal Communication , Sep. 2003, pp. 93–105

    2003

  11. [19]

    Sustaining interaction dynamics and engage- ment in dyadic child-robot interaction kinesics: lessons learnt from an exploratory study,

    B. Robins, K. Dautenhahn, C. L. Nehaniv, N. A. Mirza, D. Fran- cois, and L. Olsson, “Sustaining interaction dynamics and engage- ment in dyadic child-robot interaction kinesics: lessons learnt from an exploratory study,” in ROMAN 2005. IEEE Interna- tional Workshop on Robot an...

    2005

  12. [20]

    The effects of interrupting behavior on interpersonal attitude and engagement in dyadic in- teractions,

    A. Cafaro, N. Glas, and C. Pelachaud, “The effects of interrupting behavior on interpersonal attitude and engagement in dyadic in- teractions,” in Proceedings of the 2016 International Conference on Autonomous Agents & Multiagent Systems, 2016, pp. 911–920

    2016

  13. [21]

    Now or when? interrup- tion timing prediction in dyadic interaction,

    L. Yang, C. Achard, and C. Pelachaud, “Now or when? interrup- tion timing prediction in dyadic interaction,” inProceedings of the 23rd ACM International Conference on Intelligent Virtual Agents, 2023, pp. 1–4

    2023

  14. [22]

    How turn-taking strategies influence users’ impressions of an agent,

    M. ter Maat, K. P. Truong, and D. Heylen, “How turn-taking strategies influence users’ impressions of an agent,” in Intelligent Virtual Agents, ser. Lecture notes in computer science. Berlin, Heidelberg: Springer Berlin Heidelberg, 2010, pp. 441–453

    2010

  15. [23]

    Timing in turn-taking and its im- plications for processing models of language,

    S. Levinson and F. Torreira, “Timing in turn-taking and its im- plications for processing models of language,” Front. Psychol., vol. 6, Jun. 2015

    2015

  16. [24]

    Timing in conversation,

    A. S. Meyer, “Timing in conversation,” J. Cogn. , vol. 6, no. 1, p. 20, Apr. 2023

    2023

  17. [25]

    Can I finish?: learning when to respond to incremental interpretation results in interac- tive dialogue,

    D. DeVault, K. Sagae, and D. Traum, “Can I finish?: learning when to respond to incremental interpretation results in interac- tive dialogue,” in Proceedings of the SIGDIAL 2009 Conference on The 10th Annual Meeting of the Special Interest Group on Dis- course and Dialogue - S...

    2009

  18. [26]

    Investigating fluidity for human- robot interaction with real-time, real-world grounding strategies,

    J. Hough and D. Schlangen, “Investigating fluidity for human- robot interaction with real-time, real-world grounding strategies,” in Proceedings of the 17th Annual Meeting of the Special Inter- est Group on Discourse and Dialogue , R. Fernandez, W. Minker, G. Carenini, R. Higa...

    2016

  19. [27]

    Attentive listening system with backchanneling, response generation and flexible turn-taking,

    D. Lala, P. Milhorat, K. Inoue, M. Ishida, K. Takanashi, and T. Kawahara, “Attentive listening system with backchanneling, response generation and flexible turn-taking,” in Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue . Stroudsburg, PA, USA: Associa...

    2017

  20. [28]

    V oice activity projection: Self- supervised learning of turn-taking events,

    E. Ekstedt and G. Skantze, “V oice activity projection: Self- supervised learning of turn-taking events,” in Interspeech 2022 . ISCA: ISCA, Sep. 2022, pp. 5190–5194

    2022

  21. [29]

    Pre- dicting next speaker and timing from gaze transition patterns in multi-party meetings,

    R. Ishii, K. Otsuka, S. Kumano, M. Matsuda, and J. Yamato, “Pre- dicting next speaker and timing from gaze transition patterns in multi-party meetings,” in Proceedings of the 15th ACM on Inter- national conference on multimodal interaction . New York, NY , USA: ACM, Dec. 2013

    2013

  22. [30]

    Predicting turn-taking by compact gazing transition patterns in multiparty conversation,

    L. Tian, Q. Jia, and Z. Zhu, “Predicting turn-taking by compact gazing transition patterns in multiparty conversation,” in Image and Video Technology , ser. Lecture notes in computer science. Cham: Springer International Publishing, 2018, pp. 437–447

    2018

  23. [31]

    Prediction of who will be next speaker and when using mouth- opening pattern in multi-party conversation,

    R. Ishii, K. Otsuka, S. Kumano, R. Higashinaka, and J. Tomita, “Prediction of who will be next speaker and when using mouth- opening pattern in multi-party conversation,”Multimodal Technol. Interact., vol. 3, no. 4, p. 70, Oct. 2019

    2019

  24. [32]

    Multimodal voice ac- tivity prediction: Turn-taking events detection in expert-novice conversation,

    K. Onishi, H. Tanaka, and S. Nakamura, “Multimodal voice ac- tivity prediction: Turn-taking events detection in expert-novice conversation,” in International Conference on Human-Agent In- teraction. New York, NY , USA: ACM, Dec. 2023, pp. 13–21

    2023

  25. [33]

    Turn-taking and backchannel prediction with acoustic and large language model fusion,

    J. Wang, L. Chen, A. Khare, A. Raju, P. Dheram, D. He, M. Wu, A. Stolcke, and V . Ravichandran, “Turn-taking and backchannel prediction with acoustic and large language model fusion,” arXiv [cs.CL], Jan. 2024

    2024

  26. [34]

    Predictive turn-taking: Leveraging language models to anticipate turn transitions in human-robot di- alogue,

    M. J. Pinto and T. Belpaeme, “Predictive turn-taking: Leveraging language models to anticipate turn transitions in human-robot di- alogue,” in 2024 33rd IEEE International Conference on Robot and Human Interactive Communication (ROMAN) . IEEE, Aug. 2024, pp. 1733–1738

    2024

  27. [35]

    Mini-omni: Language models can hear, talk while thinking in streaming,

    Z. Xie and C. Wu, “Mini-omni: Language models can hear, talk while thinking in streaming,” arXiv [cs.AI], Aug. 2024

    2024

  28. [36]

    Moshi: a speech-text foundation model for real-time dialogue,

    A. D ´efossez, L. Mazar´e, M. Orsini, A. Royer, P. P´erez, H. J´egou, E. Grave, and N. Zeghidour, “Moshi: a speech-text foundation model for real-time dialogue,” arXiv [eess.AS], Sep. 2024

    2024

  29. [37]

    [Online]

    Google. [Online]. Available: https://support.google.com/gemini/ answer/15274899?hl=en&co=GENIE.Platform%3DAndroid

  30. [38]

    [Online]

    OpenAI. [Online]. Available: https://help.openai.com/en/articles/ 8400625-voice-mode-faq

  31. [39]

    Real-time and continuous turn-taking prediction using voice ac- tivity projection,

    K. Inoue, B. Jiang, E. Ekstedt, T. Kawahara, and G. Skantze, “Real-time and continuous turn-taking prediction using voice ac- tivity projection,” arXiv [cs.CL], Jan. 2024

    2024

  32. [40]

    Multilingual turn-taking prediction using voice activity projection,

    ——, “Multilingual turn-taking prediction using voice activity projection,” arXiv [cs.CL], Mar. 2024

    2024

  33. [41]

    How much does prosody help turn- taking? investigations using voice activity projection models,

    E. Ekstedt and G. Skantze, “How much does prosody help turn- taking? investigations using voice activity projection models,” arXiv [eess.AS], Sep. 2022

    2022

  34. [42]

    OpenFace: An open source facial behavior analysis toolkit,

    T. Baltrusaitis, P. Robinson, and L.-P. Morency, “OpenFace: An open source facial behavior analysis toolkit,” in2016 IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE, Mar. 2016, pp. 1–10

    2016

  35. [43]

    Open- pose: Realtime multi-person 2d pose estimation using part affinity fields,

    Z. Cao, G. Hidalgo, T. Simon, S.-E. Wei, and Y . Sheikh, “Open- pose: Realtime multi-person 2d pose estimation using part affinity fields,” IEEE Transactions on Pattern Analysis and Machine In- telligence, vol. 43, no. 1, pp. 172–186, 2021

    2021

  36. [44]

    Former-dfer: Dynamic facial expression recognition transformer,

    Z. Zhao and Q. Liu, “Former-dfer: Dynamic facial expression recognition transformer,” in Proceedings of the 29th ACM Inter- national Conference on Multimedia, 2021, pp. 1553–1561

    2021

  37. [45]

    Unsu- pervised pretraining transfers well across languages,

    M. Rivi `ere, A. Joulin, P.-E. Mazar ´e, and E. Dupoux, “Unsu- pervised pretraining transfers well across languages,” in ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020, pp. 7414–7418

    2020

  38. [46]

    Dlib-ml: A machine learning toolkit,

    D. E. King, “Dlib-ml: A machine learning toolkit,” J. Mach. Learn. Res., vol. 10, no. 60, pp. 1755–1758, Dec. 2009

    2009

  39. [47]

    The NoXi database: multimodal recordings of mediated novice-expert interactions,

    A. Cafaro, J. Wagner, T. Baur, S. Dermouche, M. Torres Torres, C. Pelachaud, E. Andr ´e, and M. Valstar, “The NoXi database: multimodal recordings of mediated novice-expert interactions,” in Proceedings of the 19th ACM International Conference on Multi- modal Interaction. New ...

    2017

  40. [48]

    PyTorch Lightning,

    W. Falcon and T. P. L. team, “PyTorch Lightning,” 12

  41. [50]

    Discourse as an interactional achievement iii: The omnirelevance of action,

    E. A. Schegloff, “Discourse as an interactional achievement iii: The omnirelevance of action,” Research on Language and Social Interaction , vol. 28, no. 3, pp. 185–211, 1995. [Online]. Available: https://doi.org/10.1207/s15327973rlsi2803 2

    1995 doi

  42. [51]

    Between and within: Alternative sequential treat- ments of continuers and assessments,

    C. Goodwin, “Between and within: Alternative sequential treat- ments of continuers and assessments,”Hum. Stud., vol. 9, no. 2-3, pp. 205–217, 1986

    1986

  43. [52]

    Yeah, un, oh: Continuous and real-time backchannel prediction with fine-tuning of voice activity projection,

    K. Inoue, D. Lala, G. Skantze, and T. Kawahara, “Yeah, un, oh: Continuous and real-time backchannel prediction with fine-tuning of voice activity projection,” arXiv [cs.CL], Oct. 2024

    2024

  44. [2019]

    Available: https://github.com/Lightning-AI/ pytorch-lightning

    [Online]. Available: https://github.com/Lightning-AI/ pytorch-lightning

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.