Pith. sign in

REVIEW 3 major objections 4 minor 68 references

Multimodal Transformer Models for Turn-taking Prediction: Effects on Conversational Dynamics of Human-Agent Interaction during Cooperative Gameplay

T0 review · 3 major / 4 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A model that fuses text, vision, audio, and game state predicts avatar speaking moments one second ahead and changes live conversation dynamics without changing talk volume.

desk verdict The user-study claim is overstated—offline model is a decent extension but the deployed effect is unverified and all perception results are null. read the letter →

arxiv 2503.16432 v1 pith:34ONMB5U submitted 2025-02-05 cs.HC cs.AIcs.CL

Young-Ho Bae , Casey C. Bennett This is my paper · ORCID
classification cs.HCcs.AIcs.CL
keywords Human-AgentInteractionConversationalAgentsTurn-takingUserstudySocialCognitionmultimodaltransformercooperativegameplayspokendialoguesystems
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish two connected claims: a transformer model that fuses text, vision, audio, and in-game context can predict, one second ahead, when a virtual avatar should begin speaking, and deploying that model in a cooperative game changes how people interact with the avatar. On the modeling side, the paper reports 87.3% accuracy and 83.0% macro F1 on a held-out benchmark, several points above early- and late-fusion LSTM baselines and a standard multimodal transformer. On the interaction side, the paper reports that the model kept utterance counts about the same, modestly lowered the avatar's interruption ratio, and left participants' questionnaire ratings of the avatar statistically unchanged. A sympathetic reader would care because turn-taking is a longstanding weak point of spoken dialogue systems: if an agent can time its speech better without talking more or less, voice assistants, game characters, and robots could feel more natural without retraining for every situation.

What carries the argument

The load-bearing object is the Crossmodal Transformer, a set of twelve pairwise cross-attention blocks one for each ordered pair of the four modalities built on the style of the multimodal transformer in [47]. Each modality's features are first encoded by 1D temporal convolution and a bidirectional LSTM (a recurrent network that processes a sequence forward and backward), then enriched by attention from the other three modalities, concatenated, passed through another bidirectional LSTM, and mapped to a sigmoid probability. The mechanism that makes the model continuous rather than pause-based is a sliding window: the previous five seconds of text, vision, audio, and game state predict a turn-taking event in the next second, with the one-second horizon chosen to span language-production latency (600-1500 ms) and typical gaps between turns (100-300 ms). Turn-taking labels are defined by speaker transitions within that one-second window, following the continuous turn-taking method of [21].

What would settle it

Count, per session, how many times the turn-taking model vetoed an avatar utterance that the Social AI's internal logic wanted to produce, and how many times it triggered an utterance that logic would not have produced. If those counts are near zero, or if the avatar's speech-onset times are statistically indistinguishable between the Turn-taking and Control conditions, the user-study comparison does not test the model. For the benchmark side, an independent re-run on the same held-out split should reproduce the reported 87.3% accuracy within a point or two.

Watch

Extended reading notes

Core claim

The central claim is that continuous, context-aware turn-taking prediction is both learnable and deployable. The paper's Crossmodal Transformer cross-attends between four modalities text, vision, audio, and 40 in-game state variables over a sliding five-second window and outputs the probability that the next second offers a turn-taking opportunity for the avatar. On the benchmark this gives 87.3% accuracy and 83.0% macro F1 (the unweighted average of the per-class F1 scores), beating the strongest baseline by 4.5 accuracy points and 6.0 F1 points; quantization cuts inference time to 25.8 ms on a desktop CPU without changing accuracy. In the user study, 40 participants played a cooperative survival game with the model active and 20 played against the same avatar without it. The paper reports that utterance counts stayed statistically flat, the avatar's interruption percentage dropped in both Korean and English without reaching significance, the one significant difference was Korean participants interrupting the model-driven avatar more often (5.72% versus 2.53% of avatar utterances), and the Godspeed and Networked Minds ratings did not differ between conditions. The paper interprets this as evidence that multimodal turn-taking prediction can smooth interaction rhythm without distorting dialogue volume, and that the Korean/English asymmetry points toward language-specific turn-taking cues.

Load-bearing premise

The user-study conclusion assumes the turn-taking model actually changed when the avatar spoke during the sessions, but the paper reports no count of how often the model vetoed or triggered utterances, so the avatar's timing may have been nearly identical in the two conditions.

Editorial extensions

If this is right

  • If the benchmark result holds, the model can decide whether to speak in about 26 ms after optimization on a desktop CPU, leaving most of the roughly 200 ms gap between turns for content generation and speech synthesis.
  • The in-game modality is load-bearing for the claimed accuracy: removing it drops accuracy from 87.3% to 85.1%, so environment state adds signal beyond speech and face.
  • Deploying the model keeps utterance counts statistically unchanged in Korean and English, which the paper takes as evidence that the agent can time its turns without suppressing conversation volume.
  • The avatar's interruption percentage falls in both languages under the model but not significantly, while Korean participants interrupt the model-driven avatar significantly more often, which the paper reads as a possible language-specific mismatch in turn-taking cues.
  • Perception ratings on the two instruments show no significant differences, so within a 30-minute session reduced interruption frequency alone did not move perceived intelligence, likeability, animacy, or social presence.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: a pairwise ablation of the twelve crossmodal attention blocks would identify which direction of information flow matters most, since the reported ablation only removes whole modalities rather than individual crossmodal paths.
  • Beyond the paper: porting the architecture to a second cooperative task with its own logged context features would test whether the in-game modality's contribution is task-general or specific to the survival-game state space.
  • Beyond the paper: the one-second prediction horizon could be made adaptive and learned from data, which would test whether production latency or short turn gaps dominate the timing signal.
  • Beyond the paper: measuring inference on mobile or embedded hardware would extend the real-time deployability claim beyond the reported desktop CPU timing.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This paper proposes a multimodal transformer model that fuses text, vision, audio, and in-game data to predict turn-taking opportunities in human-agent dialogue. The model is first evaluated offline on a held-out test set from the authors' previous HAI dataset, reporting 87.3% accuracy and 83.0% macro F1, with an ablation study and a TFLite optimization showing 25.8 ms inference. The paper then reports a user study (n=60) in which the model is deployed as a veto gate on an avatar's speech during cooperative gameplay in Don't Starve Together, comparing a Turn-taking condition to a Control condition on utterance counts, interruption rates, and Godspeed/Networked Minds perceptions. The paper claims the model outperforms baselines and that the user study shows enhanced fluidity and naturalness, but the reported statistics are largely null or mixed.

Significance. The offline benchmark is a useful contribution: the model achieves strong accuracy on a realistic, multimodal HAI dataset, the ablation study indicates that all modalities contribute, and the TFLite optimization demonstrates real-time feasibility. These results are reproducible from the described pipeline, although no code is provided. The user study is well motivated by the cross-linguistic comparison, but the evidence for the headline claim is weak: the key interaction effects are null or in the opposite direction, and there is no manipulation check to confirm that the model actually changed avatar behavior. As a result, the paper is significant mainly for its model design and benchmark, not for the user-study conclusions in their current form.

major comments (3)
  1. [§3.4.1, §3.3.4] Section 3.4.1 and Section 3.3.4 describe the turn-taking model as a veto gate on avatar speech, but the paper provides no manipulation check: there are no statistics on how often the model vetoed or approved utterances, no comparison of avatar speech-onset timing between conditions, and no log of model predictions during the user study. If the Social AI's internal timing already approximated the model's decisions, or if the model blocked only a handful of utterances, then the null and mixed results in Section 4.2 would not test the turn-taking model at all. Please report deployment statistics (e.g., veto/approval counts, prediction-to-speech alignment, timing distributions) or temper the user-study claims accordingly.
  2. [§4.2.2, Table 8; §4.2.3, Tables 9-10] The abstract and Section 5.1 state that the model 'enhances the fluidity and naturalness of human-agent conversations' and maintains 'a balanced conversational dynamic,' but the paper's own statistics contradict this: Table 8 shows no significant avatar interruption reduction (Korean p=0.0596, English p=0.2650), a significant increase in Korean human interruptions in the Turn-taking condition (5.72% vs 2.53%, p=0.0253), and Table 10 shows a significant decrease in English Animacy (2.90 vs 3.26, p=0.0454) with all other Godspeed and Networked Minds subscales null. Table 7 shows no significant utterance-count differences, which supports only the weaker claim that dialogue frequency was unchanged. The paper should be revised to present these results as null and mixed rather than as evidence of enhanced fluidity and naturalness.
  3. [§4.1.1, Table 4] Section 4.1.1 and Table 4 claim 'superior performance' over baselines based on single point estimates (87.3 vs 82.8 accuracy), but the text states that each model underwent 10 training iterations without reporting the variance across those iterations. Given that the two best baselines differ by only 0.1 accuracy points (82.7 vs 82.8), it is unclear whether the reported advantage is stable. Please report the mean and standard deviation (or confidence intervals) across runs for all models, or state how many independent training runs the reported numbers are based on.
minor comments (4)
  1. [§3.3.4] The rule that short ASR speech segments are always classified as 'class1' is stated without a quantitative definition of 'short'; please specify the duration threshold and discuss how many samples are affected, since backchannels are later acknowledged as a limitation.
  2. [§4.2.3] The significant English Animacy decrease (p=0.0454) is not mentioned in the text or discussion; please include it in the interpretation of instrumental results.
  3. [§5.2] The phrase 'predicting enhancing HAI' appears to be a typo for 'predicting and enhancing HAI.'
  4. [Equation (5)] The notation in Equation (5) is unclear: C_G appears without a definition, and the subscripts on the layer-normalized inputs should be checked for consistency.

Circularity Check

0 steps flagged · score 0.0 of 10

No meaningful circularity: the held-out benchmark and the separate user study keep the central claims from reducing to the model's training inputs.

full rationale

The model-development claim (87.3% accuracy, 83.0% macro F1) is evaluated on a held-out 20% of the benchmarking dataset (Section 3.3.8), so the headline numbers are not fitted values from the test set. The target label is defined behaviorally as speaker transitions within a one-second window (Section 3.3.4), while the model inputs are features from the preceding five seconds (Section 3.3.1); the prediction is therefore not the same quantity as the input by construction. Baselines (EF-LSTM, LF-LSTM, Mult) are trained on identical data (Section 4.1.1), so the comparison is not self-fulfilling. The paper does cite the authors' prior work [31] for the training dataset and [48] for additional architecture details, but those are normal provenance citations rather than load-bearing uniqueness arguments; the current paper specifies its own architecture equations, hyperparameters, and evaluation. The user study is an independent comparison of model-on versus model-off conditions with 60 new participants, and no user-study outcome is fed back into the model. A legitimate validity concern exists: Section 3.3.4 says the model only vetoes avatar speech, and no deployment statistics are reported, so the manipulation may have been weak. That is an evidentiary weakness, not a circular derivation. Under the stated rules, no step in the claimed derivation chain reduces to its own input.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The prediction threshold, context window, focal loss coefficients, and sequence lengths are hand-set and not derived from data, and no sensitivity analysis is reported. The key domain axioms are that one-second-ahead events are the right target, the heuristic labels are valid ground truth, and the questionnaires can detect naturalness changes.

free parameters (5)
  • Prediction window threshold = 1 second
    A turn-taking event is defined as a speaker transition within a fixed one-second window, chosen to balance language production latency and typical turn-taking gaps, not derived from data.
  • Context window length = 5 seconds
    The model uses the preceding five seconds of multimodal data; no ablation over the window length is reported.
  • Focal loss alpha = 0.3
    Balancing factor in the focal loss, set by hand with no sensitivity analysis.
  • Focal loss gamma = 2
    Focusing parameter in the focal loss, a standard default, not tuned in this paper.
  • Sequence lengths T/V/A/G = 35/5/10/5
    Fixed sequence lengths per modality, chosen for efficiency; no sensitivity analysis is provided.
assumptions (6)
  • standard math Transformer, LSTM, and focal loss formulations are accepted as given.
    The paper uses standard deep learning components without proving or questioning their behavior.
  • domain assumption A one-second lookahead window is the correct timescale for turn-taking opportunities.
    Section 3.3.1 introduces the one-second prediction window; this is load-bearing for the entire label definition.
  • domain assumption The manually defined turn-taking labels from IPUs and speaker transitions are a valid ground truth for appropriate avatar speech timing.
    Section 3.3.4 describes the labeling rule; the model's target variable inherits any flaws in this heuristic.
  • ad hoc to paper Short speech segments from ASR are always classified as class1, meaning no turn-taking event.
    Section 3.3.4 states this rule; it shapes the training labels and may suppress genuine turn-taking events that happen to be brief.
  • domain assumption The Godspeed and Networked Minds instruments are sensitive enough to detect turn-taking effects on perceived naturalness.
    Section 4.2.3 discusses possible insensitivity of the instruments, which weakens any null interpretation.
  • domain assumption English and Korean participant groups are comparable despite different recruitment timelines and L1/L2 mix.
    Section 3.4.2 notes the English speakers were added later via IRB addendum and include both L1 and L2 speakers; this complicates cross-language comparisons.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Multimodal Transformer Models for Turn-taking Prediction: Effects on Conversational Dynamics of Human-Agent Interaction during Cooperative Gameplay." pith.science (2026). https://pith.science/paper/34ONMB5U

@misc{pith2026250316432,
  author       = {Pith},
  title        = {Pith review of: Multimodal Transformer Models for Turn-taking Prediction: Effects on Conversational Dynamics of Human-Agent Interaction during Cooperative Gameplay},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/34ONMB5U}},
  note         = {Machine review of arXiv:2503.16432}
}
read the original abstract

This study investigates multimodal turn-taking prediction within human-agent interactions (HAI), particularly focusing on cooperative gaming environments. It comprises both model development and subsequent user study, aiming to refine our understanding and improve conversational dynamics in spoken dialogue systems (SDSs). For the modeling phase, we introduce a novel transformer-based deep learning (DL) model that simultaneously integrates multiple modalities - text, vision, audio, and contextual in-game data to predict turn-taking events in real-time. Our model employs a Crossmodal Transformer architecture to effectively fuse information from these diverse modalities, enabling more comprehensive turn-taking predictions. The model demonstrates superior performance compared to baseline models, achieving 87.3% accuracy and 83.0% macro F1 score. A human user study was then conducted to empirically evaluate the turn-taking DL model in an interactive scenario with a virtual avatar while playing the game "Dont Starve Together", comparing a control condition without turn-taking prediction (n=20) to an experimental condition with our model deployed (n=40). Both conditions included a mix of English and Korean speakers, since turn-taking cues are known to vary by culture. We then analyzed the interaction quality, examining aspects such as utterance counts, interruption frequency, and participant perceptions of the avatar. Results from the user study suggest that our multimodal turn-taking model not only enhances the fluidity and naturalness of human-agent conversations, but also maintains a balanced conversational dynamic without significantly altering dialogue frequency. The study provides in-depth insights into the influence of turn-taking abilities on user perceptions and interaction quality, underscoring the potential for more contextually adaptive and responsive conversational agents.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

68 extracted references · 60 canonical work pages

  1. [1]

    S. C. Levinson, Turn-taking in human communication –origins and im - plications for language processing, Trends in cognitive sciences 20 (2016) 6–14

    work page 2016

  2. [2]

    Skantze, Turn-taking in conversational systems and human-robot interaction: a review, Computer Speech & Language 67 (2021) 101178

    G. Skantze, Turn-taking in conversational systems and human-robot interaction: a review, Computer Speech & Language 67 (2021) 101178

    work page 2021

  3. [3]

    Khouzaimi, R

    H. Khouzaimi, R. Laroche, F. Lef`evre, A methodology for turn -taking capabilities enhancement in spoken dialogue systems using reinforce - ment learning, Computer Speech & Language 47 (2018) 93–111

    work page 2018

  4. [4]

    Heldner, J

    M. Heldner, J. Edlund, Pauses, gaps and overlaps in conversations, Journal of Phonetics 38 (2010) 555–568

    work page 2010

  5. [5]

    N. Ward, W. Tsukahara, Prosodic features which cue back -channel responses in english and japanese, Journal of pragmatics 32 (2000) 1177– 1207

    work page 2000

  6. [6]

    E. A. Schegloff, Overlapping talk and the organization of turn -taking for conversation, Language in society 29 (2000) 1–63

    work page 2000

  7. [7]

    A. Raux, M. Eskenazi, A finite -state turn-taking model for spoken dialog systems, in: Proceedings of human language technologies: The 2009 annual conference of the North American chapter of the association for computational linguistics, 2009, pp. 629–637

    work page 2009

  8. [8]

    Sacks, E

    H. Sacks, E. A. Schegloff, G. Jefferson, A simplest systematics for the organization of turn -taking for conversation, Language 50 (1974) 696– 735. URL: http://www.jstor.org/stable/412243

    work page 1974

Show all 68 references
  1. [9]

    Gravano, J

    A. Gravano, J. Hirschberg, Turn-taking cues in task -oriented dialogue, Computer Speech & Language 25 (2011) 601–634

    2011

  2. [10]

    Lundholm Fors, Production and perception of pauses in speech (2015)

    K. Lundholm Fors, Production and perception of pauses in speech (2015)

    2015

  3. [11]

    Clemens, C

    C. Clemens, C. Diekhaus, Prosodic turn -yielding cues with and without 32 optical feedback, in: Proceedings of the SIGDIAL 2009 Conference, 2009, pp. 107–110

    2009

  4. [12]

    Schlangen, G

    D. Schlangen, G. Skantze, A general, abstract model of incremental dialogue processing, Dialogue & Discourse 2 (2011) 83–111

    2011

  5. [13]

    Dua, Akanksha, S

    M. Dua, Akanksha, S. Dua, Noise robust automatic speech recognition: review and analysis, International Journal of Speech Technology 26 (2023) 475–519

    2023

  6. [14]

    Morbini, K

    F. Morbini, K. Audhkhasi, K. Sagae, R. Artstein, D. Can, P. Georgiou, S. Narayanan, A. Leuski, D. Traum, Which asr should i choose for my dialogue system?, in: Proceedings of the SIGDIAL 2013 Conference, 2013, pp. 394–403

    2013

  7. [15]

    Khouzaimi, R

    H. Khouzaimi, R. Laroche, F. Lefevre, Optimising turn-taking strategies with reinforcement learning, in: Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue, 2015, pp. 315– 324

    2015

  8. [16]

    Maier, J

    A. Maier, J. Hough, D. Schlangen, et al., Towards deep end -of-turn prediction for situated spoken dialogue systems (2017)

    2017

  9. [17]

    Ferrer, E

    L. Ferrer, E. Shriberg, A. Stolcke, Is the speaker done yet? faster and more accurate end -of-utterance detection using prosody, in: Seventh international conference on spoken language processing, 2002

    2002

  10. [18]

    Roddy, G

    M. Roddy, G. Skantze, N. Harte, Investigating speech features for continuous turn-taking prediction using lstms, arXiv preprint arXiv:1806.11461 (2018)

    2018 arXiv

  11. [19]

    Roddy, G

    M. Roddy, G. Skantze, N. Harte, Multimodal continuous turn -taking prediction using multiscale rnns, in: Proceedings of the 20th ACM International Conference on Multimodal Interaction, 2018, pp. 186–190

    2018

  12. [20]

    N. G. Ward, D. Aguirre, G. Cervantes, O. Fuentes, Turn -taking predic - tions across languages and genres using an lstm recurrent neural net - work, in: 2018 IEEE Spoken Language Technology Workshop (SLT), IEEE, 2018, pp. 831–837

    2018

  13. [21]

    G. Skantze, Towards a general, continuous model of turn-taking in spoken dialogue using lstm recurrent neural networks, in: Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, 2017, pp. 220–230

    2017

  14. [22]

    Al Moubayed, J

    S. Al Moubayed, J. Beskow, G. Skantze, B. Granstr¨om, Furhat: a back- projected human -like robot head for multiparty human -machine inter - action, in: Cognitive Behavioural Systems: COST 2102 International Training School, Dresden, Germany, February 21 -26, 2011, Revised Se ...

    2011

  15. [23]

    L. P. Morency, I. De Kok, J. Gratch, Predicting listener backchannels: A probabilistic multimodal approach, in: International workshop on intelligent virtual agents, Springer, 2008, pp. 176–190

    2008

  16. [24]

    Baltruˇsaitis, C

    T. Baltruˇsaitis, C. Ahuja, L.-P. Morency, Multimodal machine learning: A survey and taxonomy, IEEE transactions on pattern analysis and machine intelligence 41 (2018) 423–443

    2018

  17. [25]

    P. K. Atrey, M. A. Hossain, A. El Saddik, M. S. Kankanhalli, Multi - modal fusion for multimedia analysis: a survey, Multimedia systems 16 (2010) 345–379

    2010

  18. [26]

    S. Wang, M. Dash, L.-T. Chia, M. Xu, Efficient sampling of training set in large and noisy multimedia data, ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) 3 (2007) 14 – es

    2007

  19. [27]

    Noulas, G

    A. Noulas, G. Englebienne, B. J. Krose, Multimodal speaker diarization, IEEE Transactions on Pattern Analysis and Machine Intelligence 34 (2011) 79–93

    2011

  20. [28]

    W. Zhu, D. Boxer, Turn-taking and disagreement: A comparison of american english and mandarin chinese, Contrastive Pragmatics 2 (2021) 227–257

    2021

  21. [29]

    S. C. Levinson, F. Torreira, Timing in turn-taking and its implications for processing models of language, Frontiers in psychology 6 (2015) 731

    2015

  22. [30]

    Stivers, N

    T. Stivers, N. J. Enfield, P. Brown, C. Englert, M. Hayashi, T. Heine - mann, G. Hoymann, F. Rossano, J. P. De Ruiter, K. -E. Yoon, et al., Universals and cultural variation in turn -taking in conversation, Pro - ceedings of the National Academy of Sciences 106 (2009) 10587–10592

    2009

  23. [31]

    C. C. Bennett, Y.-H. Bae, J. H. Yoon, Y. Chae, E. Yoon, S. Lee, U. Ryu, S. Y. Kim, B. Weiss, Effects of cross-cultural language differences on social cognition during human -agent interaction in cooperative game environments, Computer Speech & Language 81 (2023) 101521

    2023

  24. [32]

    C. C. Bennett, B. Weiss, J. Suh, E. Yoon, J. Jeong, Y. Chae, Explori ng data-driven components of socially intelligent ai through cooperative game paradigms, Multimodal Technologies and Interaction 6 (2022) 16

    2022

  25. [33]

    J. Suh, C. C. Bennett, B. Weiss, E. Yoon, J. Jeong, Y. Chae, Develop - ment of speech dialogue systems for social ai in cooperative game envi- ronments, in: 2021 IEEE Region 10 Symposium (TENSYMP), IEEE, 2021, pp. 1–4

    2021

  26. [34]

    V. Sanh, L. Debut, J. Chaumond, T. Wolf, Distilbert, a distilled 34 version of bert: Smaller, faster, cheaper and lighter. arxiv 2019, arXiv preprint arXiv:1910.01108 (2019)

    2019 arXiv

  27. [35]

    Wu, Google’s neural machine translation system: Bridging the gap between human and machine translation, arXiv preprint arXiv:1609.08144 (2016)

    Y. Wu, Google’s neural machine translation system: Bridging the gap between human and machine translation, arXiv preprint arXiv:1609.08144 (2016)

    2016 arXiv

  28. [36]

    Kendon, Gesture: Visible action as utterance, Cambridge University Press, 2004

    A. Kendon, Gesture: Visible action as utterance, Cambridge University Press, 2004

    2004

  29. [37]

    Ekman, W

    P. Ekman, W. V. Friesen, Constants across cultures in the face and emotion., Journal of personality and social psychology 17 (1971) 124

    1971

  30. [38]

    Zhong, W

    Y. Zhong, W. Deng, J. Hu, D. Zhao, X. Li, D. Wen, Sface: Sigmoi d- constrained hypersphere loss for robust face recognition, IEEE Trans - actions on Image Processing 30 (2021) 2587–2598

    2021

  31. [39]

    Schuller, S

    B. Schuller, S. Steidl, A. Batliner, J. Hirschberg, J. K. Burgoon, A. Baird, A. Elkins, Y. Zhang, E. Coutinho, K. Evanini, The interspeech 2016 computational paralinguistics challenge: Deception, sincerity and native language (2016)

    2016

  32. [40]

    LeCun, L

    Y. LeCun, L. Bottou, Y. Bengio, P. Haffner, Gradient-based learning applied to document recognition, Proceedings of the IEEE 86 (1998) 2278–2324

    1998

  33. [41]

    Bengio, I

    Y. Bengio, I. Goodfellow, A. Courville, Deep learning, volume 1, MIT press Cambridge, MA, USA, 2017

    2017

  34. [42]

    S. Bai, J. Z. Kolter, V. Koltun, An empirical evaluation of generic con- volutional and recurrent networks for sequence modeling, arXiv preprint arXiv:1803.01271 (2018)

    2018 arXiv

  35. [43]

    Graves, J

    A. Graves, J. Schmidhuber, Framewise phoneme classification with bidi- rectional lstm and other neural network architectures, Neural networks 18 (2005) 602–610

    2005

  36. [44]

    Hochreiter, Long short-term memory, Neural Computation MIT - Press (1997)

    S. Hochreiter, Long short-term memory, Neural Computation MIT - Press (1997)

    1997

  37. [45]

    Vaswani, Attention is all you need, Advances in Neural Information Processing Systems (2017)

    A. Vaswani, Attention is all you need, Advances in Neural Information Processing Systems (2017)

    2017

  38. [46]

    Lei Ba, J

    J. Lei Ba, J. R. Kiros, G. E. Hinton, Layer normalization, ArXiv e - prints (2016) arXiv–1607

    2016

  39. [47]

    Y.-H. H. Tsai, S. Bai, P. P. Liang, J. Z. Kolter, L.-P. Morency, R. Salakhutdinov, Multimodal transformer for unaligned multimodal language sequences, in: Proceedings of the conference. Association for 35 computational linguistics. Meeting, volume 2019, NIH Public Access, 2019...

    2019

  40. [48]

    Y.-H. Bae, C. C. Bennett, Real-time multimodal turn -taking predic - tion to enhance cooperative dialogue during human -agent interaction, in: 2023 32nd IEEE International Conference on Robot and Human Interactive Communication (RO-MAN), IEEE, 2023, pp. 2037–2044

    2023

  41. [49]

    S. Z. Razavi, B. Kane, L. K. Schubert, Investigating linguistic and semantic features for turn -taking prediction in open -domain human - computer conversation., in: INTERSPEECH, 2019, pp. 4140 –4144

    2019

  42. [50]

    A. Raux, M. Eskenazi, Optimizing endpointing thresholds using dia - logue features in a spoken dialogue system, in: Proceedings of the 9th SIGdial Workshop on Discourse and Dialogue, 2008, pp. 1–10

    2008

  43. [51]

    Meena, G

    R. Meena, G. Skantze, J. Gustafson, Data-driven models for timing feedback responses in a map task dialogue system, Computer Speech & Language 28 (2014) 903–922

    2014

  44. [52]

    Johansson, G

    M. Johansson, G. Skantze, Opportunities and obligations to take turns in collaborative multi -party human-robot interaction, in: Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue, 2015, pp. 305–314

    2015

  45. [53]

    J. Yang, P. Wang, Y. Zhu, M. Feng, M. Chen, X. He, Gated multimodal fusion with contrastive learning for turn-taking prediction in human- robot dialogue, in: ICASSP 2022 -2022 IEEE Internat i onal Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, 2022, pp. ...

    2022

  46. [54]

    T.-Y. Lin, P. Goyal, R. Girshick, K. He, P. Doll´ar, Focal loss for dense object detection, in: Proceedings of the IEEE international conference on computer vision, 2017, pp. 2980–2988

    2017

  47. [55]

    Bartneck, D

    C. Bartneck, D. Kuli´c, E. Croft, S. Zoghbi, Measureme nt instruments for the anthropomorphism, animacy, likeability, perceived intelligence, and perceived safety of robots, International journal of social robotics 1 (2009) 71–81

    2009

  48. [56]

    Biocca, C

    F. Biocca, C. Harms, J. Gregg, The networked minds measure of social presence: Pilot test of the factor structure and concurrent validity, in: 4th annual international workshop on presence, Philadelphia, PA, 2001, pp. 1–9

    2001

  49. [57]

    C. S. Oh, J. N. Bailenson, G. F. Welch, A systematic review of so - cial presence: Definition, antecedents, and implications, Frontiers in Robotics and AI 5 (2018) 409295. 36

    2018

  50. [58]

    C. C. Bennett, M. Lee, Would people mumble rap to alexa?, in: Pro- ceedings of the 5th International Conference on Conversational User Interfaces, 2023, pp. 1–5

    2023

  51. [59]

    Ekstedt, G

    E. Ekstedt, G. Skantze, Turngpt: a transformer -based language model for predicting turn-taking in spoken dialog, arXiv preprint arXiv:2010.10874 (2020)

    2020 arXiv

  52. [60]

    De Ruiter, H

    J.-P. De Ruiter, H. Mitterer, N. J. Enfield, Projecting the end of a speaker’s turn: A cognitive cornerstone of conversation, Language 82 (2006) 515–535

    2006

  53. [61]

    Ehret, A

    J. Ehret, A. B¨onsch, P. Nossol, C. A. Ermert, C. Mohanathasan, S. J. Schlittmeier , J. Fels, T. W. Kuhlen, Who’s next? integrating non - verbal turn-taking cues for embodied conversational agents, in: Proceedings of the 23rd ACM International Conference on Intelligent Virtual...

    2023

  54. [62]

    Mlakar, M

    I. Mlakar, M. Rojc, D. Verdonik, S. Majheniˇc, Chapter can turn-taking highlight the nature of non-verbal behavior: A case study (2021)

    2021

  55. [63]

    T. Zhou, J. P. Wachs, Spiking neural networks for early prediction in human –robot collaboration, The International Journal of Robotics Research 38 (2019) 1619–1643

    2019

  56. [64]

    D. Lala, K. Inoue, T. Kawahara, Evaluation of real -time deep learning turn-taking models for multiple dialogue scenarios, in: Proceedings of the 20th ACM International Conference on Multimodal Interaction (ICMI), 2018, pp. 78–86

    2018

  57. [65]

    T. Zhou, J. P. Wachs, Early turn -taking prediction with spiking neural networks for human robot collaboration, in: 2018 IEEE International Conference on Robotics and Automation (ICRA), IEEE, 2018, pp. 3250 – 3256

    2018

  58. [66]

    C. C. Bennett, Findings from Studies on English -Based Conversational AI Agents (including ChatGPT) Are Not Universal. In: Proceedings of the 6th ACM Conference on Conversational User Interfaces (CUI), 2024, pp. 1-5

    2024

  59. [67]

    Irfan, S.-M

    B. Irfan, S.-M. Kuoppam¨aki, G. Skantze, Between reality and delusion: challenges of applying large language models to companion robots for open-domain dialogues with older adults (2023)

    2023

  60. [68]

    Y. Zhu, J. R. A. Moniz, S. Bhargava, J. Lu, D. Piraviperumal, S. Li, Y. Zhang, H. Yu, B.-H. Tseng, Can large language models understand context?, arXiv preprint arXiv:2402.00858 (2024)

    2024 arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.