REVIEW 4 major objections 4 minor 34 references
Gaze-Enhanced Multimodal Turn-Taking Prediction in Triadic Conversations
T0 review · 4 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read Gaze becomes a reliable turn-taking cue when anchored to the active speaker's location, and more participants' gaze helps more.
desk verdict Gaze's real contribution is unproven because the gaze models also add ASL audio spatial features; the paper needs ablations before its central claim can be trusted. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the pairing of two spatial heatmaps: a gaze heatmap built from six frames of azimuth/elevation samples and an active-speaker heatmap produced by an egocentric audio active-speaker localization model. Each user's two heatmaps pass through separate CNN feature extractors and a ConvLSTM to form a user feature vector; the multi-user model then combines the three users' vectors with a user-attention mechanism and concatenates the target user's voice-activity feature. This arrangement turns raw gaze direction into a spatially grounded cue, effectively letting the model read gaze as "looking at the current speaker," while keeping the whole pipeline lightweight enough for low-latency use.
What would settle it
Take the trained single- or multi-user model on held-out sessions and replace the active-speaker heatmaps with random or mislocalized speaker positions (or with the true speaker location measured independently); if turn-taking F1 stays as high as with the model's own heatmaps, the claim that spatial anchoring is what makes gaze useful is false, while if F1 falls to the voice-activity-only level, the claim is supported.
Extended reading notes
Core claim
The paper's central discovery is that gaze, though unreliable on its own, becomes a dependable predictive signal when placed in a spatial frame defined by the active speaker. The single-user model processes a 0.2-second gaze heatmap and an active-speaker heatmap from an audio-based localization model through CNN extractors and a ConvLSTM, then appends the target user's binary voice-activity feature. Reported F1 on role-based classification rises from 0.704 with voice activity alone to 0.746 with single-user gaze and 0.765 with multi-user gaze; on behavior-based transition-level analysis, turn-taking F1 rises from 0.081 to 0.218 to 0.400. A gaze-only model without voice activity failed to converge, so voice activity remains necessary; the improvement comes from combining it with spatially constrained gaze, not from gaze alone.
Load-bearing premise
The paper assumes, without validating it on its own recordings, that the pretrained active-speaker localization model correctly determines where the current speaker is from each user's egocentric audio; if its heatmaps point to the wrong location, gaze anchored to them is no longer a reliable cue.
Editorial extensions
If this is right
- In role-based classification, single-user gaze lifts overall F1 from 0.704 (voice activity only) to 0.746, and multi-user gaze lifts it to 0.765; the main-speaker class shows the clearest gain, from 0.761 to 0.835 to 0.869.
- At the transition level, where turn changes actually happen, gaze has its largest effect: turn-taking F1 rises from 0.081 with voice activity alone to 0.218 with single-user gaze and 0.400 with multi-user gaze.
- Gaze alone is not sufficient: a gaze-only model without voice activity did not converge, so the practical system still needs binary voice activity to separate silence from speech.
- Multi-user gaze captures joint attention: reference users looking at the target helps predict the target as main speaker earlier, and sustained gaze from reference users supports turn-keeping predictions.
- Because the inputs are lightweight heatmaps, voice activity, and gaze angles, the trained models fit the latency and compute budget of adaptive directional sound control for hearing assistance on smart glasses.
Reading between the lines
- If the spatial-anchoring story is right, the same recipe should transfer beyond triads: any directional attention signal (gaze, head orientation, or even pointing) anchored to a localized speaker could predict turn changes in dyads and larger groups, and the active-speaker heatmap could be replaced by any reliable localization source.
- The paper compares against voice activity only, not against unanchored gaze; a direct control that feeds the same gaze heatmaps with the active-speaker heatmaps shuffled or replaced by random locations would isolate whether the spatial constraint, rather than gaze motion alone, drives the reported gains.
- The multi-user model as described needs the three users' gaze features at inference time; an implicit engineering consequence is that on-device hearing assistance would require either short-range sharing of gaze features among users or a decision to launch with the single-user model, which already improves over voice activity alone.
- The paper notes one case where the model's gaze-based prediction disagreed with the manual label during laughter; that suggests gaze may track conversational attention more faithfully than voice-activity-derived labels, which could make gaze a useful auxiliary signal for cleaning turn-taking ground truth.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper proposes lightweight single- and multi-user models for continuous turn-taking prediction in triadic conversations, using egocentric gaze heatmaps, audio-based active speaker localization (ASL) heatmaps, and binary voice activity detection (VAD) from Aria glasses. Labels are role-based (main/non-main/observer) and behavior-based (turn-taking, turn-keeping, back-channel, silence). Evaluated on a subset of RLR-CHAT (6 algorithmically labeled sessions for pretraining, 4 manually labeled sessions for fine-tuning and testing), the authors report F1 improvements of the single-user and multi-user models over a target-user VAD-only baseline, including a transition-level turn-taking F1 increase from 0.081 to 0.400 for the multi-user model. The central claim is that gaze, spatially anchored to speaker location, is a reliable predictive cue.
Significance. If the causal claim were established, this would be a useful practical contribution: a lightweight, privacy-conscious turn-taking predictor for hearing-assistance smart glasses, with continuous prediction rather than discrete pre-labeled intervals. The paper has genuine strengths: use of a naturalistic egocentric dataset, a plausible architecture, and transition-level evaluation that captures real-time relevance. The authors also report full per-class F1 scores rather than only aggregates, which is helpful. However, the significance rests on an ablation that is currently missing, so the contribution is not yet demonstrated.
major comments (4)
- [§4.1 and §5.1] The reported comparison does not isolate gaze. The single-user model adds both gaze heatmaps and ASL audio-derived spatial heatmaps relative to the 'Target V AD' baseline, yet the baseline uses only binary VAD features, and no ablation removes gaze while keeping ASL (or vice versa). The transition-level turn-taking improvement in Table 2 (a.1), from 0.081 to 0.400, could therefore be driven by the ASL audio localization features, which encode when and where reference users speak in the target user's egocentric frame. The statement in the abstract that 'incorporating gaze data ... improves prediction performance' is underdetermined by the current experiments.
- [§3 and Tables 1-2] The label construction is partially circular with respect to the VAD input. Role-based observer labels and behavior-based silence labels are assigned from the same binary VAD signal that is an input feature, and the silence/observer columns report perfect F1=1.000 by construction. This inflates the macro-averaged 'All' F1 and makes the headline comparisons partly self-fulfilling. The paper should either exclude the silence/observer classes from the macro-average or use manually annotated labels that do not derive from the input VAD.
- [§5.1] The test set is small and there are no uncertainty estimates. Testing uses the final 20% of each of the 4 manually labeled sessions only, and the reported differences are sometimes small (e.g., behavior-based All F1 0.661, 0.670, 0.682 in Table 2). Without confidence intervals, session-level standard errors, or significance tests, 'significantly improves' cannot be assessed, especially for the role-based and group-level comparisons.
- [§4.1] The spatial constraint relies on the ASL inference model producing accurate active-speaker heatmaps from egocentric audio, but the paper provides no validation of ASL accuracy on this dataset or per-participant analysis. If the heatmaps mislocalize the active speaker, the interpretation that gaze is being spatially anchored to the speaker is unsupported. A control analysis comparing ASL heatmaps to ground-truth speaker positions (or to head-azimuth measurements) would substantiate the mechanism.
minor comments (4)
- [Figure 2] The caption label 'Head-locked to world-locked' is unclear and is not explained in the text.
- [§3] The sentence 'Back-channeling occurred when The V AD indicates speaking' contains a capitalization error in 'The'.
- [§5.3 and Table 2] The relationship between the 'original evaluation' method and the (a.1)/(a.2) sub-analyses, and how the 1-second window is aligned to ground-truth turn changes, should be stated explicitly.
- [§5.1] The paper states that weighted cross-entropy loss is used, but the class weights are not specified; please state the weighting rule or provide a reference for the chosen scheme.
Circularity Check
Self-definitional silence/observer class disclosed by the paper; central gaze-improvement claim is not circular, though its attribution to gaze is confounded by simultaneously added ASL audio features.
-
self definitional
[Section 3 (behavior-based label definition) and Section 5.2 (role-based results, Table 1)]
"Silence was assigned when the V AD indicates silence. ... Since the observer role is directly determined by binary V AD indicating silence, the F1 score remains 100% across all."
For the silence/observer class, the ground-truth label is defined as 'VAD indicates silence,' and the same binary VAD is a direct input feature to every model. The reported 1.000 F1 is therefore a read-off of the input rather than a learned prediction. This is self-definitional by construction. It is not load-bearing for the paper's central gaze claim because the perfect score is a constant across the baseline, single-user, and multi-user columns, and the reported gaze improvements come from the other classes, whose labels depend on additional turn-taking rules rather than the VAD value alone.
full rationale
The paper's central claim is that adding gaze data improves turn-taking prediction. That claim does not reduce to the construction of the labels: gaze is not used in the VAD-based label-generation algorithm, and the non-silence classes (main/non-main speaker, turn-taking, turn-keeping, back-channel) are functions of VAD onsets and speaker-role rules rather than simple copies of the input features. No fitted parameter is renamed as a prediction, and no uniqueness theorem from the authors' prior work is invoked to force the model choice. The ASL features come from a cited CVPR paper (Jiang et al., including co-author Murdock); that is a self-citation to a real, externally evaluated model, not an unverified premise. The one genuine circular element is the silence/observer class, which the paper itself explicitly identifies as 'directly determined by binary V AD.' Because this perfect score is constant across all compared models, it inflates macro-F1 equally and does not drive the reported gaze gains. A separate validity concern, that the single-user model adds ASL audio spatial features at the same time as gaze so no ablation isolates gaze, is a confounding/ablation issue rather than a circularity, and for that reason it does not further raise the circularity score.
Assumptions & free parameters
free parameters (8)
- VAD context window =
5 seconds
- Gaze accumulation window =
0.2 seconds (6 frames at 30 Hz)
- Audio segment window =
0.2 seconds
- IPU pause smoothing threshold =
0.5 seconds
- CNN kernel and channels =
3x3, 32 channels
- ConvLSTM kernel and channels =
3x3, 64 channels
- MLP hidden units =
128 and 32
- Learning rate =
1e-2
assumptions (4)
- domain assumption Gaze direction, after world-locked correction, reflects participants' intended focus points.
- domain assumption The ASL model of Jiang et al. [14] provides accurate active speaker localization heatmaps from 7-channel egocentric audio.
- domain assumption Manual re-labeling of 4 sessions yields ground truth that is more accurate than the algorithmic labels.
- standard math Standard ML operations (CNN, ConvLSTM, attention) are well-defined and correctly implemented.
Cite this review
Pith. "Pith review of Gaze-Enhanced Multimodal Turn-Taking Prediction in Triadic Conversations." pith.science (2026). https://pith.science/paper/S7VC5BJJ
@misc{pith2026250513688,
author = {Pith},
title = {Pith review of: Gaze-Enhanced Multimodal Turn-Taking Prediction in Triadic Conversations},
year = {2026},
howpublished = {\url{https://pith.science/paper/S7VC5BJJ}},
note = {Machine review of arXiv:2505.13688}
}
read the original abstract
Turn-taking prediction is crucial for seamless interactions. This study introduces a novel, lightweight framework for accurate turn-taking prediction in triadic conversations without relying on computationally intensive methods. Unlike prior approaches that either disregard gaze or treat it as a passive signal, our model integrates gaze with speaker localization, structuring it within a spatial constraint to transform it into a reliable predictive cue. Leveraging egocentric behavioral cues, our experiments demonstrate that incorporating gaze data from a single-user significantly improves prediction performance, while gaze data from multiple-users further enhances it by capturing richer conversational dynamics. This study presents a lightweight and privacy-conscious approach to support adaptive, directional sound control, enhancing speech intelligibility in noisy environments, particularly for hearing assistance in smart glasses.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
Introduction Turn-taking is a fundamental mechanism in triadic conversa- tions, enabling smooth speaker transitions and preventing inter- ruptions. Accurate turn-taking prediction enhances conversa- tional flow by ensuring timely responses and sustained engage- ment [1, 2, 3]. Non-verbal cues, particularly eye gaze, play a crucial role in regulating turn ...
-
[2]
Related Work 2.1. Gaze-driven approaches to turn-taking prediction Several studies have explored the integration of gaze data for turn-taking prediction. Lee et al. [12] developed a binary arXiv:2505.13688v3 [cs.HC] 29 May 2025 Target user’s binary V AD MLP model User feature model User feature model User feature model User-attention valueUser attention m...
arXiv 2025
-
[3]
Dataset We used a subset of the Reality Labs Research Conversa- tions for Hearing Augmentation Technology (RLR-CHAT) dataset [15, 16, 17], collected using Aria glasses [18]. The dataset consists of 30 hours triadic conversations engaged in free-form conversations around a round table. Participants aged 20 to 60 years, including those without and with mild...
-
[4]
An overview of these models is presented in Fig
Methodology We propose two models for turn-taking prediction: a single- user model that focuses on the target user’s gaze and audio data, and a multi-user model that incorporated gaze features from all participants. An overview of these models is presented in Fig. 2. 4.1. Single-user model As illustrated in Fig. 2 (a), the single-user model processes mul-...
-
[5]
Experiments and Results 5.1. Training setup and evaluation metrics The user feature model consists of a 2D CNN block (kernel size: 3, 32 dimensions) followed by a ConvLSTM layer (ker- nel size: 3, 64 dimensions). The binary V AD model employs a two-layer MLP with 128 and 32 units. We used the Adam optimizer (learning rate: 1e-2) and trained the model usin...
-
[6]
Conclusion We introduced a lightweight framework for turn-taking predic- tion, demonstrating that gaze, despite its inherent complexity, is a reliable predictive cue when structured with spatial con- straints. Unlike prior work that overlooked or passively used gaze, our method integrates speaker localization and speech ac- tivity to extract meaningful pr...
-
[7]
A simplest systematics for the organization of turn tak- ing for conversation,
H. Sacks, “A simplest systematics for the organization of turn tak- ing for conversation,” 1978
work page 1978
-
[8]
On the structure of speaker–auditor interaction dur- ing speaking turns1,
S. Duncan Jr, “On the structure of speaker–auditor interaction dur- ing speaking turns1,”Language in society, vol. 3, no. 2, pp. 161– 180, 1974
work page 1974
Show all 34 references
-
[9]
Anticipation in turn-taking: mechanisms and information sources,
C. Riest, A. B. Jorschick, and J. P. de Ruiter, “Anticipation in turn-taking: mechanisms and information sources,”Frontiers in Psychology, vol. 6, p. 89, 2015
2015
-
[10]
The role of eye gaze in regulating turn taking in conversations: a systematized review of methods and findings,
Z. Degutyte and A. Astell, “The role of eye gaze in regulating turn taking in conversations: a systematized review of methods and findings,”Frontiers in Psychology, vol. 12, p. 616471, 2021
2021
-
[11]
Gaze and turn-taking behavior in casual conversational interactions,
K. Jokinen, H. Furukawa, M. Nishida, and S. Yamamoto, “Gaze and turn-taking behavior in casual conversational interactions,” ACM Transactions on Interactive Intelligent Systems (TiiS), vol. 3, no. 2, pp. 1–30, 2013
2013
-
[12]
Turn-taking in human face-to-face interaction is multimodal: gaze direction and manual gestures aid the coordination of turn transitions,
K. H. Kendrick, J. Holler, and S. C. Levinson, “Turn-taking in human face-to-face interaction is multimodal: gaze direction and manual gestures aid the coordination of turn transitions,”Philo- sophical Transactions of the Royal Society B, vol. 378, no. 1875, p. 20210473, 2023
2023
-
[13]
Influence of turn-taking in a two- person conversation on the gaze of a viewer,
L. Hirvenkari, J. Ruusuvuori, V .-M. Saarinen, M. Kivioja, A. Per ¨akyl¨a, and R. Hari, “Influence of turn-taking in a two- person conversation on the gaze of a viewer,”PloS One, vol. 8, no. 8, p. e71569, 2013
2013
-
[14]
Turngpt: a transformer-based lan- guage model for predicting turn-taking in spoken dialog,
E. Ekstedt and G. Skantze, “Turngpt: a transformer-based lan- guage model for predicting turn-taking in spoken dialog,”arXiv preprint arXiv:2010.10874, 2020
2010 arXiv
-
[15]
Turn-taking prediction for natural conver- sational speech,
S.-y. Chang, B. Li, T. N. Sainath, C. Zhang, T. Strohman, Q. Liang, and Y . He, “Turn-taking prediction for natural conver- sational speech,”arXiv preprint arXiv:2208.13321, 2022
2022 arXiv
-
[16]
Multimodal turn-taking model using visual cues for end-of-utterance predic- tion in spoken dialogue systems,
F. Kurata, M. Saeki, S. Fujie, and Y . Matsuyama, “Multimodal turn-taking model using visual cues for end-of-utterance predic- tion in spoken dialogue systems,”Proc. Interspeech 2023, pp. 2658–2662, 2023
2023
-
[17]
Prediction of who will be the next speaker and when using gaze behavior in multi- party meetings,
R. Ishii, K. Otsuka, S. Kumano, and J. Yamato, “Prediction of who will be the next speaker and when using gaze behavior in multi- party meetings,”ACM Transactions on Interactive Intelligent Sys- tems (TIIS), vol. 6, no. 1, pp. 1–31, 2016
2016
-
[18]
Multimodal turn analysis and prediction for multi-party conversations,
M.-C. Lee, M. Trinh, and Z. Deng, “Multimodal turn analysis and prediction for multi-party conversations,” inProceedings of the 25th International Conference on Multimodal Interaction, 2023, pp. 436–444
2023
-
[19]
Predicting and understanding turn-taking behavior in open-ended group activities in virtual reality,
P. Wang, E. Han, A. Queiroz, C. DeVeaux, and J. N. Bailen- son, “Predicting and understanding turn-taking behavior in open-ended group activities in virtual reality,”arXiv preprint arXiv:2407.02896, 2024
2024 arXiv
-
[20]
Egocentric deep multi- channel audio-visual active speaker localization,
H. Jiang, C. Murdock, and V . K. Ithapu, “Egocentric deep multi- channel audio-visual active speaker localization,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 10 544–10 552
2022
-
[21]
Hearing loss detection from facial expressions in one-on-one conversations,
Y . Yin, I. Ananthabhotla, V . K. Ithapu, S. Petridis, Y .-H. Wu, and C. Miller, “Hearing loss detection from facial expressions in one-on-one conversations,” inICASSP 2024-2024 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp...
2024
-
[22]
Self- motion as supervision for egocentric audiovisual localization,
C. Murdock, I. Ananthabhotla, H. Lu, and V . K. Ithapu, “Self- motion as supervision for egocentric audiovisual localization,” in ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 7835– 7839
2024
-
[23]
Spherical world-locking for audio-visual localization in egocentric videos,
H. Yun, R. Gao, I. Ananthabhotla, A. Kumar, J. Donley, C. Li, G. Kim, V . K. Ithapu, and C. Murdock, “Spherical world-locking for audio-visual localization in egocentric videos,” inEuropean Conference on Computer Vision. Springer, 2025, pp. 256–274
2025
-
[24]
Project aria: A new tool for egocentric multi-modal ai research,
J. Engel, K. Somasundaram, M. Goesele, A. Sun, A. Gamino, A. Turner, A. Talattof, A. Yuan, B. Souti, B. Meredithet al., “Project aria: A new tool for egocentric multi-modal ai research,” arXiv preprint arXiv:2308.13561, 2023
2023 arXiv
-
[25]
How pause duration influences impressions of english speech: Comparison between native and non-native speakers,
S. Liu, Y . Nakajima, L. Chen, S. Arndt, M. Kakizoe, M. A. Elliott, and G. B. Remijn, “How pause duration influences impressions of english speech: Comparison between native and non-native speakers,”Frontiers in Psychology, vol. 13, p. 778018, 2022
2022
-
[26]
A large-scale multilingual study of silent pause duration,
E. Campione and J. V ´eronis, “A large-scale multilingual study of silent pause duration,” inSpeech Prosody 2002, International Conference, 2002
2002
-
[27]
Convolutional lstm network: A machine learning ap- proach for precipitation nowcasting,
X. Shi, Z. Chen, H. Wang, D.-Y . Yeung, W.-K. Wong, and W.- c. Woo, “Convolutional lstm network: A machine learning ap- proach for precipitation nowcasting,”Advances in Neural Infor- mation Processing Systems, vol. 28, 2015
2015
-
[28]
Attention is all you need,
A. Vaswani, “Attention is all you need,”Advances in Neural In- formation Processing Systems, 2017
2017
-
[29]
Structured atten- tion networks,
Y . Kim, C. Denton, L. Hoang, and A. M. Rush, “Structured atten- tion networks,”arXiv preprint arXiv:1702.00887, 2017
2017 arXiv
-
[30]
Learning from imbalanced data sets with weighted cross- entropy function,
Y . S. Aurelio, G. M. De Almeida, C. L. de Castro, and A. P. Braga, “Learning from imbalanced data sets with weighted cross- entropy function,”Neural Processing Letters, vol. 50, pp. 1937– 1949, 2019
1937
-
[31]
Macro f1 and macro f1,
J. Opitz and S. Burst, “Macro f1 and macro f1,”arXiv preprint arXiv:1911.03347, 2019
1911 arXiv
-
[32]
Using respiration to predict who will speak next and when in multiparty meetings,
R. Ishii, K. Otsuka, S. Kumano, and J. Yamato, “Using respiration to predict who will speak next and when in multiparty meetings,” ACM Transactions on Interactive Intelligent Systems (TiiS), vol. 6, no. 2, pp. 1–20, 2016
2016
-
[33]
End-to-end eye movement de- tection using convolutional neural networks,
S. Hoppe and A. Bulling, “End-to-end eye movement de- tection using convolutional neural networks,”arXiv preprint arXiv:1609.02452, 2016
2016 arXiv
-
[34]
Online eye-movement classification with temporal convolutional networks,
C. Elmadjian, C. Gonzales, R. L. d. Costa, and C. H. Morimoto, “Online eye-movement classification with temporal convolutional networks,”Behavior Research Methods, vol. 55, no. 7, pp. 3602– 3620, 2023
2023
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.