{"id":"496fe289-c9a2-4ab3-a605-bd171f031b71","arxiv_id":"2505.13688","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"A gaze-enhanced multimodal model predicts turn-taking in triadic conversations better than voice activity alone, and multi-user gaze gives the largest gains.","lead":"This paper presents a lightweight deep learning system that uses eye gaze and audio to predict who will speak next in three-person conversations. It reports that gaze improves turn-taking prediction, which could help future smart glasses aim sound toward the likely next speaker.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported gaze gains are confounded: the single- and multi-user models add ASL audio spatial features at the same time as gaze, and no ablation isolates gaze, so the central claim is not yet demonstrated.","rationale":"The strongest_claim is empirical, so the load-bearing condition is not the mechanism but the attribution of the measured improvement to gaze. The architecture confounds gaze with ASL audio spatial features. The reader's identified assumption (ASL accuracy) is a legitimate secondary risk, and my concern overlaps with it: if ASL is inaccurate, the spatial anchoring is meaningless. But the deeper problem is that even if ASL were perfect, the reported results would not prove the gaze contribution, because the comparison adds gaze and ASL simultaneously. This is an internal-validity issue rather than a consensus disagreement. The ablation test would settle it directly. The lack of error bars and significance tests is also real, but it acts as an additional reason to demand the ablation and repeated-seed reporting, not as a separate fatal flaw. Dataset availability is secondary to attribution. The verdict should remain CONDITIONAL: the approach is plausible and lightweight, but the central claim should not be accepted until gaze is isolated from ASL in the evaluation.","tokens_in":6917,"tokens_out":7749,"duration_ms":75301,"concrete_test":"On the 4 manually labeled sessions, train and test four single-user variants with identical splits, architecture, and loss: (1) VAD only; (2) VAD + ASL audio heatmaps, no gaze; (3) VAD + gaze heatmaps, no ASL; (4) full VAD + gaze + ASL. Repeat over at least 5 random seeds and report mean ± std macro-F1. If condition (2) matches or exceeds condition (4), the headline gain is attributable to ASL audio, not gaze. For the multi-user claim, run the analogous variant with ASL from all users but gaze only from the target user; if it matches the full multi-user model, the multi-user gaze claim also fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim attributes the F1 improvements to gaze, but the experimental design never isolates gaze. In Section 4.1, the single-user model takes binary VAD features, gaze heatmaps, and audio ASL-derived spatial heatmaps; 'Both heatmaps are then fed into a CNN-based feature extractor' and combined. The baseline in Section 5.1 is 'a model using only the target user's binary VAD features.' Therefore the reported single-user gain over baseline is the joint effect of gaze plus ASL audio localization. The ASL audio features encode where and when reference users are speaking in the target user's egocentric frame, which is itself highly informative for turn-taking. The same confound affects the multi-user model, which adds gaze and ASL features from all users; the 'richer conversational dynamics' could come from additional audio streams rather than joint gaze. No ablation removes gaze while keeping ASL, or removes ASL while keeping gaze. The failed gaze-only model without VAD does not address this, since it lacks the VAD and ASL context. Thus the abstract's causal claim that 'incorporating gaze data ... improves' is underdetermined by the current comparisons.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes lightweight single- and multi-user models for continuous turn-taking prediction in triadic conversations, using egocentric gaze heatmaps, audio-based active speaker localization (ASL) heatmaps, and binary voice activity detection (VAD) from Aria glasses. Labels are role-based (main/non-main/observer) and behavior-based (turn-taking, turn-keeping, back-channel, silence). Evaluated on a subset of RLR-CHAT (6 algorithmically labeled sessions for pretraining, 4 manually labeled sessions for fine-tuning and testing), the authors report F1 improvements of the single-user and multi-user models over a target-user VAD-only baseline, including a transition-level turn-taking F1 increase from 0.081 to 0.400 for the multi-user model. The central claim is that gaze, spatially anchored to speaker location, is a reliable predictive cue.","tokens_in":7198,"tokens_out":4023,"duration_ms":35127,"significance":"If the causal claim were established, this would be a useful practical contribution: a lightweight, privacy-conscious turn-taking predictor for hearing-assistance smart glasses, with continuous prediction rather than discrete pre-labeled intervals. The paper has genuine strengths: use of a naturalistic egocentric dataset, a plausible architecture, and transition-level evaluation that captures real-time relevance. The authors also report full per-class F1 scores rather than only aggregates, which is helpful. However, the significance rests on an ablation that is currently missing, so the contribution is not yet demonstrated.","major_comments":[{"comment":"The reported comparison does not isolate gaze. The single-user model adds both gaze heatmaps and ASL audio-derived spatial heatmaps relative to the 'Target V AD' baseline, yet the baseline uses only binary VAD features, and no ablation removes gaze while keeping ASL (or vice versa). The transition-level turn-taking improvement in Table 2 (a.1), from 0.081 to 0.400, could therefore be driven by the ASL audio localization features, which encode when and where reference users speak in the target user's egocentric frame. The statement in the abstract that 'incorporating gaze data ... improves prediction performance' is underdetermined by the current experiments.","section":"§4.1 and §5.1"},{"comment":"The label construction is partially circular with respect to the VAD input. Role-based observer labels and behavior-based silence labels are assigned from the same binary VAD signal that is an input feature, and the silence/observer columns report perfect F1=1.000 by construction. This inflates the macro-averaged 'All' F1 and makes the headline comparisons partly self-fulfilling. The paper should either exclude the silence/observer classes from the macro-average or use manually annotated labels that do not derive from the input VAD.","section":"§3 and Tables 1-2"},{"comment":"The test set is small and there are no uncertainty estimates. Testing uses the final 20% of each of the 4 manually labeled sessions only, and the reported differences are sometimes small (e.g., behavior-based All F1 0.661, 0.670, 0.682 in Table 2). Without confidence intervals, session-level standard errors, or significance tests, 'significantly improves' cannot be assessed, especially for the role-based and group-level comparisons.","section":"§5.1"},{"comment":"The spatial constraint relies on the ASL inference model producing accurate active-speaker heatmaps from egocentric audio, but the paper provides no validation of ASL accuracy on this dataset or per-participant analysis. If the heatmaps mislocalize the active speaker, the interpretation that gaze is being spatially anchored to the speaker is unsupported. A control analysis comparing ASL heatmaps to ground-truth speaker positions (or to head-azimuth measurements) would substantiate the mechanism.","section":"§4.1"}],"minor_comments":[{"comment":"The caption label 'Head-locked to world-locked' is unclear and is not explained in the text.","section":"Figure 2"},{"comment":"The sentence 'Back-channeling occurred when The V AD indicates speaking' contains a capitalization error in 'The'.","section":"§3"},{"comment":"The relationship between the 'original evaluation' method and the (a.1)/(a.2) sub-analyses, and how the 1-second window is aligned to ground-truth turn changes, should be stated explicitly.","section":"§5.3 and Table 2"},{"comment":"The paper states that weighted cross-entropy loss is used, but the class weights are not specified; please state the weighting rule or provide a reference for the chosen scheme.","section":"§5.1"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the venue's scope and the dataset is valuable, but the missing gaze/ASL ablation is load-bearing for the central claim. I recommend requiring an ablation study that separates gaze from ASL features, plus uncertainty or significance reporting on the 4-session test set, before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick read on arXiv:2505.13688. The one thing to know before you spend time: the experiments never isolate gaze. The single-user model takes the target user's binary VAD, gaze heatmaps, and ASL audio-derived spatial heatmaps together; the baseline is VAD only. So the reported F1 gain over baseline is the joint contribution of gaze plus active-speaker localization. The same is true for the multi-user model, which adds ASL audio features from every participant. No ablation removes gaze while keeping ASL, or removes ASL while keeping gaze. The gaze-only model reportedly failed to converge, but that doesn't tell you whether gaze adds anything on top of VAD+ASL. So the abstract's causal claim—'incorporating gaze data significantly improves'—is underdetermined.\n\nWhat the paper does well: it frames turn-taking prediction as a continuous egocentric task on triadic conversations, uses a real dataset (RLR-CHAT) with synchronized gaze and audio, and evaluates both role-based and behavior-based labels, including transition-level and group-level metrics. The architecture is lightweight and the writing is clear. The idea of anchoring gaze to the active speaker's spatial location via ASL is a sensible way to give gaze meaning.\n\nBeyond the confound, the evaluation is thin: no confidence intervals or significance tests, test data comes from the last 20% of four sessions, and the macro-F1 averages include a silence class that is 1.0 by construction. The ASL model itself is not validated on this dataset; if its heatmaps are wrong, the spatial constraint doesn't help. And in Figure 4 (a.3) the authors dismiss a prediction error as mislabeling—that's exactly the kind of post-hoc reasoning that needs care, not assertion.\n\nOverall: the paper is a reasonable engineering contribution with a promising idea, but the headline result is not yet supported. The fix is straightforward—run ablations that hold ASL fixed and vary gaze, and vice versa, and report per-condition significance. If that comes out clean, the result would be interesting. As it stands, it deserves a serious referee, but the referee should treat the central claim skeptically until the ablations are done.","headline":"Gaze's real contribution is unproven because the gaze models also add ASL audio spatial features; the paper needs ablations before its central claim can be trusted.","tokens_in":7700,"tokens_out":2394,"would_cite":false,"duration_ms":21431,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Gaze becomes a reliable turn-taking cue when anchored to the active speaker's location, and more participants' gaze helps more.","keywords":["turn-taking prediction","gaze","active speaker localization","triadic conversation","multimodal interaction","egocentric behavior","voice activity detection","hearing assistance"],"falsifier":"Take the trained single- or multi-user model on held-out sessions and replace the active-speaker heatmaps with random or mislocalized speaker positions (or with the true speaker location measured independently); if turn-taking F1 stays as high as with the model's own heatmaps, the claim that spatial anchoring is what makes gaze useful is false, while if F1 falls to the voice-activity-only level, the claim is supported.","tokens_in":6737,"feed_emoji":"👀","tokens_out":8656,"duration_ms":75417,"temperature":0.7,"pith_summary":"Turn-taking prediction is usually built on language or image models that are heavy and privacy-invasive. This paper argues that eye gaze becomes a strong predictive cue once it is spatially anchored to the active speaker: the model pairs a gaze heatmap with an active-speaker heatmap derived from egocentric audio, so the network can learn patterns like \"the target is about to speak because they are looking at the current speaker.\" On a naturalistic triadic-conversation dataset, adding the target user's gaze to voice-activity features improves F1 for role-based and behavior-based labels, and adding the other two participants' gaze improves it further. The appeal is practical: if correct, gaze gives a lightweight, privacy-conscious signal for anticipating turn changes, which supports adaptive directional sound control for hearing-assist smart glasses.","feed_headline":"Gaze plus speaker location predicts who speaks next in triads","feed_subtitle":"One user's gaze beats voice activity alone for turn-taking F1; all three users' gaze is better still.","key_machinery":"The load-bearing mechanism is the pairing of two spatial heatmaps: a gaze heatmap built from six frames of azimuth/elevation samples and an active-speaker heatmap produced by an egocentric audio active-speaker localization model. Each user's two heatmaps pass through separate CNN feature extractors and a ConvLSTM to form a user feature vector; the multi-user model then combines the three users' vectors with a user-attention mechanism and concatenates the target user's voice-activity feature. This arrangement turns raw gaze direction into a spatially grounded cue, effectively letting the model read gaze as \"looking at the current speaker,\" while keeping the whole pipeline lightweight enough for low-latency use.","core_discovery":"The paper's central discovery is that gaze, though unreliable on its own, becomes a dependable predictive signal when placed in a spatial frame defined by the active speaker. The single-user model processes a 0.2-second gaze heatmap and an active-speaker heatmap from an audio-based localization model through CNN extractors and a ConvLSTM, then appends the target user's binary voice-activity feature. Reported F1 on role-based classification rises from 0.704 with voice activity alone to 0.746 with single-user gaze and 0.765 with multi-user gaze; on behavior-based transition-level analysis, turn-taking F1 rises from 0.081 to 0.218 to 0.400. A gaze-only model without voice activity failed to converge, so voice activity remains necessary; the improvement comes from combining it with spatially constrained gaze, not from gaze alone.","pith_inferences":["If the spatial-anchoring story is right, the same recipe should transfer beyond triads: any directional attention signal (gaze, head orientation, or even pointing) anchored to a localized speaker could predict turn changes in dyads and larger groups, and the active-speaker heatmap could be replaced by any reliable localization source.","The paper compares against voice activity only, not against unanchored gaze; a direct control that feeds the same gaze heatmaps with the active-speaker heatmaps shuffled or replaced by random locations would isolate whether the spatial constraint, rather than gaze motion alone, drives the reported gains.","The multi-user model as described needs the three users' gaze features at inference time; an implicit engineering consequence is that on-device hearing assistance would require either short-range sharing of gaze features among users or a decision to launch with the single-user model, which already improves over voice activity alone.","The paper notes one case where the model's gaze-based prediction disagreed with the manual label during laughter; that suggests gaze may track conversational attention more faithfully than voice-activity-derived labels, which could make gaze a useful auxiliary signal for cleaning turn-taking ground truth."],"forward_implications":["In role-based classification, single-user gaze lifts overall F1 from 0.704 (voice activity only) to 0.746, and multi-user gaze lifts it to 0.765; the main-speaker class shows the clearest gain, from 0.761 to 0.835 to 0.869.","At the transition level, where turn changes actually happen, gaze has its largest effect: turn-taking F1 rises from 0.081 with voice activity alone to 0.218 with single-user gaze and 0.400 with multi-user gaze.","Gaze alone is not sufficient: a gaze-only model without voice activity did not converge, so the practical system still needs binary voice activity to separate silence from speech.","Multi-user gaze captures joint attention: reference users looking at the target helps predict the target as main speaker earlier, and sustained gaze from reference users supports turn-keeping predictions.","Because the inputs are lightweight heatmaps, voice activity, and gaze angles, the trained models fit the latency and compute budget of adaptive directional sound control for hearing assistance on smart glasses."],"supporting_citations":[{"why":"Supplies the active-speaker localization model whose spatial heatmaps anchor gaze to the current speaker; it is the core of the spatial constraint.","marker":"[14]"},{"why":"Provides the role-based and behavior-based turn-taking labeling framework and the 0.5-second VAD smoothing used to define ground-truth classes.","marker":"[13]"},{"why":"Is one of the sources of the naturalistic triadic-conversation dataset used for training and evaluation.","marker":"[15]"},{"why":"Describes the egocentric smart-glasses platform that recorded the synchronized audio, gaze, and head-motion streams.","marker":"[18]"},{"why":"Provides the ConvLSTM layer that models spatiotemporal dynamics of the heatmap features.","marker":"[21]"},{"why":"Provides the attention mechanism the multi-user model uses to weight each participant's gaze contribution.","marker":"[22]"},{"why":"A prior gaze-based turn-taking model requiring pre-labeled information; the contrast establishes the need for a practical continuous approach.","marker":"[12]"}],"fun_headline_variants":["Spatial gaze anchoring turns gaze into a turn-taking predictor","Single user's gaze in speaker frame lifts turn-taking F1","Gaze becomes predictive when anchored to speaker's position","Lightweight gaze+location model predicts turns in triads","Gaze helps predict next speaker only when aligned with speaker location"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes, without validating it on its own recordings, that the pretrained active-speaker localization model correctly determines where the current speaker is from each user's egocentric audio; if its heatmaps point to the wrong location, gaze anchored to them is no longer a reliable cue.","fun_headline_variants_meta":{"raw":{"variants":["Spatial gaze anchoring turns gaze into a turn-taking predictor","Single user's gaze in speaker frame lifts turn-taking F1","Gaze becomes predictive when anchored to speaker's position","Lightweight gaze+location model predicts turns in triads","Gaze helps predict next speaker only when aligned with speaker location"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000783,"raw_usage":{"total_tokens":3403,"prompt_tokens":835,"completion_tokens":2568,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":451,"completion_tokens_details":{"reasoning_tokens":2485}},"tokens_in":451,"tokens_out":2568,"duration_ms":17107,"temperature":1.0,"reasoning_tokens":2485,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:10:40.976628+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the trained single- or multi-user model on held-out sessions and replace the active-speaker heatmaps with random or mislocalized speaker positions (or with the true speaker location measured independently); if turn-taking F1 stays as high as with the model's own heatmaps, the claim that spatial anchoring is what makes gaze useful is false, while if F1 falls to the voice-activity-only level, the claim is supported.","supporting_citations":[{"cited_title":"Influence of turn-taking in a two- person conversation on the gaze of a viewer,","cited_arxiv_id":null,"evidence_quote":"Provides the role-based and behavior-based turn-taking labeling framework and the 0.5-second VAD smoothing used to define ground-truth classes."},{"cited_title":"Multimodal turn analysis and prediction for multi-party conversations,","cited_arxiv_id":null,"evidence_quote":"Describes the egocentric smart-glasses platform that recorded the synchronized audio, gaze, and head-motion streams."},{"cited_title":"Hearing loss detection from facial expressions in one-on-one conversations,","cited_arxiv_id":null,"evidence_quote":"Provides the ConvLSTM layer that models spatiotemporal dynamics of the heatmap features."},{"cited_title":"Self- motion as supervision for egocentric audiovisual localization,","cited_arxiv_id":null,"evidence_quote":"Provides the attention mechanism the multi-user model uses to weight each participant's gaze contribution."},{"cited_title":"Turn-taking in human face-to-face interaction is multimodal: gaze direction and manual gestures aid the coordination of turn transitions,","cited_arxiv_id":null,"evidence_quote":"A prior gaze-based turn-taking model requiring pre-labeled information; the contrast establishes the need for a practical continuous approach."}],"review_version":1}