Pith. sign in

REVIEW 4 major objections 4 minor 53 references

This paper claims that adding gaze, pointing, and object cues to VR speech transcripts raises a language model's correct coreference resolution from 40.6% to 67.1% in VR design conversations.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Augmenting VR speech transcripts with gaze and pointing cues improved GPT-4 coreference resolution from 40.6% to 67.1% accuracy.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection The 26.5-point headline gain is real but mostly reflects injecting the object name; the gaze/pointing-specific contribution is untested, yet the system and dataset are worth engaging. the 4 major comments →

arxiv 2509.08689 v1 pith:75KQI6PN submitted 2025-09-10 cs.HC

Augmenting speech transcripts of VR recordings with gaze, pointing, and visual context for multimodal coreference resolution

Riccardo Bovo , Frederik Brudy , George Fitzmaurice , Fraser Anderson This is my paper
classification cs.HC
keywords multimodal coreference resolutionvirtual realityeye-trackingpointing gesturesspeech transcript augmentationreferring expressionsLLM evaluationcollaborative design review
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that speech-only transcripts of VR conversations are missing the information that makes short references like 'this' or 'there' meaningful: where a person is looking and what they are pointing at. The authors built a system that turns those non-verbal cues into short text notes—'P1 was pointing at the lamp'—and inserts them into the transcript at the right moment. In a study of six pairs reviewing virtual apartment models, adding these notes let a GPT-4 model resolve 67.1% of implicit spatial referring expressions correctly, versus 40.6% with speech alone. If the result holds, recorded immersive meetings become substantially easier for machines to summarize, search, and make accessible.

Core claim

The central discovery is that non-verbal cues from a VR session can be reduced to textual metadata and used as a retrieval signal for the object behind an ambiguous expression. For each implicit spatial referring expression, the system identifies a candidate object from eye-gaze and laser-pointer fixations, using concurrent, recurrent, and individual behavior across the two conversation partners, then applies a fixed priority: pointing before gaze, and shared attention before individual attention. The resulting object-of-interest identification reaches 0.875 precision in the authors' 12-participant study, and the augmented transcript lifts GPT-4's coreference F1 score from 0.507 to 0.723, wi

What carries the argument

The load-bearing component is a deterministic hierarchical selection algorithm that converts noisy 120 Hz VR logs into one text annotation per ambiguous expression. Fixations are first extracted with an I-DT dispersion-threshold method applied to both eye rays and laser-pointer rays in the 3D scene. The algorithm then computes, per object, individual fixation time, concurrent fixation overlap between partners, and recurrent fixation where one partner follows the other's attention. Selection follows a fixed order—pointing before gaze, synergistic before individual—so the language model never sees raw numbers; it receives an appended note such as '[P1 was pointing at the sofa]', which supplies

Load-bearing premise

The system assumes that when someone says 'this' or 'it', the thing they mean is the object they are pointing at or looking at, with pointing trusted first; a misleading gesture, or a pointer that lands on part of a larger place (the fridge when they mean the kitchen), leads to the wrong choice.

What would settle it

Run the same pipeline on a held-out set of VR conversations where a speaker points at one object while looking at another and say which one they mean; the hierarchy predicts the pointed object wins every time, so any majority favoring the gazed object would falsify the mechanism. A simpler check: if adding gaze data alone fails to beat the speech-only baseline on endophoric references, the claim that non-verbal cues carry coreference information loses its support.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Recorded VR design reviews and spatial meetings become readable by language models without full visual scene understanding, because behavior is converted into text.
  • Summarization, search, and accessibility tools that consume meeting transcripts inherit the coreference accuracy gain directly.
  • The hierarchy offers a design rule for future systems: trust deliberate pointing over reflexive gaze when they conflict, and trust shared attention over individual attention.
  • Place references remain the known boundary: when 'this' means the kitchen but the laser hits the fridge, the augmentation can mislead, so higher-level spatial scene models would be needed.
  • The 26.5-point gain was shown on a 12-participant sample, so replicating the pipeline on larger and more varied VR conversation corpora is the natural next test.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • This text-augmentation strategy should transfer to AR or video conversations wherever gaze and pointing can be estimated against segmented objects, since the paper's pre-segmented VR scene is an implementation convenience, not a conceptual requirement.
  • The success of handing the LLM the outcome of spatial reasoning rather than raw time series suggests that future systems could replace the hand-built hierarchy with learned rankers once more labeled data exist.
  • A testable extension is to personalize the pointing-versus-gaze priority per speaker, since people differ in how often they gesture versus rely on gaze.
  • The place-reference weakness points directly to a fix: a hierarchical scene graph linking objects to the places they belong to should recover most of the lost accuracy, and that can be checked by rerunning the same pipeline on annotated scene hierarchies.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes a pipeline that ingests VR session audio, eye-gaze, and laser-pointer data, transcribes and diarizes speech with Whisper, uses GPT-4 to identify implicit spatial referring expressions (REs), exploits gaze/pointing fixations and their concurrent/recurrent/individual patterns to select an object of interest, and appends a textual description such as '[P1 was pointing at the sofa]' to the transcript. GPT-4 then resolves coreferences on this augmented transcript. In a 12-participant (six-pair) design-review study, the system resolves 235/350 implicit REs versus 142/350 for a speech-only baseline, with non-overlapping 95% CIs (baseline 0.507–0.584, system 0.675–0.770). The paper also reports object-identification precision of 0.875 and analyzes the precision of individual cue combinations. The core end-to-end claim is plausible, but the experimental design does not isolate the specific contribution of gaze/pointing cues from the injected object label, and several internal numerical inconsistencies affect the headline results.

Significance. If the end-to-end result holds, this is a useful empirical demonstration that augmenting VR transcripts with scene and behavioral metadata can substantially improve LLM coreference resolution in a collaborative task. The collected paired gaze/pointing/speech dataset with manual labels is a contribution, and the use of bootstrap confidence intervals is a strength relative to typical HCI evaluations. The object-identification ablation and the finding that pointing is more precise than gaze are informative. However, the central claim as worded in the title and abstract—that gaze and pointing cues improve coreference resolution—is not isolated from the injection of the object name into the augmented transcript. The internal count inconsistencies and the under-specified GPT-4 evaluation protocol must be addressed before the magnitude of the reported gain can be fully trusted.

major comments (4)
  1. [§3.4.6, §5.4.1, §7] The reported 26.5pp gain conflates the object-identification algorithm with the effect of the injected object label. As described in §3.4.6, the augmentation appends a deterministically selected object name (e.g., 'P1 was pointing at the sofa') to the sentence containing the implicit RE, while the baseline receives only the speech transcript. Therefore, the improvement from 142/350 to 235/350 could be driven largely by GPT-4 seeing the referent's name, not by the temporal alignment or non-verbal nature of the gaze/pointing cue. The ablation in Figure 7 evaluates object-identification precision only; it does not include a coreference condition that removes the object name while keeping the non-verbal description, nor a control that inserts an object label without temporal alignment to the RE (e.g., the most-fixated object in the room). Without such conditions, the data support only the fu
  2. [§5.1, §5.4.1, §6.1] The counts around the 350 implicit REs are internally inconsistent. §5.1 reports 'correctly identified 318 and misclassified 82' (sum 400); §6.1 reports 'successfully identified 318 but misclassified 63' (sum 381); §5.4.1 reports 'there were 318 correctly identified' out of 350 and then gives baseline/system resolutions as 142 (40.6%) and 235 (67.1%) 'of them.' Since 142/350 = 40.6% and 235/350 = 67.1%, the percentages use 350 as the denominator, but the text says 'of them,' and the relationship between the 318 correctly identified REs and the 350 ground-truth REs is unclear. Please report the full confusion matrix (true positives, false negatives, false positives) and state the exact denominator for each reported accuracy.
  3. [§5.4.1] The performance metric is described inconsistently. The text says 'resulting in a .21 increase in the F1 score when comparing baseline with a precision of .507 to system with a precision of .723,' but the headline percentages are 40.6% and 67.1%, and the confidence intervals are 0.507–0.584 and 0.675–0.770. It is unclear whether 0.507 and 0.723 are F1 scores, precision values, or something else, and the .21 increase is written as if 'precision' and 'F1 score' are interchangeable. Please choose one metric, define how it is averaged (micro/macro), and present consistent values across text, tables, and figures.
  4. [§3.5, §5.4] The GPT-4 evaluation does not report the exact model version, temperature/sampling settings, or number of independent runs. If the API responses are stochastic, the 26.5 percentage-point difference could be sensitive to a particular random draw; the bootstrap CIs as computed appear to capture participant variability but not LLM stochasticity. At minimum, the authors should report deterministic decoding (e.g., temperature 0) or run the evaluation multiple times and report variance. This is needed to assess the stability of the headline improvement.
minor comments (4)
  1. [Abstract] 'coreferences resolution' should be 'coreference resolution'.
  2. [§1, §3.5] The system is referred to as 'Chat GPT4' in the introduction and 'GPT-4' elsewhere; please standardize to the exact API model name and date (e.g., gpt-4-0613) for reproducibility.
  3. [Figure 7] The caption is dense and groups several subplots with different meanings; the text should define the six behavior measures, the units of each subplot, and how 'correct/incorrect/none' are counted. In particular, some subplots are labeled as percentages and one as precision, but the caption gives a single y/x description for all.
  4. [§3.4.1] The I-DT thresholds (0.5°, 100 ms) are imported from 2D screen eye-tracking [35] and applied to VR gaze and laser pointing. Since laser-pointing adjustment behavior differs from saccadic eye movement, a sentence justifying or empirically checking these thresholds would strengthen the work.

Circularity Check

0 steps flagged

No significant circularity: the coreference gain is an empirical pipeline result, with only minor self-citations and an ablation gap that is a limitation, not a logical reduction.

full rationale

The paper's derivation chain is: (i) Whisper transcribes audio; (ii) GPT-4 detects implicit spatial referring expressions; (iii) a deterministic fixation/hierarchy algorithm converts gaze/pointing time series into an object name; (iv) that name is inserted into the transcript as a textual cue; (v) GPT-4 resolves the referring expression, and the result is compared with human labels. At no point is the ground-truth referent used to define the system's inputs. The object name inserted in Section 3.4.6 is not drawn from the human labels; it is produced by the I-DT fixation algorithm and the hand-specified hierarchy of Section 3.4.5. Thus the reported improvement from 142/350 (40.6%) to 235/350 (67.1%) is an empirical outcome of a pipeline, not a quantity forced by construction. The self-citations to Bovo et al. [6,7] are background motivation for gaze/pointing as cues and are not load-bearing; the hierarchy's ordering is justified by the deliberate/reflexive distinction and by the data analysis, not by appeal to an unverified self-citation. The skeptic's concern that the object label, rather than the raw gaze/pointing descriptions, drives the coreference gain is a genuine ablation gap: the paper does not include a condition that removes the object name while keeping the non-verbal descriptions. However, this is an experimental-attribution limitation, not circularity, because the label itself is the system's inference from independently measured gaze/pointing. Similarly, the hierarchy was informed by observed precision differences (Section 6.2), which raises an in-sample evaluation concern, but it does not make the prediction equivalent to the input by definition. The internal count inconsistencies (82 vs 63 misclassified; 318+82=400 vs 350) are correctness risks, not circular steps. On balance, the central claim is self-contained against independently captured multimodal data, so no circular step is flagged.

Axiom & Free-Parameter Ledger

1 free parameters · 4 axioms · 0 invented entities

The central contribution is an application pipeline, not a new mathematical or physical entity. The only design choices without independent justification are the fixation thresholds (borrowed from prior work) and the hierarchical selection ordering (introduced by the authors).

free parameters (1)
  • Fixation thresholds (0.5 degree, 100 ms) = 0.5 deg, 100 ms
    Adopted from Salvucci and Goldberg (Section 3.4.1), not fitted to this dataset.
axioms (4)
  • domain assumption Gaze and pointing behavior reliably indicates the intended referent of an implicit spatial referring expression.
    The whole object-identification stage (Section 3.4) rests on this; the paper's own place-reference analysis (Section 6.5) shows it is imperfect.
  • ad hoc to paper The hierarchical ordering (pointing over gaze, synergistic over individual) is the correct way to select the object of interest.
    Section 3.4.5 asserts this ordering without comparative evaluation of alternative hierarchies; it directly determines the output.
  • domain assumption The manual labels produced by the three authors, after discussion to reach consensus, constitute a correct ground truth.
    Section 4.6.5 reports 80.7% initial agreement and 100% after discussion; the final labels are treated as ground truth for all evaluations.
  • domain assumption GPT-4 and Whisper perform adequately for transcription, RE detection, and coreference resolution in this pipeline.
    No separate audit of transcription or RE-detection errors is provided beyond the final F1 numbers; the system depends on these black-box models.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Augmenting speech transcripts of VR recordings with gaze, pointing, and visual context for multimodal coreference resolution." pith.science (2026). https://pith.science/paper/75KQI6PN

@misc{pith2026250908689,
  author       = {Pith},
  title        = {Pith review of: Augmenting speech transcripts of VR recordings with gaze, pointing, and visual context for multimodal coreference resolution},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/75KQI6PN}},
  note         = {Machine review of arXiv:2509.08689}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Understanding transcripts of immersive multimodal conversations is challenging because speakers frequently rely on visual context and non-verbal cues, such as gestures and visual attention, which are not captured in speech alone. This lack of information makes coreferences resolution-the task of linking ambiguous expressions like ``it'' or ``there'' to their intended referents-particularly challenging. In this paper we present a system that augments VR speech transcript with eye-tracking laser pointing data, and scene metadata to generate textual descriptions of non-verbal communication and the corresponding objects of interest. To evaluate the system, we collected gaze, gesture, and voice data from 12 participants (6 pairs) engaged in an open-ended design critique of a 3D model of an apartment. Our results show a 26.5\% improvement in coreference resolution accuracy by a GPT model when using our multimodal transcript compared to a speech-only baseline.

Figures

Figures reproduced from arXiv: 2509.08689 by Fraser Anderson, Frederik Brudy, George Fitzmaurice, Riccardo Bovo.

Figure 1
Figure 1. Figure 1: Depiction of our system performing coreference resolution by leveraging non verbal cues such as pointing and [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The system architecture, as depicted in the diagram, consists of four main components: Transcript Generation [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The two apartment scenes reviewed by participants. Participants were asked to collaboratively review these two [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: This snapshot shows participant’s in the environment while the transcript captures the dialogue. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: An example of the manual annotation process. The transcript corresponds to the snapshot in Figure 4 and the manual [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: An example of the system’s output showing the transcript augmentation corresponding to the snapshot in Figure 4, [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Identified Object of Interest: (a) Plot depicting the performance of each behaviour combination in terms of identifying [PITH_FULL_IMAGE:figures/full_fig_p010_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: (a) Plot depicting the count of explicit and implicit REs labelled for each participant (y-axis: count per participant, [PITH_FULL_IMAGE:figures/full_fig_p011_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: (a) Plot illustrating the count of implicit REs categorized as endophora or exophora for each participant (y-axis: [PITH_FULL_IMAGE:figures/full_fig_p012_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

53 extracted references · 3 canonical work pages

  1. [1]

    Arkio ehf. 2023. Arkio. https://www.arkio.is/. Accessed: [2023-12-09]. (2023)

    2023

  2. [2]

    Autodesk. 2023. The Wild. https://thewild.com/. Accessed: [2023-12-09]. (2023)

    2023

  3. [3]

    Autodesk. 2023. WorkshopXR. https://workshopxr.autodesk.com/. Accessed: [2023-12-09]. (2023)

    2023

  4. [4]

    Jiaxin Bai, Hongming Zhang, Yangqiu Song, and Kun Xu. 2021. Joint Corefer- ence Resolution and Character Linking for Multiparty Conversation. InEACL. https://aclanthology.org/2021.eacl-main.43.pdf

    2021

  5. [5]

    Put-That-There

    Richard A. Bolt. 1980. “Put-That-There”: Voice and Gesture at the Graphics Interface. InProceedings of the 7th Annual Conference on Computer Graphics and Interactive Techniques(SIGGRAPH ’80). Association for Computing Machinery, Seattle, Washington, USA, 262–270.isbn: 0897910214. doi:10.1145/800250.8075 03

    arXiv 1980

  6. [6]

    Riccardo Bovo, Daniele Giunchi, Alebri Muna, Anthony Steed, Enrico Costanza, and Thomas Heinis. 2022. Cone of Vision as a Behavioural Cue for VR Collab- oration.Taiepei 2022: Conference on Computer Supported Cooperative Work and Social Computing, November 12-16, 2022, Taiepei, Taiwan, 1, 1. doi:10.1145/3555 615

    work page doi:10.1145/3555 2022

  7. [8]

    Sarah D’Angelo and Andrew Begel. 2017. Improving communication between pair programmers using shared gaze awareness.Conference on Human Factors in Computing Systems - Proceedings, 2017-Janua, 6245–6255.isbn: 9781450346559. doi:10.1145/3025453.3025573

    arXiv 2017

  8. [9]

    Arushi Goel, Basura Fernando, Frank Keller, and Hakan Bilen. 2023. Who Are You Referring To? Coreference Resolution In Image Narrations. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). (Oct. 2023), 15247–15258

    2023

  9. [10]

    Danfeng Guo, Arpit Gupta, Sanchit Agarwal, Jiun-Yu Kao, Shuyang Gao, Arijit Biswas, Chien-Wei Lin, Tagyoung Chung, and Mohit Bansal. 2022. GRAVL- BERT: Graphical Visual-Linguistic Representations for Multimodal Coreference Resolution. InProceedings of the 29th International Conference on Computational Linguistics. International Committee on Computational ...

    2022

  10. [11]

    Jon Hindmarsh, Mike Fraser, Christian Heath, Steve Benford, and Chris Green- halgh. 1998. Fragmented Interaction: Establishing Mutual Orientation in Virtual Environments. InProceedings of the 1998 ACM Conference on Computer Sup- ported Cooperative Work(CSCW ’98). Association for Computing Machinery, Seattle, Washington, USA, 217–226.isbn: 1581130090. doi:...

    arXiv 1998

  11. [12]

    Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 2023. 3D-LLM: Injecting the 3D World into Large Language Models, (July 2023). http://arxiv.org/abs/2307.12981

    Pith/arXiv arXiv 2023

  12. [13]

    Allison Jing, Kieran William May, Mahnoor Naeem, Gun Lee, and Mark Billinghurst

  13. [14]

    Seungwon Kim, Gun Lee, Mark Billinghurst, and Weidong Huang. 2020. The Combination of Visual Communication Cues in Mixed Reality Remote Collab- oration.Journal on Multimodal User Interfaces, (July 2020), 1–15. doi:10.1007/s1 2193-020-00335-x

    doi:10.1007/s1 2020

  14. [15]

    Chen Kong, Dahua Lin, Mohit Bansal, Raquel Urtasun, and Sanja Fidler. 2014. What Are You Talking About? Text-to-Image Coreference. In2014 IEEE Con- ference on Computer Vision and Pattern Recognition, 3558–3565. doi:10.1109 /CVPR.2014.455

    2014

  15. [16]

    Chen Kong, Dahua Lin, Mohit Bansal, Raquel Urtasun, and Sanja Fidler. [n. d.] What are you talking about? Text-to-Image Coreference. Tech. rep

  16. [17]

    Satwik Kottur, Seungwhan Moon, Alborz Geramifard, and Babak Damavandi

  17. [18]

    Jaewook Lee, Jun Wang, Elizabeth Brown, Liam Chu, Sebastian S Rodriguez, and Jon E Froehlich. 2024. Gazepointar: a context-aware multimodal voice assistant for pronoun disambiguation in wearable augmented reality

    2024

  18. [19]

    SIMMC 2.0: A Task-oriented Dialog Dataset for Immersive Multimodal Conversations, (Apr. 2021). http://arxiv.org/abs/2104.08667

    Pith/arXiv arXiv 2021

  19. [20]

    Yifan Liu, Jennifer Lather, and John Messner. 2014. Virtual Reality to Support the Integrated Design Process: A Retrofit Case Study. InComputing in civil and building engineering (2014), 801–808

    2014

  20. [21]

    Yifan Liu, Fadi Castronovo, John Messner, and Robert Leicht. 2020. Evaluating the Impact of Virtual Reality on Design Review Meetings.Journal of Computing in Civil Engineering, 34, 1, 04019045

    2020

  21. [22]

    Pan Lu, Liang Qiu, Wenhao Yu, Sean Welleck, and Kai-Wei Chang. 2022. A sur- vey of deep learning for mathematical reasoning.arXiv preprint arXiv:2212.10535

    Pith/arXiv arXiv 2022

  22. [23]

    Juan López-Tarruella Maldonado, Juan Luis Higuera Trujillo, Susana Iñarra Abad, MªCarmen Llinares Millán, Jaime Guixeres Provinciales, and Mariano Alcañiz Raya. 2018. Virtual Reality as a Tool for Emotional Evaluation of Architectural Environments. InArchitectural Draughtsmanship: From Analog to Digital Narratives 16. Springer, 889–903

    2018

  23. [24]

    Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J

    Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David McClosky. 2014. The Stanford CoreNLP Natural Language Processing Toolkit. InAssociation for Computational Linguistics (ACL) System Demonstrations, 55–60. http://www.aclweb.org/anthology/P/P14/P14-5010

    2014

  24. [25]

    Karthik Mahadevan, Qian Zhou, George Fitzmaurice, Tovi Grossman, and Fraser Anderson. 2023. Tesseract: Querying Spatial Design Recordings by Ma- nipulating Worlds in Miniature. InConference on Human Factors in Computing Systems - Proceedings. Association for Computing Machinery, (Apr. 2023).isbn: 9781450394215. doi:10.1145/3544548.3580876

    arXiv 2023

  25. [26]

    Sven Mayer, Valentin Schwind, Robin Schweigert, and Niels Henze. 2018. The Effect of Offset Correction and Cursor on Mid-Air Pointing in Real and Virtual Environments. InProceedings of the 2018 CHI Conference on Human Factors in Computing Systems(CHI ’18). Association for Computing Machinery, Montreal QC, Canada, 1–13.isbn: 9781450356206. doi:10.1145/3173...

    arXiv 2018

  26. [27]

    Sven Mayer, Gierad Laput, and Chris Harrison. 2020. Enhancing Mobile Voice Assistants with WorldGaze. InConference on Human Factors in Computing Systems - Proceedings. Association for Computing Machinery, (Apr. 2020).isbn: 9781450367080. doi:10.1145/3313831.3376479

    arXiv 2020

  27. [28]

    Robert Moulder, Brandon Booth, Angelina Abitino, and Sidney D’Mello. 2023. Recurrence Quantification Analysis of Eye Gaze Dynamics during Team Col- laboration. In Association for Computing Machinery, (Mar. 2023), 430–440. isbn: 9781450398657. doi:10.1145/3576050.3576113

    arXiv 2023

  28. [29]

    Scott MacKenzie

    Darius Miniotas, Oleg Špakov, Ivan Tugoy, and I. Scott MacKenzie. 2006. Speech- Augmented Eye Gaze Interaction with Small Closely Spaced Targets. InPro- ceedings of the 2006 Symposium on Eye Tracking Research & Applications(ETRA ’06). Association for Computing Machinery, San Diego, California, 67–72.isbn: 1595933050. doi:10.1145/1117309.1117345

    arXiv 2006

  29. [30]

    Anna Penzkofer, Philipp Müller, Felix Bühler, Sven Mayer, and Andreas Bulling

  30. [31]

    Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023. The RefinedWeb Dataset for Falcon LLM: Outper- forming Curated Corpora with Web Data, and Web Data Only, (June 2023). http://arxiv.org/abs/2306.01116

    Pith/arXiv arXiv 2023

  31. [32]

    Thammathip Piumsomboon, Arindam Dey, Barrett Ens, Gun Lee, and Mark Billinghurst. 2019. The Effects of Sharing Awareness Cues in Collaborative Mixed Reality.Frontiers Robotics AI, 6, FEB. doi:10.3389/frobt.2019.00005

    arXiv 2019

  32. [33]

    InProceedings of the 2021 International Conference on Multimodal Interaction, 341–351

    Conan: a usable tool for multimodal conversation analysis. InProceedings of the 2021 International Conference on Multimodal Interaction, 341–351

    2021

  33. [34]

    Sami Pietinen, Roman Bednarik, Tatiana Glotova, Vesa Tenhunen, and Markku Tukiainen. 2008. A Method to Study Visual Attention Aspects of Collaboration: Eye-Tracking Pair Programmers Simultaneously. InProceedings of the 2008 Symposium on Eye Tracking Research & Applications(ETRA ’08). Association for Computing Machinery, Savannah, Georgia, 39–42.isbn: 9781...

    arXiv 2008

  34. [35]

    Salvucci and Joseph H

    Dario D. Salvucci and Joseph H. Goldberg. 2000. Identifying Fixations and Saccades in Eye-Tracking Protocols. InProceedings of the 2000 Symposium on Eye Tracking Research & Applications(ETRA ’00). Association for Computing Machinery, Palm Beach Gardens, Florida, USA, 71–78.isbn: 1581132808. doi:1 0.1145/355017.355028

    arXiv 2000

  35. [36]

    RealWear, Inc. 2023. RealWear. https://www.realwear.com/. Accessed: [2023- 12-09]. (2023)

    2023

  36. [37]

    Yevhen Romaniak, Anastasiia Smielova, Yevhenii Yakishyn, Valerii Dziubliuk, Mykhailo Zlotnyk, and Oleksandr Viatchaninov. 2020. Nimble: Mobile Inter- face for a Visual Question Answering Augmented by Gestures. InAdjunct Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology(UIST ’20 Adjunct). Association for Computing Machi...

    arXiv 2020

  37. [38]

    Hugo Touvron et al. 2023. LLaMA: Open and Efficient Foundation Language Models, (Feb. 2023). http://arxiv.org/abs/2302.13971

    Pith/arXiv arXiv 2023

  38. [39]

    Bertrand Schneider and Roy Pea. 2013. Real-Time Mutual Gaze Perception Enhances Collaborative Learning and Collaboration Quality.International Journal of Computer-Supported Collaborative Learning, 8, 4, 375–397.isbn: 1141201391814. doi:10.1007/s11412-013-9181-4

    doi:10.1007/s11412-013-9181-4 2013

  39. [40]

    Maurício Sousa, Rafael Kuffner Dos Anjos, Daniel Mendes, Mark Billinghurst, and Joaquim Jorge. 2019. Warping deixis: Distorting Gestures to Enhance Col- laboration. InConference on Human Factors in Computing Systems - Proceedings. Vol. 12. Association for Computing Machinery, New York, NY, USA, (May 2019), 1–12.isbn: 9781450359702. doi:10.1145/3290605.3300838

    arXiv 2019

  40. [41]

    Stewart, and Sidney K

    Hana Vrzakova, Mary Jean Amon, Angela E.B. Stewart, and Sidney K. D’Mello

  41. [42]

    Maureen Villamor and Ma Mercedes Rodrigo. 2018. Predicting Successful Collaboration in a Pair Programming Eye Tracking Experiment. InUMAP 2018 - Adjunct Publication of the 26th Conference on User Modeling, Adaptation and Personalizationnumber July, 263–268.isbn: 9781450357845. doi:10.1145/32135 86.3225234

    work page doi:10.1145/32135 2018

  42. [43]

    VRChat Inc. 2023. VRChat. https://vrchat.com/. Accessed: [2023-12-09]. (2023)

    2023

  43. [44]

    Nelson Wong and Carl Gutwin. 2010. Where Are You Pointing? The Accuracy of Deictic Pointing in CVEs. InConference on Human Factors in Computing Systems - Proceedings. Vol. 2. ACM Press, New York, New York, USA, 1029–1038. isbn: 9781605589299. doi:10.1145/1753326.1753480

    arXiv 2010

  44. [45]

    Xintong Yu, Hongming Zhang, Ruixin Hong, Yangqiu Song, and Changshui Zhang. 2022. VD-PCR: Improving Visual Dialog with Pronoun Coreference Resolution.Pattern Recognition, 125, (May 2022). doi:10.1016/j.patcog.2022.108 540

    work page doi:10.1016/j.patcog.2022.108 2022

  45. [46]

    Eric Wallace, Yizhong Wang, Sujian Li, Sameer Singh, and Matt Gardner

  46. [47]

    Do nlp models know numbers? probing numeracy in embeddings.arXiv preprint arXiv:1909.07940

    Pith/arXiv arXiv 1909

  47. [48]

    Nelson Wong and Carl Gutwin. 2014. Support for Deictic Pointing in CVEs: Still Fragmented after All These Years’. InProceedings of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing(CSCW ’14). Association for Computing Machinery, Baltimore, Maryland, USA, 1377–1387. isbn: 9781450325400. doi:10.1145/2531602.2531691

    arXiv 2014

  48. [51]

    Xintong Yu, Hongming Zhang, Yangqiu Song, Yan Song, and Changshui Zhang

  49. [52]

    What You See is What You Get: Visual Pronoun Coreference Resolution in Dialogues, (Sept. 2019). http://arxiv.org/abs/1909.00421

    Pith/arXiv arXiv 2019

  50. [53]

    Xiaoyu Zhang, Jianping Li, Po Wei Chi, Senthil Chandrasegaran, and Kwan Liu Ma. 2023. ConceptEVA: Concept-Based Interactive Exploration and Customiza- tion of Document Summaries. InConference on Human Factors in Computing Systems - Proceedings. Association for Computing Machinery, (Apr. 2023).isbn: 9781450394215. doi:10.1145/3544548.3581260

    arXiv 2023

  51. [54]

    Yanxia Zhang, Ken Pfeuffer, Ming Ki Chong, Jason Alexander, Andreas Bulling, and Hans Gellersen. 2017. Look together: using gaze for assisting co-located collaborative search.Personal and Ubiquitous Computing, 21, 1, 173–186. doi:10 .1007/s00779-016-0969-x. Received 20 February 2007; revised 12 March 2009; accepted 5 June 2009

    2017

  52. [2019]

    doi:10.11 45/3290605.3300572

    Dynamics of Visual Aention in Multiparty Collaborative Problem Solving using Multidimensional Recurrence antification Analysis.Conference on Human Factors in Computing Systems - Proceedings, 14.isbn: 9781450359702. doi:10.11 45/3290605.3300572

    arXiv

  53. [2021]

    InConference on Human Factors in Com- puting Systems - Proceedings

    EyemR-Vis: Using Bi-Directional Gaze Behavioural Cues to Improve Mixed Reality Remote Collaboration. InConference on Human Factors in Com- puting Systems - Proceedings. Association for Computing Machinery, (May 2021).isbn: 9781450380959. doi:10.1145/3411763.3451844

    arXiv 2021

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.