Pith. sign in

REVIEW 2 major objections 6 minor 111 references

Can You Move These Over There? An LLM-based VR Mover for Supporting Object Manipulation

T0 review · 2 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that a VR interface which lets users speak and point naturally, driven by a large language model, outperforms standard gizmo and voice-command methods for moving multiple objects, based on a 24-participant user study.

desk verdict A genuinely useful LLM+gesture VR manipulation system with a substantial user study, but the claimed LLM-specific benefit is undercut by a confounded ablation and faulty statistical reporting. read the letter →

arxiv 2502.02201 v1 pith:ZVRWNMDY submitted 2025-02-04 cs.HC cs.AIcs.CLcs.ET

Xiangzhi Eric Wang , Zackary P. T. Sin , Ye Jia , Daniel Archer , Wynonna H. Y. Fong , Qing Li , Chen Li This is my paper · ORCID
classification cs.HCcs.AIcs.CLcs.ET
keywords virtualrealityobjectmanipulationlargelanguagemodelnaturaluserinterfacemultimodalinteractionstudyarmfatiguemulti-object
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes VR Mover, a virtual-reality interface that lets users move, rotate, and place objects by speaking naturally and pointing at targets while a large language model interprets the intent and issues the manipulation commands. The central claim is that this LLM-based natural interface improves usability, user experience, and performance on multi-object manipulation tasks, and reduces workload and arm fatigue, relative to the standard gizmo-and-virtual-hand technique and to a grammar-based voice command variant. The authors aim to show that transposing real-world speech-and-gesture communication into VR is not just feasible but practically better for coarse placement, with users then switching to hand controls for fine adjustments. If true, this offers designers a concrete alternative to menu-and-controller manipulation in VR scene editing and a template for LLM-driven multimodal interaction.

What carries the argument

The carrying mechanism is a user-centric augmentation loop that serializes what the user says, where they look, and what they point or draw into a timestamped JSON prompt, plus an LLM processing module that translates that prompt into line-wise API calls (CREATE, MOVE, FORWARD, LOOKAT, SCALE, DELETE, MESSAGE) executed one by one as they stream, giving roughly a two-second response delay. The LLM's system prompt embeds a JSON scene model (oriented bounding boxes, prefab descriptions, room and environment objects) and instructs the model to respect physical constraints, user perspective, and spatial relations such as chairs facing a table. Gestural input is injected into the transcribed speech text so the model can resolve deictic references like 'here' and 'this' without a structured command grammar.

What would settle it

Take the Voice Command interface, add line-drawing and a free-form instruction parser that uses the same system-prompt knowledge but no learned language model, and rerun Task 1B. If the manipulation-time and fatigue advantages of VR Mover disappear or shrink to non-significance, the LLM itself is not the cause of the reported gains; if they persist, the claim that the LLM is load-bearing is supported.

Watch

Extended reading notes

Core claim

The core discovery is that VR Mover, built on GPT-4o with a streaming response pipeline, significantly shortens coarse and fine manipulation time for multi-object placement (Task 1B), cuts total hand movement distance, lowers Borg C10 arm fatigue and NASA-TLX workload, and raises UEQ-S experience scores compared with both a gizmo/virtual-hand control and a Voice Command variant that maps a fixed grammar to the same APIs. The advantage appears specifically when multiple objects are involved: for a single mid-air object (Task 1A), VR Mover shows no improvement. Users ranked VR Mover first for both the performance task and the free-form sandbox task, and qualitative comments emphasize conversational naturalness, no need to memorize grammar, and a coarse-to-fine workflow where the LLM places objects roughly and the user fine-tunes with a virtual hand.

Load-bearing premise

The comparison that isolates the language model also changes other things: the Voice Command version has no line-drawing and only accepts a fixed set of phrases, so the measured benefit might come from the extra gesture channel and the flexibility of the prompts rather than from the LLM as such.

Editorial extensions

If this is right

  • VR Mover reduces coarse manipulation time in multi-object tasks by at least 40% on average compared to control and voice command, and halves hand movement distance.
  • Users can issue unstructured, incomplete, or context-dependent instructions (e.g., "move it back", "put four chairs around the table") and see the scene update within about 2 seconds.
  • The design supports a coarse-to-fine workflow: users delegate rough placement to the LLM and switch to gizmos or virtual hands for fine-tuning, matching how visual working memory chunks objects.
  • The LLM is not a full autonomous agent; it needs human fine-tuning for precise and mid-air placement, and the paper does not find significant presence gains.
  • The interface shows no benefit for single-object mid-air manipulation, indicating its value is concentrated in multi-object and ground-plane scenarios.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the benefit is driven mainly by the added line-drawing gesture and the freedom to phrase instructions, then extending the Voice Command baseline with line-input and a more flexible (but still grammar-based) parser might close much of the measured gap, which would be a cheap way to test whether the LLM itself is load-bearing.
  • A promising extension would be to replace the text-only scene model with visual prompting for collision-aware placement; the authors note this is a limitation because of response-time constraints, and faster multimodal models may enable it without losing interactivity.
  • The two-second streaming-response design suggests a general recipe for real-time LLM-in-the-loop interfaces: keep prompts text-only, emit line-oriented executable commands, and stream the output so the first effect appears before the full response finishes.
  • The results imply that the gorilla-arm effect in VR scene editing could be mitigated by letting users delegate motion to an LLM while retaining direct manipulation only for the final adjustment, which is exactly the division users spontaneously adopted.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper proposes VR Mover, an LLM-based VR object manipulation interface that combines speech with pointing and line-drawing (lining) gestures. The system converts the scene and the user's head, gaze-like focus frames, gestures, and transcribed speech into JSON prompts for GPT-4o, which returns streaming API calls to create, move, rotate, scale, and delete objects. A user study with 24 participants compares VR Mover with a Control condition (gizmos plus virtual hand) and a Voice Command condition (regular-expression grammar, no lining) on single-object mid-air manipulation (Task 1A), multi-object manipulation (Task 1B), and a free-form room furnishing task (Task 2). The results show significant advantages for VR Mover over both baselines in Task 1B coarse and fine manipulation time and hand movement, in Borg C10 arm fatigue, in NASA-TLX workload, and in UEQ-S user experience, with no significant benefit in Task 1A. The paper concludes that the Voice Command comparison demonstrates the importance of the LLM.

Significance. If the results hold after the statistical reporting is corrected, this is a useful systems contribution to VR object manipulation. The paper gives an unusually complete technical description, including the full system prompts, the Voice Command grammar, and a worked input/output example, which is valuable for reproducibility. The streaming API-call design and the roughly two-second response delay are practical contributions, and the user study uses standard instruments (Borg C10, NASA-TLX, UEQ-S, SUS, PQ) alongside objective timing and hand-movement measures. The main limitations are that the Voice Command condition is not a clean ablation of the LLM, and that the reported ANOVA degrees of freedom and at least one p-value appear inconsistent; both issues must be resolved before the paper's central claims can be accepted at face value.

major comments (2)
  1. [5.1, 9] The Voice Command condition is described as an 'LLM-removed' variant (Section 5.1, Fig. 12c), but it differs from VR Mover in at least two respects: the LLM is replaced by a regular-expression grammar, and the line-drawing (lining) gesture is not implemented because the variant 'only includes pointing, but not aligning'. Consequently, the significant advantages of VR Mover over Voice Command in Task 1B manipulation time and hand movement (Sections 6.1-6.2) and in Borg C10, NASA-TLX, and UEQ-S (Sections 6.3-6.5) cannot be attributed specifically to the LLM; they may equally be driven by the additional line-drawing input channel, by the LLM's ability to handle asynchronous multi-object instructions, or by the richer system prompt. The concluding sentence in Section 9, which says the comparison 'shows the importance of an LLM', therefore overreaches. Please reframe the contribution as an evaluation of the complete multimodal LLM-based interface, add a controlled condition that isolates the LLM (e.g., VR Mover without lining, or an LLM-based variant with the same pointing-only input), or explicitly acknowledge this confound as a central limitation.
  2. [6.1-6.7] The repeated-measures ANOVA results are reported with F(2,23) throughout, but with N=24 participants and one within-subjects factor at three levels the denominator degrees of freedom should be (N-1)(k-1)=46 for a standard repeated-measures model. In Section 6.2, F(2,23)=1.083 is paired with p=0.582, which is not the p-value for that statistic (approximately 0.35 for either df), and in Section 6.7, F(2,23)=4.748 with p=0.0134 appears inconsistent with the stated degrees of freedom. Because these statistics carry the paper's main quantitative claims, please provide corrected F, p, and effect-size values, or clearly specify the statistical model if it differs from a standard repeated-measures ANOVA, and confirm that all post-hoc tests were Bonferroni-corrected as stated.
minor comments (6)
  1. [6.1] For Task 1A, the text reports 'a significant main effect for coarse and fine manipulation time' but provides no omnibus F or p values, and the pairwise tests show VR Mover is not significantly different from Control (p=1.00 for both coarse and fine time) while Voice Command is significantly slower; please add the missing omnibus statistics and state explicitly that VR Mover did not improve single-object mid-air performance over Control.
  2. [6.4] There are missing equals signs in 'F(2,23)18.024' and 'F(2,23)22.747', and the direction of the 'performance' subscale (higher is better) should be stated explicitly, since the other NASA-TLX subscales are all lower-is-better.
  3. [6.7] The sentence 'for Virtual Assistant, Voice Command and Control group' mislabels the conditions; it should read 'Control, Voice Command and VR Mover'.
  4. [6.8] The duplicated phrase 'For Task 2, For Task 2' should be removed.
  5. [Appendix C, C.1] There is a typo in the system prompt: 'tne End's normal' should be 'the End's normal'; the manuscript also uses 'manipulatable' and 'manipulable' inconsistently.
  6. [7.3] The statement that the lower mental demand 'can be indicative of validation of this belief' overinterprets the NASA-TLX subscale as support for the visual working memory account; the study does not directly measure chunking or coarse-to-fine processing, so this claim should be softened.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the paper's claims rest on a user study with externally measured outcomes, not on derived predictions or self-citation chains.

full rationale

This paper makes no derivation claim that could reduce to its own inputs. The central claims are empirical: VR Mover improved multi-object manipulation time, hand movement distance, arm fatigue, workload, and user experience relative to a gizmo/virtual-hand control and a grammar-based voice command variant. These outcomes are measured user behavior, not constants recovered from a theory or parameters fitted to the outcome being predicted. The visual working memory discussion in Section 3.2 is used only as design motivation (chunking and coarse-to-fine processing), and the measured task times, Borg C10 scores, NASA-TLX ratings, SUS scores, and UEQ-S scores are external to that discussion; no equation defines an outcome in terms of the theory. The system does use an LLM with a crafted system prompt, but the paper does not claim to derive the LLM's behavior from first principles; it reports the observed performance of the deployed system. No fitted parameter is renamed as a prediction: coarse and fine manipulation thresholds (0.3m and 0.12m) are defined independently of the comparison results, and no result is computed from those thresholds in a way that would force the reported differences. The references cited for prior techniques are external works, and none of the paper's load-bearing claims rests on a self-citation chain. The Voice Command variant lacks the lining gesture and uses a rigid grammar, which is a genuine experimental confound that weakens the claim that the LLM specifically caused the advantage, but confounded comparison is a validity concern, not circularity: the measured advantages are not identical to the design by construction. For these reasons, the appropriate finding is no significant circularity, with a score of 0.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The only hand-tuned numbers are the task completion thresholds, which determine the dependent variables but are not fitted to produce significance. The core assumptions are the reliability of the external LLM, the validity of the head-motion proxy, and the generalizability of the 24-person study.

free parameters (2)
  • Coarse target threshold = 0.3 m
    Average bounding-box point distance used to define 'coarse' completion; chosen from early testing (Section 5.4).
  • Fine target threshold = 0.12 m
    Average bounding-box point distance defining 'fine' completion; chosen from early testing (Section 5.4).
assumptions (5)
  • domain assumption GPT-4o reliably generates valid API calls from the prompt context
    The entire system depends on the hosted GPT-4o model parsing user requests into API calls (Section 4.3).
  • domain assumption Head motion is a sufficient proxy for gaze in focus frames
    Section 4.2.2 states the system tracks head motion instead of gaze due to headset limitations.
  • domain assumption OBB text serialization preserves enough spatial information for placement decisions
    Scene modelling in Section 4.1 converts objects to text-based oriented bounding boxes; if the LLM cannot reason spatially from this text, placements fail.
  • domain assumption The within-subjects user study with 24 participants generalizes
    Section 5.3 describes the recruitment; the paper assumes this sample is representative for drawing conclusions.
  • standard math Standard repeated-measures statistics are appropriate
    Section 6 states Shapiro-Wilk, ANOVA/Friedman and post-hoc tests with Bonferroni correction.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Can You Move These Over There? An LLM-based VR Mover for Supporting Object Manipulation." pith.science (2026). https://pith.science/paper/ZVRWNMDY

@misc{pith2026250202201,
  author       = {Pith},
  title        = {Pith review of: Can You Move These Over There? An LLM-based VR Mover for Supporting Object Manipulation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZVRWNMDY}},
  note         = {Machine review of arXiv:2502.02201}
}
read the original abstract

In our daily lives, we can naturally convey instructions for the spatial manipulation of objects using words and gestures. Transposing this form of interaction into virtual reality (VR) object manipulation can be beneficial. We propose VR Mover, an LLM-empowered solution that can understand and interpret the user's vocal instruction to support object manipulation. By simply pointing and speaking, the LLM can manipulate objects without structured input. Our user study demonstrates that VR Mover enhances user usability, overall experience and performance on multi-object manipulation, while also reducing workload and arm fatigue. Users prefer the proposed natural interface for broad movements and may complementarily switch to gizmos or virtual hands for finer adjustments. These findings are believed to contribute to design implications for future LLM-based object manipulation interfaces, highlighting the potential for more intuitive and efficient user interactions in VR environments.

Figures

Figures reproduced from arXiv: 2502.02201 by the authors.

Figure 1
Figure 1. We propose VR Mover, an LLM-based interface for supporting object manipulation. It aggregates user-centric information such as what the user is saying, seeing and pointing at to decide how to assist them in the placement of objects. In our daily lives, we can naturally convey instructions for the spatial manipulation of objects using words and gestures. Transposing this form of interaction into virtual reality (VR) … view at source ↗
Figure 2
Figure 2. To move an object, (a) the user can first specify an object and its target by pointing. However, as [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. VR Mover can handle complex instructions such as the user using asynchronous multi-object manipulation where objects are applied with different manipulation (e.g. different movement) while mixing different manipulation operations (e.g. moving and rotating). • Lining: Aside from pointing, we have different ways to gesturally indicate spatial information. Another method is by drawing a line (lining) to indicate direct… view at source ↗
Figures from the paper (19 more)
Figure 4
Figure 4. Figure 4: By drawing a line (lining), a user can express different manipulations. Here, we show the user using (a) a line to represent a [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Empowered by LLM, VR Mover can demonstrate intelligent responses in some instances. (a) When the user requests four chairs and a table in the middle of the room, VR Mover is aware of the environment and able to place the objects in the room’s center. Further, it has sp…
Figure 6
Figure 6. Figure 6: Different interaction methods can be used to engage with [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: The system overview of VR Mover. The scene modelling component, maintains environmental and manipulatable object data, [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: Scene modelling component in VR Mover. Manipulatable prefabs, manipulatable objects, and environmental objects can be [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]
Figure 9
Figure 9. Figure 9: User-centric augmentation module: integrates speech-to-text, focus frames from head motion, and gesture cues from the [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]
Figure 10
Figure 10. Figure 10: LLM processing module: Prompt engineering combines user inputs with system prompts. The context window manages [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 11
Figure 11. Figure 11: Scene update module: processes LLM responses through text API and parser, queries scene modelling for object data, and [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]
Figure 12
Figure 12. Figure 12: The (a) gizmo and (b) virtual hand are used in all three experimental techniques. The [PITH_FULL_IMAGE:figures/full_fig_p017_12.png]
Figure 13
Figure 13. Figure 13: The environment of (a) Task 1A and (b) 1B. The goal of the user is to move the object(s) to the (semi-transparent) target(s). [PITH_FULL_IMAGE:figures/full_fig_p018_13.png]
Figure 14
Figure 14. Figure 14: In Task 2, given an (a) empty room the user is instructed to populate it with object according to the (b) mini-room. [PITH_FULL_IMAGE:figures/full_fig_p018_14.png]
Figure 15
Figure 15. Figure 15: The upper row shows Task 1A’s (ai) coarse manipulation time, (aii) fine manipulation time and (aiii) hand movement distance. [PITH_FULL_IMAGE:figures/full_fig_p020_15.png]
Figure 16
Figure 16. Figure 16: The result from (a) the borg C10, (b) SUS and (c) presence. [PITH_FULL_IMAGE:figures/full_fig_p021_16.png]
Figure 17
Figure 17. Figure 17: The (a) overall mean score of NASA-TLX and its subscales, (bi) mental demand, (bii) physical demand, (biii) temporal demand [PITH_FULL_IMAGE:figures/full_fig_p022_17.png]
Figure 18
Figure 18. Figure 18: The (a) overall, (b) pragmatic, and (c) hedonic user experience result from UEQ-S. [PITH_FULL_IMAGE:figures/full_fig_p023_18.png]
Figure 19
Figure 19. Figure 19: The preference ranking for (a) Task 1 and (b) Task 2. [PITH_FULL_IMAGE:figures/full_fig_p024_19.png]
Figure 20
Figure 20. Figure 20: The menu control for task two in the user study. [PITH_FULL_IMAGE:figures/full_fig_p031_20.png]
Figure 21
Figure 21. Figure 21: User built rooms via different techniques [PITH_FULL_IMAGE:figures/full_fig_p033_21.png]
Figure 22
Figure 22. Figure 22: The task two scene before (left) and after (right) the user request has been processed by the [PITH_FULL_IMAGE:figures/full_fig_p041_22.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

111 extracted references · 73 canonical work pages

  1. [1]

    Rania Abdelghani, Yen-Hsiang Wang, Xingdi Yuan, and et al. 2024. GPT-3-Driven Pedagogical Agents to Train Children’s Curious Question-Asking Skills. International Journal of Artificial Intelligence in Education 34 (2024), 483–518. https://doi.org/10.1007/s40593-023-00340-7

    doi:10.1007/s40593-023-00340-7 2024

  2. [2]

    Anthropic. 2024. Meet Claude. https://www.anthropic.com/claude Accessed: 2024-9-13. Manuscript submitted to ACM 28 Xiangzhi Eric Wang, Zackary P. T. Sin, Ye Jia, Daniel Archer, Wynonna H. Y. Fong, Qing Li, and Chen Li

    2024

  3. [3]

    Elizabeth E Austin and Naomi Sweller. 2014. Presentation and production: The role of gesture in spatial communication. Journal of experimental child psychology 122 (2014), 92–103

    2014

  4. [4]

    Edward Awh, Brian Barton, and Edward K. Vogel. 2007. Visual Working Memory Represents a Fixed Number of Items Regardless of Complexity. Psychological Science 18, 7 (2007), 622–628. https://doi.org/10.1111/j.1467-9280.2007.01949.x PMID: 17614871

    work page arXiv 2007

  5. [5]

    Farkhandah Aziz, Chris Creed, Sayan Sarcar, Maite Frutos-Pascual, and Ian Williams. 2022. Voice Snapping: Inclusive Speech Interaction Techniques for Creative Object Manipulation. In Proceedings of the 2022 ACM Designing Interactive Systems Conference . 1486–1496

    2022

  6. [6]

    Alan Baddeley. 2003. Working memory: looking back and looking forward. Nature reviews neuroscience 4, 10 (2003), 829–839

    2003

  7. [7]

    Yiwei Bao, Jiaxi Wang, Zhimin Wang, and Feng Lu. 2023. Exploring 3d interaction with gaze guidance in augmented reality. In 2023 IEEE Conference Virtual Reality and 3D User Interfaces (VR) . IEEE, 22–32

    2023

  8. [8]

    Miguel Angel Bautista, Pengsheng Guo, Samira Abnar, Walter Talbott, Alexander Toshev, Zhuoyuan Chen, Laurent Dinh, Shuangfei Zhai, Hanlin Goh, Daniel Ulbricht, et al. 2022. Gaudi: A neural architect for immersive 3d scene generation. Advances in Neural Information Processing Systems 35 (2022), 25102–25116

    2022

Show all 111 references
  1. [9]

    Rojin Bayat, Elios De Maio, Jacopo Fiorenza, Massimo Migliorini, and Fabrizio Lamberti. 2024. Exploring Methodologies to Create a Unified VR User-Experience in the Field of Virtual Museum Experiences. In 2024 IEEE Gaming, Entertainment, and Media Conference (GEM) . IEEE, 1–4

    2024

  2. [10]

    Gunnar A Borg. 1982. Psychophysical bases of perceived exertion. Medicine and science in sports and exercise 14, 5 (1982), 377–381

    1982

  3. [11]

    J Brooke. 1996. SUS: A quick and dirty usability scale. Usability Evaluation in Industry (1996)

    1996

  4. [12]

    Erin A Carroll, Celine Latulipe, Richard Fung, and Michael Terry. 2009. Creativity factor evaluation: towards a standardized survey metric for creativity support. In Proceedings of the seventh ACM conference on Creativity and cognition . 127–136

    2009

  5. [13]

    Zhaoxi Chen, Guangcong Wang, and Ziwei Liu. 2023. Scenedreamer: Unbounded 3d scene generation from 2d image collections. IEEE transactions on pattern analysis and machine intelligence (2023)

    2023

  6. [14]

    Alan Y Cheng, Meng Guo, Melissa Ran, Arpit Ranasaria, Arjun Sharma, Anthony Xie, Khuyen N Le, Bala Vinaithirthan, Shihe Luan, David Thomas Henry Wright, et al. 2024. Scientific and Fantastical: Creating Immersive, Culturally Relevant Learning Experiences with Augmented Reality...

    2024

  7. [15]

    Tor-Salve Dalsgaard, Jarrod Knibbe, and Joanna Bergström. 2021. Modeling pointing for 3D target selection in VR. In Proceedings of the 27th ACM Symposium on Virtual Reality Software and Technology . 1–10

    2021

  8. [16]

    Fernanda De La Torre, Cathy Mengying Fang, Han Huang, Andrzej Banburski-Fahey, Judith Amores Fernandez, and Jaron Lanier. 2024. LLMR: Real-time Prompting of Interactive Worlds using Large Language Models. In Proceedings of the CHI Conference on Human Factors in Computing Syste...

    2024

  9. [17]

    Giuseppe Desolda, Andrea Esposito, Florian Müller, and Sebastian Feger. 2023. Digital Modeling for Everyone: Exploring How Novices Approach Voice-Based 3D Modeling. In IFIP Conference on Human-Computer Interaction . Springer, 133–155

    2023

  10. [18]

    Tobias Drey, Michael Montag, Andrea Vogt, Nico Rixen, Tina Seufert, Steffi Zander, Michael Rietzler, and Enrico Rukzio. 2023. Investigating the effects of individual spatial abilities on virtual reality object manipulation. In Proceedings of the 2023 CHI Conference on Human Fa...

    2023

  11. [19]

    Jorge Askur Vazquez Fernandez, Jae Joong Lee, Santiago Andrés Serrano Vacca, Alejandra Magana, Bedrich Benes, and Voicu Popescu. 2024. Hands-Free VR. arXiv preprint arXiv:2402.15083 (2024)

    2024 arXiv

  12. [20]

    Franz, Sasa Junuzovic, and Martez Mott

    Rachel L. Franz, Sasa Junuzovic, and Martez Mott. 2024. A Virtual Reality Scene Taxonomy: Identifying and Designing Accessible Scene-Viewing Techniques. ACM Trans. Comput.-Hum. Interact. 31, 2, Article 23 (feb 2024), 44 pages. https://doi.org/10.1145/3635142

    2024 doi

  13. [21]

    Markus Friedrich, Stefan Langer, and Fabian Frey. 2021. Combining Gesture and Voice Control for Mid-air Manipulation of CAD Models in VR Environments.. In VISIGRAPP (2: HUCAPP). 119–127

    2021

  14. [22]

    Zaifeng Gao and Shlomo Bentin. 2011. Coarse-to-fine encoding of spatial frequency information into visual short-term memory for faces but impartial decay. Journal of Experimental Psychology: Human Perception and Performance 37, 4 (2011), 1051

    2011

  15. [23]

    Zaifeng Gao, Xiaowei Ding, Tong Yang, Junying Liang, and Rende Shui. 2013. Coarse-to-fine construction for high-resolution representation in visual working memory. PloS one 8, 2 (2013), e57913

    2013

  16. [24]

    Daniele Giunchi, Nels Numan, Elia Gatti, and Anthony Steed. 2024. DreamCodeVR: Towards Democratizing Behavior Design in Virtual Reality with Speech-Driven Programming. In2024 IEEE Conference Virtual Reality and 3D User Interfaces (VR). 579–589. https://doi.org/10.1109/VR58804....

    2024

  17. [25]

    P Christopher Gloumeau, Wolfgang Stuerzlinger, and JungHyun Han. 2020. Pinnpivot: Object manipulation using pins in immersive virtual environments. IEEE transactions on visualization and computer graphics 27, 4 (2020), 2488–2494

    2020

  18. [26]

    Barbara Göbl, Simone Kriglstein, and Helmut Hlavacs. 2021. Conversational Interfaces in Serious Games: Identifying Potentials and Future Research Directions based on a Systematic Literature Review. CSEDU (1) (2021), 108–115

    2021

  19. [27]

    Ankur Handa, Viorica Pătrăucean, Simon Stent, and Roberto Cipolla. 2016. Scenenet: An annotated model generator for indoor scene understanding. In 2016 IEEE International Conference on Robotics and Automation (ICRA) . IEEE, 5737–5743

    2016

  20. [28]

    Jeffrey T Hansberger, Chao Peng, Shannon L Mathis, Vaidyanath Areyur Shanthakumar, Sarah C Meacham, Lizhou Cao, and Victoria R Blakely

  21. [29]

    Sandra G Hart. 2006. NASA-task load index (NASA-TLX); 20 years later. In Proceedings of the human factors and ergonomics society annual meeting , Vol. 50. Sage publications Sage CA: Los Angeles, CA, 904–908

    2006

  22. [30]

    John B Haviland. 2000. Pointing, gesture spaces, and mental maps. Language and gesture 2 (2000), 13

    2000

  23. [31]

    Jan Hombeck, Henrik Voigt, Timo Heggemann, Rabi R Datta, and Kai Lawonn. 2023. Tell me where to go: Voice-controlled hands-free locomotion for virtual reality systems. In 2023 IEEE Conference Virtual Reality and 3D User Interfaces (VR) . IEEE, 123–134

    2023

  24. [32]

    Robert JK Jacob. 1990. What you look at is what you get: eye movement-based interaction techniques. In Proceedings of the SIGCHI conference on Human factors in computing systems . 11–18

    1990

  25. [33]

    Maurice Jakesch, Advait Bhat, Daniel Buschek, Lior Zalmanson, and Mor Naaman. 2023. Co-writing with opinionated language models affects users’ views. In Proceedings of the 2023 CHI conference on human factors in computing systems . 1–15

    2023

  26. [34]

    Sujin Jang, Wolfgang Stuerzlinger, Satyajit Ambike, and Karthik Ramani. 2017. Modeling cumulative arm fatigue in mid-air interaction based on perceived exertion and kinetics of arm motion. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems . 3328–3339

    2017

  27. [35]

    Kevin Sujith John, G Abin Roy, and PS Bindhya. 2024. LLM Based 3D Avatar Assistant. In 2024 1st International Conference on Trends in Engineering Systems and Technologies (ICTEST). IEEE, 1–5

    2024

  28. [36]

    Majeed Kazemitabaar, Xinying Hou, Austin Henley, Barbara Jane Ericson, David Weintrop, and Tovi Grossman. 2023. How novices use LLM-based code generators to solve CS1 coding tasks in a self-paced learning environment. In Proceedings of the 23rd Koli Calling International Confe...

    2023

  29. [37]

    Ahmet Baki Kocaballi, Liliana Laranjo, and Enrico Coiera. 2019. Understanding and measuring user experience in conversational interfaces. Interacting with Computers 31, 2 (2019), 192–207

    2019

  30. [38]

    Xiaolong Liu, Lili Wang, Wei Ke, and Sio-Kei Im. 2024. Object manipulation based on the head manipulation space in VR. International Journal of Human-Computer Studies (2024), 103346

    2024

  31. [39]

    Robert H Logie. 2003. Spatial and visual working memory: A mental workspace. In Psychology of learning and motivation . Vol. 42. Elsevier, 37–78

    2003

  32. [40]

    Rui Ma, Akshay Gadi Patil, Matthew Fisher, Manyi Li, Sören Pirk, Binh-Son Hua, Sai-Kit Yeung, Xin Tong, Leonidas Guibas, and Hao Zhang. 2018. Language-driven synthesis of 3D scenes from scene databases. ACM Transactions on Graphics (TOG) 37, 6 (2018), 1–16

    2018

  33. [41]

    Mitchell W McEwan, Alethea L Blackler, Daniel M Johnson, and Peta A Wyeth. 2014. Natural mapping and intuitive interaction in videogames. In Proceedings of the first ACM SIGCHI annual symposium on Computer-human interaction in play . 191–200

    2014

  34. [42]

    Daniel Mendes, Fabio Marco Caputo, Andrea Giachetti, Alfredo Ferreira, and Joaquim Jorge. 2019. A survey on 3d virtual object manipulation: From the desktop to immersive virtual environments. In Computer graphics forum, Vol. 38. Wiley Online Library, 21–45

    2019

  35. [43]

    Daniel Mendes, Maurício Sousa, Rodrigo Lorena, Alfredo Ferreira, and Joaquim Jorge. 2017. Using custom transformation axes for mid-air manipulation of 3D virtual objects. In Proceedings of the 23rd ACM Symposium on Virtual Reality Software and Technology . 1–8

    2017

  36. [44]

    Meta. 2024. Llama 3.1. https://llama.meta.com/ Accessed: 2024-9-13

    2024

  37. [45]

    Microsoft. 2024. How to generate reproducible output with Azure OpenAI Service - Azure OpenAI. https://learn.microsoft.com/en-us/azure/ai- services/openai/how-to/reproducible-output Accessed: 2024-09-11

    2024

  38. [46]

    Microsoft Corporation. 2024. Microsoft Azure. https://azure.microsoft.com Accessed: 2024-09-11

    2024

  39. [47]

    George A Miller. 1956. The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological review 63, 2 (1956), 81

    1956

  40. [48]

    Elena Morotti, Lorenzo Donatiello, and Gustavo Marfia. 2020. Fostering fashion retail experiences through virtual reality and voice assistants. In 2020 IEEE conference on virtual reality and 3D user interfaces abstracts and workshops (VRW) . IEEE, 338–342

    2020

  41. [49]

    Başak Melis Öcal, Maxim Tatarchenko, Sezer Karaoglu, and Theo Gevers. 2024. SceneTeller: Language-to-3D Scene Generation. arXiv preprint arXiv:2407.20727 (2024)

    2024 arXiv

  42. [50]

    OpenAI. 2024. ChatGPT. https://openai.com/chatgpt/ Accessed: 2024-9-13

    2024

  43. [51]

    OpenAI. 2024. Hello GPT-4o. https://openai.com/index/hello-gpt-4o/ Accessed: 2024-9-13

    2024

  44. [52]

    Ivan Poupyrev, Mark Billinghurst, Suzanne Weghorst, and Tadao Ichikawa. 1996. The go-go interaction technique: non-linear mapping for direct manipulation in VR. In Proceedings of the 9th annual ACM symposium on User interface software and technology . 79–80

    1996

  45. [53]

    Siyuan Qi, Yixin Zhu, Siyuan Huang, Chenfanfu Jiang, and Song-Chun Zhu. 2018. Human-centric indoor scene synthesis using stochastic grammar. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . 5899–5908

    2018

  46. [54]

    Jesse Sargent, Stephen Dopkins, John Philbeck, and David Chichka. 2010. Chunking in spatial memory. Journal of Experimental Psychology: Learning, memory, and cognition 36, 3 (2010), 576

    2010

  47. [55]

    Jeff Sauro. 2018. Can You Use a Single Item to Predict SUS Scores? Retrieved Sep 13, 2024 from https://measuringu.com/single-item-sus/

    2018

  48. [56]

    SCHNEIDER and DAVID H

    PHILIP J. SCHNEIDER and DAVID H. EBERLY. 2003. CHAPTER 11 - INTERSECTION IN 3D. In Geometric Tools for Computer Graphics , PHILIP J. SCHNEIDER and DAVID H. EBERLY (Eds.). Morgan Kaufmann, San Francisco, 481–662. https://doi.org/10.1016/B978-155860594-7/50014-X

    2003 doi

  49. [57]

    Martin Schrepp, Andreas Hinderks, et al. 2017. Design and evaluation of a short version of the user experience questionnaire (UEQ-S). (2017)

    2017

  50. [58]

    Bradford L Schroeder, Shannon KT Bailey, Cheryl I Johnson, and Emily Gonzalez-Holland. 2017. Presence and usability do not directly predict procedural recall in virtual reality training. In HCI International 2017–Posters’ Extended Abstracts: 19th International Conference, HCI ...

    2017

  51. [59]

    Alon Shoa, Ramon Oliva, Mel Slater, and Doron Friedman. 2023. Sushi with Einstein: Enhancing Hybrid Live Events with LLM-Based Virtual Humans. In Proceedings of the 23rd ACM International Conference on Intelligent Virtual Agents . 1–6. Manuscript submitted to ACM 30 Xiangzhi E...

    2023

  52. [60]

    Yasuko Sugase, Shigeru Yamane, Shoogo Ueno, and Kenji Kawano. 1999. Global and fine information coded by single neurons in the temporal visual cortex. Nature 400, 6747 (1999), 869–873

    1999

  53. [61]

    Mirko Thalmann, Alessandra S Souza, and Klaus Oberauer. 2019. How does chunking help working memory? Journal of Experimental Psychology: Learning, Memory, and Cognition 45, 1 (2019), 37

    2019

  54. [62]

    Leslie G Ungerleider, Susan M Courtney, and James V Haxby. 1998. A neural system for human visual working memory. Proceedings of the National Academy of Sciences 95, 3 (1998), 883–890

    1998

  55. [63]

    Hongyu Wan, Jinda Zhang, Abdulaziz Arif Suria, Bingsheng Yao, Dakuo Wang, Yvonne Coady, and Mirjana Prpa. 2024. Building LLM-based AI Agents in Social Virtual Reality. In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems . 1–7

    2024

  56. [64]

    Can Wang, Hongliang Zhong, Menglei Chai, Mingming He, Dongdong Chen, and Jing Liao. 2024. Chat2Layout: Interactive 3D Furniture Layout with a Multimodal LLM. arXiv preprint arXiv:2407.21333 (2024)

    2024 arXiv

  57. [65]

    Zhan Wang, Lin-Ping Yuan, Liangwei Wang, Bingchuan Jiang, and Wei Zeng. 2024. Virtuwander: Enhancing multi-modal interaction for virtual tour guidance through large language models. In Proceedings of the CHI conference on human factors in computing systems . 1–20

    2024

  58. [66]

    Yuxi Wei, Zi Wang, Yifan Lu, Chenxin Xu, Changxing Liu, Hao Zhao, Siheng Chen, and Yanfeng Wang. 2024. Editable scene simulation for autonomous driving via collaborative llm-agents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 15077–15087

    2024

  59. [67]

    Zehao Wen, Zichen Liu, Srinath Sridhar, and Rao Fu. 2023. Anyhome: Open-vocabulary generation of structured and textured 3d homes. arXiv preprint arXiv:2312.06644 (2023)

    2023 arXiv

  60. [68]

    Matt Whitlock, Ethan Harnner, Jed R Brubaker, Shaun Kane, and Danielle Albers Szafir. 2018. Interacting with distant objects in augmented reality. In 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR) . IEEE, 41–48

    2018

  61. [69]

    Adam S Williams, Jason Garcia, and Francisco Ortega. 2020. Understanding multimodal user gesture and speech behavior for object manipulation in augmented reality using elicitation. IEEE Transactions on Visualization and Computer Graphics 26, 12 (2020), 3479–3489

    2020

  62. [70]

    Bob G Witmer, Christian J Jerome, and Michael J Singer. 2005. The factor structure of the presence questionnaire. Presence: Teleoperators & Virtual Environments 14, 3 (2005), 298–312

    2005

  63. [71]

    Chaoxiong Ye, Tengfei Liang, Yin Zhang, Qianru Xu, Yongjie Zhu, and Qiang Liu. 2020. The two-stage process in visual working memory consolidation. Scientific reports 10, 1 (2020), 13564

    2020

  64. [72]

    Difeng Yu, Xueshi Lu, Rongkai Shi, Hai-Ning Liang, Tilman Dingler, Eduardo Velloso, and Jorge Goncalves. 2021. Gaze-supported 3d object manipulation in virtual reality. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems . 1–13

    2021

  65. [73]

    Put 4 carpet touching each other under the table and chairs

    Xiaoyan Zhou, Adam Sinclair Williams, and Francisco Raul Ortega. 2022. Eliciting multimodal gesture+ speech interactions in a multi-object augmented reality environment. In Proceedings of the 28th ACM Symposium on Virtual Reality Software and Technology . 1–10. Manuscript subm...

    2022

  66. [75]

    Interpret user 's requirements to manipulate objects in the room

  67. [76]

    Help with move/rotate/scale the object(s) to meet user 's request

  68. [80]

    Send a debug text message: EXPLAIN(string reason); # Important Factors When placing objects, you need to consider the following factors:

  69. [81]

    For example, a bookshelf will not overlap with a table or wall

    **Physical constraints:** The objects have physical properties that must be considered. For example, a bookshelf will not overlap with a table or wall

  70. [82]

    a chair on the left

    **User centric:** The objects should be placed according to the user 's perspective. e.g. When the user says "a chair on the left". The chair should be placed left in his view *But this shouldn 't break factor 1 # User Prompt Formats The user prompt are JSONs containing follow...

  71. [84]

    A list of objects in the scene for you to manipulate, please note: Manuscript submitted to ACM 36 Xiangzhi Eric Wang, Zackary P

    Objects. A list of objects in the scene for you to manipulate, please note: Manuscript submitted to ACM 36 Xiangzhi Eric Wang, Zackary P. T. Sin, Ye Jia, Daniel Archer, Wynonna H. Y. Fong, Qing Li, and Chen Li - position is showing the bottom center anchor of the object, consi...

  72. [87]

    Drawing Lines. A line starting from an environment element or object indicating a direction or magnitude, please note: - A drawing line indicates two possible lines, one is from Start to End, another is from Start to Project. End infers to the drawing end point while Project i...

  73. [92]

    Move / Rotate the object to the asked destination

  74. [93]

    - Keep two decimals; Manuscript submitted to ACM Can You Move These Over There? An LLM-based VR Mover for Supporting Object Manipulation 37 - No math expressions in your responses

    Repeat 2, 3, and 4, until request met; **Notes:** - Call SCALE only when explicitly required; - Call MESSAGE only when the player asks questions, or you cannot perform the required task; - Each line should end with ";". - Keep two decimals; Manuscript submitted to ACM Can You ...

  75. [94]

    Interpret user 's requirements to manipulating objects in the room

  76. [95]

    Help with design of the room with your creativity and insights

  77. [96]

    Fulfill the user 's needs and provide suggestions (if asked)

  78. [97]

    ONLY REPLY formatted API calls for the system to parse # Available APIs

  79. [98]

    Create an instance based on the prefab ID: CREATE(string prefab_id)

  80. [99]

    Set the position/rotation/scale of an object: MOVE(string object_id, float? x = null, float? y = null, float? z = null); // Set the forward direction of the object in Unity style FORWARD(string object_id, float x = 0, float y = 0, float z = 0); // Set a position for the object...

  81. [100]

    Delete an object by its ID: DELETE(string object_id)

  82. [101]

    Send a text message: MESSAGE(string content)

  83. [102]

    Send a debug text message: EXPLAIN(string reason); * object ID can be set as \"crt\" to refer to the object that called in CREATE / MOVE / ROTATE / LOOKAT / SCALE just before # Important Factors When placing objects, you need to consider the following factors:

  84. [103]

    For example, a bookshelf will not overlap with a table or wall; Or a picture will not fly midair

    **Physical constraints:** The objects have physical properties that must be considered. For example, a bookshelf will not overlap with a table or wall; Or a picture will not fly midair

  85. [104]

    a chair on the left

    **User centric:** The objects should be placed according to the user 's perspective. e.g. When the user says "a chair on the left". The chair should be placed left in his view *But this shouldn 't break factor 1

  86. [105]

    **Spatial relation:** You should consider the relations between objects. e.g. when placing chairs and a table. The chairs should be reasonably close to the table and face the table; Manuscript submitted to ACM Can You Move These Over There? An LLM-based VR Mover for Supporting...

  87. [106]

    The transform info about the player 's head

    Player. The transform info about the player 's head

  88. [107]

    Objects. A list of objects in the scene for you to manipulate, please note: - position is showing the bottom center anchor of the object except the picture, consider collision when placing objects - rotation of the object is represented by forward, right, and up directions - b...

  89. [108]

    2. Objects

    Head Stay Frames. The frame that recording where the player 's looking for a while, please note: - The object and environment objects are sorted, the higher weight it is, the more important it maybe - Infer the objects/places the player mentioned based on the frames, instead o...

  90. [109]

    Hit Points. The recorded click points by the player for indicating a place to perform tasks, please note: - The point is either at the environment surface or on the objects, take the normal into consideration when placing objects - The position IS inaccurate, do not directly u...

  91. [110]

    Drawing Lines. A line starting from an environment element or object indicating a direction or magnitude, please note: - A drawing line indicates two possible lines, one is from Start to End, another is from Start to Project. End infers to the drawing end point while Project i...

  92. [111]

    Direct request of the player recognized from voice, might be inaccurate on words

    User Request. Direct request of the player recognized from voice, might be inaccurate on words

  93. [112]

    here" /

    User Request with Actions Inserted. The user request inserted with the IDs of the hit points or drawing lines, indicating what is the player talking about when doing the actions: - The user may say "here" / "there" / "this place" / etc. to indicate the actions

  94. [113]

    Telling you want API is allowed to call or not by that request # Replay Principle ## Format Available API calls with correct syntax and no comments ## Process Steps:

    Enable Actions. Telling you want API is allowed to call or not by that request # Replay Principle ## Format Available API calls with correct syntax and no comments ## Process Steps:

  95. [114]

    Infer the objectiveness of the player

  96. [115]

    Create a new object (if requested)

  97. [116]

    Move the object to the reasonable/asked destination; Manuscript submitted to ACM 40 Xiangzhi Eric Wang, Zackary P. T. Sin, Ye Jia, Daniel Archer, Wynonna H. Y. Fong, Qing Li, and Chen Li

  98. [117]

    Rotate or make the object look at in the reasonable/asked direction

  99. [118]

    Repeat 2, 3, and 4, until request met; ## Principles:

  100. [119]

    Objects cannot in the air, placed at the same position, or overlapped with each other - Simply move objects to the same or close position will cause SERIOUS overlapping problem - Use the bounding box 's orientation and dimension to check the collision is a must

  101. [120]

    Objects completely stay inside the room - There are environment objects describing the room, walls, floor, and ceiling is the boundary of the room - To ensure staying in the room entirely, collision check via the bounding box is a must

  102. [121]

    prefabs": [ {

    Orientation of the objects must be proper - Objects align with the wall has the same forward direction with the wall - Object face to another object when there is a relationship (e.g., table vs. chair, TV vs. couch/sofa) **Notes:** - Call SCALE only when explicitly required; -...

  103. [2017]

    In Virtual, Augmented and Mixed Reality: 9th International Conference, V AMR 2017, Held as Part of HCI International 2017, Vancouver, BC, Canada, July 9-14, 2017, Proceedings 9

    Dispelling the gorilla arm syndrome: the viability of prolonged gesture interactions. In Virtual, Augmented and Mixed Reality: 9th International Conference, V AMR 2017, Held as Part of HCI International 2017, Vancouver, BC, Canada, July 9-14, 2017, Proceedings 9 . Springer, 50...

    2017

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.