pith. sign in

arxiv: 2606.18511 · v1 · pith:WMHROAB3new · submitted 2026-06-16 · 💻 cs.HC

Stitching the Divide: Investigating Mixed Reality as a Bridge Between Paper-Based and Digital Artifacts in UI/UX Design

Abidullah Khan , Jinghui Cheng This is my paper

Pith reviewed 2026-06-26 22:23 UTC · model grok-4.3

classification 💻 cs.HC
keywords mixed realityUI/UX designhybrid workflowspaper artifactsdigital artifactsdesign dimensionsconceptual probesuser study
0
0 comments X

The pith

Mixed reality can integrate paper sketches and digital prototypes for UI/UX designers by supporting hybrid workflows and spatial collaboration.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

UI/UX designers currently switch between paper and digital tools without seamless connections, forcing repeated manual work and separate processes. The paper examines whether mixed reality can close this gap through interviews with 19 professionals and nine conceptual-probe sessions using an MR system that merges the two mediums. Participants reported gains in continuous workflows, less reconstruction, anchored workspaces, and real-time cross-medium teamwork, plus ideas for AI support and unified artifact handling. From these results the authors extract four design dimensions to shape future MR tools. A sympathetic reader cares because the findings outline concrete steps toward more fluid design practice that keeps the tactile advantages of paper while gaining digital flexibility.

Core claim

Through interviews and conceptual-probe sessions, the study shows that mixed reality supports continuous hybrid design workflows, reduces manual reconstruction, provides spatially anchored workspaces, and enables real-time cross-medium collaboration; designers also envision AI assistance, richer interactive content, and unified management of diverse artifacts, from which four design dimensions for future MR systems are derived to enable more fluid, creative, and collaborative practices.

What carries the argument

The MR-probe that merges paper and digital prototyping processes, used in conceptual-probe sessions to surface valued features and derive four design dimensions.

If this is right

  • Hybrid workflows become continuous without repeated manual copying between paper and digital.
  • Spatially anchored workspaces organize artifacts in physical space around the designer.
  • Real-time collaboration occurs across paper sketches and digital screens simultaneously.
  • Future systems add AI assistance and dynamic content while keeping all artifacts in one environment.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The four dimensions could guide MR applications in architecture or product design that also mix physical sketches with digital models.
  • Education programs might introduce MR early so students learn hybrid methods before professional habits form.
  • Over time, long-term use could surface new issues around version control when physical marks and digital edits coexist.

Load-bearing premise

The conceptual-probe sessions with the MR-probe accurately capture how designers would value and use fully realized mixed-reality tools in sustained professional practice.

What would settle it

A multi-week deployment study measuring reconstruction time, workflow continuity, and collaboration quality with a production MR tool versus current paper-digital methods would falsify the benefits if no measurable gains appear.

Figures

Figures reproduced from arXiv: 2606.18511 by Abidullah Khan, Jinghui Cheng.

Figure 1
Figure 1. Figure 1: Illustration of tasks supported by the MR conceptual probe. The images were taken through HoloLens 2. (a) The user [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
read the original abstract

UI/UX designers work with both paper-based and digital artifacts but lack tools that seamlessly integrate the two. Mixed Reality (MR) offers under-explored opportunities to combine the strengths of both design environments. To examine these opportunities, we first conducted interviews with 19 professional UI/UX designers to understand their current experiences using paper and digital artifacts. Motivated and informed by the interview insights, we organized nine conceptual-probe user study sessions in which designers engaged with a MR-probe that combined paper and digital prototyping processes and brainstormed MR's potential in UI/UX design. We found that participants valued MR for enabling continuous hybrid design workflows, reducing manual reconstruction, supporting spatially anchored workspaces, and facilitating real-time cross-medium collaboration. They also envisioned future MR tools with AI assistance, richer interactive and dynamic content, and the ability to manage diverse design artifacts within a unified environment. From these findings, we derive four design dimensions for future MR systems that could enable more fluid, creative, and collaborative design practices.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The manuscript reports results from interviews with 19 professional UI/UX designers on their current use of paper-based and digital artifacts, followed by nine conceptual-probe sessions in which designers interacted with an MR-probe that combined paper and digital prototyping processes. Participants expressed value in MR for enabling continuous hybrid workflows, reducing manual reconstruction, supporting spatially anchored workspaces, and facilitating real-time cross-medium collaboration. They also described visions for future MR tools incorporating AI assistance, richer interactive content, and unified management of diverse artifacts. From these data the authors derive four design dimensions intended to guide development of MR systems for more fluid, creative, and collaborative design practices.

Significance. If the empirical findings are robust, the work contributes exploratory HCI insights into an under-explored application area—using mixed reality to bridge longstanding paper-digital divides in UI/UX practice. The conceptual-probe method is well-suited to early-stage ideation, and the derivation of four design dimensions supplies concrete, falsifiable guidance for subsequent tool-building efforts. The study is appropriately scoped to participant reactions to the probe rather than claims about production-grade systems.

major comments (2)
  1. [Methods] Methods section: the manuscript supplies no information on participant recruitment criteria, interview protocol structure, data analysis procedure (e.g., coding scheme or inter-rater reliability), or the technical and interaction details of the MR-probe itself. These omissions are load-bearing because the central claims consist of participant valuations and derived design dimensions that cannot be evaluated for rigor, bias, or transferability without them.
  2. [Results/Discussion] Results and Discussion: the four design dimensions are presented as direct outputs of the probe sessions, yet the manuscript does not report how many participants contributed to each dimension or whether any dimensions emerged from only a minority of sessions. This weakens the claim that the dimensions represent broadly shared participant visions.
minor comments (1)
  1. [Abstract] Abstract: the phrase 'conceptual-probe user study sessions' is used without a brief parenthetical gloss on what a conceptual probe entails, which may confuse readers unfamiliar with the method.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive comments, which identify key areas where additional detail will strengthen the manuscript's transparency and evaluability. We respond to each major comment below.

read point-by-point responses

  1. Referee: [Methods] Methods section: the manuscript supplies no information on participant recruitment criteria, interview protocol structure, data analysis procedure (e.g., coding scheme or inter-rater reliability), or the technical and interaction details of the MR-probe itself. These omissions are load-bearing because the central claims consist of participant valuations and derived design dimensions that cannot be evaluated for rigor, bias, or transferability without them.

    Authors: We agree that the current Methods section is insufficiently detailed for assessing rigor and transferability. In the revised manuscript we will add: (1) recruitment criteria and channels for the 19 interviewees and nine probe-session participants; (2) the full structure of the semi-structured interview protocol; (3) the thematic-analysis procedure, including the coding scheme, number of coders, and steps taken to establish consistency; and (4) the technical implementation, hardware, software, and interaction mechanics of the MR-probe. These additions will directly address the load-bearing concerns. revision: yes

  2. Referee: [Results/Discussion] Results and Discussion: the four design dimensions are presented as direct outputs of the probe sessions, yet the manuscript does not report how many participants contributed to each dimension or whether any dimensions emerged from only a minority of sessions. This weakens the claim that the dimensions represent broadly shared participant visions.

    Authors: The dimensions were synthesized from recurring themes across the nine probe sessions. We acknowledge that the original manuscript does not quantify participant or session contributions per dimension. In revision we will add a table or textual breakdown indicating the number of sessions and participants that informed each dimension, while clarifying that the dimensions are exploratory, generative insights rather than claims of universal consensus. This will allow readers to judge the breadth of support for each dimension. revision: yes

Circularity Check

0 steps flagged

No significant circularity: empirical qualitative study grounded in participant data

full rationale

The paper reports an interview study (N=19) followed by nine conceptual-probe sessions whose outputs are summarized and used to derive four design dimensions. All load-bearing claims are explicitly tied to participant statements and probe interactions rather than any equation, fitted parameter, or self-citation chain. No mathematical derivation, uniqueness theorem, or ansatz is invoked. The central result is therefore a direct reporting and thematic synthesis of external data, not a reduction to the authors' own prior inputs.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

No mathematical models, fitted parameters, axioms, or invented entities are present; the work is a qualitative empirical study in human-computer interaction.

pith-pipeline@v0.9.1-grok · 5710 in / 1120 out tokens · 49191 ms · 2026-06-26T22:23:21.619278+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

36 extracted references · 32 canonical work pages

  1. [1]

    Judith Amores, Xavier Benavides, and Pattie Maes. 2015. ShowMe: A Remote Collaboration System that Supports Immersive Gestural Communication. In Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems(Seoul Republic of Korea). ACM, New York, NY, USA, 1343–1348. doi:10.1145/2702613.2732927

    work page doi:10.1145/2702613.2732927 2015

  2. [2]

    Jodi Aronson. 1993. A Pragmatic View of Thematic Analysis.The Qualitative Report2, 1 (1993), 1–3. doi:10.46743/2160-3715/1995.2069

    work page doi:10.46743/2160-3715/1995.2069 1993

  3. [3]

    Narges Ashtari, Andrea Bunt, Joanna McGrenere, Michael Nebeling, and Parmit K. Chilana. 2020. Creating Augmented and Virtual Reality Applications: Current Practices, Challenges, and Opportunities. InProceedings of the 2020 CHI Conference on Human Factors in Computing Systems(2020-04-21). ACM, Honolulu HI USA, 1–13. doi:10.1145/3313831.3376722

    work page doi:10.1145/3313831.3376722 2020

  4. [4]

    Jonas Auda, Roman Heger, Uwe Gruenefeld, and Stefan Schneegass. 2021. VRS- ketch: Investigating 2D Sketching in Virtual Reality with Different Levels of Hand and Pen Transparency. InHuman-Computer Interaction – INTERACT 2021, Carmelo Ardito, Rosa Lanzilotti, Alessio Malizia, Helen Petrie, Antonio Piccinno, Giuseppe Desolda, and Kori Inkpen (Eds.). Vol. 12...

    work page doi:10.1007/978-3-030-85607-6_14 2021

  5. [5]

    Julian Baader, Mats Ole Ellenberg, and Marc Satkowski. 2025. The Invisible Hand of the Context: Authoring of Context-Aware Mixed Reality Labels. InProceedings of the Mensch Und Computer 2025 (MuC ’25). ACM, New York, NY, USA, 738–743. doi:10.1145/3743049.3748552

    work page doi:10.1145/3743049.3748552 2025

  6. [6]

    Radka Bauerová, Michal Halaška, and Veronika Kopřivová. 2025. User Experience with Virtual Reality in Team-Based Prototyping and Brainstorming.International Journal of Human–Computer Interaction0, 0 (2025), 1–19. doi:10.1080/10447318. 2025.2474462

    work page doi:10.1080/10447318 2025

  7. [7]

    Christensen, Kim Halskov, and Clemens N

    Bo T. Christensen, Kim Halskov, and Clemens N. Klokmose. 2020. Chapter 1 - The properties of sticky notes for collaborative creativity: An introduction. In Sticky Creativity, Bo T. Christensen, Kim Halskov, and Clemens N. Klokmose (Eds.). Academic Press, 1–16. doi:10.1016/B978-0-12-816566-9.00001-X

    work page doi:10.1016/b978-0-12-816566-9.00001-x 2020

  8. [8]

    Tim Coughlan and Peter Johnson. 2008. Idea management in creative lives. InCHI ’08 Extended Abstracts on Human Factors in Computing Systems(Florence, Italy) (CHI EA ’08). ACM, New York, NY, USA, 3081–3086. doi:10.1145/1358628.1358811

    work page doi:10.1145/1358628.1358811 2008

  9. [9]

    Davis, T

    Richard C. Davis, T. Scott Saponas, Michael Shilman, and James A. Landay

  10. [10]

    Wobbrock, Andrew D

    SketchWizard: Wizard of Oz prototyping of pen-based user interfaces. In Proceedings of the 20th Annual ACM Symposium on User Interface Software and Technology(Newport, Rhode Island, USA)(UIST ’07). ACM, New York, NY, USA, 119–128. doi:10.1145/1294211.1294233

    work page doi:10.1145/1294211.1294233

  11. [11]

    Tobias Drey, Jan Gugenheimer, Julian Karlbauer, Maximilian Milo, and Enrico Rukzio. 2020. VRSketchIn: Exploring the Design Space of Pen and Tablet Interac- tion for 3D Sketching in Virtual Reality. InProceedings of the 2020 CHI Conference on Human Factors in Computing Systems(Honolulu HI USA, 2020-04-21). ACM, New York, NY, USA, 1–14. doi:10.1145/3313831.3376628

    work page doi:10.1145/3313831.3376628 2020

  12. [12]

    Hatana El-Jarn and Glen Southern. 2020. Can co-creation in extended reality technologies facilitate the design process?Journal of Work-Applied Management 12, 2 (2020), 191–205. doi:10.1108/JWAM-04-2020-0022

    work page doi:10.1108/jwam-04-2020-0022 2020

  13. [13]

    K. J. Kevin Feng, Tony W Li, and Amy X. Zhang. 2023. Understanding Collabo- rative Practices and Tools of Professional UX Practitioners in Software Organi- zations. InProceedings of the 2023 CHI Conference on Human Factors in Comput- ing Systems(Hamburg Germany, 2023-04-19). ACM, New York, NY, USA, 1–20. doi:10.1145/3544548.3581273

    work page doi:10.1145/3544548.3581273 2023

  14. [14]

    Mafalda Gamboa. 2022. Conversations with Myself: Sketching Workshop Ex- periences in Design Epistemology. InCreativity and Cognition(Venice Italy, 2022-06-20). ACM, New York, NY, USA, 71–82. doi:10.1145/3527927.3531450

    work page doi:10.1145/3527927.3531450 2022

  15. [15]

    Florian Geyer, Jochen Budzinski, and Harald Reiterer. 2012. IdeaVis: a hybrid workspace and interactive visualization for paper-based collaborative sketching sessions. InProceedings of the 7th Nordic Conference on Human-Computer Inter- action: Making Sense Through Design(Copenhagen Denmark). ACM, New York, NY, USA, 331–340. doi:10.1145/2399016.2399069

    work page doi:10.1145/2399016.2399069 2012

  16. [16]

    Jens Emil Sloth Grønbæk, Juan Sánchez Esquivel, Germán Leiva, Eduardo Velloso, Hans Gellersen, and Ken Pfeuffer. 2024. Blended Whiteboard: Physicality and Reconfigurability in Remote Mixed Reality Collaboration. InProceedings of the 2024 CHI Conference on Human Factors in Computing Systems(Honolulu)(CHI ’24). ACM, New York, USA, Article 798, 16 pages. doi...

    work page doi:10.1145/3613904.3642293 2024

  17. [17]

    Klemmer, Michael Bernstein, Leith Abdulla, Bran- don Burr, Avi Robinson-Mosher, and Jennifer Gee

    Björn Hartmann, Scott R. Klemmer, Michael Bernstein, Leith Abdulla, Bran- don Burr, Avi Robinson-Mosher, and Jennifer Gee. 2006. Reflective physical prototyping through integrated design, test, and analysis. InProceedings of the 19th annual ACM symposium on User interface software and technology. ACM, Montreux Switzerland, 299–308. doi:10.1145/1166253.1166300

    work page doi:10.1145/1166253.1166300 2006

  18. [18]

    Mads Møller Jensen, Sarah-Kristin Thiel, Eve Hoggan, and Susanne Bødker. 2018. Physical Versus Digital Sticky Notes in Collaborative Ideation.Comput. Supported Coop. Work27, 3 (2018), 609–645. doi:10.1007/s10606-018-9325-1

    work page doi:10.1007/s10606-018-9325-1 2018

  19. [19]

    Jervis and Masood Masoodian

    Matthew G. Jervis and Masood Masoodian. 2010. SOPHYA: a system for digital management of ordered physical document collections. InProceedings of the Fourth International Conference on Tangible, Embedded, and Embodied Interaction (Cambridge, Massachusetts, USA)(TEI ’10). ACM, New York, NY, USA, 33–40. doi:10.1145/1709886.1709894

    work page doi:10.1145/1709886.1709894 2010

  20. [20]

    Ben Jonson. 2005. Design ideation: the conceptual sketch in the digital age. Design Studies26, 6 (2005), 613–624. doi:10.1016/j.destud.2005.03.001

    work page doi:10.1016/j.destud.2005.03.001 2005

  21. [21]

    Lee Kent, Chris Snider, James Gopsill, and Ben Hicks. 2021. Mixed reality in design prototyping: A systematic review.Design Studies77 (2021), 101046. doi:10. 1016/j.destud.2021.101046

    arXiv 2021

  22. [22]

    Klemmer, Katherine M

    Scott R. Klemmer, Katherine M. Everitt, and James A. Landay. 2008. Integrating Physical and Digital Interactions on Walls for Fluid Design Collaboration.Hu- man–Computer Interaction23, 2 (2008), 138–213. doi:10.1080/07370020802016399

    work page doi:10.1080/07370020802016399 2008

  23. [23]

    Klemmer, Mark W

    Scott R. Klemmer, Mark W. Newman, Ryan Farrell, Mark Bilezikjian, and James A. Landay. 2001. The designers’ outpost: a tangible interface for collaborative web site. InProceedings of the 14th annual ACM symposium on User interface software and technology. ACM, Orlando Florida, 1–10. doi:10.1145/502348.502350

    work page doi:10.1145/502348.502350 2001

  24. [24]

    Zhen Li, Michelle Annett, Ken Hinckley, Karan Singh, and Daniel Wigdor. 2019. HoloDoc: Enabling Mixed Reality Workspaces that Harness Physical and Digital Content. InProceedings of the 2019 CHI Conference on Human Factors in Computing Systems(Glasgow, Scotland Uk)(CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–14. doi:10.1145/3290605.3300917

    work page doi:10.1145/3290605.3300917 2019

  25. [25]

    Zhipeng Li, Christoph Gebhardt, Yves Inglin, Nicolas Steck, Paul Streli, and Christian Holz. 2024. SituationAdapt: Contextual UI Optimization in Mixed Reality with Situation Awareness via LLM Reasoning. InProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology(Pittsburgh) (UIST ’24). ACM, New York, Article 43, 13 pages. doi:...

    work page doi:10.1145/3654777.3676470 2024

  26. [26]

    Youn-Kyung Lim, Erik Stolterman, and Josh Tenenberg. 2008. The anatomy of prototypes: Prototypes as filters, prototypes as manifestations of design ideas. ACM Trans. Comput.-Hum. Interact.15, 2, Article 7 (July 2008), 27 pages. doi:10. 1145/1375761.1375762

    arXiv 2008

  27. [27]

    Newman, Jason I

    James Lin, Mark W. Newman, Jason I. Hong, and James A. Landay. 2000. DENIM: finding a tighter fit between tools and practice for Web site design. InProceedings of the SIGCHI Conference on Human Factors in Computing Systems(The Hague, The Netherlands)(CHI ’00). ACM, New York, NY, USA, 510–517. doi:10.1145/ 332040.332486

    arXiv 2000

  28. [28]

    Michael McCurdy, Christopher Connors, Guy Pyrzak, Bob Kanefsky, and Alonso Vera. 2006. Breaking the fidelity barrier: an examination of our current charac- terization of prototypes and an example of a mixed-fidelity success. InPro- ceedings of the SIGCHI Conference on Human Factors in Computing Systems (Montréal, Québec, Canada)(CHI ’06). ACM, New York, N...

    work page doi:10.1145/1124772.1124959 2006

  29. [29]

    Inoussa Ouedraogo, Huyen Nguyen, and Patrick Bourdot. 2024. Where to Draw the Line: Physical Space Partitioning and View Privacy in AR-based Co-located Collaboration for Immersive Analytics. InProceedings of the 2024 ACM Symposium on Spatial User Interaction(Trier, Germany)(SUI ’24). ACM, New York, NY, USA, Article 20, 12 pages. doi:10.1145/3677386.3682085

    work page doi:10.1145/3677386.3682085 2024

  30. [30]

    Shwetha Rajaram and Michael Nebeling. 2022. Paper Trail: An Immersive Author- ing System for Augmented Reality Instructional Experiences. InCHI Conference on Human Factors in Computing Systems(2022-04-29). ACM, New Orleans LA USA, 1–16. doi:10.1145/3491102.3517486

    work page doi:10.1145/3491102.3517486 2022

  31. [31]

    Reinhard Sefelin, Manfred Tscheligi, and Verena Giller. 2003. Paper prototyping - what is it good for? a comparison of paper- and computer-based low-fidelity prototyping. InCHI ’03 Extended Abstracts on Human Factors in Computing Systems(Ft. Lauderdale, Florida, USA)(CHI EA ’03). ACM, New York, NY, USA, 778–779. doi:10.1145/765891.765986

    work page doi:10.1145/765891.765986 2003

  32. [32]

    Hall, and Michael Nebeling

    Maximilian Speicher, Brian D. Hall, and Michael Nebeling. 2019. What is Mixed Reality?. InProceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, New York, NY, USA, 1–15. doi:10.1145/3290605.3300767

    work page doi:10.1145/3290605.3300767 2019

  33. [33]

    Mojtaba Vaismoradi, Hannele Turunen, and Terese Bondas. 2013. Content analy- sis and thematic analysis: Implications for conducting a qualitative descriptive study.Nursing & Health Sciences15, 3 (2013), 398–405. doi:10.1111/nhs.12048 Investigating Mixed Reality as a Bridge Between Paper-Based and Digital Artifacts in UI/UX Design GI ’26, ,

    work page doi:10.1111/nhs.12048 2013

  34. [34]

    Remko van der Lugt. 2002. Functions of sketching in design idea generation meet- ings. InProceedings of the 4th Conference on Creativity & Cognition(Loughborough, UK)(C&C ’02). ACM, New York, NY, USA, 72–79. doi:10.1145/581710.581723

    work page doi:10.1145/581710.581723 2002

  35. [35]

    Hongbo Zhang, Pei Chen, Xuelong Xie, Zhaoqu Jiang, Zihong Zhou, and Lingyun Sun. 2024. A Hybrid Prototype Method Combining Physical Models and Genera- tive Artificial Intelligence to Support Creativity in Conceptual Design.ACM Trans. Comput.-Hum. Interact.31, 5, Article 65 (Nov. 2024), 34 pages. doi:10.1145/3689433

    work page doi:10.1145/3689433 2024

  36. [36]

    Shih, Ellen Yi-Luen Do, Malte F Jung, Wendy Ju, and Daniel Leithinger

    Yuzhen Zhang, Ruixiang Han, Ran Zhou, Peter Gyory, Clement Zheng, Patrick C. Shih, Ellen Yi-Luen Do, Malte F Jung, Wendy Ju, and Daniel Leithinger. 2024. Wizard of Props: Mixed Reality Prototyping with Physical Props to Design Re- sponsive Environments. InProceedings of the Eighteenth International Conference on Tangible, Embedded, and Embodied Interactio...

    work page doi:10.1145/3623509.3633395 2024