Pith. sign in

REVIEW 3 major objections 5 minor 91 references

Do It For Me vs. Do It With Me: Investigating User Perceptions of Different Paradigms of Automation in Copilots for Feature-Rich Software

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Semi-automated copilots beat full automation in user control and learning, study finds

desk verdict A solid HCI study of a real system whose paradigm-level claim is undercut by an engineered-failure baseline and a confound between automation level and guidance; the within-study result is believable, the generalization is not. read the letter →

arxiv 2504.15549 v1 pith:PSCMT3G2 submitted 2025-04-22 cs.HC cs.AIcs.LG

Anjali Khurana , Xiaotian Su , April Yi Wang , Parmit K Chilana This is my paper · ORCID
classification cs.HCcs.AIcs.LG
keywords softwarecopilotssemi-automationfullautomationusercontrollearnabilityfeature-richLLMassistantsvisualguidance
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that for feature-rich software like spreadsheets and design tools, a copilot that does routine steps automatically while showing visual step-by-step instructions beats a copilot that tries to do the whole task by itself. The authors build two working copilots, embed them in Google Sheets and Figma, and run a controlled study with 20 users. The semi-automated one led to much higher task completion and accuracy, stronger feelings of control and utility, and better software learnability, while the fully automated one only won praise from a few technical users for saving time on simple tasks. A follow-up design study with 10 users shows that adding task- and state-aware preview clips and adaptive instructions makes the semi-automated approach even more useful. If the result holds, the default for copilot design should shift from full automation toward user-controlled, guided semi-automation.

What carries the argument

The central object is GuidedCopilot, a semi-automated copilot that automates only trivial or repetitive steps (e.g., creating a frame in Figma, sorting in Sheets) while giving step-by-step guidance with in-context visual references. Users can initiate automation themselves, edit the entities the LLM extracts from their query before automation runs, and receive follow-up answers that mix web data with software documentation. The architecture uses GraphRAG over software documentation and Q&A forums to fetch text, images, and scripted automation functions, and an LLM agent that assembles these into a mixed-medium response. AutoCopilot is the contrasting fully automated system, built to mirror real-world copilot behavior, including occasional incorrect automation when it fails to map multiple intents or entities to a single automation function.

What would settle it

Run the same tasks with a fully automated copilot whose failure rate is measured from real deployments of tools like Microsoft 365 Copilot or Figma AI rather than designed into the prototype, and check whether the completion and accuracy gaps between the two paradigms shrink to insignificance.

Watch

Extended reading notes

Core claim

The core claim is that the 'Do It With Me' paradigm, embodied in GuidedCopilot, outperforms the 'Do It For Me' paradigm, embodied in AutoCopilot, along user control, software utility, and learnability, with significantly higher task completion and accuracy in both Google Sheets and Figma. In Sheets, participants completed on average 88.5% of the task with GuidedCopilot versus 35.0% with AutoCopilot; in Figma the numbers were 55.0% versus 20.0%. Task accuracy similarly favored GuidedCopilot (82.0% vs 12.0% in Sheets; 40.0% vs 5.0% in Figma). The paper also reports that users engaged in roughly twice as many trial-and-error attempts with AutoCopilot, and that several users abandoned it entirely. A minority, mostly male computer-science professionals, preferred full automation for its time savings on simpler visual tasks.

Load-bearing premise

The comparison assumes that AutoCopilot's full-automation behavior, including its deliberately included incorrect executions, fairly represents how real fully automated copilots perform.

Editorial extensions

If this is right

  • Copilot designers should treat full automation as a feature for simple, repetitive tasks rather than as the default interaction paradigm for complex software.
  • Semi-automated copilots that show steps and let users trigger automation can reduce debugging effort, as indicated by the trial-and-error counts (192 attempts with AutoCopilot vs 89 with GuidedCopilot).
  • Learnability is a measurable benefit of the guided approach: users reported transferring skills to new tasks, and the paper documents a concrete example of a participant who could only copy-paste prompts with AutoCopilot but succeeded with GuidedCopilot.
  • Task- and state-aware features, such as preview clips that track user progress and adaptive instructions that skip completed steps, can address the main complaints users had about the basic GuidedCopilot (overly detailed instructions and difficulty mapping chat visuals to the interface).
  • The three-dimensional framework proposed in the paper, with axes for automation level, adaptive guidance, and factors of familiarity, task type, and user intent, gives designers a checklist for choosing how much automation to offer.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A fair test of the paradigm claim requires an AutoCopilot whose failure rate matches real production copilots; the paper's baseline was deliberately weakened by including incorrect automation, so the size of the completion and accuracy gaps is likely inflated even if the direction of the effect is real.
  • The preference of expert users for full automation suggests that copilots should adapt not just to task state but to user identity over longer time scales, something the follow-up study only begins to explore.
  • The same semi-automation-plus-visual-guidance structure could be tested in adjacent domains such as code editors, CAD tools, or data science notebooks, where LLM copilots already exist but rarely offer step-by-step in-context visual walkthroughs.
  • Measuring whether GuidedCopilot's learnability gains persist after a delay (e.g., one week later) would separate true skill acquisition from in-the-moment guidance, a distinction the current studies do not address.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper compares two copilot paradigms for feature-rich software: AutoCopilot, which fully automates a user's task from a natural-language prompt, and GuidedCopilot, which automates only trivial steps and provides step-by-step visual guidance. The authors implemented both as in-application assistants for Google Sheets and Figma, ran a within-subject controlled experiment with 20 participants, and report higher task completion and accuracy plus more positive perceptions of control, utility, and learnability for GuidedCopilot. A follow-up Wizard-of-Oz design exploration with 10 participants evaluates two adaptive extensions in Photoshop. The paper concludes that semi-automated copilots with visual step-by-step guidance are a better default than full automation for complex tasks, and proposes a three-dimensional framework for choosing automation levels.

Significance. If the comparison were fair, the result would be a valuable empirical challenge to the current push toward fully automated in-application copilots. The paper contributes two working prototype copilots, rich think-aloud and interview data, a concrete framework for automation-level decisions, and a useful follow-up design exploration. These contributions are real and the qualitative findings about user control and learnability are plausible on their own. However, the paper's central paradigm-level claim—that semi-automation outperforms full automation—is not yet established because the full-automation baseline appears to be constructed so that it fails systematically on the chosen experimental tasks, and because the reported statistics pool observations in a way that may inflate significance. The significance is therefore conditional on a fair baseline and appropriate statistical analysis.

major comments (3)
  1. [Section 3.1] The AutoCopilot baseline is deliberately engineered to fail on the very tasks used in the experiment. The paper states that AutoCopilot 'occasionally encounters incorrect full automation when it fails to identify a single automation function from the multiple intents in the user's query for performing complex tasks outlined in the study, or when it cannot accurately map multiple entities.' The experimental tasks are explicitly multi-stage and multi-entity (Section 5.3), so the observed completion gaps (35.0% vs 88.5% in Sheets; 20.0% vs 55.0% in Figma) and accuracy gaps (12.0% vs 82.0%; 5.0% vs 40.0%) are substantially determined by the researchers' implementation decisions rather than by the paradigm itself. No calibration data are given to show that AutoCopilot's failure rate is representative of production full-automation copilots, and the code and prompts are not publicly released. This undermines the abstract's and Section 6.1.1's claim that semi-automation outperforms full automation as a paradigm. The authors should either compare against an existing production copilot, calibrate their baseline's failure rate to a published/measured rate, or reframe the conclusion to be about this specific AutoCopilot implementation rather than the paradigm.
  2. [Section 6.1.1] The statistical results are reported in a way that is inconsistent with the stated analysis and appears to treat repeated measures as independent observations. The text says a 'paired-sample t-test' was used, yet the reported degrees of freedom (t(37.6), t(37.9), t(30.7)) are consistent with Welch's independent-samples t-test on roughly 40 observations per group, not a paired test on N=20 participants (which would give df=19). In addition, each participant contributed multiple measurements (two tasks per application across two applications), so task-level observations are nested within participants and are not independent. The Pearson chi-square tests in Sections 6.1.2, 6.2.1, and 6.3 pool Likert responses across both applications and both tasks for the same participants, also violating independence. This inflates the apparent strength of evidence. The authors should analyze the data with per-participant averages or mixed-effects models, and should report effect sizes and confidence intervals. The significant p-values may still hold, but they must be recomputed.
  3. [Section 9 and Sections 3.1–4.1] The comparison confounds the automation level with several other interaction-design features: GuidedCopilot provides step-by-step visuals, user-initiated automation, editable LLM-extracted entities, and mixed-medium responses, whereas AutoCopilot provides none of these. The limitations section explicitly concedes that 'the independent contributions of specific features (e.g., automation, step-by-step guidance)' were not assessed. Since the paper's central claim is about automation paradigms, this confound is not a peripheral limitation: the reported differences could be due to the presence of guidance and user control affordances rather than to semi-automation per se. The authors should either run a factorial design that varies automation level and guidance independently, or substantially narrow the claim to 'a semi-automated copilot with visual guidance is preferred over this particular fully automated implementation.'
minor comments (5)
  1. [Section 5.4] The trial-and-error analysis is described as manually annotated, but no inter-rater reliability or coding scheme details are reported; adding these would strengthen the reproducibility of the 192 vs. 89 trial-and-error counts in Section 6.2.2.
  2. [Abstract and Section 6.1.2] The abstract states that AutoCopilot 'saved time for simpler visual tasks,' but the paper does not report any direct time measurements; the support for this statement is participants' perceived time savings (6/20 users), which is a weaker claim and should be phrased accordingly.
  3. [Section 4.1.3] The code and prompts are described as 'available upon request'; for a CHI paper presenting a novel system comparison, a public repository link would substantially aid reproduction and allow reviewers to scrutinize the AutoCopilot failure-injection logic.
  4. [Section 6.1.1] When reporting the paired t-tests, the paper does not state whether the data were checked for normality or whether the tests were two-tailed in each case; the 'two-tailed' designation appears only once and should be applied consistently.
  5. [Figure 5] The trial-and-error figure is informative but the percentages in Section 6.2.2 (e.g., 'about 75% of AutoCopilot attempts involved undoing') are presented without the denominator for each attempt category; clarifying the denominators would avoid ambiguity.

Circularity Check

1 steps flagged · score 5.0 of 10

AutoCopilot's failure modes are engineered around the study's multi-intent tasks, so the headline completion/accuracy gap is partly built into the comparison.

  1. self definitional [Section 3.1, 'AutoCopilot: User Interface Design and Implementation'; see also Section 5.3, 'Choice of Tasks and Applications']
    "To mirror real-world automation performance, AutoCopilot occasionally encounters incorrect full automation when it fails to identify a single automation function from the multiple intents in the user's query for performing complex tasks outlined in the study, or when it cannot accurately map multiple entities from the user's query to the appropriate automation function."

    Section 5.3 says 'we selected tasks that would require multi-stage help and prompts' (e.g., 'analyze the top 5 products and visualize their sales across regions'; 'create a webpage that includes a login section and displays products'). Section 3.1 then defines AutoCopilot's failure mode as an inability to handle 'multiple intents in the user's query for performing complex tasks outlined in the study.' Thus the baseline is operationalized as a system that fails on exactly the multi-intent task structure used in the evaluation. The reported completion/accuracy gaps (35.0% vs 88.5% in Sheets; 20.0% vs 55.0% in Figma) are therefore partly determined by the designers' decisions about when to inject errors, not by an independent measurement of full-automation copilots.

full rationale

This is an empirical HCI study, not a formal derivation, so most circularity categories do not apply. The authors' self-citations (e.g., ChatrEx and prior IUI work) are background and not load-bearing, and the follow-up Wizard-of-Oz design exploration is explicitly a prototype usability probe rather than a predictive derivation. The one substantive circularity is the construction of the AutoCopilot baseline: its failure behavior is explicitly tied to 'complex tasks outlined in the study,' while the study tasks are deliberately chosen to be multi-stage and multi-intent. The central completion and accuracy advantages of GuidedCopilot are therefore partly built into the experimental design rather than discovered empirically. However, participants' perceptions of control, learnability, and utility are not logically entailed by the design and could have favored AutoCopilot; this keeps the overall circularity partial rather than total. Score 5 reflects one central, partially constructive circularity.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central comparison rests on three domain assumptions: small-sample generalizability, researcher-defined ground truth, and a fair baseline. The free-parameter and invented-entity lists are empty because the paper introduces prototype systems and fitting-free empirical measures rather than theoretical constructs.

assumptions (3)
  • domain assumption Likert-scale responses from 20 participants can support generalizable conclusions about copilot design paradigms.
    The study uses a small convenience sample recruited from university mailing lists and snowball sampling; some chi-square cells have N=38 with 5 response levels, so statistical power is limited.
  • domain assumption The researcher-defined ground-truth step sequences represent the correct way to complete tasks.
    Task completion and accuracy are measured against Appendix A definitions; a step-by-step guided copilot is structurally aligned with this scoring, which may not capture valid alternative approaches.
  • domain assumption The two copilot implementations are comparable apart from the automation paradigm.
    Both use the same documentation for retrieval, but AutoCopilot was deliberately designed to fail on complex queries, so the conditions differ in more than just automation level.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Do It For Me vs. Do It With Me: Investigating User Perceptions of Different Paradigms of Automation in Copilots for Feature-Rich Software." pith.science (2026). https://pith.science/paper/PSCMT3G2

@misc{pith2026250415549,
  author       = {Pith},
  title        = {Pith review of: Do It For Me vs. Do It With Me: Investigating User Perceptions of Different Paradigms of Automation in Copilots for Feature-Rich Software},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PSCMT3G2}},
  note         = {Machine review of arXiv:2504.15549}
}
read the original abstract

Large Language Model (LLM)-based in-application assistants, or copilots, can automate software tasks, but users often prefer learning by doing, raising questions about the optimal level of automation for an effective user experience. We investigated two automation paradigms by designing and implementing a fully automated copilot (AutoCopilot) and a semi-automated copilot (GuidedCopilot) that automates trivial steps while offering step-by-step visual guidance. In a user study (N=20) across data analysis and visual design tasks, GuidedCopilot outperformed AutoCopilot in user control, software utility, and learnability, especially for exploratory and creative tasks, while AutoCopilot saved time for simpler visual tasks. A follow-up design exploration (N=10) enhanced GuidedCopilot with task-and state-aware features, including in-context preview clips and adaptive instructions. Our findings highlight the critical role of user control and tailored guidance in designing the next generation of copilots that enhance productivity, support diverse skill levels, and foster deeper software engagement.

Figures

Figures reproduced from arXiv: 2504.15549 by the authors.

Figure 1
Figure 1. GuidedCopilot, a novel semi-automatic copilot: (a) Copilot assistance is structured to provide step-by-step guidance along with semi-automation for only repetitive or trivial steps in the task; (b) Visual references of the UI elements in-context to user’s tasks and application are provided within the step-by-step guidance; (c) Users have control over editing the LLM extracted entities from their query before the sem… view at source ↗
Figure 2
Figure 2. AutoCopilot: (a) Fully automates the user’s task (e.g., creating a webpage that includes a login and product page); (b) Similar to state-of-the-art copilots, demonstrates incorrect automation (such as color coding the entire sheet instead of values greater than 40 in column C); (c) Provides follow-up textual response based on context from software documentation and task-specific terminology [39, 74]. Recent literatu… view at source ↗
Figure 3
Figure 3. GuidedCopilot Architecture: The user’s query is used to initiate a conversation about automating software tasks, which is then transmitted to the query understanding and text retrieval module (Section 4.1.1). This module interprets the query and performs a contextual search across documentation and web data. The extracted intent and relevant excerpts are processed by GPT-4o to generate text-based procedural steps. T… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Overview of participants’ responses to the post-task questionnaire. The Pearson Chi-Squared test showed a significant [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Trial-and-Error Differences in AutoCopilot vs GuidedCopilot: This figure illustrates the case of Participant P14, a computer science professional. Despite P14’s technical expertise, they encountered higher trial-and-error with AutoCopilot, primarily focused on customiz…
Figure 6
Figure 6. Figure 6: Illustration of Transfer Learning in Software Usage with [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Design exploration building on GuidedCopilot to offer targeted, task- and state-aware assistance in Photoshop through two key features: (a, b) GuidedCopilotVisual integrates in-context preview clips within the software interface. When the user clicks “Show in the inter…
Figure 8
Figure 8. Figure 8: Dimensional framework describing key factors to consider when determining levels of automation and step-by-step [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

91 extracted references · 30 canonical work pages

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

    arXiv 2023

  2. [2]

    Adobe. 2023. Adobe unveils Firefly, a family of new creative generative ai. https://news.adobe.com/news/news-details/2023/Adobe-Unveils-Firefly- a-Family-of-new-Creative-Generative-AI/default.aspx

    2023

  3. [3]

    Open AI. 2022. Introducing chatgpt. https://openai.com/blog/chatgpt

    2022

  4. [4]

    Naser Al Madi. 2023. How Readable is Model-generated Code? Examining Readability and Visual Inspection of GitHub Copilot. In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering(Rochester, MI, USA) (ASE ’22). Association for Computing Machinery, New York, NY, USA, Article 205, 5 pages. https://doi.org/10.1145/3551349.3560438

    arXiv 2023

  5. [5]

    Allen, C.I

    J.E. Allen, C.I. Guinn, and E. Horvtz. 1999. Mixed-initiative interaction. IEEE Intelligent Systems and their Applications 14, 5 (1999), 14–23. https://doi.org/10. 1109/5254.796083

    arXiv 1999

  6. [6]

    Bennett, Kori Inkpen, Jaime Teevan, Ruth Kikin-Gil, and Eric Horvitz

    Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, Paul N. Bennett, Kori Inkpen, Jaime Teevan, Ruth Kikin-Gil, and Eric Horvitz. 2019. Guidelines for Human- AI Interaction. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Asso...

    arXiv 2019

  7. [7]

    2023.Personalized human- computer interaction

    Mirjam Augstein, Eelco Herder, and Wolfgang Wörndl. 2023.Personalized human- computer interaction. Walter de Gruyter GmbH & Co KG

    2023

  8. [8]

    Lisanne Bainbridge. 1983. Ironies of automation. InAnalysis, design and evaluation of man–machine systems. Elsevier, 129–135

    1983

Show all 91 references
  1. [9]

    Gagan Bansal, Jennifer Wortman Vaughan, Saleema Amershi, Eric Horvitz, Adam Fourney, Hussein Mozannar, Victor Dibia, and Daniel S. Weld. 2024. Chal- lenges in Human-Agent Communication. Technical Report MSR-TR-2024-53. Mi- crosoft. https://www.microsoft.com/en-us/research/publ...

    2024

  2. [10]

    James, and Nadia Polikarpova

    Shraddha Barke, Michael B. James, and Nadia Polikarpova. 2023. Grounded Copilot: How Programmers Interact with Code-Generating Models. Proc. ACM Program. Lang. 7, OOPSLA1, Article 78 (apr 2023), 27 pages. https://doi.org/10. 1145/3586030

    2023

  3. [12]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey W...

    2020

  4. [13]

    Donald T Campbell. 1960. Blind variation and selective retentions in creative thought as in other knowledge processes. Psychological review 67, 6 (1960), 380

    1960

  5. [14]

    John M Carroll and Caroline Carrithers. 1984. Training wheels in a user interface. Commun. ACM 27, 8 (1984), 800–806

    1984

  6. [15]

    Carroll and Mary Beth Rosson

    John M. Carroll and Mary Beth Rosson. 1987. Paradox of the Active User . MIT Press, Cambridge, MA, USA, 80–111

    1987

  7. [16]

    Chilana, Amy J

    Parmit K. Chilana, Amy J. Ko, and Jacob O. Wobbrock. 2012. LemonAid: Selection- Based Crowdsourced Contextual Help for Web Applications. InProceedings of the SIGCHI Conference on Human Factors in Computing Systems (Austin, Texas, USA) (CHI ’12). Association for Computing Machi...

    2012

  8. [17]

    Microsoft Power Platform Community. 2024. Forums | Microsoft Power Platform Community — powerusers.microsoft.com. https: //powerusers.microsoft.com/t5/General-Power-Automate/What-am-I-doing- wrong-Is-Co-Pilot-just-useless-First-time-user/td-p/2701867. [Accessed 12-09-2024]

    2024

  9. [18]

    Juliet M Corbin and Anselm Strauss. 1990. Grounded theory research: Procedures, canons, and evaluative criteria. Qualitative sociology 13, 1 (1990), 3–21

    1990

  10. [19]

    Justin Cranshaw, Emad Elwany, Todd Newman, Rafal Kocielnik, Bowen Yu, Sandeep Soni, Jaime Teevan, and Andrés Monroy-Hernández. 2017. Calen- dar.help: Designing a Workflow-Based Scheduling Agent with Humans in the Loop. ACM CHI Conference on Human Factors in Computing Systems (...

    2017

  11. [20]

    Nils Dahlbäck, Arne Jönsson, and Lars Ahrenberg. 1993. Wizard of Oz studies: why and how. In Proceedings of the 1st International Conference on Intelligent User Interfaces (Orlando, Florida, USA)(IUI ’93). Association for Computing Machinery, New York, NY, USA, 193–200. https:...

    1993

  12. [21]

    Sylvain Delisle and Bernard Moulin. 2002. User interfaces and help systems: from helplessness to intelligent assistance. Artif. Intell. Rev. 18, 2 (oct 2002), 117–157. https://doi.org/10.1023/A:1015179704819

    2002 doi

  13. [22]

    Françoise Détienne. 2001. Software design–cognitive aspect. Springer Science & Business Media

    2001

  14. [23]

    Mica R Endsley. 2017. From here to autonomy: lessons learned from human– automation research. Human factors 59, 1 (2017), 5–27

    2017

  15. [24]

    Figma. 2024. Figma AI: Your Creativity, unblocked with Figma AI — figma.com. https://www.figma.com/ai/. [Accessed 11-09-2024]

    2024

  16. [25]

    Adobe Firefly. 2024. https://firefly.adobe.com/

    2024

  17. [26]

    Microsoft Forum. 2024. Redirecting — answers.microsoft.com. https://answers.microsoft.com/en-us/msoffice/forum/all/title-a-nightmare- experience-with-microsoft/a01f65b8-62bb-43aa-bda5-7ae48fac9095. [Accessed 11-09-2024]

    2024

  18. [27]

    Then click ok!

    Adam Fourney, Ben Lafreniere, Richard Mann, and Michael Terry. 2012. " Then click ok!" extracting references to interface elements in online documentation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems . CHI ’25, April 26-May 1, 2025, Yokohama, ...

    2012

  19. [29]

    G. W. Furnas, T. K. Landauer, L. M. Gomez, and S. T. Dumais. 1987. The Vocabulary Problem in Human-System Communication. Commun. ACM 30, 11 (nov 1987), 964–971. https://doi.org/10.1145/32206.32212

    1987

  20. [30]

    L Gao, S Biderman, S Black, L Golding, T Hoppe, C Foster, J Phang, H He, A Thite, N Nabeshima, et al. 2020. The pile: An 800GB dataset of diverse text for language modeling. arXiv. arXiv preprint arXiv:2101.00027 (2020)

    2020 arXiv

  21. [31]

    McGuinness, and Michael Wolverton

    Alyssa Glass, Deborah L. McGuinness, and Michael Wolverton. 2008. To- ward Establishing Trust in Adaptive Agents. In Proceedings of the 13th Inter- national Conference on Intelligent User Interfaces (Gran Canaria, Spain) (IUI ’08). Association for Computing Machinery, New York...

    2008

  22. [32]

    Tovi Grossman and George Fitzmaurice. 2010. ToolClips: An Investigation of Contextual Video Assistance for Functionality Understanding. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Atlanta, Georgia, USA) (CHI ’10) . Association for Computing M...

    2010

  23. [33]

    Tovi Grossman, George Fitzmaurice, and Ramtin Attar. 2009. A Survey of Software Learnability: Metrics, Methodologies and Guidelines. InProceedings of the SIGCHI Conference on Human Factors in Computing Systems (Boston, MA, USA) (CHI ’09). Association for Computing Machinery, N...

    2009

  24. [34]

    Björn Hartmann, Daniel MacDougall, Joel Brandt, and Scott R. Klemmer. 2010. What Would Other Programmers Do: Suggesting Solutions to Error Messages. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Atlanta, Georgia, USA) (CHI ’10). Association for...

    2010

  25. [35]

    Hastie, M

    H.W. Hastie, M. Johnston, and P. Ehlen. 2003. Context-sensitive help for mul- timodal dialogue. In Proceedings. Fourth IEEE International Conference on Mul- timodal Interfaces. Institute of Electrical and Electronics Engineers, 93. https: //doi.org/10.1109/ICMI.2002.1166975 4t...

    2003 arXiv

  26. [36]

    Jeffrey Heer. 2019. Agency plus automation: Designing artificial intelligence into interactive systems. Proceedings of the National Academy of Sciences 116, 6 (2019), 1844–1850

    2019

  27. [37]

    Eric J. Horvitz. 2007. Reflections on Challenges and Promises of Mixed-Initiative Interaction. AI Magazine 28, 2 (Jun. 2007), 3. https://doi.org/10.1609/aimag.v28i2. 2036

    2007 doi

  28. [38]

    Amir Jahanlou and Parmit K Chilana. 2024. How Example-Based Authoring of Mo- tion Graphics Impacts Creative Expression: Differences in Perceptions of Profes- sional and Casual Motion Designers. In Proceedings of the 16th Conference on Cre- ativity & Cognition (Chicago, IL, USA...

    2024

  29. [39]

    Majeed Kazemitabaar, Jack Williams, Ian Drosos, Tovi Grossman, Austin Henley, Carina Negreanu, and Advait Sarkar. 2024. Improving Steering and Verification in AI-Assisted Data Analysis with Interactive Task Decomposition.arXiv preprint arXiv:2407.02651 (2024)

    2024 arXiv

  30. [40]

    Caitlin Kelleher and Randy Pausch. 2005. Stencils-based tutorials: design and evaluation. In Proceedings of the SIGCHI conference on Human factors in computing systems. 541–550

    2005

  31. [41]

    Anjali Khurana, Parsa Alamzadeh, and Parmit K. Chilana. 2021. ChatrEx: De- signing Explainable Chatbot Interfaces for Enhancing Usefulness, Transparency, and Trust. In 2021 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC). 1–11. https://doi.org/10.1109/V...

    2021 arXiv

  32. [42]

    Anjali Khurana, Hariharan Subramonyam, and Parmit K Chilana. 2024. Why and When LLM-Based Assistants Can Go Wrong: Investigating the Effectiveness of Prompt-Based Interactions for Software Help-Seeking. In Proceedings of the 29th International Conference on Intelligent User In...

    2024

  33. [43]

    Kimia Kiani, George Cui, Andrea Bunt, Joanna McGrenere, and Parmit K. Chilana

  34. [44]

    Guo, Robert C

    Juho Kim, Phu Tran Nguyen, Sarah Weir, Philip J. Guo, Robert C. Miller, and Krzysztof Z. Gajos. 2014. Crowdsourcing Step-by-Step Information Extraction to Enhance Existing How-to Videos. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Toronto, On...

    2014

  35. [45]

    Help Me Help the AI

    Sunnie S. Y. Kim, Elizabeth Anne Watkins, Olga Russakovsky, Ruth Fong, and Andrés Monroy-Hernández. 2023. "Help Me Help the AI": Understanding How Explainability Can Support Human-AI Interaction. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (...

    2023

  36. [46]

    Hyung-Kwon Ko, Gwanmo Park, Hyeon Jeon, Jaemin Jo, Juho Kim, and Jinwook Seo. 2023. Large-scale Text-to-Image Generation Models for Visual Artists’ Cre- ative Works. In Proceedings of the 28th International Conference on Intelligent User Interfaces (Sydney, NSW, Australia)(IUI...

    2023

  37. [47]

    Chilana, Adam Fourney, and Michael A

    Benjamin Lafreniere, Parmit K. Chilana, Adam Fourney, and Michael A. Terry

  38. [48]

    Benjamin Lafreniere, Tovi Grossman, and George Fitzmaurice. 2013. Community Enhanced Tutorials: Improving Tutorials with Multiple Demonstrations. In Pro- ceedings of the SIGCHI Conference on Human Factors in Computing Systems (Paris, France) (CHI ’13). Association for Computin...

    2013

  39. [49]

    Mitchell, and Brad A

    Toby Jia-Jun Li, Jingya Chen, Haijun Xia, Tom M. Mitchell, and Brad A. Myers

  40. [50]

    Liang, Chenyang Yang, and Brad A

    Jenny T. Liang, Chenyang Yang, and Brad A. Myers. 2024. A Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challenges. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering (Lisbon, Portugal) (ICSE ’24). Association...

    2024

  41. [51]

    Vera Liao, Daniel Gruen, and Sarah Miller

    Q. Vera Liao, Daniel Gruen, and Sarah Miller. 2020. Questioning the AI: Informing Design Practices for Explainable AI User Experiences. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20) . Association for Computing Mac...

    2020

  42. [52]

    Q Vera Liao and Jennifer Wortman Vaughan. 2023. Ai transparency in the age of llms: A human-centered research roadmap. arXiv preprint arXiv:2306.01941 (2023), 5368–5393

    2023 arXiv

  43. [53]

    Justin Matejka, Tovi Grossman, and George Fitzmaurice. 2011. IP-QAT: in-product questions, answers, & tips. In Proceedings of the 24th annual ACM symposium on User interface software and technology . 175–184

    2011

  44. [54]

    Justin Matejka, Wei Li, Tovi Grossman, and George Fitzmaurice. 2009. Com- munityCommands: command recommendations for software applications. In Proceedings of the 22nd Annual ACM Symposium on User Interface Software and Technology (Victoria, BC, Canada) (UIST ’09). Association...

    2009

  45. [55]

    Robinson Meyer. 2015. Even Early Focus Groups Hated Clippy. https://www.theatlantic.com/technology/archive/2015/06/clippy-the- microsoft-office-assistant-is-the-patriarchys-fault/396653/ [Accessed: 10-May-2020]

    2015

  46. [56]

    Tim Miller. 2019. Explanation in artificial intelligence: Insights from the social sciences. Artificial intelligence 267 (2019), 1–38

    2019

  47. [57]

    Hussein Mozannar, Gagan Bansal, Adam Fourney, and Eric Horvitz. 2024. Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Program- ming. InProceedings of the CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’24). Association ...

    2024

  48. [58]

    Brad A Myers, David A Weitzman, Amy J Ko, and Duen H Chau. 2006. Answering why and why not questions in user interfaces. In Proceedings of the SIGCHI conference on Human Factors in computing systems . 397–406

    2006

  49. [59]

    Don Norman. 2013. The design of everyday things: Revised and expanded edition . Basic books

    2013

  50. [60]

    Novick, Oscar D

    David G. Novick, Oscar D. Andrade, and Nathaniel Bean. 2009. The Micro- Structure of Use of Help. In Proceedings of the 27th ACM International Confer- ence on Design of Communication (Bloomington, Indiana, USA) (SIGDOC ’09). Association for Computing Machinery, New York, NY, U...

    2009

  51. [61]

    Novick, Oscar D

    David G. Novick, Oscar D. Andrade, Nathaniel Bean, and Edith Elizalde. 2008. Help-Based Tutorials. In Proceedings of the 26th Annual ACM International Con- ference on Design of Communication (Lisbon, Portugal) (SIGDOC ’08). Association for Computing Machinery, New York, NY, US...

    2008

  52. [62]

    Novick and Karen Ward

    David G. Novick and Karen Ward. 2006. Why Don’t People Read the Manual?. In Proceedings of the 24th Annual ACM International Conference on Design of Com- munication (Myrtle Beach, SC, USA) (SIGDOC ’06). Association for Computing Do It For Me vs. Do It With Me CHI ’25, April 26...

    2006

  53. [63]

    OpenAI. 2024. Embeddings. https://platform.openai.com/docs/guides/ embeddings. [Accessed 10-09-2024]

    2024

  54. [64]

    Susan Palmiter, Jay Elkerton, and Patricia Baggett. 1991. Animated demon- strations vs written instructions for learning procedural tasks: a preliminary investigation. International Journal of Man-Machine Studies 34, 5 (1991), 687–701

    1991

  55. [65]

    Pangoli and F

    S. Pangoli and F. Paternó. 1995. Automatic generation of task-oriented help. In Proceedings of the 8th Annual ACM Symposium on User Interface and Software Tech- nology (Pittsburgh, Pennsylvania, USA) (UIST ’95). Association for Computing Machinery, New York, NY, USA, 181–187. ...

    1995

  56. [66]

    Fischer, and Stuart Reeves

    Martin Porcheron, Joel E. Fischer, and Stuart Reeves. 2021. Pulling Back the Curtain on the Wizards of Oz. Proc. ACM Hum.-Comput. Interact. 4, CSCW3, Article 243 (jan 2021), 22 pages. https://doi.org/10.1145/3432942

    2021 doi

  57. [67]

    Microsoft Graph RAG. 2024. GitHub - microsoft/graphrag: A modular graph- based Retrieval-Augmented Generation (RAG) system — github.com. https: //github.com/microsoft/graphrag. [Accessed 11-09-2024]

    2024

  58. [68]

    Ashwin Ramachandran and R Michael Young. 2005. Providing intelligent help across applications in dynamic user and environment contexts. In Proceedings of the 10th international conference on Intelligent user interfaces . 269–271

    2005

  59. [69]

    Michael L Raulin and Anthony M Graziano. 2019. Quasi-experiments and correla- tional studies. In Companion Encyclopedia of Psychology . Routledge, 1122–1141

    2019

  60. [70]

    Marc Rettig. 1991. Nobody Reads Documentation. Commun. ACM 34, 7 (jul 1991), 19–24. https://doi.org/10.1145/105783.105788

    1991

  61. [71]

    John Rieman. 1996. A Field Study of Exploratory Learning Strategies. ACM Trans. Comput.-Hum. Interact. 3, 3 (sep 1996), 189–218. https://doi.org/10.1145/234526. 234527

    1996 doi

  62. [72]

    Advait Sarkar, Andrew D Gordon, Carina Negreanu, Christian Poelitz, Sruti Srini- vasa Ragavan, and Ben Zorn. 2022. What is it like to program with artificial intelligence? arXiv preprint arXiv:2208.06213 (2022)

    2022 arXiv

  63. [73]

    Elastic Hybrid Search. 2024. Hybrid Search — Search Labs — elas- tic.co. https://www.elastic.co/search-labs/tutorials/search-tutorial/vector- search/hybrid-search. [Accessed 11-09-2024]

    2024

  64. [74]

    Abigail Sellen and Eric Horvitz. 2024. The rise of the AI Co-Pilot: Lessons for design from aviation and beyond. Commun. ACM 67, 7 (2024), 18–23

    2024

  65. [75]

    Ben Shneiderman. 2020. Human-centered artificial intelligence: Reliable, safe & trustworthy. International Journal of Human–Computer Interaction 36, 6 (2020), 495–504

    2020

  66. [76]

    Ben Shneiderman and Pattie Maes. 1997. Direct manipulation vs. interface agents. interactions 4, 6 (1997), 42–61

    1997

  67. [77]

    Jared Spataro. 2023. Introducing Microsoft 365 copilot – your copilot for work. https://blogs.microsoft.com/blog/2023/03/16/introducing-microsoft-365- copilot-your-copilot-for-work/

    2023

  68. [78]

    Ashley Stewart. 2024. A CIO canceled a Microsoft AI deal. The rea- son should worry the entire tech industry. — businessinsider.com. https://www.businessinsider.com/pharma-cio-cancelled-microsoft-copilot-ai- tool-2024-7. [Accessed 11-09-2024]

    2024

  69. [79]

    Hari Subramonyam, Roy Pea, Christopher Pondoc, Maneesh Agrawala, and Colleen Seifert. 2024. Bridging the Gulf of Envisioning: Cognitive Challenges in Prompt Based Interactions with LLMs. In Proceedings of the CHI Conference on Human Factors in Computing Systems (Honolulu, HI, ...

    2024

  70. [80]

    Yuzhang Tian, Jianbo Zhao, Haoyu Dong, Junyu Xiong, Shiyu Xia, Mengyu Zhou, Yun Lin, José Cambronero, Yeye He, Shi Han, et al. 2024. SpreadsheetLLM: En- coding Spreadsheets for Large Language Models. arXiv preprint arXiv:2407.09025 (2024)

    2024 arXiv

  71. [81]

    Glassman

    Priyan Vaithilingam, Tianyi Zhang, and Elena L. Glassman. 2022. Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models. In Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, ...

    2022

  72. [82]

    Wang, Joanna McGrenere, and Parmit K

    Laton Vermette, Shruti Dembla, April Y. Wang, Joanna McGrenere, and Parmit K. Chilana. 2017. Social CheatSheet: An Interactive Community-Curated Informa- tion Overlay for Web Applications. Proc. ACM Hum.-Comput. Interact. 1, CSCW, Article 102 (dec 2017), 19 pages. https://doi....

    2017 doi

  73. [83]

    Xu Wang, Benjamin Lafreniere, and Tovi Grossman. 2018. Leveraging community- generated videos and command logs to classify and recommend software work- flows. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1–13

    2018

  74. [84]

    Jules White, Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olea, Henry Gilbert, Ashraf Elnashar, Jesse Spencer-Smith, and Douglas C Schmidt. 2023. A prompt pattern catalog to enhance prompt engineering with chatgpt. arXiv preprint arXiv:2302.11382 (2023)

    2023 arXiv

  75. [85]

    Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. 2023. C-Pack: Packaged Resources To Advance General Chinese Embedding. arXiv:2309.07597 [cs.CL]

    2023 arXiv

  76. [86]

    Xu, Bogdan Vasilescu, and Graham Neubig

    Frank F. Xu, Bogdan Vasilescu, and Graham Neubig. 2022. In-IDE Code Genera- tion from Natural Language: Promise and Challenges. ACM Trans. Softw. Eng. Methodol. 31, 2, Article 29 (mar 2022), 47 pages. https://doi.org/10.1145/3487569

    2022 doi

  77. [87]

    Litao Yan, Alyssa Hwang, Zhiyuan Wu, and Andrew Head. 2024. Ivie: Light- weight Anchored Explanations of Just-Generated Code. In Proceedings of the CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’24). Association for Computing Machinery, New York...

    2024

  78. [88]

    Tom Yeh, Tsung-Hsiang Chang, Bo Xie, Greg Walsh, Ivan Watkins, Krist Wong- suphasawat, Man Huang, Larry S Davis, and Benjamin B Bederson. 2011. Creating contextual help for GUIs using screenshots. In Proceedings of the 24th annual ACM symposium on User interface software and t...

    2011

  79. [89]

    Zamfirescu-Pereira, Richmond Y

    J.D. Zamfirescu-Pereira, Richmond Y. Wong, Bjoern Hartmann, and Qian Yang

  80. [2015]

    In Proceedings of the 28th Annual ACM Symposium on User Interface Software and Technology (Charlotte, NC, USA) (UIST ’15)

    These Aren’t the Commands You’re Looking For: Addressing False Feedforward in Feature-Rich Software. In Proceedings of the 28th Annual ACM Symposium on User Interface Software and Technology (Charlotte, NC, USA) (UIST ’15). Association for Computing Machinery, New York, NY, US...

  81. [2019]

    One-Size-Fits-All

    Beyond "One-Size-Fits-All": Understanding the Diversity in How Software Newcomers Discover and Make Use of Help Resources. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New...

    2019

  82. [2020]

    In Proceedings of the 33rd Annual ACM Symposium on User Interface Soft- ware and Technology (Virtual Event, USA) (UIST ’20)

    Multi-Modal Repairs of Conversational Breakdowns in Task-Oriented Dialogs. In Proceedings of the 33rd Annual ACM Symposium on User Interface Soft- ware and Technology (Virtual Event, USA) (UIST ’20). Association for Computing Machinery, New York, NY, USA, 1094–1107. https://do...

    doi

  83. [2023]

    Greater than

    Why Johnny Can’t Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI ’23). Association for Computing Machinery, New York, NY, USA, Article 437, 21 pages. https:...

    2023

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.