{"id":"a7fd319c-a50e-4c45-ae38-8b6029a139ab","arxiv_id":"2504.15549","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"In a 20-participant comparison, a semi-automated copilot with visual step-by-step guidance beat a fully automated copilot on task completion, accuracy, perceived control, utility, and learnability.","lead":"This study compares two types of AI copilots for complex software: one that automates the whole task and one that automates only small steps while showing visual, step-by-step instructions. Most of the 20 users preferred the guided version for control and learning, though the automated version saved time on simpler visual tasks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"AutoCopilot baseline is designed to fail on multi-intent tasks, so the completion/accuracy gap does not yet establish that semi-automation outperforms full automation.","rationale":"The reader's weakest assumption identifies the same load-bearing issue: the comparison is only meaningful if AutoCopilot is a fair representative of the full-automation paradigm. Section 3.1 confirms that AutoCopilot was deliberately designed to encounter incorrect automation on multi-intent queries, and the study's tasks are exactly of that kind. The quantitative superiority claims, especially the large completion and accuracy gaps, are therefore not evidence about full automation as a paradigm unless the baseline is calibrated against real-world copilots. A fair baseline or an ablation that removes the injected failures would settle whether the gap is due to the paradigm or the implementation. The paper also acknowledges in Limitations that specific features were not isolated, so the result may reflect guidance and control as much as automation level. This does not invalidate the within-study finding that users preferred this GuidedCopilot, but it does prevent the broader paradigm-level conclusion from being accepted as established. The reader's CONDITIONAL verdict remains appropriate.","tokens_in":25177,"tokens_out":6941,"duration_ms":66235,"concrete_test":"Re-run the study's Sheets tasks with AutoCopilot modified to use the correct pre-scripted automation for each multi-step task, removing the intentional entity-mapping failure described in Section 3.1, while leaving GuidedCopilot unchanged and scoring with the same ground-truth rubric. If AutoCopilot's completion/accuracy remain near 35%/12%, the baseline is not the cause; if they rise substantially, the reported paradigm-level gap is an artifact of the weakened baseline.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 explicitly constructs AutoCopilot's failure modes: it \"occasionally encounters incorrect full automation when it fails to identify a single automation function from the multiple intents in the user's query... or when it cannot accurately map multiple entities.\" The study tasks are chosen precisely to be multi-stage and multi-entity (e.g., \"analyze the top 5 products and visualize their sales across regions\"; \"create a webpage that includes a login section and displays products\"). Thus the baseline is engineered to fail on the experimental tasks. The reported completion rates (35.0% vs 88.5% in Sheets; 20.0% vs 55.0% in Figma) and accuracy rates (12% vs 82%; 5% vs 40%) are therefore not an unbiased estimate of full automation; they depend on the authors' decisions about when to inject errors. No calibration data are reported comparing AutoCopilot's failure rate with production full-automation copilots, and the code/prompts are not publicly released. The paper's within-study conclusion about this GuidedCopilot implementation is plausible, but the general claim that semi-automated copilots are a better default for feature-rich software requires a representative full-automation baseline. The limitation section also concedes that the independent contributions of automation level vs. step-by-step guidance were not isolated, which further limits the paradigm-level interpretation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper compares two copilot paradigms for feature-rich software: AutoCopilot, which fully automates a user's task from a natural-language prompt, and GuidedCopilot, which automates only trivial steps and provides step-by-step visual guidance. The authors implemented both as in-application assistants for Google Sheets and Figma, ran a within-subject controlled experiment with 20 participants, and report higher task completion and accuracy plus more positive perceptions of control, utility, and learnability for GuidedCopilot. A follow-up Wizard-of-Oz design exploration with 10 participants evaluates two adaptive extensions in Photoshop. The paper concludes that semi-automated copilots with visual step-by-step guidance are a better default than full automation for complex tasks, and proposes a three-dimensional framework for choosing automation levels.","tokens_in":25376,"tokens_out":3665,"duration_ms":34345,"significance":"If the comparison were fair, the result would be a valuable empirical challenge to the current push toward fully automated in-application copilots. The paper contributes two working prototype copilots, rich think-aloud and interview data, a concrete framework for automation-level decisions, and a useful follow-up design exploration. These contributions are real and the qualitative findings about user control and learnability are plausible on their own. However, the paper's central paradigm-level claim—that semi-automation outperforms full automation—is not yet established because the full-automation baseline appears to be constructed so that it fails systematically on the chosen experimental tasks, and because the reported statistics pool observations in a way that may inflate significance. The significance is therefore conditional on a fair baseline and appropriate statistical analysis.","major_comments":[{"comment":"The AutoCopilot baseline is deliberately engineered to fail on the very tasks used in the experiment. The paper states that AutoCopilot 'occasionally encounters incorrect full automation when it fails to identify a single automation function from the multiple intents in the user's query for performing complex tasks outlined in the study, or when it cannot accurately map multiple entities.' The experimental tasks are explicitly multi-stage and multi-entity (Section 5.3), so the observed completion gaps (35.0% vs 88.5% in Sheets; 20.0% vs 55.0% in Figma) and accuracy gaps (12.0% vs 82.0%; 5.0% vs 40.0%) are substantially determined by the researchers' implementation decisions rather than by the paradigm itself. No calibration data are given to show that AutoCopilot's failure rate is representative of production full-automation copilots, and the code and prompts are not publicly released. This undermines the abstract's and Section 6.1.1's claim that semi-automation outperforms full automation as a paradigm. The authors should either compare against an existing production copilot, calibrate their baseline's failure rate to a published/measured rate, or reframe the conclusion to be about this specific AutoCopilot implementation rather than the paradigm.","section":"Section 3.1"},{"comment":"The statistical results are reported in a way that is inconsistent with the stated analysis and appears to treat repeated measures as independent observations. The text says a 'paired-sample t-test' was used, yet the reported degrees of freedom (t(37.6), t(37.9), t(30.7)) are consistent with Welch's independent-samples t-test on roughly 40 observations per group, not a paired test on N=20 participants (which would give df=19). In addition, each participant contributed multiple measurements (two tasks per application across two applications), so task-level observations are nested within participants and are not independent. The Pearson chi-square tests in Sections 6.1.2, 6.2.1, and 6.3 pool Likert responses across both applications and both tasks for the same participants, also violating independence. This inflates the apparent strength of evidence. The authors should analyze the data with per-participant averages or mixed-effects models, and should report effect sizes and confidence intervals. The significant p-values may still hold, but they must be recomputed.","section":"Section 6.1.1"},{"comment":"The comparison confounds the automation level with several other interaction-design features: GuidedCopilot provides step-by-step visuals, user-initiated automation, editable LLM-extracted entities, and mixed-medium responses, whereas AutoCopilot provides none of these. The limitations section explicitly concedes that 'the independent contributions of specific features (e.g., automation, step-by-step guidance)' were not assessed. Since the paper's central claim is about automation paradigms, this confound is not a peripheral limitation: the reported differences could be due to the presence of guidance and user control affordances rather than to semi-automation per se. The authors should either run a factorial design that varies automation level and guidance independently, or substantially narrow the claim to 'a semi-automated copilot with visual guidance is preferred over this particular fully automated implementation.'","section":"Section 9 and Sections 3.1–4.1"}],"minor_comments":[{"comment":"The trial-and-error analysis is described as manually annotated, but no inter-rater reliability or coding scheme details are reported; adding these would strengthen the reproducibility of the 192 vs. 89 trial-and-error counts in Section 6.2.2.","section":"Section 5.4"},{"comment":"The abstract states that AutoCopilot 'saved time for simpler visual tasks,' but the paper does not report any direct time measurements; the support for this statement is participants' perceived time savings (6/20 users), which is a weaker claim and should be phrased accordingly.","section":"Abstract and Section 6.1.2"},{"comment":"The code and prompts are described as 'available upon request'; for a CHI paper presenting a novel system comparison, a public repository link would substantially aid reproduction and allow reviewers to scrutinize the AutoCopilot failure-injection logic.","section":"Section 4.1.3"},{"comment":"When reporting the paired t-tests, the paper does not state whether the data were checked for normality or whether the tests were two-tailed in each case; the 'two-tailed' designation appears only once and should be applied consistently.","section":"Section 6.1.1"},{"comment":"The trial-and-error figure is informative but the percentages in Section 6.2.2 (e.g., 'about 75% of AutoCopilot attempts involved undoing') are presented without the denominator for each attempt category; clarifying the denominators would avoid ambiguity.","section":"Figure 5"}],"recommendation":"major_revision","confidential_remarks":"This is a well-written and timely HCI paper with useful design artifacts and qualitative insights. The blockbuster claim—that semi-automated copilots are a better paradigm than full automation—is not yet supported because the full-automation baseline is engineered to fail on the experimental tasks and because the inferential statistics pool repeated measures. I believe these issues can be fixed within a revision by re-analyzing the existing data (mixed-effects models, per-participant aggregates) and by substantially softening or recalibrating the paradigm-level claim, so I recommend major revision rather than rejection. If the authors cannot add a fairer baseline, the paper could still be acceptable as a comparative study of two specific prototype designs, but not as a general paradigm comparison."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Paper you asked about: a competent HCI study with a genuine system and a plausible finding, but its headline claim—that semi-automated copilots are a better default than full automation—is not actually established by the data. The baseline is the problem. Section 3.1 says AutoCopilot 'occasionally encounters incorrect full automation when it fails to identify a single automation function from the multiple intents in the user's query... or when it cannot accurately map multiple entities.' The study tasks in Section 5.3 are chosen precisely to be multi-intent and multi-entity (analyze top 5 products and visualize across regions; create a webpage with login and product sections). So the baseline is engineered to fail on the experimental tasks. The completion gap (35% vs 88.5% in Sheets; 20% vs 55% in Figma) is partly a consequence of that design choice, and no calibration data show AutoCopilot's failure rate matches production copilots. Code and prompts are 'available upon request,' which is not the same as released.\n\nCredit where it's due. GuidedCopilot is a real system combining step-by-step visual guidance, user-initiated semi-automation, and editable LLM-extracted entities. The architecture is described in enough detail to reproduce. The within-study user perception data—control, utility, learnability—are consistent with the qualitative quotes, and the trial-and-error analysis (192 vs 89 attempts) is a nice touch. The follow-up Wizard-of-Oz exploration is clearly labeled as a design exploration, not a validation. The limitations section is honest: it concedes they did not isolate automation level from guidance. That matches my reading.\n\nThe main soft spot is generalizing from these two implementations to paradigms. GuidedCopilot differs from AutoCopilot not only in automation level but also in guidance, visuals, editability, and initiation. The authors acknowledge this in Section 9. So the evidence supports a within-system comparison, not a paradigm-level one. The 'occasionally' in Section 3.1 is doing a lot of work; if AutoCopilot's failure modes are representative of real copilots, the finding is meaningful, but the paper doesn't give the reader enough to judge.\n\nMinor: N=20 is small but fine for a qualitative-first study. The chi-square tests are fine. Ground-truth scoring is researcher-defined but the appendix rubric is clear.\n\nWho's this for? Researchers building in-application copilots and HCI folks studying automation-augmented learning. It deserves a serious referee. But it needs either a fairer baseline (calibrated against actual copilot failures) or a more modest claim. I'd send it out with a request for failure-rate calibration and released artifacts, and I would not cite the paradigm conclusion.","headline":"A solid HCI study of a real system whose paradigm-level claim is undercut by an engineered-failure baseline and a confound between automation level and guidance; the within-study result is believable, the generalization is not.","tokens_in":25944,"tokens_out":4969,"would_cite":false,"duration_ms":36617,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Semi-automated copilots beat full automation in user control and learning, study finds","keywords":["software copilots","semi-automation","full automation","user control","learnability","feature-rich software","LLM assistants","visual guidance"],"falsifier":"Run the same tasks with a fully automated copilot whose failure rate is measured from real deployments of tools like Microsoft 365 Copilot or Figma AI rather than designed into the prototype, and check whether the completion and accuracy gaps between the two paradigms shrink to insignificance.","tokens_in":24957,"feed_emoji":"🤖","tokens_out":1529,"duration_ms":15022,"temperature":0.7,"pith_summary":"This paper argues that for feature-rich software like spreadsheets and design tools, a copilot that does routine steps automatically while showing visual step-by-step instructions beats a copilot that tries to do the whole task by itself. The authors build two working copilots, embed them in Google Sheets and Figma, and run a controlled study with 20 users. The semi-automated one led to much higher task completion and accuracy, stronger feelings of control and utility, and better software learnability, while the fully automated one only won praise from a few technical users for saving time on simple tasks. A follow-up design study with 10 users shows that adding task- and state-aware preview clips and adaptive instructions makes the semi-automated approach even more useful. If the result holds, the default for copilot design should shift from full automation toward user-controlled, guided semi-automation.","feed_headline":"Semi-automated copilots beat full automation in control and learning","feed_subtitle":"Step-by-step guidance plus automation of routine steps beat handing the whole task to the AI, a 20-person study shows.","key_machinery":"The central object is GuidedCopilot, a semi-automated copilot that automates only trivial or repetitive steps (e.g., creating a frame in Figma, sorting in Sheets) while giving step-by-step guidance with in-context visual references. Users can initiate automation themselves, edit the entities the LLM extracts from their query before automation runs, and receive follow-up answers that mix web data with software documentation. The architecture uses GraphRAG over software documentation and Q&A forums to fetch text, images, and scripted automation functions, and an LLM agent that assembles these into a mixed-medium response. AutoCopilot is the contrasting fully automated system, built to mirror real-world copilot behavior, including occasional incorrect automation when it fails to map multiple intents or entities to a single automation function.","core_discovery":"The core claim is that the 'Do It With Me' paradigm, embodied in GuidedCopilot, outperforms the 'Do It For Me' paradigm, embodied in AutoCopilot, along user control, software utility, and learnability, with significantly higher task completion and accuracy in both Google Sheets and Figma. In Sheets, participants completed on average 88.5% of the task with GuidedCopilot versus 35.0% with AutoCopilot; in Figma the numbers were 55.0% versus 20.0%. Task accuracy similarly favored GuidedCopilot (82.0% vs 12.0% in Sheets; 40.0% vs 5.0% in Figma). The paper also reports that users engaged in roughly twice as many trial-and-error attempts with AutoCopilot, and that several users abandoned it entirely. A minority, mostly male computer-science professionals, preferred full automation for its time savings on simpler visual tasks.","pith_inferences":["A fair test of the paradigm claim requires an AutoCopilot whose failure rate matches real production copilots; the paper's baseline was deliberately weakened by including incorrect automation, so the size of the completion and accuracy gaps is likely inflated even if the direction of the effect is real.","The preference of expert users for full automation suggests that copilots should adapt not just to task state but to user identity over longer time scales, something the follow-up study only begins to explore.","The same semi-automation-plus-visual-guidance structure could be tested in adjacent domains such as code editors, CAD tools, or data science notebooks, where LLM copilots already exist but rarely offer step-by-step in-context visual walkthroughs.","Measuring whether GuidedCopilot's learnability gains persist after a delay (e.g., one week later) would separate true skill acquisition from in-the-moment guidance, a distinction the current studies do not address."],"forward_implications":["Copilot designers should treat full automation as a feature for simple, repetitive tasks rather than as the default interaction paradigm for complex software.","Semi-automated copilots that show steps and let users trigger automation can reduce debugging effort, as indicated by the trial-and-error counts (192 attempts with AutoCopilot vs 89 with GuidedCopilot).","Learnability is a measurable benefit of the guided approach: users reported transferring skills to new tasks, and the paper documents a concrete example of a participant who could only copy-paste prompts with AutoCopilot but succeeded with GuidedCopilot.","Task- and state-aware features, such as preview clips that track user progress and adaptive instructions that skip completed steps, can address the main complaints users had about the basic GuidedCopilot (overly detailed instructions and difficulty mapping chat visuals to the interface).","The three-dimensional framework proposed in the paper, with axes for automation level, adaptive guidance, and factors of familiarity, task type, and user intent, gives designers a checklist for choosing how much automation to offer."],"supporting_citations":[{"why":"Defines the copilot concept and argues that excessive automation can reduce agency; supplies the design premise that balance between guidance and automation matters.","marker":"[74]"},{"why":"Shows why LLM-based help can fail and why users benefit from prompts and visuals; grounds the paper's argument that text-only copilot output is hard to apply.","marker":"[42]"},{"why":"ChatrEx provides evidence that combining text with visual cues improves usefulness and trust in help-seeking systems, a direct basis for GuidedCopilot's mixed-medium design.","marker":"[41]"},{"why":"Appinite shows the value of visual, step-by-step task guidance and its limits, informing the design of the visual step-through feature.","marker":"[49]"},{"why":"Carroll and Rosson's active-user paradox justifies the goal of letting users learn by doing rather than passively receiving finished automation.","marker":"[15]"},{"why":"The GraphRAG framework is the retrieval backbone used to fetch context-relevant documentation, visuals, and automation functions for GuidedCopilot.","marker":"[67]"},{"why":"The ironies of automation literature supplies the theoretical risk that full automation reduces user skill and control, motivating the semi-automation paradigm.","marker":"[8]"}],"fun_headline_variants":["Step-by-step copilot beats full automation in user study","Do It With Me copilot beats Do It For Me in control and learning","Guided copilot outperforms fully automated copilot in learnability","Semi-automated copilot gives more control and learning than full automation","Copilot that guides beats copilot that does everything for you"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes that AutoCopilot's full-automation behavior, including its deliberately included incorrect executions, fairly represents how real fully automated copilots perform.","fun_headline_variants_meta":{"raw":{"variants":["Step-by-step copilot beats full automation in user study","Do It With Me copilot beats Do It For Me in control and learning","Guided copilot outperforms fully automated copilot in learnability","Semi-automated copilot gives more control and learning than full automation","Copilot that guides beats copilot that does everything for you"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.002072,"raw_usage":{"total_tokens":8061,"prompt_tokens":943,"completion_tokens":7118,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":559,"completion_tokens_details":{"reasoning_tokens":7026}},"tokens_in":559,"tokens_out":7118,"duration_ms":42103,"temperature":1.0,"reasoning_tokens":7026,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:23:20.557349+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same tasks with a fully automated copilot whose failure rate is measured from real deployments of tools like Microsoft 365 Copilot or Figma AI rather than designed into the prototype, and check whether the completion and accuracy gaps between the two paradigms shrink to insignificance.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the copilot concept and argues that excessive automation can reduce agency; supplies the design premise that balance between guidance and automation matters."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The GraphRAG framework is the retrieval backbone used to fetch context-relevant documentation, visuals, and automation functions for GuidedCopilot."}],"review_version":1}