Pith. sign in

REVIEW 3 major objections 6 minor 26 references

System-driven Interactive Design Support for Cloud Architecture: A Qualitative User Experience Study with Novice Engineers

T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that a system-driven, workflow-based AI assistant, CA-Buddy, helps novice engineers design cloud architectures more effectively and learn in the process, based on a qualitative study of 60 newly hired engineers.

desk verdict Useful qualitative data on novices using a system-driven cloud design tool, but the paper's central comparative claim outruns a design with no baseline and no objective outcome. read the letter →

arxiv 2508.12385 v1 pith:T3XSD5RU submitted 2025-08-17 cs.HC cs.SE

Ryosuke Kohita , Akira Kasuga This is my paper · ORCID
classification cs.HCcs.SE
keywords system-drivendesignsupportcloudarchitecturenoviceengineersLLM-basedtoolsqualitativeuserexperiencestudythematicanalysisinteractiveeducationaltechnology
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to show that a system-driven, workflow-based assistant helps novice engineers design cloud architectures better than open-ended chat-based AI tools, and that the support doubles as a learning aid. The authors analyze free-text feedback from 60 newly hired engineers with limited cloud experience who used the tool for a 30-minute design task. Participants reported that the tool's proactive steps—proposing an initial architecture, summarizing and inspecting it, and asking clarifying questions—made it easier to start, removed the burden of writing prompts, and helped them compare alternatives. The authors read these reports as evidence that structured guidance reduces cognitive load and fosters incidental learning about cloud services and trade-offs. If right, the result argues for designing AI assistance around explicit workflows and state management rather than leaving interaction entirely to the user.

What carries the argument

The machinery is CA-Buddy's system-driven workflow, built on two structured state models. UserState records the project's goals, constraints, preferences, and answers to system questions; ArchitectureState tracks the current service configuration, design summary, identified risks, and open questions. Around these states, the system runs a fixed loop powered by a large language model: propose or update the architecture, summarize and inspect it, generate targeted questions, and refine after user responses, with user-initiated chat, pinned services, and feedback on alternatives layered on top. The loop is the mechanism that converts unstructured requirements into a progressively more complete design without requiring the user to direct the interaction.

What would settle it

An experiment that scores the architectures novices produce on a realistic scenario with an expert rubric, comparing CA-Buddy against a plain chat-based LLM, would settle the claim; if the two tools produce designs of equal quality, the reported benefits are likely novelties of LLM assistance rather than the workflow. A second check is whether novices can spot deliberately injected errors in generated architectures; high acceptance rates would contradict the claim that the tool effectively supports comprehension.

Watch

Extended reading notes

Core claim

The paper's central claim is that structured and proactive system guidance helps novices engage more effectively in cloud architecture design, especially where knowledge gaps are largest. CA-Buddy walks users through an explicit loop: initial requirements produce a proposed architecture, which the system summarizes, inspects for issues, and uses to generate clarifying questions; user answers and choices update a structured representation of requirements and architecture until the design stabilizes. In the study, the most reported benefit was that the system's initial draft gave novices a place to start, followed by the value of inspecting trade-offs and avoiding oversights, and the relief of not having to craft prompts. Many participants also described the iterative, simulation-like process as educational, giving them a way to compare architectures and discover unfamiliar services. The authors conclude that system-driven support both reduces cognitive load and supports learning, while noting that novices wanted output validation, less information density, and integration with implementation workflows such as infrastructure-as-code generation.

Load-bearing premise

The study relies on self-reported reactions from 60 newly hired engineers at a single company after a 30-minute design task as evidence that the tool improves design quality and learning; no objective measure of design correctness or learning gain was collected.

Editorial extensions

If this is right

  • Novice engineers can produce an initial cloud architecture by reacting to system questions and drafts, without knowing how to prompt an LLM or where to start.
  • A simulation-like loop of proposing, comparing, and revising architectures can double as a low-stakes learning environment for cloud services and design trade-offs.
  • Users at different skill levels need different information density; one-size-fits-all detailed explanations overwhelm novices.
  • Tools that generate designs must offer verification, such as rationale, documentation links, and checkable outputs, to counter novices' inability to detect errors.
  • Closing the gap to implementation—infrastructure-as-code, cost estimates, and deployment steps—is the natural next step for such tools to be useful beyond conceptual design.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, a direct quantitative test would be a randomized comparison in which novices design the same scenario with CA-Buddy versus a chat-only LLM, scoring outputs against an expert rubric; the paper's qualitative data predicts CA-Buddy would win on completeness and correctness, but leaves this unmeasured.
  • The reported uncertainty about output correctness suggests that without an independent validation mechanism, the same proactive workflow could teach novices to trust flawed architectures; measuring over-reliance is a natural follow-up.
  • The same workflow pattern—structured states plus proactive questioning—could plausibly transfer to other ill-structured design tasks, such as data-pipeline design or API design, where novices face similar ambiguity and trade-off problems.
  • Adaptive information delivery was requested by users and is consistent with cognitive-load theory; an A/B test on information density could turn this request into a design guideline.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper reports a qualitative user experience study of CA-Buddy, a system-driven, workflow-based LLM tool for cloud architecture design. Sixty novice engineers from a single Japanese company each completed a 20-minute orientation, a 30-minute design task in one of three scenarios, and a 10-minute open-ended questionnaire. The authors apply thematic analysis to the free-text responses and report themes of cognitive support (e.g., no need to craft prompts, smooth progress by answering questions), architectural support (e.g., diagrams, trade-off identification), learning through iterative exploration, and areas for improvement (e.g., output validation, information overload, IaC/deployment features). The paper concludes that structured and proactive system guidance helps novices engage more effectively in architectural design and has educational value.

Significance. The study addresses a timely and underexplored question: whether system-driven LLM support, as opposed to open-ended chat, helps novice cloud architects. Its strengths are a clear, reproducible study protocol; a relatively large qualitative sample (N=60) for a CHI-style UX study; the use of a standard thematic-analysis method with independent coding by two authors; and candid reporting of negative findings such as FB13, in which participants expressed uncertainty about design correctness. The authors also explicitly acknowledge single-company and short-task limitations. However, the significance is limited by the gap between the evidence collected and the claims made: without a comparison condition or objective outcome measures, the results support a description of perceived experiences but not a comparative claim about design quality or learning effectiveness. If reframed accordingly, the paper offers a useful descriptive account and a set of concrete design implications for system-driven AI design tools.

major comments (3)
  1. [Abstract and Section 3.1 (Study Design)] The central comparative claim is not supported by the study design. The abstract states that the findings indicate system-driven support 'helps novices engage more effectively in architectural design' and, in the final paragraph of the Introduction, that 'system-driven approaches not only improve conceptual design quality.' However, Section 3.1 describes a single-arm study with no baseline or control condition: participants used CA-Buddy for 30 minutes and then answered a questionnaire. All positive themes (no prompt crafting, smooth progress, useful initial draft) are consistent with the generic effect of any LLM that can produce a starting architecture, with a novelty effect, or with demand characteristics. The authors should either add a comparison condition (e.g., an open-ended chat tool with the same underlying LLM) or rephrase the claims to describe what was actually measured: participants' self-reported experiences with CA-Buddy.
  2. [Section 3.2 (Feedback Analysis) and Table 1] No objective outcome measure of design quality or learning is reported. The paper's claim of 'educational value' and 'improved conceptual design quality' rests entirely on participant opinion. Design quality is never rated by experts or checked against the stated requirements, and learning gains are not measured. In fact, FB13 shows that novices explicitly lacked confidence in whether the final design was correct. The Discussion (Section 5) acknowledges some limitations but does not concede that the absence of outcome measures prevents the comparative and quality-related conclusions in the abstract and introduction. The authors should soften these claims to 'participants reported that...' or add expert ratings, pre/post tests, or artifact-quality evaluation.
  3. [Section 3.1 (Study Design) and Section 3.2] The thematic analysis would benefit from reporting inter-rater reliability or a resolved-coding procedure. The text says 'Two authors independently coded the responses using an inductive approach' but gives no agreement metric (e.g., Cohen's kappa) and no description of how disagreements were resolved. This matters because the paper presents numeric mention counts (e.g., drafting 32, comparison 10) as evidence of prevalence. Also, the sample consists of newly hired engineers from one company with a 30-minute task, which the authors list as a limitation in Section 5, but the limitation section should additionally state that the single-company sample cannot support claims about generalizable effectiveness beyond perceived experiences.
minor comments (6)
  1. [Figure 2] The phrase 'No needs for trial-and-error with prompts' should be 'No need for trial-and-error with prompts.'
  2. [Section 3.2 (Feedback Analysis)] The sentence 'As feature requests, user mentioned a desire...' should read 'As feature requests, users mentioned a desire...'
  3. [Figure 2] The label 'Infrastructure-as-a-Code' is nonstandard; the usual term is 'Infrastructure as Code' or 'IaC.'
  4. [References] The reference list contains a journal/magazine name 'Proceedings of .' with empty fields in the ACM Reference Format block; this should be completed before submission.
  5. [Section 2] The sentence 'This section outlines the system’s workflow-based guidance and its hybrid integration of chat-based interactions, as informed by prior research' is slightly vague; it would help to state explicitly which prior research informed the chat integration (e.g., the previous CA-Buddy paper).
  6. [Section 5] The Discussion acknowledges single-company and short-task limitations but does not mention the absence of a control condition; this should be added for completeness.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the study's claims are qualitative summaries of independently collected user feedback, with only a non-load-bearing self-citation.

full rationale

The paper is a qualitative UX study, not a derivation with fitted parameters or predictive claims. The findings are thematic summaries of free-text responses from 60 novice engineers who used CA-Buddy (Section 3.1), and no outcome is defined as a function of an input in a way that would make a conclusion true by construction. The only self-citation is to the authors' prior CA-Buddy paper [8], used to introduce the system and prior role-playing results; the current conclusions rest on the newly collected questionnaire data and inductive coding, so the citation is not load-bearing. The real weakness is external validity: Section 3.1 describes a single-arm study with no control condition, and Section 5 acknowledges only single-company and short-task limitations, not the absence of a baseline. The Introduction's claim that results indicate 'system-driven approaches not only improve conceptual design quality' overreaches the self-reported measures, but that is a validity/interpretation concern, not a circular reduction of inputs to outputs. No equation, fitted parameter, or definitional identity links the tool's design to the measured outcome, so the circularity score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No numerical fitting is involved. The load-bearing assumptions are methodological: self-report validity, sample representativeness, thematic analysis suitability, and LLM output reliability.

assumptions (4)
  • domain assumption Self-reported user experience is a valid proxy for design support effectiveness and learning.
    The paper's central claims rest on participant feedback reported in Section 3.2; no objective measures of design quality or learning are collected.
  • domain assumption Participants classified as novices by being newly hired at one company represent the broader population of novice engineers.
    Section 3.1 defines novices by employment status at a single company; generalizability is assumed but not empirically established.
  • domain assumption Thematic analysis by two authors is a sufficient method for drawing valid conclusions from the feedback.
    The paper adopts Braun and Clarke's thematic analysis as a standard qualitative method without reporting inter-rater reliability or a full audit trail.
  • domain assumption LLM-generated architectures and summaries are accurate enough that user perceptions reflect the system-driven workflow, not systematic model errors.
    Section 2 states all system actions are powered by LLMs; the paper does not evaluate the correctness of generated architectures.

how reviews work

0 comments
Cite this review

Pith. "Pith review of System-driven Interactive Design Support for Cloud Architecture: A Qualitative User Experience Study with Novice Engineers." pith.science (2026). https://pith.science/paper/T3XSD5RU

@misc{pith2026250812385,
  author       = {Pith},
  title        = {Pith review of: System-driven Interactive Design Support for Cloud Architecture: A Qualitative User Experience Study with Novice Engineers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/T3XSD5RU}},
  note         = {Machine review of arXiv:2508.12385}
}
read the original abstract

Cloud architecture design presents significant challenges due to the necessity of clarifying ambiguous requirements and systematically addressing complex trade-offs, especially for novice engineers with limited cloud experience. While recent advances in the use of AI tools have broadened available options, system-driven approaches that offer explicit guidance and step-by-step information management may be especially effective in supporting novices during the design process. This study qualitatively examines the experiences of 60 novice engineers using such a system-driven cloud design support tool. The findings indicate that structured and proactive system guidance helps novices engage more effectively in architectural design, especially when addressing tasks where knowledge and experience gaps are most critical. For example, participants found it easier to create initial architectures and did not need to craft prompts themselves. In addition, participants reported that the ability to simulate and compare multiple architecture options enabled them to deepen their understanding of cloud design principles and trade-offs, demonstrating the educational value of system-driven support. The study also identifies areas for improvement, including more adaptive information delivery tailored to user expertise, mechanisms for validating system outputs, and better integration with implementation workflows such as infrastructure-as-code generation and deployment guidance. Addressing these aspects can further enhance the educational and practical value of system-driven support tools for cloud architecture design.

Figures

Figures reproduced from arXiv: 2508.12385 by the authors.

Figure 1
Figure 1. Overview of CA-Buddy, a workflow-based tool that guides users through cloud architecture design via a system-driven flow (left; green), structured state management (top right; blue), user actions (bottom right; red), and a UI snapshot (far right). 2 System Overview: CA-Buddy In this study, we use CA-Buddy [8], a system-driven cloud archi￾tecture design support tool. CA-Buddy proactively guides users through a struct… view at source ↗
Figure 2
Figure 2. summarizes the main themes from user feedback. Feedback can be grouped into two categories: Cognitive Support enabling smooth and efficient design and Architectural Support ensuring clear and comprehensive design. Users reported benefits such as user-friendly UI/UX (22 mentions), smooth design progress through answering system-generated questions (20), and no need to write detailed prompts (10). The tool further aid… view at source ↗

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 18 canonical work pages

  1. [1]

    C., Arun Iyer, Suresh Parthasarathy, Sriram Rajamani, B

    Ramakrishna Bairi, Atharv Sonwane, Aditya Kanade, Vageesh D. C., Arun Iyer, Suresh Parthasarathy, Sriram Rajamani, B. Ashok, and Shashank Shet. 2024. CodePlan: Repository-Level Coding Using LLMs and Planning. Proceedings of the ACM on Software Engineering 1, FSE (2024), 675–698

    work page 2024

  2. [2]

    Virginia Braun and Victoria Clarke. 2006. Using Thematic Analysis in Psychology. Qualitative Research in Psychology 3, 2 (2006), 77–101

    2006

  3. [3]

    Andrés Díaz-Pace, Antonela Tommasel, and Rafael Capilla

    J. Andrés Díaz-Pace, Antonela Tommasel, and Rafael Capilla. 2024. Helping Novice Architects to Make Quality Design Decisions Using an LLM-Based As- sistant. In Software Architecture: 18th European Conference, ECSA 2024 . Springer- Verlag, 324–332. doi:10.1007/978-3-031-70797-1_21

    doi:10.1007/978-3-031-70797-1_21 2024

  4. [4]

    Siqi Han, Wanting Li, En Zhang, Jilin Shi, Wei Wang, and Xuesong Lu. 2022. MLadder: An Online Training System for Machine Learning and Data Science Education. In Proceedings of the 31st ACM International Conference on Information and Knowledge Management (CIKM ’22)

    work page 2022

  5. [5]

    Arto Hellas, Juho Leinonen, Sami Sarsa, Charles Koutchéme, Lilja Kujanpää, and Juha Sorva. 2023. Exploring the Responses of Large Language Models to Beginner Programmers’ Help Requests. InProceedings of the 2023 ACM Conference on International Computing Education Research

    work page 2023

  6. [6]

    Majeed Kazemitabaar, Justin Chow, Carl K. T. Ma, Barbara J. Ericson, David Weintrop, and Tovi Grossman. 2023. Studying the Effect of AI Code Generators on Supporting Novice Learners in Introductory Programming. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI ’23)

    work page 2023

  7. [7]

    Henley, Paul Denny, Michelle Craig, and Tovi Grossman

    Majeed Kazemitabaar, Runlong Ye, Xiaoning Wang, Austin Z. Henley, Paul Denny, Michelle Craig, and Tovi Grossman. 2024. CodeAid: Evaluating a Classroom Deployment of an LLM-based Programming Assistant that Balances Student and Educator Needs. In CHI 2024

    work page 2024

  8. [8]

    Ryosuke Kohita and Akira Kasuga. 2025. System-driven Cloud Architecture De- sign Support with Structured State Management and Guided Decision Assistance. arXiv:2505.20701 https://arxiv.org/abs/2505.20701

    work page Pith review arXiv 2025

Show all 26 references
  1. [9]

    Mrityunjay Kumar and Venkatesh Choppella. 2023. A modeling language for novice engineers to design well at SaaS product companies. In Proceedings of the 16th Innovations in Software Engineering Conference . 1–5. doi:10.1145/3578527. 3578548

    2023 doi

  2. [10]

    Hellendoorn, Bogdan Vasilescu, and Brad A

    Daye Nam, Andrew Macvean, Vincent J. Hellendoorn, Bogdan Vasilescu, and Brad A. Myers. 2024. Using an LLM to Help With Code Understanding. In Proceedings of the 46th International Conference on Software Engineering (ICSE ’24). IEEE / ACM, 97:1–97:13. doi:10.1145/3597503.3639187

    2024

  3. [11]

    Babe, Yangtian Zi, Arjun Guha, Carolyn J

    Sydney Nguyen, Hannah M. Babe, Yangtian Zi, Arjun Guha, Carolyn J. Anderson, and Molly Q. Feldman. 2024. How Beginning Programmers and Code LLMs (Mis)read Each Other. InProceedings of the 2024 CHI Conference on Human Factors in Computing Systems (CHI ’24)

    2024

  4. [12]

    Claus Pahl, Pooyan Jamshidi, and Olaf Zimmermann. 2018. Architectural Princi- ples for Cloud Software. ACM Transactions on Internet Technology 18, 2, Article 17 (Feb. 2018), 23 pages. doi:10.1145/3104028

    2018 doi

  5. [13]

    James Prather, Brent Reeves, Juho Leinonen, Stephen MacNeil, Arisoa Randri- anasolo, Brett Becker, Bailey Kimmel, Jared Wright, and Ben Briggs. 2024. The Widening Gap: The Benefits and Harms of Generative AI for Novice Program- mers. In Proceedings of the 2024 ACM Conference o...

    2024

  6. [14]

    It’s Weird That it Knows What I Want

    James Prather, Brent N. Reeves, Paul Denny, Brett A. Becker, Juho Leinonen, Andrew Luxton-Reilly, Garrett Powell, James Finnie-Ansley, and Eddie Antonio Santos. 2023. “It’s Weird That it Knows What I Want”: Usability and Interactions with Copilot for Novice Programmers. ACM Tr...

    2023 doi

  7. [15]

    Sangho Suh, Meng Chen, Bryan Min, Toby Jia-Jun Li, and Haijun Xia. 2024. Luminate: Structured Generation and Exploration of Design Space with Large Language Models for Human-AI Co-Creation. In Proceedings of the 2024 CHI Con- ference on Human Factors in Computing Systems (CHI ...

    2024

  8. [16]

    Shangqing Tu, Zheyuan Zhang, Jifan Yu, Chunyang Li, Siyu Zhang, Zijun Yao, Lei Hou, and Juanzi Li. 2023. LittleMu: Deploying an Online Virtual Teaching Assistant via Heterogeneous Sources Integration and Chain of Teach Prompts. In Proceedings of the 32nd ACM International Conf...

    2023

  9. [17]

    Jules White, Sam Hays, Quchen Fu, Jesse Spencer-Smith, and Douglas C. Schmidt

  10. [18]

    Yeh, Karena Tran, Ge Gao, Tyler Yu, Wai On Fong, and Tzu-Yi Chen

    Thomas Y. Yeh, Karena Tran, Ge Gao, Tyler Yu, Wai On Fong, and Tzu-Yi Chen

  11. [19]

    Zamfirescu-Pereira, Richmond Wong, Bjoern Hartmann, and Qian Yang

    J.D. Zamfirescu-Pereira, Richmond Wong, Bjoern Hartmann, and Qian Yang

  12. [20]

    J. D. Zamfirescu-Pereira, Eunice Jun, Michael Terry, Qian Yang, and Björn Hart- mann. 2025. Beyond Code Generation: LLM-supported Exploration of the Pro- gram Design Space. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (CHI ’25)

    2025

  13. [21]

    Zhuoxun Zheng, Baifan Zhou, Dongzhuoran Zhou, Ahmet Soylu, and Evgeny Kharlamov. 2022. ExeKG: Executable Knowledge Graph System for User-friendly Data Analytics. In Proceedings of the 31st ACM International Conference on Infor- mation and Knowledge Management (CIKM ’22)

    2022

  14. [22]

    Yangtian Zi, Luisa Li, Arjun Guha, Carolyn Jane Anderson, and Molly Q. Feldman

  15. [26]

    I Would Have Written My Code Differently

    “I Would Have Written My Code Differently”: Beginners Struggle to Un- derstand LLM-Generated Code. In Companion Proceedings of the 37th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE ’25) . https://arxiv.or...

    arXiv

  16. [2023]

    In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI ’23)

    Why Johnny Can’t Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI ’23) . 1–21. doi:10.1145/3544548.3581388

    2023

  17. [2024]

    In Generative AI for Effective Soft- ware Development

    ChatGPT Prompt Patterns for Improving Code Quality, Refactoring, Re- quirements Elicitation, and Software Design. In Generative AI for Effective Soft- ware Development. Springer, 71–108

  18. [2025]

    In Proceedings of the 56th ACM Technical Symposium on Computer Science Education (SIGCSE ’25)

    Bridging Novice Programmers and LLMs with Interactivity. In Proceedings of the 56th ACM Technical Symposium on Computer Science Education (SIGCSE ’25). ACM, 1295–1301. doi:10.1145/3641554.3701867

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.