Pith. sign in

REVIEW 2 major objections 47 references

Developers' Experience with Generative AI Beyond Productivity Assessment -- Insights from an Empirical Mixed-Methods Field Study

T0 review · 2 major / 0 minor · reviewed 2026-07-12 · grok-4.5

Pith's one-line read Using either in-code suggestions or chat alone speeds coding work and lowers load; mixing them in one task cancels the gains.

desk verdict Solid mixed-methods field study with a usable interaction-type finding; free-choice confounding is real but does not sink the practical contribution. read the letter →

arxiv 2607.02337 v2 pith:B2RVE3X3 submitted 2026-07-02 cs.SE

classification cs.SE
keywords developerexperienceGenerativeAImixed-methodsstudyfieldsoftwareengineeringdeveloper-GeninteractionGitHubCopilotcognitiveload
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This field study of 22 professional developers follows them through controlled coding sessions and ordinary workdays to learn how they actually experience generative AI coding tools, not just whether they ship more code. Developers report high satisfaction, especially for repetitive structured work such as boilerplate, tests, and debugging, and they see efficiency gains. The central finding is sharper: both in-code suggestions and chat-based prompting independently shorten task time and reduce perceived workload, yet switching between those two modes inside a single task erases the benefits and can raise cognitive overhead. The authors therefore offer a simple rule of thumb for matching interaction style to task type and complexity. On everyday development-heavy work, extra cognitive load comes mainly from the act of interacting with AI, while any productivity lift depends on whether the output is judged helpful. Taking part in the study itself raised developers’ awareness and made their subsequent tool use more intentional. The work shows that real-world mixed-methods designs can surface interaction patterns that pure productivity metrics miss.

What carries the argument

The contrast between interaction types (in-code suggestions alone, chat prompts alone, both, or neither) measured by mixed-effects models on task duration, completion, and NASA-TLX workload across controlled HumanEval-X tasks and self-reported everyday work, which isolates the switching-cost effect that cancels efficiency gains.

What would settle it

A replication with a larger, multi-language, multi-firm sample that measures the same four interaction conditions and finds either no efficiency or workload penalty for mixed interaction, or no benefit for single-type use.

Watch

Extended reading notes

Core claim

Both in-code suggestions and chat-based prompting independently improve task efficiency and reduce perceived workload for professional developers, but combining the two interaction types within a single task diminishes those benefits. The paper therefore proposes a rule-of-thumb for selecting interaction type according to whether the task is coding or non-coding, how much codebase context is required, and whether explanations are needed.

Load-bearing premise

That the six controlled tasks and the developers’ self-categorized everyday work for twenty-two Java developers at one firm are representative enough of professional software practice that the interaction-type effects will hold elsewhere.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 0 minor

Summary. This mixed-methods field study examines professional SAP developers’ (n=22) experience with GenAI (primarily GitHub Copilot) beyond pure productivity metrics. Over four days it combines two controlled sessions (six HumanEval-X-derived tasks: coding, debugging, documentation, testing, summary, brainstorming; A/B Copilot exposure) with three days of naturalistic work (445 self-documented tasks). Multimodal data include questionnaires (NASA-TLX, Likert, open-ended), ~66 h of annotated screen recordings, and (unused here) physiological signals. Main claims: developers are satisfied with GenAI especially for monotonous/structured tasks; both in-code suggestions and chat independently improve efficiency and reduce perceived workload relative to no Copilot, while combining them within a task yields no benefit (and higher workload than in-code alone); a task-characteristic rule-of-thumb is proposed; in everyday development-heavy work, AI interaction raises cognitive load while productivity gains track perceived output helpfulness; study participation itself increased intentional GenAI use.

Significance. The paper addresses a genuine gap: developer-centered, ecologically valid evidence on GenAI interaction experience rather than benchmark correctness or laboratory productivity alone. Strengths include the rare combination of controlled and uncontrolled phases inside a firm, explicit linkage of subjective NASA-TLX/Likert data to manually annotated interaction behavior, and a concrete, falsifiable rule-of-thumb that practitioners can test. The finding that single-mode use helps while mode-switching does not is practically actionable if it holds. The modest sample and free-choice design limit causal strength, yet the work still supplies one of the more carefully instrumented field pictures currently available and motivates better-controlled follow-ups.

major comments (2)
  1. §5.3 / Figure 7 (and the companion analysis [7]): the central claim that “combining interaction types diminishes benefits” rests on free-choice observational data. Interaction type was not assigned; the “both” condition is therefore self-selected and likely enriched for harder tasks or for developers who switched after an initial mode failed. The (G)LMMs treat type as a fixed effect with participant random intercepts but report no residual diagnostics, no propensity-style adjustment for task difficulty/order, and no within-task sequence analysis that would isolate switching cost from selection. With n=22 this residual confounding is a plausible alternative explanation; if the penalty is largely selection rather than causal, the proposed rule-of-thumb loses its causal warrant. At minimum the manuscript must (a) quantify how task category, duration, and completion rate differ across the fo
  2. §4.1–4.4 and §5.4: generalizability of both the controlled-task set and the development-heavy / collaboration-heavy / other taxonomy is asserted rather than demonstrated. All controlled coding tasks are Java-only HumanEval-X items; the 445 everyday tasks are self-reported and manually categorized by three authors with only brief reference to Meyer et al. Without inter-rater reliability statistics, sensitivity checks on the taxonomy, or evidence that the six task types span the complexity distribution of real SAP work, the claim that the observed interaction-type and cognitive-load patterns generalize beyond this cohort remains under-supported. A short robustness subsection or explicit limitation statement is needed.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: empirical mixed-methods findings from screen annotations, NASA-TLX and self-reports do not reduce by construction to fitted parameters or author-supplied definitions; self-citations to companion design paper supply methods/background only.

full rationale

This is an observational/empirical HCI field study (22 SAP developers, controlled HumanEval-X tasks + uncontrolled workdays, multimodal data). There is no first-principles derivation, uniqueness theorem, ansatz, or parameter fit that is later re-labeled a prediction. The load-bearing claim (single interaction type improves efficiency/workload; combining diminishes benefits) rests on (generalized) linear mixed-effects models of annotated screen recordings and NASA-TLX scores (Section 5.3 / Figure 7, summarizing prior analysis in [7]). Interaction type was free-choice, not experimentally assigned, but that is a confounding/selection threat, not circularity by construction. Self-citations ([7] for design + behavioral results, [6] for task categorization, [5] for supplemental) provide background methods and earlier analyses of the same study; they do not define the measured effects or force the rule-of-thumb. The rule-of-thumb is an interpretive recommendation after the data, not a tautological restatement of inputs. No self-definitional loops, no fitted-input-as-prediction, no uniqueness imported from authors. Score 1 only for the minor, non-load-bearing self-citation pattern that is normal in multi-paper study series.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The work is empirical; it introduces no free parameters fitted to produce the central claim, no novel physical or mathematical entities, and only standard domain assumptions of mixed-methods HCI/SE research. The ledger is therefore nearly empty, reflecting that the contribution is observational rather than theoretical.

assumptions (3)
  • domain assumption NASA-TLX raw scores and self-reported Likert ratings of cognitive load and productivity are valid proxies for developers’ subjective experience.
    Invoked throughout Sections 5.3–5.4 when interpreting workload and productivity differences; standard in HCI but still an assumption about construct validity.
  • domain assumption The six HumanEval-X-derived tasks plus the three-way categorization of everyday work (development-heavy / collaboration-heavy / other) adequately sample professional software engineering activity.
    Underpins generalizability claims in Sections 4.4 and 5.4; the mapping is partly interpretive and performed by three authors.
  • standard math Linear mixed-effects models with participant as random effect correctly capture the hierarchical structure of repeated measures.
    Used for all inferential claims in Sections 5.3–5.4; standard statistical practice.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Developers' Experience with Generative AI Beyond Productivity Assessment -- Insights from an Empirical Mixed-Methods Field Study." pith.science (2026). https://pith.science/paper/B2RVE3X3

@misc{pith2026260702337,
  author       = {Pith},
  title        = {Pith review of: Developers' Experience with Generative AI Beyond Productivity Assessment -- Insights from an Empirical Mixed-Methods Field Study},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/B2RVE3X3}},
  note         = {Machine review of arXiv:2607.02337}
}
read the original abstract

With the growing adoption of AI-powered coding assistants, organizations and developers are increasingly seeking to optimize their interaction with these tools. Prior research has largely focused on output quality and productivity gains, with limited attention paid to developers' well-being and interaction experiences. This paper presents a developer-centered empirical mixed-methods study to investigate how professional developers engage with Generative AI (GenAI) in their natural work environment. Controlled data collection sessions are combined with natural work periods. Results show that developers are generally satisfied with GenAI, particularly for monotonous, repetitive, and structured tasks, and report perceived efficiency and productivity gains. Copilot interaction type preferences differ by task type and complexity: While both in-code suggestions and chat-based prompting independently improve task efficiency and reduce perceived workload, combining these interaction types within a single task diminishes benefits. We propose a rule-of-thumb for selecting an interaction type based on task characteristics. During development-heavy tasks, results indicate that perceived cognitive load arises from AI interaction, while perceived productivity depends on AI output quality. Participation in this study positively influenced developers' awareness and intentional use of GenAI tools. These findings demonstrate the value of real-world, mixed-methods study designs to understand GenAI tools and developers' experiences with them.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

47 extracted references · 3 linked inside Pith

  1. [7]

    Charlotte Brandebusemeyer, Tobias Schimmer, and Bert Arnrich. 2025. Developers’ Experience with Generative AI - First Insights from an Empirical Mixed-Methods Field Study. ArXiv (December 2025). Retrieved April 18, 2026 from https://doi.org/10.48550/arXiv.2512.19926

  2. [1]

    Gal Bakal, Ali Dasdan, Yaniv Katz, Michael Kaufman, and Guy Levin. 2025. Experience with GitHub Copilot for Developer Productivity at Zoominfo. ArXiv (January 2025). https://doi.org/10.48550/arXiv.2501.13282

  3. [2]

    James, and Nadia Polikarpova

    Shraddha Barke, Michael B. James, and Nadia Polikarpova. 2023. Grounded Copilot: How Programmers Interact with Code -Generating Models. Proceedings of the ACM on Programming Languages 7, OOPSLA1 (April 2023), 85–111. https://doi.org/10.1145/3586030

  4. [3]

    Alexander Barker. 2022. JNativeHook: Global Keyboard and Mouse Listener for Java. Retrieved July 29, 2025 from https://github.com/kwhat/jnativehook

  5. [4]

    Christian Bird, Denae Ford, Thomas Zimmermann, Nicole Forsgren, Eirini Kalliamvakou, Travis Lowdermilk, and Idan Gazit. 2023. Taking Flight with Copilot. Commun. ACM 66, 6 (May 2023), 56–62. https://doi.org/10.1145/3589996

  6. [5]

    Charlotte Brandebusemeyer, Tobias Schimmer, and Bert Arnrich. 2025. Supplemental Material for Developers’ Experience with Gen erative AI - First Insights from an Empirical Mixed-Methods Field Study. Zenodo. https://doi.org/10.5281/zenodo.17818081

  7. [6]

    Charlotte Brandebusemeyer, Tobias Schimmer, and Bert Arnrich. 2025. Wearables to Measure Developer Experience at Work. In 2025 IEEE/ACM 47th International Conference on Software Engineering: Software Engineer ing in Practice (ICSE -SEIP), April 27, 2025. IEEE, 23 –33. https://doi.org/10.1109/ICSE-SEIP66354.2025.00008

  8. [8]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian...

Show all 47 references
  1. [9]

    Mariana Coutinho, Lorena Marques, Anderson Santos, Marcio Dahia, Cesar França, and Ronnie de Souza Santos. 2024. The Role of Generative AI in Software Development Productivity: A Pil ot Case Study. In Proceedings of the 1st ACM International Conference on AI -Powered Software ...

  2. [10]

    Mihaly Czikszentmihalyi. 1990. Flow: The psychology of optimal experience. Harper & Row, New York

  3. [11]

    Nicole Forsgren, Eirini Kalliamvakou, Abi Noda, Michaela Greiler, Brian Houck, and Margaret Anne Storey. 2023. DevEx in Action: A study of its tangible impacts. Queue 21, 6 (December 2023), 47–77. https://doi.org/10.1145/3639443

  4. [12]

    Nicole Forsgren, Margaret-Anne Storey, Chandra Maddila, Thomas Zimmermann, Brian Houck, and Jenna Butler. 2021. The SPACE of Developer Productivity: There’s more to it than you think. Queue 19, 1 (February 2021), 20–48. https://doi.org/10.1145/3454122.3454124

  5. [13]

    Povilas Godliauskas and Darja Šmite. 2025. The well -being of software engineers: a systematic literature review and a theory. Empir. Softw. Eng. 30, 1 (January 2025), 35. https://doi.org/10.1007/s10664 -024-10543-8

  6. [14]

    Lewis R Goldberg. 1992. The development of markers for the Big -Five factor structure. Psychol. Assess. 4, 1 (1992), 26–42

  7. [15]

    Lewis R Goldberg. 2024. International Personality Item Pool: A Scientific Collaboratory for the Development of Advanced Measu res of Personality Traits and Other Individual Differences. Retrieved July 29, 2025 from https://ipip.ori.org/new_ipip -50-item-scale.htm

  8. [16]

    Michaela Greiler, Margaret -Anne Storey, and Abi Noda. 2023. An Actionable Framework for Understanding and Improving Developer Experience. IEEE Transactions on Software Engineering 49, 4 (April 2023), 1411–1425. https://doi.org/10.1109/TSE.2022.3175660

  9. [17]

    Sandra G. Hart. 2006. Nasa -Task Load Index (NASA -TLX); 20 Years Later. Proceedings of the Human Factors and Ergonomics Society Annu al Meeting 50, 9 (October 2006), 904–908. https://doi.org/10.1177/154193120605000909

  10. [18]

    Headfulness - Luke Horton. 2024. Calming 4 -7-8 Breathing (5 Minutes). Retrieved July 29, 2025 from https://www.youtube.com/watch?v=DAp3aiC57ZQ&t=17s

  11. [19]

    Haritha Khandabattu. 2 025. The 2025 Hype Cycle for Artificial Intelligence Goes Beyond GenAI. Gartner. Retrieved September 13, 2025 from https://www.gartner.com/en/articles/hype-cycle-for-artificial-intelligence

  12. [20]

    Mansi Khemka and Brian Houck. 2024. Toward Effective AI Support fo r Developers. Commun. ACM 67, 11 (November 2024), 42 –49. https://doi.org/10.1145/3690928

  13. [21]

    Shuang Li, Yuntao Cheng, Jinfu Chen, Jifeng Xuan, Sen He, and Weiyi Shang. 2024. Assessing the Performance of AI -Generated Code: A Case Study on GitHub Copilot. In 2024 IEEE 35th International Symposium on Software Reliability Engineering (ISSRE) , October 28, 2024. IEEE, 216–

  14. [22]

    https://doi.org/10.1109/ISSRE62328.2024.00030 39

  15. [23]

    Philipp Mayring and Thomas Fenzl. 2019. Qualitative Inhaltsanalyse. In Handbuch Methoden der emp irischen Sozialforschung . Springer Fachmedien Wiesbaden, Wiesbaden, 633–648. https://doi.org/10.1007/978-3-658-21308-4_42

  16. [24]

    Meyer, Earl T

    Andre N. Meyer, Earl T. Barr, Christian Bird, and Thomas Zimmermann. 2021. Today Was a Good Day: The Daily Life of Software Developers . IEEE Transactions on Software Engineering 47, 5 (May 2021), 863–880. https://doi.org/10.1109/TSE.2019.2904957

  17. [25]

    Microsoft and LinkedIn. 2024. 2024 Work Trend Index Annual Report - AI at Work is Here. Now Comes the Hard Part. Retrieved July 29, 2025 from https://www.microsoft.com/en-us/worklab/work-trend-index/ai-at-work-is-here-now-comes-the-hard-part

  18. [26]

    Giovanni B. Moneta. 2021. On the conceptualization and measurement of flow. In Advances in Flow Research . Springer Science, New York, 31–

  19. [27]

    https://doi.org/10.1007/978-3-030-53468-4_2

  20. [28]

    Desmarais, and Zhen Ming (Jack) Jiang

    Arghavan Moradi Dakhel, Vahid Majdinasab, Amin Nikanjam, Foutse Khomh, Michel C. Desmarais, and Zhen Ming (Jack) Jiang. 2023. GitHub Copilot AI pair programmer: Asset or Liability? Journal of Syste ms and Software 203, (September 2023), 111734. https://doi.org/10.1016/j.jss.20...

  21. [29]

    Hussein Mozannar, Gagan Bansal, Adam Fourney, and Eric Horvitz. 2024. Reading Between the Lines: Modeling User Behavior and C osts in AI- Assisted Programming. In Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI ’24) , May 11, 2024. ACM, Honolulu, H...

  22. [30]

    Nhan Nguyen and Sarah Nadi. 2022. An empirical evaluation of GitHub copilot’s code suggestions. In Proceedings of the 19th International Conference on Mining Software Repositories (MSR ’22) , May 23, 2022. ACM, Pittsburgh, PA, USA, 1 –5. https://doi.org/10.1145/3524842.3528470

  23. [31]

    Ojelanki Ngwenyama, Nada Kanita, and Frantz Rowe. 2025. Can Generative AI C ontribute to Both Productivity Gains and Human Flourishing, and in Fine Satisfaction at Work? Research on GitHub Copilot Use in Software Development. In Hawaii International Conference on System Scienc...

  24. [32]

    Abi Noda, Margaret Anne Storey, Nicole Forsgren, and Michaela Greiler. 2023. DevEx: What Actually Drives Productivity. Queue 21, 2 (April 2023), 35–53. https://doi.org/10.1145/3595878

  25. [33]

    OBS Project. 2025. OBS - Open Broadcaster Software. Retrieved July 29, 2025 from https://obsproject.com/

  26. [34]

    Sida Peng, Eirini Kalliamvakou, Peter Cihon, and Mert Demirer. 2023. The Impact of AI on Developer Productivity: Evidence from GitHub Copilot. ArXiv (February 2023). https://doi.org/10.48550/arXiv.2302.06590

  27. [35]

    Paige S Rutner, Bill C Hardgrave, and D Harrison Mcknight. 2008. Emotional Dissonance and the Information Technology Professi onal. MIS Quarterly 32, 3 (September 2008), 635–652. https://doi.org/10.2307/25148859

  28. [36]

    Gordon, Carina Negreanu, Christian Poelitz, Sruti Srinivasa Ragavan, and Ben Zorn

    Advait Sarkar, Andrew D. Gordon, Carina Negreanu, Christian Poelitz, Sruti Srinivasa Ragavan, and Ben Zorn. 2022. What is it like to program with artificial intelligence? ArXiv (2022), 1–27. https://doi.org/10.48550/arXiv.2208.06213

  29. [37]

    Stanford University Human-Centered Artificial Intelligence. 2025. Artificial Intelligence Index Report 2025

  30. [38]

    Margaret Anne Storey, Thomas Zimmermann, Christian Bird, Jacek Czerwonka, Brendan Murphy, and Eirini Kalliamvakou. 2021. Towa rds a Theory of Software Developer Job Satisfactio n and Perceived Productivity. IEEE Transactions on Software Engineering 47, 10 (October 2021), 2125–...

  31. [39]

    Margaret-Anne Storey, T Zimmermann, C Bird, J Czerwonka, B Murphy, and E Kalliamvakou. 2019. Supplemental material for towards a theory of software developer job satisfaction and perceived productivity. Zenodo. Retrieved from https://zenodo.org/records/3451354#.XYUr - OdKjOQ

  32. [40]

    Viktoria Stray, Nils Brede Moe, Nivethika Ganeshan, and Simon Kobbenes. 2025. Generative AI and Developer Workflows: How GitHub Copilot and ChatGPT Influence Solo and Pair Programming. In Proceedings of the 58th Hawaii International Conference on System Sciences (HICSS ’25) , ...

  33. [41]

    Laura Tacho. 2024. Introducing Core 4: The best way to measure and improve your product velocity. Retrieved July 29, 2025 from https://www.lennysnewsletter.com/p/introducing -core-4-the-best-way-to

  34. [42]

    Ningzhi Tang, Meng Chen, Zheng Ning, Aakash Bansal, Yu Huang, Collin McMillan, and Toby Jia-Jun Li. 2024. Developer Behaviors in Validating and Repairing LLM-Generated Code Using IDE and Eye Tracking. In 2024 IEEE Symposium on Visual Languages and Human -Centric Computing (VL/...

  35. [43]

    Glassman

    Priyan Vaithilingam, Tianyi Zhang, and Elena L. Glassman. 2022. Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models. In CHI Conference on Human Factors in Computing Systems Extended Abstracts (CHI ’22 Extended Abstract...

  36. [44]

    Burak Yetistiren, Isik Ozsoy, and Eray Tuzun. 2022. Assessing the quality of GitHub copilot’s code ge neration. In Proceedings of the 18th International Conference on Predictive Models and Data Analytics in Software Engineering (PROMISE ’22) , November 07, 2022. ACM, Singapore...

  37. [45]

    Burak Yetiştiren, Işık Özsoy, Miray Ayerdem, and Eray Tüzün. 2023. Evaluating the Code Quality of AI -Assisted Code Generation Tools: An Empirical Study on GitHub Copilot, Amazon CodeWhisperer, and ChatGPT. arXiv preprint (April 2023)

  38. [46]

    Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Lei Shen, Zihan Wang, Andi Wang, Yang Li, Teng Su, Zhilin Yang, a nd Jie Tang. 2023. CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval -X. In Proceedings of the 29th ...

  39. [47]

    Alice Li, Andrew Rice, Devon Rifkin, Shawn Simister, Ganesh Sittampalam, and Edward Aftandilian

    Albert Ziegler, Eirini Kalliamvakou, X. Alice Li, Andrew Rice, Devon Rifkin, Shawn Simister, Ganesh Sittampalam, and Edward Aftandilian. 2022. Productivity assessment of neural code completion. In MAPS 2022: Proceedings of the 6th ACM SIGPLAN International Symposium on Machine...

Pith tools

Reviewed July 12, 2026 · model on record in the stance chip above.