Pith. sign in

REVIEW 4 major objections 5 minor 51 references

Creating benchmarkable components to measure the quality ofAI-enhanced developer tools

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that AI coding tools can be benchmarked on developer experience using reusable, modular components, and it provides the surveys, the standardized task, and the experimental design to do so.

desk verdict A useful methods case study that actually ships reusable instruments, but the 'robust/validated' language outruns the evidence and the single-task foundation is the load-bearing assumption. read the letter →

arxiv 2504.12211 v1 pith:IML5I7R5 submitted 2025-04-16 cs.SE cs.HC

Elise Paradis , Ambar Murillo , Maulishree Pandey , Sarah D'Angelo , Matthew Hughes , Andrew Macvean , Ben Ferrari-Church This is my paper · ORCID
classification cs.SEcs.HC
keywords AIUXmetricsbenchmarkinghumancomputerinteractionevaluationimpactofdeveloperexperiencegenerativecognitivetesting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Generative-AI coding assistants are usually compared on model benchmarks, not on how developers actually experience using them. This paper argues that product teams need a parallel standard: reusable, enterprise-grade components for benchmarking the developer-experience (DX) dimensions of AI-for-code products. The authors share the process they used to build such components at a large software company: selecting metrics from a developer-productivity framework, drafting and cognitively testing surveys on demographics, AI attitudes, task experience, and feature experience; designing a realistic C++ data-logging server task; and piloting that task until it could be completed in under 1.5 hours three times in a row without researcher help. They also share the resulting component kit and a between-subject randomized controlled design in which time on task is measured with and without AI. If the approach holds, teams can borrow these instruments instead of building their own, making AI coding products comparable across vendors and release cycles.

What carries the argument

The load-bearing mechanism is the benchmarkable task together with the validated survey set. The task is a C++ data-logging server that participants must implement, build, and test; it is designed to be feasible asynchronously, realistic for back-end engineers, and complex enough to require real code changes. The surveys measure covariates such as programming experience, tenure, daily coding hours, language familiarity, AI tool experience, attitudes toward AI, trust in AI accuracy, and post-task and per-feature perceptions. The pilot feasibility rule—three consecutive completions under 1.5 hours with no researcher intervention—is what makes the task safe to administer unmoderated and comparable across conditions.

What would settle it

Run the same component kit with a different enterprise-grade task, such as a Python web service or a debugging-heavy change, and check whether the estimated AI velocity gain matches the C++ task's estimate; a materially different gain would show the benchmark task, not the AI tool, drives the comparison.

Watch

Extended reading notes

Core claim

The paper's central claim is that a rigorous DX benchmark for AI coding products can be decomposed into modular, reusable components: validated questionnaires, a standardized enterprise-grade coding task, and a study design that ties them together. It describes the components in detail, including the demographics and AI-attitudes survey, the post-task survey, per-feature surveys, and the task stimulus: a C++ service that receives log messages from a fake product and writes them to per-application files, with an append flag that controls whether new messages truncate or extend the file. The paper also documents the process of cognitive testing and piloting, including the rule that the task must be completed in under 1.5 hours three consecutive times without researcher intervention. Using these instruments in the accompanying randomized trial, the authors report their estimate that developers are 21% faster with AI, a result that motivates the benchmark's value for product and investment decisions.

Load-bearing premise

The benchmark's validity rests on a single C++ data-logging task being representative enough of enterprise-grade engineering work that speed comparisons made on it generalize to other developers and other AI coding tools.

Editorial extensions

If this is right

  • Teams can reuse the published surveys and task to benchmark their own AI coding products and compare results with the published 21% velocity estimate.
  • The between-subject randomized design with a control group supports causal interpretation of AI's impact on end-to-end velocity, not just correlation.
  • The same component kit can evaluate multiple AI features in one study, giving an objective speed measure and subjective sentiment measure for each feature.
  • Adopting such components would let the industry move from ad hoc Copilot studies to repeatable, comparable DX benchmarks.
  • The paper's process—metric selection, literature review, cognitive testing, task identification, and piloting—provides a template for teams benchmarking complex AI products in other technical domains.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If these components gain adoption, previously isolated DX studies become commensurable, enabling pooled comparisons across tools and vendors; the paper gestures at this benefit but does not work out a concrete aggregation method.
  • Because the benchmark's comparability over time depends on the C++ task remaining equally hard as IDEs and AI assistants evolve, the pilot criterion would likely need periodic recalibration; the paper does not address task-difficulty drift.
  • A natural test of the component kit's generality is to swap the task for another enterprise-grade coding task, such as a Python service or a debugging-heavy change, and check whether the estimated AI velocity gain changes materially.
  • The same survey machinery could extend to non-code technical roles, such as data scientists or cloud operators, by replacing only the task stimulus while keeping the attitude and feature questionnaires; the paper mentions this possibility but does not validate it.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This CHI EA case study from Google describes a process for creating benchmark components for evaluating the developer experience (DX) of AI-enhanced coding tools. The authors report five steps: selecting benchmarkable metrics (sentiment and productivity, based on a SPACE variant), reviewing literature and internal sources for attitude questions, conducting two rounds of cognitive testing with 17 engineers, designing a single enterprise-grade benchmarking task (a C++ data-logging server), and piloting that task for feasibility. They then present the experimental design: a randomized controlled trial comparing developers with and without access to three AI features, measuring time on task and subjective sentiment via post-task and feature surveys. The paper includes the full questionnaires and task description in appendices, reports internal impact including an estimated 21% velocity gain from the companion paper [20], and offers adoption guidance for other teams. The central claim is that the shared components are 'robust, enterprise-grade and modular' and can support benchmarking of genAI code products across products and over time.

Significance. If the components were as robust and portable as claimed, this would be a genuinely useful contribution to an underdeveloped area: it offers a transparent, step-by-step process for building DX benchmarks, openly shares survey items and a task stimulus, and provides a real-world example of an RCT design. The paper is also honest about practical constraints, such as the need for large samples (>100) for statistical significance and the importance of precise time-on-task logging. These strengths make the case study a valuable starting point for teams wanting to benchmark AI coding tools. However, the central claims currently outpace the evidence: survey 'validation' rests solely on cognitive testing with 17 engineers, the single C++ task is not shown to be representative of enterprise-grade work, the components depend heavily on Google-internal infrastructure, and no reliability or validity statistics are reported. The significance of the paper therefore depends on whether the authors can either provide additional validation evidence or carefully scope the claims to what the current evidence supports.

major comments (4)
  1. [§3.4–3.5] The representativeness of the single benchmarking task is not established. In §3.4, 'high fidelity and enterprise grade' is defined only by exclusion ('not LeetCode, not PyBench') and by assertion that Google back-end engineers could reasonably complete it. In §3.5, feasibility is operationalized solely as completion in under 1.5 hours three times in a row without researcher intervention; this is a timeability criterion, not evidence of content validity. Because every downstream comparison — including the 21% velocity estimate from companion paper [20] — rests on this one C++ logging task, the paper's claim that these components enable valid cross-product and cross-time DX comparisons is not supported. Please provide evidence on task representativeness (e.g., expert judgment of task alignment with authentic work, multiple task sampling, or comparison with real-world task distributions) or explicitly limit the claim to 'a demonstration component' rather than a general benchmark.
  2. [§1 contribution bullet and §3.3] The term 'validated surveys' is used in the contributions list, but the only validation evidence reported is two rounds of cognitive testing with a total of 17 engineers (§3.3). No test-retest reliability, internal consistency (e.g., alpha or omega), or convergent/discriminant validity data are provided. For instruments intended for cross-team and cross-time comparison, the word 'validated' is load-bearing. Please either add psychometric evidence (from this study or a cited source) or replace 'validated' with a more precise descriptor such as 'cognitively tested' or 'qualitatively piloted.'
  3. [Appendices B–D and §5.2] The claimed portability of the components is not currently actionable outside Google. The task description (Appendix D) depends on Google-internal IDE settings, source control, storage paths, data retention policies, and redacted feature documentation links; the surveys (Appendices B and C) reference internal repositories, internal job titles, and Google-specific response options. The paper's stated aim to 'lower the barrier' and invite others to 'borrow our components' is therefore only partially realized. The paper does acknowledge in §5.2 that adjustments are needed, but the claim of 'modular' and 'ready-to-use' components requires either de-identified, tool-agnostic versions (e.g., a generic task with explicit adaptation instructions) or a clear statement that the task is an exemplar requiring substantial reimplementation in another environment.
  4. [§5.1 and §2.2] The impact discussion in §5.1 relies on the 21% velocity estimate from the authors' own companion paper [20], which uses the same benchmark components. This is not an independent validation of the components, and the current text in §2.2 presents the result without noting that it derives from the very instruments being introduced here. The paper should explicitly state that the impact evidence is internal and shares the same unresolved representativeness and reliability limitations as the components themselves, so that readers can calibrate the strength of the 'impact' claims.
minor comments (5)
  1. [Abstract] The abstract contains a grammatical error: 'product team have struggled' should be 'product teams have struggled.'
  2. [Table 7] In the demographics questionnaire, the question 'Where is the code you've worked on in the last 3 months for Google hosted? Select all that apply' is listed as 'Single choice' in the Question type column, which conflicts with the 'Select all that apply' instruction and the 'All but last' randomization note.
  3. [§2.2] The sentence 'Results seem similar at a more micro level' is awkward; consider rephrasing to 'Results at a more micro level are similar' for clarity.
  4. [Appendix D] The task instruction 'If the file does not exist and needs to be created, it needs to have [our data retention policy added to the file name]' is unclear even as a redaction; consider explaining what the redaction is intended to convey to external readers.
  5. [§4] The text says the study totals 'at most four hours over three activities,' but Activity 2 alone is described as up to 1.5 to 2.5 hours in §3.4 and §3.5; clarifying the relationship between the task time limit and the total time budget would help readers replicate the design.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark components are developed and piloted in-paper; self-citations are contextual and not load-bearing.

full rationale

This paper is a case study describing the creation of benchmark components (surveys, a coding task, and study protocol) and does not claim to derive a prediction from fitted inputs. The central claim is that the components are reusable, feasible, and benchmarkable, and this is supported by in-paper processes: two rounds of cognitive testing with 17 engineers (Section 3.3), explicit task-selection criteria (Section 3.4), and an iterative pilot with a defined feasibility rule (completed under 1.5 hours three times in a row, Section 3.5). The task stimulus and survey instruments are shared in the appendices so that external teams can exercise them. The 21% velocity estimate is cited from the authors' companion paper [20], but that estimate is presented as prior work and as an application of the benchmark, not as evidence for the validity of the components. Similarly, the internal survey repository [7] and the SPACE framework [9] are sources for item selection, not circular justifications. The pilot criterion is a design target, not a fitted parameter renamed as a prediction, and the paper does not claim statistical generalization beyond the pilot. Any concerns about single-task representativeness or the strength of the term 'validated' are correctness or validity limitations, not circular reasoning. The derivation chain is therefore self-contained with respect to circularity, and the minor self-citations do not bear the weight of the central contribution.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central claim depends on task feasibility criteria, task representativeness, measurement precision, and survey validity rather than on fitted parameters. No new physical or theoretical entities are introduced.

free parameters (1)
  • Task feasibility threshold = under 1.5 hours, three consecutive completions without researcher intervention
    Hand-set criterion used to stop pilot iteration and declare the task feasible. It calibrates task difficulty to the study time window, not to a theoretical or externally benchmarked target.
assumptions (4)
  • domain assumption Time on task, measured precisely by logging infrastructure, is a valid outcome for developer velocity.
    The main outcome is time on task (Section 4, Activity 2). The paper itself notes at Section 5.2 that the study is not rigorous if time on task relies on self-report, so measurement precision is load-bearing.
  • domain assumption A single C++ data-logging server task is representative of enterprise-grade engineering work and of the tasks AI coding features support.
    Defined as a task 'most Google back-end engineers could reasonably be expected to complete' (Section 3.4), with no external evidence of representativeness for other products or developer populations.
  • domain assumption Cognitive interviewing with small volunteer samples establishes that survey items are robust across the target population.
    Two rounds with 9 and 8 Google engineers (Section 3.3) are treated as sufficient for robust instruments. No reliability or validity statistics are reported.
  • domain assumption Random assignment to AI/no-AI with unmoderated self-administration yields unbiased causal estimates.
    The design relies on RCT logic [6] and assumes no differential noncompliance, such as control-group participants enabling other AI tools, and assumes accurate time logging.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Creating benchmarkable components to measure the quality ofAI-enhanced developer tools." pith.science (2026). https://pith.science/paper/IML5I7R5

@misc{pith2026250412211,
  author       = {Pith},
  title        = {Pith review of: Creating benchmarkable components to measure the quality ofAI-enhanced developer tools},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IML5I7R5}},
  note         = {Machine review of arXiv:2504.12211}
}
read the original abstract

In the AI community, benchmarks to evaluate model quality are well established, but an equivalent approach to benchmarking products built upon generative AI models is still missing. This has had two consequences. First, it has made teams focus on model quality over the developer experience, while successful products combine both. Second, product team have struggled to answer questions about their products in relation to their competitors. In this case study, we share: (1) our process to create robust, enterprise-grade and modular components to support the benchmarking of the developer experience (DX) dimensions of our team's AI for code offerings, and (2) the components we have created to do so, including demographics and attitudes towards AI surveys, a benchmarkable task, and task and feature surveys. By doing so, we hope to lower the barrier to the DX benchmarking of genAI-enhanced code products.

Figures

Figures reproduced from arXiv: 2504.12211 by the authors.

Figure 1
Figure 1. Experimental design and components a between-subject design for multiple reasons. First, it would have been impossible to re-use the task in a within-subject design and make participants complete it with and without AI, because of un￾avoidable learning effects. Second, it was very difficult to design even one task that would be feasible by a broad-enough swath of developers with varied previous knowledge and skills.… view at source ↗
Figure 2
Figure 2. Architecture provided to research participants [PITH_FULL_IMAGE:figures/full_fig_p016_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

51 extracted references · 37 canonical work pages

  1. [20]

    Elise Paradis, Kate Grey, Quinn Madison, Daye Nam, Andrew Macvean, Vahid Meimand, Nan Zhang, Ben Ferrari-Church, and Satish Chandra. 2024. How much does AI impact development speed? An enterprise-based randomized controlled trial. https://arxiv.org/pdf/2410.12944. , 12 pages

    arXiv 2024

  2. [1]

    Beatty and Gordon B

    Paul C. Beatty and Gordon B. Willis. 2007. Research Synthesis: The Practice of Cognitive Interviewing. Public Opinion Quarterly 71, 2 (05 2007), 287–311. https://doi.org/10.1093/poq/nfm006 arXiv:https://academic.oup.com/poq/article- pdf/71/2/287/5436696/nfm006.pdf

    work page doi:10.1093/poq/nfm006 2007

  3. [2]

    Jenna Bergdahl, Rita Latikka, Magdalena Celuch, Iina Savolainen, Eerik Soares Mantere, Nina Savela, and Atte Oksanen. 2023. Self-determination and attitudes toward artificial intelligence: Cross-national and longitudinal perspectives.Telem- atics and Informatics 82 (2023), 102013. https://doi.org/10.1016/j.tele.2023.102013

    arXiv 2023

  4. [3]

    Satish Chandra and Maxim Tabachnyk. 2024. AI in software engineering at Google: Progress and the path ahead. https://research.google/blog/ai-in-software- engineering-at-google-progress-and-the-path-ahead/. Accessed: 2024-07-10

    work page 2024

  5. [4]

    CoderPad. 2024. CoderPad State of Tech Hiring. https://coderpad.io/survey- reports/coderpad-and-codingame-state-of-tech-hiring-2024/. Accessed: 2024- 07-10

    work page 2024

  6. [5]

    Zheyuan Kevin Cui, Mert Demirer, Sonia Jaffe, Leon Musolff, Sida Peng, and Tobias Salz. 2024. The Effects of Generative AI on High Skilled Work: Evidence from Three Field Experiments with Software Developers. https://papers.ssrn. com/sol3/papers.cfm?abstract_id=4945566. , 22 pages

    work page 2024

  7. [6]

    Angus Deaton and Nancy Cartwright. 2018. Understanding and misunderstand- ing randomized controlled trials. Social Science & Medicine 210 (2018), 2–21. https://doi.org/10.1016/j.socscimed.2017.12.005 Randomized Controlled Trials and Evidence-based Policy: A Multidisciplinary Dialogue

    doi:10.1016/j.socscimed.2017.12.005 2018

  8. [7]

    Sarah D’Angelo, Jessica Lin, Jill Dicker, Carolyn Egelman, Maggie Hodges, Collin Green, and Ciera Jaspan. 2024. Measuring Developer Experience With a Longitu- dinal Survey. IEEE Software 41, 4 (2024), 19–24. https://doi.org/10.1109/MS.2024. 3386027

    doi:10.1109/ms.2024 2024

Show all 51 references
  1. [8]

    Steve W Edison and Gary L Geissler. 2003. Measuring attitudes towards general technology: Antecedents, hypotheses and scale development.Journal of Targeting, Measurement and Analysis for Marketing 12 (2003), 137–156. https://doi.org/10. 1057/palgrave.jt.5740104

    2003

  2. [9]

    Nicole Forsgren, Margaret-Anne Storey, Chandra Maddila, Thomas Zimmermann, Brian Houck, and Jenna Butler. 2021. The SPACE of Developer Productivity: There’s more to it than you think. Queue 19, 1 (2021), 20–48

    2021

  3. [10]

    Ya Gao and Github Customer Research. 2024. Research: Quanti- fying GitHub Copilot’s impact in the enterprise with Accenture. Creating benchmarkable components to measure the quality of AI-enhanced developer tools CHI EA ’25, April 26-May 1, 2025, Yokohama, Japan https://github...

    2024

  4. [11]

    William Harding and Matthew Kloster. 2024. Coding on Copilot: 2023 Data Suggests Downward Pressure on Code Quality. https://www.gitclear.com/coding_ on_copilot_data_shows_ais_downward_pressure_on_code_quality/. Accessed: 2024-07-10

    2024

  5. [12]

    Saki Imai. 2022. Is GitHub copilot a substitute for human pair-programming? an empirical study. In Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings (Pittsburgh, Pennsylvania) (ICSE ’22). Association for Computing Machiner...

    2022

  6. [13]

    Sage Kelly, Sherrie-Anne Kaye, and Oscar Oviedo-Trespalacios. 2023. What factors contribute to the acceptance of artificial intelligence? A systematic review. Telematics and Informatics 77 (2023), 101925

    2023

  7. [14]

    JetBrains Tech Insights Lab. 2023. Jetbrains The State of Developer Ecosystem

    2023

  8. [15]

    LeetCode. 2015. LeetCode - The World’s Leading Online Programming Learning Platform. https://leetcode.com/. Accessed: 2024-06-20

    2015

  9. [16]

    McIntosh, Teo Susnjak, Nalin Arachchilage, Tong Liu, Paul Watters, and Malka N

    Timothy R. McIntosh, Teo Susnjak, Nalin Arachchilage, Tong Liu, Paul Watters, and Malka N. Halgamuge. 2024. Inadequacies of Large Language Model Bench- marks in the Era of Generative Artificial Intelligence. arXiv:2402.09880 [cs.AI] https://arxiv.org/abs/2402.09880

    2024 arXiv

  10. [17]

    Lisa-Maria Neudert, Aleksi Knuutila, and Philip N. Howard. 2023. Global attitudes towards AI, Machine Learning & Automated Decision Making–Implications for involving artificial intelligence in public service and good governance. https: //perma.cc/6PB6-X56B. Accessed: 2024-07-10

    2023

  11. [18]

    Stack Overflow. 2023. Stack Overflow, 2023 Developer Survey, AI sentiment and usage. https://survey.stackoverflow.co/2023/#ai-sentiment-and-usage. Accessed: 2024-07-10

    2023

  12. [19]

    Oxford University Press. 2024. benchmark, n. https://languages.oup.com/google- dictionary-en/. Accessed: 2024-07-10

    2024

  13. [21]

    Sida Peng, Eirini Kalliamvakou, Peter Cihon, and Mert Demirer. 2023. The Impact of AI on Developer Productivity: Evidence from GitHub Copilot. arXiv:2302.06590 [cs.SE] https://arxiv.org/abs/2302.06590

    2023 arXiv

  14. [22]

    and Phoronix Media. 2010. PyBench Benchmark. https://openbenchmarking.org/ test/pts/pybench. Accessed: 2024-06-20

    2010

  15. [23]

    Meta AI Research. 2018. Papers with Code. https://paperswithcode.com/. Ac- cessed: 2024-06-24

    2018

  16. [24]

    Astrid Schepman and Paul Rodway. 2020. Initial validation of the general attitudes towards Artificial Intelligence Scale. Computers in Human Behavior Reports 1 (2020), 100014. https://doi.org/10.1016/j.chbr.2020.100014

    2020

  17. [25]

    Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, and et al

  18. [26]

    Catherine Thorbecke. 2023. A year after ChatGPT’s release, the AI revolu- tion is just beginning. https://www.cnn.com/2023/11/30/tech/chatgpt-openai- revolution-one-year/index.html. Accessed: 2024-06-24

    2023

  19. [27]

    arXiv:2206.04615 [cs.CL] https://arxiv.org/abs/2206.04615

    Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models. arXiv:2206.04615 [cs.CL] https://arxiv.org/abs/2206.04615

    arXiv

  20. [28]

    Yanlin Wang, Wanjun Zhong, Yanxian Huang, Ensheng Shi, Min Yang, Jiachi Chen, Hui Li, Yuchi Ma, Qianxiang Wang, and Zibin Zheng. 2024. Agents in Software Engineering: Survey, Landscape, and Vision. arXiv:2409.09030 [cs.SE] https://arxiv.org/abs/2409.09030

    2024 arXiv

  21. [29]

    Glassman

    Priyan Vaithilingam, Tianyi Zhang, and Elena L. Glassman. 2022. Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models. In Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, ...

    2022

  22. [30]

    Beiqi Zhang, Peng Liang, Xiyu Zhou, Aakash Ahmad, and Muhammad Waseem

  23. [31]

    Gomez, Łukasz Kaiser, and Illia Polosukhin

    Ashish Waswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems . NeurIPS Pro- ceedings, Online, 11. https://proceedings.neurips....

    2017

  24. [33]

    In Proceedings of the 35th International Conference on Software Engi- neering and Knowledge Engineering (SEKE2023, Vol

    Practices and Challenges of Using GitHub Copilot: An Empirical Study. In Proceedings of the 35th International Conference on Software Engi- neering and Knowledge Engineering (SEKE2023, Vol. 2023) . KSI Research Inc., url=http://dx.doi.org/10.18293/SEKE2023-077, 124–129. https:...

    2023 doi

  25. [34]

    No Unlikely to be differentiating within study population What do you use AI for?

  26. [35]

    No Developers might not know what features are powered by AI How frequently do you use the follow- ing features of the existing AI assis- tants for coding?

  27. [36]

    No Developers might not know what features are powered by AI In which parts of your development workflow are you currently using AI tools and which are you interested in using AI tools for over the next year?

  28. [37]

    Table 3: Questions about openness to AI tools Question Source Cog tested? Rationale How favorable is your stance on using AI tools as part of your development workflow?

    Yes Broad swath of workflow areas CHI EA ’25, April 26-May 1, 2025, Yokohama, Japan Paradis et al. Table 3: Questions about openness to AI tools Question Source Cog tested? Rationale How favorable is your stance on using AI tools as part of your development workflow?

    2025

  29. [38]

    No Biased towards the positive and unlikely to be differentiating within study population How likely is it that you would delegate the following activ- ities to an AI assistant (in an ideal world where the perfor- mance of an AI assistant is humanlike)?

  30. [39]

    I have security concerns about using AI generation services

    Yes Signals openness to AI delegation across workflow areas Would you like to use more AI as part of your job? [4] No Biased towards the positive Do you agree with the following statements? e.g. “I have security concerns about using AI generation services”, “I am ready to use ...

  31. [40]

    No Biased towards the positive and many items are irrelevant in local context Table 4: Questions about the impact of AI tools Question Source Cog tested? Rationale For the AI tools you use as part of your development work- flow, what are the MOST important benefits you are hop...

  32. [41]

    No Biased towards the positive, nei- ther focused on experiences or attitudes How similar or different is your current development work- flow compared before you started using AI tools?

  33. [42]

    Yes Straightforward assessment of impact, especially when followed by open-ended question Do you think that AI-assisted tools will help reduce your workload?

  34. [43]

    No Biased towards the positive, hypo- thetical and future-oriented Table 5: Questions about the trust in accuracy of AI tools Question Source Cog tested? Rationale How much do you trust the accuracy of the output from AI tools as part of your development workflow?

  35. [44]

    Yes Explicit definition of where trust should be evaluated In the last 3 months, how much did you trust the quality of the output from AI powered tools as part of your develop- ment work?

  36. [45]

    Yes Same as above, plus time scoping Creating benchmarkable components to measure the quality of AI-enhanced developer tools CHI EA ’25, April 26-May 1, 2025, Yokohama, Japan Table 6: Questions about general attitudes towards technology and AI Question Source Cog tested? Ratio...

    2025

  37. [46]

    No Hypothetical and future-oriented Which option most closely describes your current outlook on generative AI?

  38. [47]

    Yes Broad range of emotions Affinity for technology scale (10 items) [8] No Unlikely to be differentiating within study population If ‘technophobia’ is defined as feeling discomfort about com- puters or any new technology, which of the following best describes you?

  39. [48]

    No Unlikely to be differentiating within study population General attitudes towards Artificial Intelligence scale (20 items)

  40. [49]

    No Unlikely to be differentiating within study population General attitudes towards Artificial Intelligence scale (8 items)

  41. [50]

    If you don’t have an opinion about this, please just say so

    No Unlikely to be differentiating within study population Please tell me whether you think each of the following will mostly help or mostly harm people in this country in the next 20 years. If you don’t have an opinion about this, please just say so

  42. [51]

    AI” and “no AI

    No Unlikely to be differentiating within study population Creating benchmarkable components to measure the quality of AI-enhanced developer tools CHI EA ’25, April 26-May 1, 2025, Yokohama, Japan B DEMOGRAPHICS & ATTITUDES QUESTIONNAIRES Table 7: Demographics questionnaire (Bo...

    2025

  43. [2023]

    Accessed: 2024-07-10

    https://www.jetbrains.com/lp/devecosystem-2023/. Accessed: 2024-07-10

    2023

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.