Pith. sign in

REVIEW 2 major objections 5 minor 48 references

How Do Programming Students Use Generative AI?

T0 review · 2 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Most students who used ChatGPT ended up asking it to solve the task

desk verdict Solid behavioral evidence that novices default to code-generation prompts once they hit a wall, but the abstract's productivity claim outruns the data. read the letter →

arxiv 2501.10091 v2 pith:Z27N232R submitted 2025-01-17 cs.HC cs.AIcs.CY

Christian Rahe , Walid Maalej This is my paper · ORCID
classification cs.HCcs.AIcs.CY
keywords generativeAIprogrammingeducationstudentbehaviorChatGPTcodecomprehensionover-reliancepromptstrategiesreuse
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper reports an experiment in which 37 first-semester programming students solved a code-comprehension exercise with monitored access to a ChatGPT-like chatbot. The task was designed so ChatGPT could not reliably solve it, forcing students to read and reason about code. Of the 23 students who used the chatbot, most eventually asked it to generate a full solution, and only a minority used it to understand concepts or their own mistakes. The authors argue this pattern supports educator concerns about over-reliance: students often copied generated code, submitted it, and on failure fed the error back to the bot in a repeat loop, without stepping back to evaluate the output.

What carries the argument

The experiment's load-bearing object is the comprehension task itself: a Java exercise about null dereferencing and short-circuit evaluation that the authors verified GPT-3.5 could not solve reliably, so any correct answer required the student to trace code and reason about object state. Around it, the study records every prompt, chatbot response, copy/paste event and submission; prompts are labelled as Codegen (Solve, Fix) or Support (Hint, Inform, Explain), and responses as Informs, Solves or Explains. Similarity between submissions and generated code is measured with the Ratcliff/Obershelp algorithm, and code diffs are manually labelled (Copy All, Idea, Syntax, Explanation, None). Sequence pattern mining then exposes the recurring loop of incorrect submission, code-generation prompt, paste, and another incorrect submission.

What would settle it

A replication with unobtrusive logging of students' own ChatGPT use during real homework, or an immediate post-task test showing that students who requested full solutions could still explain and modify the code, would contradict the paper's conclusion that unguided access leads to uncritical over-reliance.

Watch

Extended reading notes

Core claim

The central discovery is that when novices are free to consult a generative AI assistant, the default behaviour is delegation rather than comprehension. Most students first attempted the exercise on their own and did not ask for code immediately, but once they hit a difficulty, a majority shifted to requesting a complete solution; none asked the chatbot to explain why their own solution was failing. Comparing submissions to generated code shows the bot's output reshaped students' work: in 62% of cases with a generation in between, the modified submission was more similar to the generated code than to the previous submission, and 54.3% of such submissions were semantically identical to generated code. Students who self-reported regular GenAI use were more likely to send code-generation prompts, suggesting that experience with these tools strengthens rather than corrects the delegation habit.

Load-bearing premise

The study assumes that students' behaviour in a monitored lab session, where they knew their actions were recorded and participation was voluntary, matches how they actually use ChatGPT in their coursework.

Editorial extensions

If this is right

  • In courses where submissions are the only assessment, AI-generated code is nearly indistinguishable from a good student's work, so submission-based grading loses much of its diagnostic value.
  • Interview-like reviews and tutor code reviews become a more reliable check, because a student presenting generated code must be able to justify and correct it on the spot.
  • Teaching GenAI use explicitly, including its limitations and non-code strategies, may be more productive than banning it.
  • Students who already use GenAI regularly are more likely to delegate code generation, so early habits matter.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the observed cycle generalizes beyond the lab, it predicts that repeated ChatGPT use could train students to outsource the debugging step entirely, making it harder for them to detect when the tool is wrong.
  • A testable extension would compare retention: students who relied on generated code might score worse on a delayed, no-AI comprehension test than students who solved the exercise independently.
  • The similarity measure used here suggests a cheap, scalable detector: a submission whose distance to a prior bot response is smaller than its distance to the student's own previous submission would flag likely delegation.
  • The chatbot's apologetic and confident tone may have kept students in the error-submission loop, so an experiment with a less reassuring bot could test whether trust calibration alone reduces over-reliance.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. This paper investigates how programming students use generative AI tools in a monitored experimental setting. The authors first evaluate GPT-3.5 and GPT-4 on the exercises of an introductory programming course (RQ1), then conduct a controlled experiment in which 37 students solve a code comprehension and authoring exercise while having access to a ChatGPT-like chatbot (RQ2, RQ3). The task was deliberately designed to be unsolvable by GPT-3.5. The paper reports that among the 23 students who used the chatbot, a majority (73.9%) eventually submitted a Solve prompt, that 54.3% of submissions after chatbot interactions were semantically identical to generated code (Copy All), and that in 62% of cases with generations the modified submission was more similar to the generated code than to the previous submission. Students who reported regular GenAI use submitted relatively more Codegen prompts. The abstract concludes that concerns about decreased agency and productivity are justified. The paper discusses implications for assessment and education.

Significance. If the findings hold, the paper provides a valuable fine-grained behavioral account of how novices delegate coding work to a chatbot. Its strengths include the monitored experimental design with recorded code changes, clipboard events, and chat logs; the manual labeling of prompts and code changes with reported inter-rater agreement; the use of sequence pattern mining to identify interaction patterns; and the public availability of the data. The 'vicious cycle' of submitting incorrect generated code and asking for fixes is a concrete, observable phenomenon that extends prior work on novice-Copilot interaction. However, the productivity generalization in the abstract is not supported by the paper's own data, and the regularity-usage correlation is marginal. The core delegation findings are robust enough to merit publication, but the overreach must be corrected.

major comments (2)
  1. [Abstract and Section 3.2.2] The abstract and Section 6 conclude that 'concerns about potential decrease in programmers' agency and productivity with Generative AI are justified.' The agency part is supported by the delegation evidence (54.3% Copy All, 62% similarity), but the productivity part is not. The only productivity-relevant measure is session duration, which was significantly longer for chatbot users overall (26:15 vs 14:36, t=3.3, p=0.003), yet Section 3.2.2 states that 'when comparing the duration only among completed attempts, the difference is not significant (t=1.9, p>0.05).' The longer duration is driven by participants who failed, and the failure loop is a direct consequence of the experimental task being engineered to be unsolvable by GPT-3.5 (Section 2.2.2; only 8.8% of 80 generated P1 solutions were correct in Section 3.2.4). The paper's own RQ1 results show that GPT-4 solves the large majority of course exercises, so in ordinary coursework the 'vicious cycle' would not occur and students might complete tasks faster, not slower. The caveat in Section 4.3.2 that only a single complex exercise was tested does not repair the abstract's unconditional productivity claim.
  2. [Section 3.2.3 / Table 4] The abstract states that 'those who self-reported using generative AI regularly were more likely to prompt the bot to generate a solution.' This is based on the category-level correlation between GenAI usage and Codegen vs. Support prompts (Kendall's tau=0.45, p=0.03 in Table 4). However, the individual Solve prompt correlation is not significant (tau=0.12, p=0.58), and the Fix correlation (tau=0.42, p=0.05) is not significant after the paper's own Bonferroni correction (alpha=0.01). The category-level result is thus driven by the combination of labels, and with multiple comparisons the evidence is thin. The conclusion in Section 4.1.1 that 'as students self-learn and gain experience with LLMs, they become more reliant on its code output' goes beyond what this correlation supports. Please temper the claim or present the category-level analysis with a clear multiplicity caveat.
minor comments (5)
  1. [Figure 10] The two panels in Figure 10 appear to be duplicated in the rendering, which makes it hard to compare the left and right results; please ensure the figure legend and panels are visually distinct.
  2. [Table 1] The example for the Solve label, '[question text] How do I produce the NullPointerException here?', reads more like a question than a direct request for a full solution; consider aligning the example with the label definition.
  3. [Section 3.2.2] The sentence 'On average, each participant submitted 4.57 prompts (M = 2, σ = 5.96)' is confusing because 'on average' and 'M' are redundant; clarify that M is the median.
  4. [Section 3.3.1] The 90% similarity threshold is justified by manual review, but it would strengthen the paper to report the threshold's sensitivity (e.g., prevalence of close matches at 80% and 95%) given that the sequence-mining results depend on it.
  5. [Section 4.2.1] The anecdote about participant GCL includes direct quotes from the chat log; consider adding a note about consent for quoting individual participant messages.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the findings are empirical observations from a transparently designed experiment, not derivations from assumptions that already contain the conclusions.

full rationale

This paper is an empirical study, not a derivation, and I found no step in which a claimed result reduces to its own inputs by construction. RQ1 evaluates GPT-3.5 and GPT-4 against externally defined tutor criteria for course exercises, which is an independent benchmark. RQ2 and RQ3 analyze observed student behavior: recorded chat logs, clipboard events, submission attempts, code diffs, and manually labeled prompt and response categories. The headline behavioral claims (majority of chatbot users eventually request a full solution; 62% of modified submissions are more similar to generated code than to the prior submission; frequent incorrect-submission/fix-prompt cycles) are measured outcomes, not fitted parameters renamed as predictions. The paper deliberately selected a task that GPT-3.5 could not reliably solve, and it states this design choice explicitly, including that it was confirmed in initial testing. This means the observed failure loops are partly a consequence of the task design, but that is a transparent experimental manipulation, not a circular argument: the paper does not use the inability of the model as both an assumption and a conclusion. The generalization about decreased productivity is vulnerable to the critique that the longer duration for chatbot users was not significant among successful completers and that the task was adversarial, but that is an external-validity or correctness concern, not circularity. The paper's self-citations (e.g., Häring and Maalej 2019 for course structure, Kruse et al. 2024 and Wei et al. 2024 for related observations) are contextual and not load-bearing for the central empirical claims. No uniqueness theorem or imported ansatz is invoked to force the conclusions. Therefore the appropriate finding is no significant circularity.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claims rest on a handful of behavioral and measurement assumptions. No formal axioms are introduced; the free parameters are hand-chosen thresholds in the analysis pipeline, and the domain assumptions are standard for empirical work of this kind and are explicitly acknowledged in the paper's limitations.

free parameters (3)
  • similarity threshold for close match = 0.90
    Hand-chosen after manual review of submissions; used to classify submissions as close matches to generated code (Section 3.3.1). Central to RQ3 claims about code reuse.
  • sequence pattern mining minimum support = 10
    Chosen for the NOSEP algorithm; determines which interaction subsequences are reported as common (Section 2.1.2, Section 3.3.3).
  • sequence pattern mining gap range = [0, 2]
    Set to allow minor tolerance in pattern mining; affects which patterns are detected (Section 2.1.2).
assumptions (5)
  • domain assumption Monitored lab behavior reflects natural ChatGPT use
    The paper creates a controlled environment to elicit typical behavior but acknowledges that participants may be influenced by being recorded (Section 4.3.1, 4.3.2). If students behave differently when observed, the central claims about how students actually use GenAI are weakened.
  • domain assumption Self-reported GenAI usage is sufficiently accurate
    The correlation between reported GenAI use and code-generation prompting relies on honest self-report; the paper notes students may under-report due to perceived cheating stigma (Section 4.3.1).
  • domain assumption Ratcliff/Obershelp similarity is a valid proxy for code reuse
    Used to quantify similarity between submissions and generated code; the 90% threshold is manually validated, but the measure is a heuristic (Section 2.1.2, 3.3.1).
  • domain assumption The exercise is representative of introductory programming tasks
    The study uses a single custom exercise about null handling; the authors note that propensity to use GenAI varies by topic (Section 4.3.2). Generalization to other tasks is assumed.
  • domain assumption GPT-3.5 (gpt-3.5-turbo-1106) represents the ChatGPT experience students would have
    The experiment used a specific model version; behavior may differ with GPT-4 or later models, as the authors note for the task-solving evaluation (Section 4.3.1).

how reviews work

0 comments
Cite this review

Pith. "Pith review of How Do Programming Students Use Generative AI?." pith.science (2026). https://pith.science/paper/Z27N232R

@misc{pith2026250110091,
  author       = {Pith},
  title        = {Pith review of: How Do Programming Students Use Generative AI?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Z27N232R}},
  note         = {Machine review of arXiv:2501.10091}
}
read the original abstract

Programming students have a widespread access to powerful Generative AI tools like ChatGPT. While this can help understand the learning material and assist with exercises, educators are voicing more and more concerns about an overreliance on generated outputs and lack of critical thinking skills. It is thus important to understand how students actually use generative AI and what impact this could have on their learning behavior. To this end, we conducted a study including an exploratory experiment with 37 programming students, giving them monitored access to ChatGPT while solving a code authoring exercise. The task was not directly solvable by ChatGPT and required code comprehension and reasoning. While only 23 of the students actually opted to use the chatbot, the majority of those eventually prompted it to simply generate a full solution. We observed two prevalent usage strategies: to seek knowledge about general concepts and to directly generate solutions. Instead of using the bot to comprehend the code and their own mistakes, students often got trapped in a vicious cycle of submitting wrong generated code and then asking the bot for a fix. Those who self-reported using generative AI regularly were more likely to prompt the bot to generate a solution. Our findings indicate that concerns about potential decrease in programmers' agency and productivity with Generative AI are justified. We discuss how researchers and educators can respond to the potential risk of students uncritically over-relying on Generative AI. We also discuss potential modifications to our study design for large-scale replications.

Figures

Figures reproduced from arXiv: 2501.10091 by the authors.

Figure 1
Figure 1. The chatbot UI participants were shown during the study. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. The exercise text, the provided source code (left) and the solution (right) to problem P1. The source [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Performance in coding exercises. GPT3.5 Learn Facts Reqs Answers GPT4 0% 50% 100% Learn Facts Reqs Answers correct incorrect undetermined [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: Total number of prompts by type (top) and [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 8
Figure 8. Figure 8: Submission attempts (left) and elapsed time [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]
Figure 9
Figure 9. Figure 9: Interaction logs for the entire session of participant [XKW]. [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

48 extracted references · 10 canonical work pages

  1. [2]

    In 2023 IEEE 35th International Conference on Software Engineering Education and Training (CSEE&T)

    Recommendations to create programming exercises to overcome ChatGPT. In 2023 IEEE 35th International Conference on Software Engineering Education and Training (CSEE&T) . IEEE, 147–151. doi:10.1109/CSEET58097.2023.00031 Boxi Cao, Hongyu Lin, Xianpei Han, Le Sun, Lingyong Yan, Meng Liao, Tong Xue, and Jin Xu

    work page arXiv 2023

  2. [4]

    2307.09009 arXiv preprint

    How is ChatGPT’s behavior changing over time? doi:10.48550/arXiv. 2307.09009 arXiv preprint. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke ...

    arXiv doi:10.48550/arxiv.2307.09009

  3. [5]

    doi:10.48550/arXiv.2107.03374 arXiv preprint

    Evaluating Large Language Models Trained on Code. doi:10.48550/arXiv.2107.03374 arXiv preprint. Debby R. E. Cotton, Peter A. Cotton, and J. Reuben Shipway

    arXiv doi:10.48550/arxiv.2107.03374

  4. [7]

    doi:10.1016/j.jss.2023.111734 Paul Denny, Viraj Kumar, and Nasser Giacaman

    Github copilot ai pair programmer: Asset or liability? Journal of Systems and Software 203 (2023), 111734. doi:10.1016/j.jss.2023.111734 Paul Denny, Viraj Kumar, and Nasser Giacaman

    arXiv 2023

  5. [8]

    In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V

    Conversing with Copilot: Exploring Prompt Engineering for Solving CS1 Problems Using Natural Language. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 (Toronto, Canada) (SIGCSE 2023). Association for Computing Machinery, New York, NY, USA, 1136–1142. doi:10.1145/3545945.3569823 Mary T Dzindolet, Scott A Peterson, Regi...

    arXiv 2023

  6. [11]

    In Proceedings of the 25th Australasian Computing Education Conference (Melbourne, Australia) (ACE ’23)

    My AI Wants to Know If This Will Be on the Exam: Testing OpenAI’s Codex on CS2 Programming Exercises. In Proceedings of the 25th Australasian Computing Education Conference (Melbourne, Australia) (ACE ’23). Association for Computing Machinery, New York, NY, USA, 97–104. doi:10.1145/3576123.3576134 GitHub

    work page arXiv

  7. [12]

    https://web.archive.org/web/20231203005848/https: //github.com/features/copilot/)

    https://github.com/features/copilot/ (archived 2023-12-03. https://web.archive.org/web/20231203005848/https: //github.com/features/copilot/). Accessed 2023-12-03. Marlo Häring and Walid Maalej

    arXiv 2023

  8. [14]

    In 2024 IEEE/ACM 21st International Conference on Mining Proc

    Can ChatGPT Support Developers? An Empirical Evaluation of Large Language Models for Code Generation. In 2024 IEEE/ACM 21st International Conference on Mining Proc. ACM Softw. Eng., Vol. 2, No. FSE, Article FSE045. Publication date: July

    work page 2024

Show all 48 references
  1. [15]

    Applied Sciences 14, 10 (2024),

    The Impact of Large Language Models on Programming Education and Student Learning Outcomes. Applied Sciences 14, 10 (2024),

    2024

  2. [17]

    In Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI ’24)

    Is Stack Overflow Obsolete? An Empirical Study of the Characteristics of ChatGPT Answers to Stack Overflow Questions. In Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI ’24) . ACM, 1–17. doi:10.1145/3613904.3642596 Enkelejda Kasneci, Kathrin Seßler...

  3. [18]

    Learning and individual differences 103 (2023), 102274

    ChatGPT for good? On opportunities and challenges of large language models for education. Learning and individual differences 103 (2023), 102274. doi:10.1016/j.lindif.2023.102274 Majeed Kazemitabaar, Justin Chow, Carl Ka To Ma, Barbara J. Ericson, David Weintrop, and Tovi Grossman

    2023

  4. [19]

    In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI ’23)

    Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI ’23). Association for Computing Machinery, New York, NY, USA, Article 45...

    2023

  5. [20]

    In 40th International Conference on Software Maintenance and Evolution (ICSME)

    Can Developers Prompt? A Controlled Experiment for Code Documentation Generation. In 40th International Conference on Software Maintenance and Evolution (ICSME) . 574–586. doi:10.1109/ICSME58944.2024.00058 Viet Dac Lai, Nghia Trung Ngo, Amir Pouran Ben Veyseh, Hieu Man, Franck...

    2024

  6. [21]

    doi:10.48550/arXiv.2304.05613 arXiv preprint

    ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning. doi:10.48550/arXiv.2304.05613 arXiv preprint. Sam Lau and Philip Guo

    doi arXiv

  7. [24]

    In Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V

    Comparing Code Explanations Created by Students and Large Language Models. In Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 (Turku, Finland) (ITiCSE 2023). Association for Computing Machinery, New York, NY, USA, 124–130. doi...

    2023

  8. [28]

    In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V

    Experiences from using code explanations generated by large language models in a web software development e-book. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 . 931–937. doi:10.1145/3545945. 3569785 Stephen MacNeil, Andrew Tran, Dan Mog...

    doi

  9. [29]

    InProceedings of the 2022 ACM Conference on International Computing Education Research-Volume

    Generating diverse code explanations using the GPT-3 large language model. InProceedings of the 2022 ACM Conference on International Computing Education Research-Volume

    2022

  10. [30]

    doi:10.1145/3501709.3544280 Alina Mailach, Dominik Gorgosch, Norbert Siegmund, and Janet Siegmund

    37–39. doi:10.1145/3501709.3544280 Alina Mailach, Dominik Gorgosch, Norbert Siegmund, and Janet Siegmund

  11. [31]

    Ok Pal, we have to code that now

    “Ok Pal, we have to code that now”: interaction patterns of programming beginners with a conversational chatbot. Empirical Software Engineering 30, 1 (2025),

    2025

  12. [34]

    Neuendorf

    doi:10.1007/s10664-024-10561-6 Kimberly A. Neuendorf

    doi

  13. [35]

    https://openai.com/blog/chatgpt (archived 2023-11-21: https://web.archive.org/web/ 20231121185200/https://openai.com/blog/chatgpt/)

    Introducing ChatGPT. https://openai.com/blog/chatgpt (archived 2023-11-21: https://web.archive.org/web/ 20231121185200/https://openai.com/blog/chatgpt/). Accessed 2023-11-22. Yen Dieu Pham, Abir Bouraffa, Marleen Hillen, and Walid Maalej

    2023

  14. [36]

    In 2021 IEEE 29th International Requirements Engineering Conference (RE)

    The Role of Linguistic Relativity on the Identification of Sustainability Requirements: An Empirical Study. In 2021 IEEE 29th International Requirements Engineering Conference (RE). 117–127. doi:10.1109/RE51729.2021.00018 Matei-Dan Popovici

    2021

  15. [37]

    Exploring Its Potential and Limitations in a Functional Programming Course

    ChatGPT in the Classroom. Exploring Its Potential and Limitations in a Functional Programming Course. International Journal of Human–Computer Interaction (2023), 1–12. doi:10.1080/10447318.2023.2269006 James Prather, Paul Denny, Juho Leinonen, Brett A. Becker, Ibrahim Albluwi,...

    2023

  16. [38]

    It’s Weird That it Knows What I Want

    How Do Programming Students Use Generative AI? FSE045:23 Savelka. 2023a. The Robots Are Here: Navigating the Generative AI Revolution in Computing Education. InProceedings of the 2023 Working Group Reports on Innovation and Technology in Computer Science Education(Turku, Finla...

    2023

  17. [39]

    In Proceedings of the 2024 ACM Conference on International Computing Education Research - Volume 1 (Melbourne, VIC, Australia) (ICER ’24)

    The Widening Gap: The Benefits and Harms of Generative AI for Novice Programmers. In Proceedings of the 2024 ACM Conference on International Computing Education Research - Volume 1 (Melbourne, VIC, Australia) (ICER ’24). Association for Computing Machinery, New York, NY, USA, ...

    2024

  18. [40]

    Jaromir Savelka, Arav Agarwal, Marshall An, Chris Bogart, and Majd Sakr

    What is it like to program with artificial intelligence? arXiv preprint arXiv:2208.06213 (2022). Jaromir Savelka, Arav Agarwal, Marshall An, Chris Bogart, and Majd Sakr

    2022 arXiv

  19. [41]

    In Proceedings of the 2023 ACM Conference on International Computing Education Research - Volume 1 (Chicago, IL, USA) (ICER ’23)

    Thrilled by Your Progress! Large Language Models (GPT-4) No Longer Struggle to Pass Assessments in Higher Education Programming Courses. In Proceedings of the 2023 ACM Conference on International Computing Education Research - Volume 1 (Chicago, IL, USA) (ICER ’23). Associatio...

    2023

  20. [43]

    arXiv preprint (2024)

    Debugbench: Evaluating debugging capability of large language models. arXiv preprint (2024). doi:10.48550/arXiv.2401.04621 Annapurna Vadaparty, Daniel Zingaro, David H. Smith IV, Mounika Padala, Christine Alvarado, Jamie Gorson Benario, and Leo Porter

    2024 doi arXiv

  21. [44]

    In Proceedings of the 2024 on Innovation and Technology in Computer Science Education V

    CS1-LLM: Integrating LLMs into CS1 Instruction. In Proceedings of the 2024 on Innovation and Technology in Computer Science Education V. 1 (Milan, Italy) (ITiCSE 2024). Association for Computing Machinery, New York, NY, USA, 297–303. doi:10.1145/3649217.3653584 Priyan Vaithili...

    2024

  22. [45]

    Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models

    Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models. In Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI EA ’22). Association for Computing Machinery, New ...

    2022

  23. [46]

    doi:10.48550/arXiv.2406.13631 arXiv:2406.13631 [cs.HC] Youxi Wu, Cong Shen, He Jiang, and Xindong Wu

    On AI-Inspired UI-Design. doi:10.48550/arXiv.2406.13631 arXiv:2406.13631 [cs.HC] Youxi Wu, Cong Shen, He Jiang, and Xindong Wu. 2017a. Strict pattern matching under non-overlapping condition. Science China. Information Sciences 60, 1 (2017), 012101. Youxi Wu, Yao Tong, Xingqua...

    2017 doi arXiv

  24. [47]

    In 2024 IEEE/ACM 21st International Conference on Mining Software Repositories (MSR)

    DevGPT: Studying Developer-ChatGPT Conver- sations. In 2024 IEEE/ACM 21st International Conference on Mining Software Repositories (MSR) . IEEE, 227–230. Yuankai Xue, Hanlin Chen, Gina R. Bai, Robert Tairas, and Yu Huang

    2024

  25. [48]

    In Proceedings of the 46th International Conference on Software Engineering: Software Engineering Education and Training (Lisbon, Portugal) (ICSE-SEET ’24)

    Does ChatGPT Help With Introductory Programming?An Experiment of Students Using ChatGPT in CS1. In Proceedings of the 46th International Conference on Software Engineering: Software Engineering Education and Training (Lisbon, Portugal) (ICSE-SEET ’24). Association for Computin...

  26. [49]

    In 2023 IEEE Frontiers in Education Conference (FIE)

    Generative AI in Computing Education: Perspectives of Students and Instructors. In 2023 IEEE Frontiers in Education Conference (FIE) . 1–9. doi:10.1109/FIE58773.2023.10343467 Albert Ziegler, Eirini Kalliamvakou, X. Alice Li, Andrew Rice, Devon Rifkin, Shawn Simister, Ganesh Si...

    2023

  27. [50]

    Measuring GitHub Copilot’s Impact on Productivity. Commun. ACM 67, 3 (Feb. 2024), 54–63. doi:10.1145/3633453 Received 2024-09-13; accepted 2025-01-14 Proc. ACM Softw. Eng., Vol. 2, No. FSE, Article FSE045. Publication date: July 2025

    2024 doi

  28. [1988]

    Gestalt: an introduction to the Ratcliff/Obershelp pattern matching algorithm. Dr. Dobbs Journal 7 (1988),

    1988

  29. [2003]

    International journal of human-computer studies 58, 6 (2003), 697–718

    The role of trust in automation reliance. International journal of human-computer studies 58, 6 (2003), 697–718. doi:10.1016/S1071-5819(03)00038-7 James Finnie-Ansley, Paul Denny, Brett A. Becker, Andrew Luxton-Reilly, and James Prather

    2003 doi

  30. [2013]

    IEEE Transactions on software Engineering 39, 9 (2013), 1264–1282

    Patterns of knowledge in API reference documentation. IEEE Transactions on software Engineering 39, 9 (2013), 1264–1282. doi:10.1109/TSE.2013.12 Walid Maalej, Rebecca Tiarks, Tobias Roehm, and Rainer Koschke

    2013 doi

  31. [2014]

    ACM Trans

    On the Comprehension of Program Comprehension. ACM Trans. Softw. Eng. Methodol. 23, 4, Article 31 (Sept. 2014), 37 pages. doi:10.1145/2622669 Stephen MacNeil, Andrew Tran, Arto Hellas, Joanne Kim, Sami Sarsa, Paul Denny, Seth Bernstein, and Juho Leinonen

    2014 doi

  32. [2016]

    ACM Inroads 7, 1 (feb 2016), 47–51

    Coderunner: A Tool for Assessing Computer Programming Skills. ACM Inroads 7, 1 (feb 2016), 47–51. doi:10.1145/2810041 Walid Maalej and Martin P Robillard

    2016 doi

  33. [2017]

    Thousand Oaks, California

    The Content Analysis Guidebook. Thousand Oaks, California. doi:10.4135/9781071802878 OpenAI

    doi

  34. [2018]

    In 2018 IEEE International Conference on Software Maintenance and Evolution (ICSME)

    A Simple NLP-Based Approach to Support Onboarding and Retention in Open Source Communities. In 2018 IEEE International Conference on Software Maintenance and Evolution (ICSME) . 172–182. doi:10.1109/ICSME.2018.00027 Runchu Tian, Yining Ye, Yujia Qin, Xin Cong, Yankai Lin, Zhiy...

    2018

  35. [2021]

    arXiv preprint (2021)

    Knowledgeable or educated guess? revisiting language models as knowledge bases. arXiv preprint (2021). doi:10.48550/arXiv.2106.09231 Lingjiao Chen, Matei Zaharia, and James Zou

    2021 doi arXiv

  36. [2022]

    In Proceedings of the 24th Australasian Computing Education Conference (Virtual Event, Australia) (ACE ’22)

    The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming. In Proceedings of the 24th Australasian Computing Education Conference (Virtual Event, Australia) (ACE ’22). Association for Computing Machinery, New York, NY, USA, 10–19. doi:10.114...

  37. [2023]

    InProceedings of the 54th ACM Technical Symposium on Computer Science Education V

    Programming is hard-or at least it used to be: Educational opportunities and challenges of ai code generation. InProceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 . 500–506. doi:10.1145/3545945.3569759 Jonnathan Berrezueta-Guzman and Stephan Krusche

  38. [2024]

    Innovations in Education and Teaching International 61, 2 (2024), 228–239

    Chatting and cheating: Ensuring academic integrity in the era of ChatGPT. Innovations in Education and Teaching International 61, 2 (2024), 228–239. doi:10.1080/14703297.2023. 2190148 Arghavan Moradi Dakhel, Vahid Majdinasab, Amin Nikanjam, Foutse Khomh, Michel C Desmarais, an...

    2024

  39. [2025]

    In CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’25)

    The Impact of Generative AI on Critical Thinking: Self-Reported Reductions in Cognitive Effort and Confidence Effects From a Survey of Knowledge Workers. In CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’25). Association for Computing Machinery. d...

  40. [4115]

    Udo-Imeh, Bonan Kou, and Tianyi Zhang

    doi:10.3390/app14104115 Samia Kabir, David N. Udo-Imeh, Bonan Kou, and Tianyi Zhang

    doi

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.