Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

"I Would Have Written My Code Differently'': Beginners Struggle to Understand LLM-Generated Code

T0 review · 4 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Beginning programmers succeeded on only 32.5% of tasks that required predicting what LLM-generated code would do.

desk verdict Useful measurement of CS1 students' poor comprehension of LLM-generated code, but the LLM-specific claim needs a human-written control and the prompt-code decoupling muddies the conditional analysis. read the letter →

arxiv 2504.19037 v1 pith:UOUIHXDP submitted 2025-04-26 cs.SE cs.HC

Yangtian Zi , Luisa Li , Arjun Guha , Carolyn Jane Anderson , Molly Q Feldman This is my paper · ORCID
classification cs.SEcs.HC
keywords LLM-generatedcodecomprehensionCS1beginningprogrammersautomationbiasoutputpredictionPythonicsyntaxcomputingeducation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether beginning programmers can read and evaluate code produced by a large language model, and its answer is that mostly they cannot. In a lab study of 32 students who had completed one introductory Python course, across 160 tasks participants predicted the behavior of LLM-generated code correctly only 32.5% of the time, versus 59.4% for the same problems described in natural language. The difficulty persisted even when students understood the problem description, and it was worse for buggy code. If correct, this is a practical problem for putting LLMs in beginners' hands: generating code is only useful if the user can judge whether the output does what is intended.

What carries the argument

The measuring instrument is a three-part task: students first read a natural-language problem description and predict its output for three inputs, then read an LLM-generated function for the same problem and predict its actual output for the same three inputs, then review the ground truth and reflect. Because both parts use identical inputs, the design isolates the act of reading code from the act of understanding the problem; the per-part gap between prompt comprehension and code comprehension is the quantitative evidence. Code variations were systematically selected to differ in correctness, comment presence, and complexity, and prompts were filtered by a high pass@1 rate so that prompt quality would not confound the comparison.

What would settle it

Run the same output-prediction tasks with human-written Python functions matched for length, style, and bug rate; if beginners succeed no more often than with LLM-generated functions, the central claim shifts from 'LLM code is hard' to 'code reading is hard.'

Watch

Extended reading notes

Core claim

The paper claims that CS1 students, after one semester of Python, can predict what an LLM-generated function will do only 32.5% of the time across 160 tasks, compared with 59.4% for predicting what the problem description asks for. Even when the prompt was understood, code comprehension failed more often than it succeeded (failure rate 0.58 versus success 0.42). Students did significantly worse on code with bugs, and qualitative responses attribute the difficulty to unfamiliar Pythonic syntax and idioms, unknown library functions, and a tendency to assume generated code is correct. The paper concludes that reading and evaluating generated code, not just prompting, is a major barrier for beginners who want to use LLMs.

Load-bearing premise

The claim that LLM-generated code is specifically hard to understand rests on the assumption that human-written code would have been easier for these students; the study never includes a human-written code condition, so the difficulty could be about reading code in general.

Editorial extensions

If this is right

  • If students cannot reliably predict what generated code does, accepting LLM output is effectively blind, so CS1 courses that adopt LLMs need to teach code reading and tracing alongside or before writing.
  • The strong association between automation bias and wrong predictions implies that novice review of generated code must be scaffolded with explicit checks, such as running test cases and comparing outputs.
  • Unfamiliar Python idioms such as list comprehensions and slicing account for a large share of reported difficulty, so curricula that expect students to read LLM output need to cover those idioms earlier.
  • Non-native English speakers' disadvantage at the prompt stage means LLM-assisted education has an equity dimension: unclear natural-language descriptions compound code-comprehension failure.
  • Because prompt-comprehension failure makes code-comprehension failure much more likely, a student who only half-understands the task is at particular risk of accepting incorrect generated solutions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the study lacks a human-written-code control, the LLM-specific portion of the difficulty is not yet measured; a matched human-code condition would separate the effect of generation from the general difficulty of reading code.
  • The dataset deliberately includes buggy and heavily idiomatic variations, so the 32.5% figure likely reflects a stress test rather than typical everyday LLM output; real tools could score higher or lower depending on the model and prompt.
  • The three-part output-prediction format is a reusable instrument: a training intervention could use it to test whether teaching tracing or Python idioms shrinks the prompt-to-code comprehension gap.
  • The paper's evidence points to a concrete design target for LLM tools aimed at novices: surfacing generated-code behavior through examples or execution traces, not just the code itself, is worth testing as a mitigation.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper investigates whether CS1 students can understand code generated by an LLM. It reports a lab study with 32 participants who each completed five tasks (160 task instances total). Each task presents a student-written natural-language problem description and an LLM-generated Python function for the same problem; participants predict the outputs of both the described function and the shown code for three test cases, then reflect on their answers. The central quantitative result is that code comprehension succeeds in only 32.5% of tasks, versus 59.4% for prompt comprehension, and that participants often fail code comprehension even when they succeed at prompt comprehension (0.58 failure vs 0.42 success in the conditional analysis). Qualitative analysis identifies unfamiliar Python syntax, unfamiliar code style, library-function knowledge gaps, and automation bias as key challenges. The paper concludes that beginners face substantial barriers in reading, evaluating, and integrating LLM-generated code, and it recommends LLM-aware teaching strategies and tool design changes.

Significance. If the core result holds, the paper addresses an important gap in the computing-education literature: prior work on CS students and LLMs has concentrated on prompt writing and task completion, while the ability to read and evaluate generated code is comparatively understudied. The study has several strengths: it reuses a validated dataset of CS1 problems and high-quality prompts from Nguyen et al., uses a structured output-prediction design rather than self-report, includes a pilot study, reports inter-rater reliability (Cohen's kappa 0.77), and makes data and materials publicly available. The qualitative participant quotes give concrete, believable evidence of the reported mechanisms. However, the significance is conditional on two central assumptions: that the observed difficulty is specific to LLM-generated code rather than to code comprehension in general, and that the prompt shown in Part A and the code shown in Part B are sufficiently aligned that prompt comprehension should predict code comprehension. Both assumptions are currently unsupported by the experimental design, so the headline comparison must be treated with caution.

major comments (4)
  1. [§3.2 and §5.1] The central comparison in §5.1—code comprehension success 52/160 (32.5%) versus prompt comprehension success 95/160 (59.4%)—does not support the paper's conclusion that beginners struggle specifically with LLM-generated code. No human-written code condition is included, so the result could reflect general difficulty with reading code, rather than any property of LLM generation. This concern is amplified by the dataset construction in §3.2, where the eight code variations were deliberately selected to include bugs and unfamiliar idioms such as list comprehensions and slicing. To attribute the gap to LLM generation, the study would need a control condition with equivalent human-written functions, or the claims should be reframed to describe comprehension of code of the kind LLMs produce, without implying a comparison to human-written code.
  2. [§3.2 and §5.1] The dataset construction creates a prompt-code mismatch that confounds the key conditional analysis. Section 3.2 states that the code 'may not directly stem from the prompt' and that this is by design, with prompts serving only as contextual framing. The Part A prompt-comprehension task and the Part B code-comprehension task therefore do not necessarily describe the same function. The claim in §5.1 that 'even when participants understand the prompt, their failure rate in code comprehension exceeds their success rate' presupposes that understanding the prompt should help predict what the shown code does. If the code was generated from a different prompt that resolves the problem differently, a participant who fully understands the shown prompt may still be unable to predict the code's outputs. This confound also weakens the automation-bias interpretation in §5.3, because entering the prompt's expected output as the code's actual output could reflect a genuine mismatch between the prompt and code rather than blind trust in the model. The authors should use matched prompt-code pairs, or analyze only instances where the prompt used for generation is shown to participants, or at minimum measure and report the semantic overlap between the shown prompt and the prompt that generated the code.
  3. [§4 and §5.1–5.2] The statistical analyses treat the 160 data points as independent, but the design is repeated-measures: each of the 32 participants contributes five correlated tasks, and each of the 40 problem-code pairings is reused across participants. The chi-squared tests, t-tests, and ANOVA in §5.1 and §5.2 therefore risk pseudoreplication and inflated significance levels. The authors should use mixed-effects models with random intercepts for participant and for problem/code variation, or at minimum cluster-standard errors by both participant and task. In addition, the paper reports p-values without effect sizes or confidence intervals; these should be added, especially for the headline 32.5% versus 59.4% comparison and for the conditional failure rate of 0.58.
  4. [§5.2 and Abstract] The abstract states that the study shows 'indiscriminate struggles across demographic populations,' but this overstates what §5.2 and Appendix Table 2 support. The only statistically significant demographic effect is childhood language on prompt comprehension; the paper correctly notes that other demographic comparisons showed no significant differences, but also notes that effects 'may exist but were undetected due to sample size and distribution.' Several subgroup sizes are extremely small (e.g., n=2 for Black participants), so a null result is not evidence of the absence of differences. The wording should be softened to 'no demographic differences were detected' or accompanied by a power analysis showing which effect sizes the design could have detected.
minor comments (4)
  1. [§2] The sentence 'with an mean eventual success rate of only 57%' contains a grammatical error: 'an mean' should be 'a mean'.
  2. [Figure 2] The figure uses check marks, crosses, equals signs, and not-equals signs to distinguish correctness for Part A and consistency for Part B. While the text explains this, the figure would benefit from a direct legend clarifying that a Part-A response is 'correct/incorrect' while a Part-B response is 'consistent/inconsistent'.
  3. [§5.3] The claim that 16 participants 'encountered situations where they perceived a code snippet as wrong, but were unable to predict its outputs' is consistent with Appendix Table 3 ('Mistake in code: inconsistency with prompt' count 10 plus 'Mistake in code: others' count 6), but the paper does not draw the connection explicitly; a cross-reference would help the reader reconcile the numbers.
  4. [§5.1] The ANOVA and t-tests for problem choice, complexity, and comment presence report only 'no significant effect' without reporting test statistics or p-values for those non-significant results. If space permits, including these statistics would make the null findings more informative.

Circularity Check

0 steps flagged · score 1.0 of 10

Empirical study with no circular derivation; the main prompt-code mismatch is an internal validity threat, not a circularity.

full rationale

This paper is an empirical user study, not a formal derivation, so the circularity patterns that apply to deriving predictions from fitted inputs are largely inapplicable. The headline result (code-comprehension pass rate 32.5% vs prompt-comprehension 59.4%, Section 5.1) is produced by participant predictions, not by the problem dataset or by any equation that defines the result in terms of its inputs. The prompts and problems are reused from Nguyen et al. [31], whose author list overlaps with the present paper; however, this is dataset reuse, not a load-bearing theorem, and the measured outcomes are externally generated by 32 CS1 participants in a lab study. No fitted parameter is later renamed as a prediction, and no uniqueness theorem is imported from the authors' prior work. The one serious internal limitation is acknowledged in Section 3.2: 'the code may not directly stem from the prompt, this is by design; the focus of the study is on the code, with prompts acting merely as a contextual framing.' This means the comparison between prompt comprehension and code comprehension can be confounded by prompt-code semantic mismatch, since a participant who perfectly understands the prompt may still fail to predict the output of code that implements a different interpretation. That is a construct-validity threat to the paper's central comparison and should be weighed heavily in judging the strength of the claim, but it is not circularity in the technical sense: the 32.5% figure is not equivalent by construction to the prompt-selection criterion or to the Nguyen et al. dataset. Under the review rule, this limitation is flagged: Section 7 also acknowledges that output-prediction success is only 'a proxy for code comprehension' and may not perfectly correlate with understanding. These are validity caveats, not self-referential reductions. Therefore the appropriate circularity score is 1: there is a minor self-citation/dataset dependency, but the central empirical claim stands on independent participant data.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No free parameters are fitted and no new entities are postulated. The study relies on domain assumptions about measurement validity, representativeness of the code samples, an implicit no-baseline assumption, and statistical independence, all of which are identified above.

assumptions (4)
  • domain assumption Output prediction is a valid proxy for code comprehension.
    The paper uses per-part pass rates (all three predicted outputs correct) and per-test-case pass rates as the quantitative measure of comprehension, and acknowledges in Section 7 that predicting outputs may not perfectly correlate with understanding.
  • domain assumption The eight hand-selected code variations per problem represent the LLM-generated code CS1 students will encounter.
    Section 3.2 selects code to cover correctness, comment presence, and complexity classes, and only 5 of 48 problems are used; the code may not directly stem from the prompt shown to participants.
  • ad hoc to paper No human-written code baseline is needed to attribute the gap to LLM generation.
    The study compares prompt comprehension with code comprehension, so the observed gap could be explained by general code-reading difficulty; attributing it to LLM-generated code specifically requires this unstated assumption.
  • domain assumption Repeated measurements can be treated as independent observations in the statistical tests.
    Section 4 applies chi-squared, t-tests, and ANOVA to 160 task instances and 480 test cases produced by 32 participants, without mixed-effects modeling or cluster-robust inference.

how reviews work

0 comments
Cite this review

Pith. "Pith review of "I Would Have Written My Code Differently'': Beginners Struggle to Understand LLM-Generated Code." pith.science (2026). https://pith.science/paper/UOUIHXDP

@misc{pith2026250419037,
  author       = {Pith},
  title        = {Pith review of: "I Would Have Written My Code Differently'': Beginners Struggle to Understand LLM-Generated Code},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UOUIHXDP}},
  note         = {Machine review of arXiv:2504.19037}
}
read the original abstract

Large language models (LLMs) are being increasingly adopted for programming work. Prior work shows that while LLMs accelerate task completion for professional programmers, beginning programmers struggle to prompt models effectively. However, prompting is just half of the code generation process -- when code is generated, it must be read, evaluated, and integrated (or rejected). How accessible are these tasks for beginning programmers? This paper measures how well beginners comprehend LLM-generated code and explores the challenges students face in judging code correctness. We compare how well students understand natural language descriptions of functions and LLM-generated implementations, studying 32 CS1 students on 160 task instances. Our results show a low per-task success rate of 32.5\%, with indiscriminate struggles across demographic populations. Key challenges include barriers for non-native English speakers, unfamiliarity with Python syntax, and automation bias. Our findings highlight the barrier that code comprehension presents to beginning programmers seeking to write code with LLMs.

Figures

Figures reproduced from arXiv: 2504.19037 by the authors.

Figure 1
Figure 1. Two implementations of reverseWords. To ensure diversity, we selected problems that have LLM code generations representing all possible combinations of these char￾acteristic classes, resulting in 8 different function variations. See Figure 1a for an example of correct, non-complex implementation of the reverseWords problem with comments and Figure 1b for an incorrect, complex implementation with comments for the sam… view at source ↗
Figure 2
Figure 2. Procedure of a task. Prompt comprehension presents a description collected from Nguyen et al. [31] and three in￾puts. A participant enters their expected output for the de￾scribed function given the description. Code comprehension presents code generated for the problem by an LLM and the same set of inputs, but asks for the output of the code. Reflec￾tion presents the correct answer for both parts and prompts the us… view at source ↗
Figure 3
Figure 3. The Study Interface. To ensure familiarity with the study interface, participants com￾plete a tutorial task before proceeding to the main study tasks. The tutorial task is designed to be less challenging than the main tasks, and we provide additional guiding text to assist participants for understanding the workflow of the study interface. 3.5 Post-Study Survey After the main study, the participants are then present… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Counts and percentages of per-part success rates for prompt comprehension (Prompt) and code comprehension (Code) [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. System-driven Interactive Design Support for Cloud Architecture: A Qualitative User Experience Study with Novice Engineers

    cs.HC 2025-08 conditional novelty 4.0 of 10

    A qualitative study of 60 novice engineers reports that a system-driven cloud architecture tool with proactive guidance and structured states makes initial design easier, reduces prompt-writing burden, and supports le...

Reference graph

Works this paper leans on

40 extracted references · 29 canonical work pages · cited by 1 Pith paper

  1. [1]

    Q3 Earnings Call: CEO’s Remarks

    2024. Q3 Earnings Call: CEO’s Remarks. https://blog.google/inside- google/message-ceo/alphabet-earnings-q3-2024/

    work page 2024

  2. [2]

    Feitelson

    Shulamyt Ajami, Yonatan Woodbridge, and Dror G. Feitelson. 2019. Syntax, predicates, idioms — what really affects code complexity?Empirical Software Engineering24, 1 (Feb. 2019), 287–328. https://doi.org/10.1007/s10664-018-9628-3

    work page doi:10.1007/s10664-018-9628-3 2019

  3. [3]

    Feitelson

    Eran Avidan and Dror G. Feitelson. 2017. Effects of Variable Names on Compre- hension: An Empirical Study. In2017 IEEE/ACM 25th International Conference on Program Comprehension (ICPC). 55–65. https://doi.org/10.1109/ICPC.2017.27

    doi:10.1109/icpc.2017.27 2017

  4. [4]

    Feld- man, and Carolyn Jane Anderson

    Hannah McLean Babe, Sydney Nguyen, Yangtian Zi, Arjun Guha, Molly Q. Feld- man, and Carolyn Jane Anderson. 2023. StudentEval: A Benchmark of Student- Written Prompts for Large Language Models of Code. https://doi.org/10.48550/ arXiv.2306.04556 arXiv:2306.04556 [cs]

    arXiv doi:10.48550/arxiv.2306.04556 2023

  5. [5]

    Hofmeister, and Sven Apel

    Jennifer Bauer, Janet Siegmund, Norman Peitek, Johannes C. Hofmeister, and Sven Apel. 2019. Indentation: Simply a Matter of Style or Support for Program Comprehension?. In2019 IEEE/ACM 27th International Conference on Program Comprehension (ICPC). 154–164. https://doi.org/10.1109/ICPC.2019.00033

    work page arXiv 2019

  6. [6]

    Roman Bednarik and Markku Tukiainen. 2006. An eye-tracking methodology for characterizing program comprehension processes. InProceedings of the 2006 symposium on Eye tracking research & applications (ETRA ’06). Association for Computing Machinery, New York, NY, USA, 125–132. https://doi.org/10.1145/ 1117309.1117356

    work page arXiv 2006

  7. [7]

    David Carrington. 1997. Teaching Software Testing. InProceedings of the 2nd Australasian Conference on Computer Science Education (ACSE ’97). Association for Computing Machinery, New York, NY, USA, 59–64. https://doi.org/10.1145/ 299359.299369

    work page arXiv 1997

  8. [8]

    Duarte, Carlos Ferreira, Joao Duraes, Henrique Madeira, and Miguel Castelo-Branco

    Joao Castelhano, Isabel C. Duarte, Carlos Ferreira, Joao Duraes, Henrique Madeira, and Miguel Castelo-Branco. 2019. The role of the insula in intuitive expert bug detection in computer code: an fMRI study.Brain Imaging and Behavior13, 3 (June 2019), 623–637. https://doi.org/10.1007/s11682-018-9885-1

    work page doi:10.1007/s11682-018-9885-1 2019

Show all 40 references
  1. [9]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott G...

    2021 arXiv

  2. [10]

    Maria De-Arteaga, Riccardo Fogliato, and Alexandra Chouldechova. 2020. A Case for Humans-in-the-Loop: Decisions in the Presence of Erroneous Algorithmic Scores. InProceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI ’20). Association for Computing ...

    2020

  3. [13]

    Victor Dibia, Adam Fourney, Gagan Bansal, Forough Poursabzi-Sangdeh, Han Liu, and Saleema Amershi. 2023. Aligning Offline Metrics and Human Judgments of Value for Code Generation Models. , 8516–8528 pages. https://doi.org/10. 18653/v1/2023.findings-acl.540

    2023

  4. [14]

    Stephen H. Edwards. 2003. Improving Student Performance by Evaluating How Well Students Test Their Own Programs.J. Educ. Resour. Comput.3, 3 (Sept. 2003), 1–es. https://doi.org/10.1145/1029994.1029995

    2003

  5. [15]

    Molly Q Feldman and Carolyn Jane Anderson. 2024. Non-Expert Programmers in the Generative AI Future. InProceedings of the 3rd Annual Meeting of the Symposium on Human-Computer Interaction for Work. ACM, Newcastle upon Tyne United Kingdom, 1–19. https://doi.org/10.1145/3663384.3663393

    2024

  6. [16]

    2018.How to design programs: an introduction to programming and computing

    Matthias Felleisen, Robert Bruce Findler, Matthew Flatt, and Shriram Krishna- murthi. 2018.How to design programs: an introduction to programming and computing. MIT Press

    2018

  7. [17]

    Ty Feng, Sa Liu, and Dipak Ghosal. 2024. CourseAssist: Pedagogically Appropriate AI Tutor for Computer Science Education. InProceedings of the 2024 on ACM Virtual Global Computing Education Conference V. 2(Virtual Event, NC, USA) (SIGCSE Virtual 2024). Association for Computin...

    2024

  8. [18]

    National Center for Women & Information Technology. 2023. NCWIT Guide to Demographic Survey Questions. https://docs.google.com/document/d/1E_ CSANwOqbKjEG27woNbGZ09JIXUfAf4Cp9j8g5DFak

    2023

  9. [19]

    Marwa Gadala. 2017. Automation bias: exploring causal mechanisms and poten- tial mitigation strategies. https://api.semanticscholar.org/CorpusID:41123263

    2017

  10. [20]

    Roudsari, and Jeremy C

    Kate Goddard, Abdul V. Roudsari, and Jeremy C. Wyatt. 2012. Automation bias: a systematic review of frequency, effect mediators, and mitigators.Journal of the American Medical Informatics Association : JAMIA19 1 (2012), 121–7

    2012

  11. [21]

    Hart and Lowell E

    Sandra G. Hart and Lowell E. Staveland. 1988. Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research. InAdvances in Psychology, Peter A. Hancock and Najmedin Meshkati (Eds.). Human Men- tal Workload, Vol. 52. North-Holland, 139–183. https://do...

    1988 doi

  12. [23]

    Kyle Daigle, GitHub Staff. 2024. Survey: The AI Wave Continues to Grow on Software Development Teams. https://github.blog/news-insights/research/ survey-ai-wave-grows

    2024

  13. [24]

    Zhen Li, Xiaohan Xu, Tao Shen, Can Xu, Jia-Chen Gu, Yuxuan Lai, Chongyang Tao, and Shuai Ma. 2024. Leveraging Large Language Models for NLG Evaluation: Advances and Challenges. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. Associatio...

    2024 doi

  14. [25]

    Rongxin Liu, Carter Zenke, Charlie Liu, Andrew Holmes, Patrick Thornton, and David J. Malan. 2024. Teaching CS50 with AI: Leveraging Generative Artificial Intelligence in Computer Science Education. InProceedings of the 55th ACM Technical Symposium on Computer Science Educatio...

    2024

  15. [26]

    Feldman, and Carolyn Jane Anderson

    Francesca Lucchetti, Zixuan Wu, Arjun Guha, Molly Q. Feldman, and Carolyn Jane Anderson. 2025. Substance Beats Style: Why Beginning Students Fail to Code with LLMs. InProceedings of the 2025 Conference of the North American Chapter of the Association for Computational Linguist...

    2025 doi arXiv

  16. [27]

    Qianou Ma, Weirui Peng, Chenyang Yang, Hua Shen, Kenneth Koedinger, and Tongshuang Wu. 2024. What Should We Engineer in Prompts? Training Humans in Requirement-Driven LLM Use. https://doi.org/10.48550/arXiv.2409.08775 arXiv:2409.08775 [cs]

    2024 doi arXiv

  17. [28]

    Ian McChesney and Raymond Bond. 2019. Eye tracking analysis of computer program comprehension in programmers with dyslexia.Empirical Software Engineering24, 3 (June 2019), 1109–1154. https://doi.org/10.1007/s10664-018- 9649-y

    2019 doi

  18. [29]

    Ismael Villegas Molina, Audria Montalvo, Benjamin Ochoa, Paul Denny, and Leo Porter. 2024. Leveraging LLM Tutoring Systems for Non-Native English Speakers in Introductory CS Courses. https://doi.org/10.48550/arXiv.2411.02725 arXiv:2411.02725 [cs]

    2024 doi arXiv

  19. [30]

    Nelson, Benjamin Xie, and Amy J

    Greg L. Nelson, Benjamin Xie, and Amy J. Ko. 2017. Comprehension First: Evaluating a Novel Pedagogy and Tutoring System for Program Tracing in CS1. InProceedings of the 2017 ACM Conference on International Computing Education Research. ACM, Tacoma Washington USA, 2–11. https:/...

    2017 doi

  20. [31]

    Sydney Nguyen, Hannah McLean Babe, Yangtian Zi, Arjun Guha, Carolyn Jane Anderson, and Molly Q Feldman. 2024. How Beginning Programmers and Code LLMs (Mis)read Each Other. InProceedings of the CHI Conference on Human Factors in Computing Systems. ACM, Honolulu HI USA, 1–26. ht...

    2024

  21. [32]

    Hofmeister, and André Brechmann

    Norman Peitek, Janet Siegmund, Chris Parnin, Sven Apel, Johannes C. Hofmeister, and André Brechmann. 2018. Simultaneous measurement of program comprehen- sion with fMRI and eye tracking: a case study. InProceedings of the 12th ACM/IEEE International Symposium on Empirical Soft...

    2018

  22. [33]

    Becker, Bailey Kimmel, Jared Wright, and Ben Briggs

    James Prather, Brent N Reeves, Juho Leinonen, Stephen MacNeil, Arisoa S Ran- drianasolo, Brett A. Becker, Bailey Kimmel, Jared Wright, and Ben Briggs. 2024. The Widening Gap: The Benefits and Harms of Generative AI for Novice Pro- grammers. InProceedings of the 2024 ACM Confer...

    2024

  23. [34]

    Christian Rahe and Walid Maalej. 2025. How Do Programming Students Use Generative AI? https://doi.org/10.48550/arXiv.2501.10091 arXiv:2501.10091 [cs]

    2025 doi arXiv

  24. [35]

    Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Wenhan Xiong, Alexandre FSE Companion ’25, June 23–28, 2025,...

    2025

  25. [36]

    Kayla Schroeder and Zach Wood-Doughty. 2024. Can You Trust LLM Judg- ments? Reliability of LLM-as-a-Judge. https://doi.org/10.48550/arXiv.2412.12509 arXiv:2412.12509 [cs]

    2024 doi arXiv

  26. [37]

    Linda J Skita, Kathleen Mosier, and Mark D. Burdick. 2000. Accountability and automation bias.International Journal of Human-Computer Studies52, 4 (2000), 701–717. https://doi.org/10.1006/ijhc.1999.0349

    2000

  27. [38]

    Yuchen Tian, Weixiang Yan, Qian Yang, Xuandong Zhao, Qian Chen, Wen Wang, Ziyang Luo, Lei Ma, and Dawn Song. 2025. CodeHalu: Investigat- ing Code Hallucinations in LLMs via Execution-based Verification. https: //doi.org/10.48550/arXiv.2405.00253 arXiv:2405.00253 [cs]

    2025 doi arXiv

  28. [39]

    Gladys Tyen, Hassan Mansoor, Victor Carbune, Peter Chen, and Tony Mak

  29. [40]

    Smith Iv, Mounika Padala, Christine Alvarado, Jamie Gorson Benario, and Leo Porter

    Annapurna Vadaparty, Daniel Zingaro, David H. Smith Iv, Mounika Padala, Christine Alvarado, Jamie Gorson Benario, and Leo Porter. 2024. CS1-LLM: Integrating LLMs into CS1 Instruction. InProceedings of the 2024 on Innovation and Technology in Computer Science Education V. 1. AC...

    2024

  30. [41]

    Weixuan Wang, Barry Haddow, Alexandra Birch, and Wei Peng. 2024. Assessing Factual Reliability of Large Language Model Knowledge. InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (V...

    2024 doi

  31. [42]

    What happened?

    J. D. Zamfirescu-Pereira, Laryn Qi, Björn Hartmann, John DeNero, and Narges Norouzi. 2024. 61A Bot Report: AI Assistants in CS1 Save Students Homework Time and Reduce Demands on Staff. (Now What?). https://doi.org/10.1145/ 3641554.3701864 arXiv:2406.05600 [cs] A Thematic Analy...

    2024

  32. [2024]

    InFindings of the Association for Computational Linguistics: ACL 2024, Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.)

    LLMs Cannot Find Reasoning Errors, but Can Correct Them given the Error Location. InFindings of the Association for Computational Linguistics: ACL 2024, Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Computational Linguistics, Bangkok, Thailand, 13894–13...

    2024

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.