{"id":"ef1f68e8-3326-498c-8e88-cec68eacb161","arxiv_id":"2504.19037","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"CS1 students correctly predicted the output of LLM-generated Python code in 32.5% of tasks, versus 59.4% for natural-language prompts, and still failed code prediction 58% of the time when they understood the prompt.","lead":"A lab study with 32 first-year programming students found they could predict what LLM-generated code actually does only 32.5% of the time, versus 59.4% for understanding the problem in plain language. The result suggests that using AI coding assistants requires reading skills that beginners may not yet have.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The task design decouples the prompt from the code shown to participants, so the headline prompt-vs-code comparison may be confounded by prompt-code mismatch.","rationale":"The reader identified a missing human-written code control as the weakest assumption. That is a fair external-validity concern. However, the more load-bearing issue is internal: Section 3.2 explicitly decouples the prompt from the code, so the prompt comprehension and code comprehension parts are not necessarily about the same specification. The headline statistic (32.5% code-comprehension success, and 58% failure even when prompt comprehension succeeds) is interpreted as evidence that students struggle with LLM-generated code even when they understand the underlying problem. But if the code was generated from a different prompt, understanding the shown prompt does not logically imply any ability to predict the code's behavior; the comparison conflates code comprehension difficulty with prompt-code mismatch. This is directly acknowledged in the manuscript, so it is not an artifact of the review pipeline. It is testable: recovering the prompt-code generation map and re-analyzing only matched pairs would either support or undermine the central claim. The paper's qualitative findings (unfamiliar syntax, style, automation bias) may still hold, but the quantitative basis for the main claim is currently unverified. Thus the appropriate verdict remains CONDITIONAL: accept if the matching is confirmed or the analysis is redone with exactly matched prompt-code pairs; the reader's CONDITIONAL verdict is not changed, but the condition should include this internal-validity check rather than only the external-validity control.","tokens_in":17193,"tokens_out":5820,"duration_ms":62223,"concrete_test":"Check generation provenance for every code variation: for each of the 40 problem-code pairings, verify whether the exact prompt text shown in Part A was the input that generated the code shown in Part B (the OSF repository should include this mapping). Restrict the analysis to tasks with exact matches and recompute: (1) overall per-task code-comprehension success; (2) conditional P(Part B fail | Part A pass). If the conditional failure rate drops below 0.5 or the overall success rate rises substantially, the reported 'struggle despite prompt understanding' is an artifact of unmatched prompt-code pairs. If exact-match data are unavailable, the authors should rerun the study or re-select code variations so that each code is generated from the exact prompt displayed, then repeat the analysis.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 states: \"the code may not directly stem from the prompt, this is by design; the focus of the study is on the code, with prompts acting merely as a contextual framing.\" This means each LLM-generated code variation in Part B was not necessarily produced from the exact prompt text shown in Part A, but from some prompt for the same problem. The central result in Section 5.1 — that code comprehension fails even when prompt comprehension succeeds (55 of 95 prompt-pass tasks fail code, 0.58 vs 0.42) — presupposes that understanding the prompt should help predict what the code does. If the code implements a different interpretation of the problem than the prompt shown, then a participant who fully understands the prompt may still be unable to predict the code's output, not because LLM code is inherently hard to comprehend, but because the prompt and code are semantically mismatched. This would inflate the 32.5% overall failure rate and the conditional failure rate, and would also weaken the automation-bias interpretation: entering the prompt's expected output as the code's actual output could reflect a mismatch between the prompt and code rather than blind trust in the model. Because the decoupling is acknowledged in the paper itself, this is not a speculative external-validity quibble; it is an internal confound in the main comparison between prompt comprehension and code comprehension.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates whether CS1 students can understand code generated by an LLM. It reports a lab study with 32 participants who each completed five tasks (160 task instances total). Each task presents a student-written natural-language problem description and an LLM-generated Python function for the same problem; participants predict the outputs of both the described function and the shown code for three test cases, then reflect on their answers. The central quantitative result is that code comprehension succeeds in only 32.5% of tasks, versus 59.4% for prompt comprehension, and that participants often fail code comprehension even when they succeed at prompt comprehension (0.58 failure vs 0.42 success in the conditional analysis). Qualitative analysis identifies unfamiliar Python syntax, unfamiliar code style, library-function knowledge gaps, and automation bias as key challenges. The paper concludes that beginners face substantial barriers in reading, evaluating, and integrating LLM-generated code, and it recommends LLM-aware teaching strategies and tool design changes.","tokens_in":17434,"tokens_out":3815,"duration_ms":39602,"significance":"If the core result holds, the paper addresses an important gap in the computing-education literature: prior work on CS students and LLMs has concentrated on prompt writing and task completion, while the ability to read and evaluate generated code is comparatively understudied. The study has several strengths: it reuses a validated dataset of CS1 problems and high-quality prompts from Nguyen et al., uses a structured output-prediction design rather than self-report, includes a pilot study, reports inter-rater reliability (Cohen's kappa 0.77), and makes data and materials publicly available. The qualitative participant quotes give concrete, believable evidence of the reported mechanisms. However, the significance is conditional on two central assumptions: that the observed difficulty is specific to LLM-generated code rather than to code comprehension in general, and that the prompt shown in Part A and the code shown in Part B are sufficiently aligned that prompt comprehension should predict code comprehension. Both assumptions are currently unsupported by the experimental design, so the headline comparison must be treated with caution.","major_comments":[{"comment":"The central comparison in §5.1—code comprehension success 52/160 (32.5%) versus prompt comprehension success 95/160 (59.4%)—does not support the paper's conclusion that beginners struggle specifically with LLM-generated code. No human-written code condition is included, so the result could reflect general difficulty with reading code, rather than any property of LLM generation. This concern is amplified by the dataset construction in §3.2, where the eight code variations were deliberately selected to include bugs and unfamiliar idioms such as list comprehensions and slicing. To attribute the gap to LLM generation, the study would need a control condition with equivalent human-written functions, or the claims should be reframed to describe comprehension of code of the kind LLMs produce, without implying a comparison to human-written code.","section":"§3.2 and §5.1"},{"comment":"The dataset construction creates a prompt-code mismatch that confounds the key conditional analysis. Section 3.2 states that the code 'may not directly stem from the prompt' and that this is by design, with prompts serving only as contextual framing. The Part A prompt-comprehension task and the Part B code-comprehension task therefore do not necessarily describe the same function. The claim in §5.1 that 'even when participants understand the prompt, their failure rate in code comprehension exceeds their success rate' presupposes that understanding the prompt should help predict what the shown code does. If the code was generated from a different prompt that resolves the problem differently, a participant who fully understands the shown prompt may still be unable to predict the code's outputs. This confound also weakens the automation-bias interpretation in §5.3, because entering the prompt's expected output as the code's actual output could reflect a genuine mismatch between the prompt and code rather than blind trust in the model. The authors should use matched prompt-code pairs, or analyze only instances where the prompt used for generation is shown to participants, or at minimum measure and report the semantic overlap between the shown prompt and the prompt that generated the code.","section":"§3.2 and §5.1"},{"comment":"The statistical analyses treat the 160 data points as independent, but the design is repeated-measures: each of the 32 participants contributes five correlated tasks, and each of the 40 problem-code pairings is reused across participants. The chi-squared tests, t-tests, and ANOVA in §5.1 and §5.2 therefore risk pseudoreplication and inflated significance levels. The authors should use mixed-effects models with random intercepts for participant and for problem/code variation, or at minimum cluster-standard errors by both participant and task. In addition, the paper reports p-values without effect sizes or confidence intervals; these should be added, especially for the headline 32.5% versus 59.4% comparison and for the conditional failure rate of 0.58.","section":"§4 and §5.1–5.2"},{"comment":"The abstract states that the study shows 'indiscriminate struggles across demographic populations,' but this overstates what §5.2 and Appendix Table 2 support. The only statistically significant demographic effect is childhood language on prompt comprehension; the paper correctly notes that other demographic comparisons showed no significant differences, but also notes that effects 'may exist but were undetected due to sample size and distribution.' Several subgroup sizes are extremely small (e.g., n=2 for Black participants), so a null result is not evidence of the absence of differences. The wording should be softened to 'no demographic differences were detected' or accompanied by a power analysis showing which effect sizes the design could have detected.","section":"§5.2 and Abstract"}],"minor_comments":[{"comment":"The sentence 'with an mean eventual success rate of only 57%' contains a grammatical error: 'an mean' should be 'a mean'.","section":"§2"},{"comment":"The figure uses check marks, crosses, equals signs, and not-equals signs to distinguish correctness for Part A and consistency for Part B. While the text explains this, the figure would benefit from a direct legend clarifying that a Part-A response is 'correct/incorrect' while a Part-B response is 'consistent/inconsistent'.","section":"Figure 2"},{"comment":"The claim that 16 participants 'encountered situations where they perceived a code snippet as wrong, but were unable to predict its outputs' is consistent with Appendix Table 3 ('Mistake in code: inconsistency with prompt' count 10 plus 'Mistake in code: others' count 6), but the paper does not draw the connection explicitly; a cross-reference would help the reader reconcile the numbers.","section":"§5.3"},{"comment":"The ANOVA and t-tests for problem choice, complexity, and comment presence report only 'no significant effect' without reporting test statistics or p-values for those non-significant results. If space permits, including these statistics would make the null findings more informative.","section":"§5.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a close companion to Nguyen et al. [31], reusing its dataset, prompts, and survey instruments; this is a reasonable design choice, but it makes the incremental contribution rest almost entirely on the prompt-versus-code comparison. That comparison is currently confounded in two ways that are acknowledged in the manuscript itself: the absence of a human-written control and the intentional decoupling of the shown prompt from the prompt that generated the shown code. Both issues are fixable in a revision, and the qualitative material is valuable. Given the small sample size and single-institution setting, this reads more as an exploratory educational study than as a definitive result; I would support acceptance after the major issues are addressed and the claims are appropriately calibrated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read on arXiv:2504.19037. The headline number is worth taking seriously: 32 CS1 students completed 160 output-prediction tasks on LLM-generated code and managed only a 32.5% per-task success rate, despite a 59.4% success rate on the corresponding natural-language prompts. That gap is the paper's real contribution. It quantifies something educators have suspected and gives tool designers a concrete target: generated code needs to be more reviewable, or students need to be taught to read it.\n\nThe paper does a few things well. The task design—predict expected output from a prompt, then predict actual output from the code, with an explicit warning that code may be buggy—is a clean, objective measure. The qualitative data are genuinely useful: students report being tripped up by list slicing, list comprehensions, and unfamiliar idioms; many appear to assume the code matches the prompt (automation bias). The dataset and materials are on OSF, and the inter-rater reliability for the thematic analysis is solid.\n\nThe soft spots are real but not fatal. First, there's no human-written code condition, so the title's claim about LLM-generated code specifically isn't supported. The 32.5% might be what CS1 students would score on any code with the same syntactic complexity. The paper should either add such a control or soften its framing. Second, the stress-test note is on target: Section 3.2 says the code \"may not directly stem from the prompt,\" so the prompts are just contextual framing. That means the conditional finding—students fail code comprehension even when they pass prompt comprehension—is confounded. If the code implements a different interpretation than the prompt shown, then understanding the prompt doesn't help predict the code's behavior. The overall 32.5% is still meaningful because participants can trace the code, but the gap between prompt and code, and the automation-bias interpretation, are weakened. Third, the statistics treat 160 tasks as independent though there are only 32 participants; no effect sizes or confidence intervals are reported. A mixed-effects model would be appropriate.\n\nThese are addressable. The paper is honest about its threats to validity, and the central measurement is novel. I'd send it to peer review and ask for a human-written code control and a more careful statistical treatment. This is a paper for CS-education and HCI readers, and it deserves a serious referee rather than a desk reject.","headline":"Useful measurement of CS1 students' poor comprehension of LLM-generated code, but the LLM-specific claim needs a human-written control and the prompt-code decoupling muddies the conditional analysis.","tokens_in":17964,"tokens_out":5858,"would_cite":true,"duration_ms":51969,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Beginning programmers succeeded on only 32.5% of tasks that required predicting what LLM-generated code would do.","keywords":["LLM-generated code","code comprehension","CS1","beginning programmers","automation bias","output prediction","Pythonic syntax","computing education"],"falsifier":"Run the same output-prediction tasks with human-written Python functions matched for length, style, and bug rate; if beginners succeed no more often than with LLM-generated functions, the central claim shifts from 'LLM code is hard' to 'code reading is hard.'","tokens_in":16997,"feed_emoji":"💻","tokens_out":7194,"duration_ms":69850,"temperature":0.7,"pith_summary":"This paper asks whether beginning programmers can read and evaluate code produced by a large language model, and its answer is that mostly they cannot. In a lab study of 32 students who had completed one introductory Python course, across 160 tasks participants predicted the behavior of LLM-generated code correctly only 32.5% of the time, versus 59.4% for the same problems described in natural language. The difficulty persisted even when students understood the problem description, and it was worse for buggy code. If correct, this is a practical problem for putting LLMs in beginners' hands: generating code is only useful if the user can judge whether the output does what is intended.","feed_headline":"Novices predict LLM code output only 32.5% of the time","feed_subtitle":"Even when they understood the problem, students misread generated code more often than not.","key_machinery":"The measuring instrument is a three-part task: students first read a natural-language problem description and predict its output for three inputs, then read an LLM-generated function for the same problem and predict its actual output for the same three inputs, then review the ground truth and reflect. Because both parts use identical inputs, the design isolates the act of reading code from the act of understanding the problem; the per-part gap between prompt comprehension and code comprehension is the quantitative evidence. Code variations were systematically selected to differ in correctness, comment presence, and complexity, and prompts were filtered by a high pass@1 rate so that prompt quality would not confound the comparison.","core_discovery":"The paper claims that CS1 students, after one semester of Python, can predict what an LLM-generated function will do only 32.5% of the time across 160 tasks, compared with 59.4% for predicting what the problem description asks for. Even when the prompt was understood, code comprehension failed more often than it succeeded (failure rate 0.58 versus success 0.42). Students did significantly worse on code with bugs, and qualitative responses attribute the difficulty to unfamiliar Pythonic syntax and idioms, unknown library functions, and a tendency to assume generated code is correct. The paper concludes that reading and evaluating generated code, not just prompting, is a major barrier for beginners who want to use LLMs.","pith_inferences":["Because the study lacks a human-written-code control, the LLM-specific portion of the difficulty is not yet measured; a matched human-code condition would separate the effect of generation from the general difficulty of reading code.","The dataset deliberately includes buggy and heavily idiomatic variations, so the 32.5% figure likely reflects a stress test rather than typical everyday LLM output; real tools could score higher or lower depending on the model and prompt.","The three-part output-prediction format is a reusable instrument: a training intervention could use it to test whether teaching tracing or Python idioms shrinks the prompt-to-code comprehension gap.","The paper's evidence points to a concrete design target for LLM tools aimed at novices: surfacing generated-code behavior through examples or execution traces, not just the code itself, is worth testing as a mitigation."],"forward_implications":["If students cannot reliably predict what generated code does, accepting LLM output is effectively blind, so CS1 courses that adopt LLMs need to teach code reading and tracing alongside or before writing.","The strong association between automation bias and wrong predictions implies that novice review of generated code must be scaffolded with explicit checks, such as running test cases and comparing outputs.","Unfamiliar Python idioms such as list comprehensions and slicing account for a large share of reported difficulty, so curricula that expect students to read LLM output need to cover those idioms earlier.","Non-native English speakers' disadvantage at the prompt stage means LLM-assisted education has an equity dimension: unclear natural-language descriptions compound code-comprehension failure.","Because prompt-comprehension failure makes code-comprehension failure much more likely, a student who only half-understands the task is at particular risk of accepting incorrect generated solutions."],"supporting_citations":[{"why":"supplies the 48 CS1 problems, the student-written prompts, and the prompt-writing baseline that the study reuses.","marker":"[31]"},{"why":"establishes the 'illusion of competence' phenomenon with novice LLM users that this study extends from prompting to code reading.","marker":"[33]"},{"why":"provides the claim that experts can salvage buggy LLM code, which presupposes the error-detection ability this study measures in beginners.","marker":"[13]"},{"why":"identifies the code-generation model whose outputs are the study's stimulus set.","marker":"[35]"},{"why":"documents the writing-first pedagogy that the paper argues leaves comprehension under-taught.","marker":"[30]"},{"why":"defines pass@1, the criterion that selected only prompts with at least 90% generation success.","marker":"[9]"},{"why":"reported that CS1 students submit LLM output without modifying it, motivating the comprehension measurement.","marker":"[34]"}],"fun_headline_variants":["Novices misread LLM code two-thirds of the time","Even when prompts are clear, novices still read LLM code wrong","LLM code comprehension: beginners fail 67.5% of the time","Reading LLM code is harder than prompting for novices","For CS1 students, LLM code is a guessing game they lose often"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim that LLM-generated code is specifically hard to understand rests on the assumption that human-written code would have been easier for these students; the study never includes a human-written code condition, so the difficulty could be about reading code in general.","fun_headline_variants_meta":{"raw":{"variants":["Novices misread LLM code two-thirds of the time","Even when prompts are clear, novices still read LLM code wrong","LLM code comprehension: beginners fail 67.5% of the time","Reading LLM code is harder than prompting for novices","For CS1 students, LLM code is a guessing game they lose often"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000669,"raw_usage":{"total_tokens":3007,"prompt_tokens":856,"completion_tokens":2151,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":472,"completion_tokens_details":{"reasoning_tokens":2058}},"tokens_in":472,"tokens_out":2151,"duration_ms":17133,"temperature":1.0,"reasoning_tokens":2058,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:02:28.575491+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same output-prediction tasks with human-written Python functions matched for length, style, and bug rate; if beginners succeed no more often than with LLM-generated functions, the central claim shifts from 'LLM code is hard' to 'code reading is hard.'","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the 48 CS1 problems, the student-written prompts, and the prompt-writing baseline that the study reuses."},{"cited_title":"Becker, Bailey Kimmel, Jared Wright, and Ben Briggs","cited_arxiv_id":null,"evidence_quote":"establishes the 'illusion of competence' phenomenon with novice LLM users that this study extends from prompting to code reading."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"provides the claim that experts can salvage buggy LLM code, which presupposes the error-detection ability this study measures in beginners."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"identifies the code-generation model whose outputs are the study's stimulus set."},{"cited_title":"Nelson, Benjamin Xie, and Amy J","cited_arxiv_id":null,"evidence_quote":"documents the writing-first pedagogy that the paper argues leaves comprehension under-taught."},{"cited_title":"How Do Programming Students Use Generative AI?","cited_arxiv_id":"2501.10091","evidence_quote":"reported that CS1 students submit LLM output without modifying it, motivating the comprehension measurement."}],"review_version":1}