{"id":"f6f9defe-ce97-47f2-9889-299d737c39e6","arxiv_id":"2504.14964","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"LLM code generators solve most introductory programming assignments but rarely produce fully correct solutions to advanced CS4 and CS5 assignments, often identifying the algorithm while missing problem-specific constraints.","lead":"This study tested four AI coding tools on twelve university programming assignments and found that they solve introductory problems well but mostly fail on advanced ones. The results give computer science teachers concrete evidence about where AI assistance breaks down.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"TSP verifier checks feasibility, not optimality, making two of the three advanced 100% results suspect; the central claim still holds.","rationale":"The central empirical claim, that LLMs are effective on introductory assignments but rarely solve advanced CS4/CS5 assignments, is well supported by Table 2: 29/36 CS1 tool-language-problem combinations reach 100% accuracy, versus only 3/108 for P4-P12, and one of those three is not a near-miss but a clear success (Cut Boards). The most concrete threat to the evaluation's correctness is the verifier for TSP. The text says that for multiple-answer problems, verification checks structural properties such as whether all nodes appear in a path and whether bounds are respected. For TSP, that is necessary but not sufficient to establish optimality, so the two 100% TSP cells are suspect. Removing them does not weaken the paper's qualitative conclusion; it strengthens it. However, the reader's conditional verdict remains appropriate because the paper should correct this verifier and re-run the affected solutions before its specific RQ1 numbers are cited. I therefore keep the verdict unchanged.","tokens_in":10511,"tokens_out":7101,"duration_ms":69103,"concrete_test":"Inspect the repository's verifier for P5/TSP. If it does not compare the generated tour's total cost to an exact optimum, re-run the two 100% Copilot solutions on the same 1000 instances using an independent exact TSP solver (e.g., exhaustive search for small n or Concorde) and require exact optimal cost. If either cell drops below 100%, the claimed advanced successes shrink from three to one and the RQ1 summary must be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.4 describes verification for problems with many valid answers, 'such as the Traveling Salesman Problem (TSP) problem', as checking 'all nodes exist in a path and that bounds are not exceeded.' For TSP, this verifies that the output is a feasible Hamiltonian tour, but it does not verify that the tour is optimal. The two green TSP cells in Table 2 (Copilot Python and Copilot Java) therefore do not establish that the generated code solves the traveling-salesman assignment, which asks for the shortest route. Since the paper reports only three 100% advanced combinations in total, and two of them are TSP, the specific RQ1 finding depends on an unvalidated verifier. This is a concrete weakness in the evaluation, though not one that overturns the qualitative central claim: even if both TSP cells are removed, the paper still shows a stark gap between CS1 (29/36 correct combinations) and CS4/CS5 (1/108 correct combinations), because the Cut Boards result would remain. The paper should either confirm that the verifier compares against the optimal tour length or re-run the TSP evaluation with an exact optimality check and update the affected percentages.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper evaluates the code-generation performance of four LLM tools (GitHub Copilot, CodePal, Llama3-70b, and Mistral-Nemo) on twelve programming problems: three introductory CS1 problems as a baseline and nine advanced CS4/CS5 problems, each in Java, Python, and C. For each problem, the authors generate 1000 random test cases, run the generated programs against those tests, and compare the outputs with their own reference solutions, defining accuracy as the fraction of correctly solved test cases and correctness as 100% accuracy. The main empirical finding is that LLMs solve most introductory problems but rarely solve advanced assignments correctly; however, they often produce partial solutions or recognize the underlying algorithm. The paper also reports tool- and language-level differences, concluding that GitHub Copilot outperforms the other tools and that C causes the most errors.","tokens_in":10720,"tokens_out":4559,"duration_ms":41996,"significance":"If the results hold, this is a useful empirical contribution to the growing literature on LLMs in computing education, complementing earlier work that focused on introductory programming. The study's strengths include a handpicked problem set spanning multiple institutions and levels, a handwritten baseline problem to reduce memorization risk, a public repository with prompts and data, and clear operational definitions of accuracy and correctness. The central qualitative claim—that LLMs are far more successful on introductory than advanced assignments—is visible in Table 2 and is robust to the specific verifier weaknesses discussed below. However, the quantitative strength of the claims is limited by unresolved verification and statistical issues, so the paper needs revision before its specific numbers can be taken at face value.","major_comments":[{"comment":"The Traveling Salesman Problem verifier checks feasibility only, not optimality. Section 4.4 states that for problems with many valid answers, 'such as the Traveling Salesman Problem (TSP) problem', the verification step 'checks each aspect of the given answer, such as verifying all nodes exist in a path and that bounds are not exceeded.' The TSP assignment asks for the shortest route, so a feasible Hamiltonian tour is not a correct solution. Consequently, the two 100% accuracy cells for TSP (Copilot in Python and Copilot in Java) do not establish that those programs solve the assignment. Since the paper reports only three advanced-problem combinations with 100% accuracy, and two of them are TSP, the specific RQ1 count is overstated. The paper should either verify optimality against an exact solver (e.g., exhaustive search on the same small instances) or re-classify these cells.","section":"4.4, Table 2, P5 (TSP)"},{"comment":"Accuracy is reported as a point estimate with t=1000 test cases and no variance across test-generation seeds. The test cases for all solutions use the same random seed per problem, so the reported differences between tools and languages (e.g., P5 Python: Copilot 100 vs CodePal 0; P7 Java: Copilot 0 vs Llama 77.4) could change substantially under a different seed, especially for NP-hard problems where instance-size distributions matter. This lack of error bars or confidence intervals weakens the quantitative comparisons in RQ3 and the partial-solution thresholds used in Section 5.3 (e.g., accuracy > 5%). The authors should run multiple test seeds and report variances, or at minimum justify why a single seed is representative.","section":"4.4, Definition 1, Table 2"},{"comment":"The test-suite generation process is not described in enough detail to assess whether the 1000 random cases are representative of real assignments. For NP-hard problems like Boxes and TSP, the instances must be small enough for exhaustive verification, so the difficulty distribution may differ from typical course assignments. Moreover, the reference solutions are assumed correct without supporting evidence; a bug in any reference solution would shift every accuracy in Table 2. The manuscript should document the instance-generation parameters, the input distributions, and the verification logic per problem, and should make the reference solutions and generators available for inspection alongside the existing prompt/data repository.","section":"4.4 and 6"}],"minor_comments":[{"comment":"The sentence 'Among P4 to P12, only P4,TSP and P11, Cut Boards, have two and one correct results, respectively' mislabels the TSP problem as P4; TSP is P5 according to Table 1.","section":"5.1"},{"comment":"The text 'it seems that CP in Python generates the same heuristic as CG in C' should read 'GC' instead of 'CG' for GitHub Copilot.","section":"5.3"},{"comment":"Listing 1.1 contains stray semicolons in Python code (e.g., 'return b o x e s _ n e e d e d ;'), which are likely transcription artifacts; the code should be cleaned up to match what was actually executed.","section":"Listing 1.1"},{"comment":"The sentence 'Their results show that the ACG performs better than students in this course Finnie-Ansley et al..' is grammatically incomplete and has a duplicated period; it should be rewritten and properly cited.","section":"2 Related Work"},{"comment":"The table header is visually ambiguous: the HF column is split into LL and MS subcolumns, making it look like five tools rather than four. Use a clearer multi-row header that groups Llama and Mistral under a single Hugging Face tool category.","section":"Table 2 header"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper does something straightforward and useful: it benchmarks four LLM tools (Copilot, CodePal, Llama-3-70B, Mistral-Nemo) on 12 programming assignments, from CS1 to CS4/CS5, in Python, Java, and C. The setup is honest: 1000 random tests per problem-tool-language, with a shared seed, and the data is on GitHub. The main finding—LLMs nail most intro problems but almost never fully solve advanced ones—is clear in Table 2 and robust to the issues I mention below. That is a real contribution, because prior work mostly stopped at CS2. The paper also gives teachers something to take away: LLMs often identify the right algorithm but miss constraints, so adding a twist or constraint to a known problem is a cheap way to keep assignments LLM-resistant.\n\nThe soft spots are, in order of seriousness. First, the TSP verifier. Section 4.4 says that for problems with many valid answers, 'such as TSP,' verification checks that nodes exist in a path and bounds are not exceeded—but not that the tour is optimal. The two TSP cells at 100% (Copilot Python and Java) therefore only prove feasibility, not correctness for the assignment's actual request (shortest route). The stress-test note is right; this is a concrete flaw. It shrinks the advanced 100% count from 3 to 1 (Cut Boards). The qualitative conclusion doesn't change, but the RQ1 numbers should be corrected and the verifier either upgraded to exact optimality or those cells relabeled.\n\nSecond, the 'minor changes' to generated code (Appendix A) are documented, which is good, but their effect isn't discussed. Lowering constants on P7 and P9 could affect results in non-obvious ways; changing output format is benign, but a reader can't tell how many wrong answers were actually fixed by these edits. A sensitivity check or at least a comment on which changes could flip a result would address this.\n\nThird, there are no error bars or seed variance. The paper uses one seed and reports point accuracies. Given the small number of problems and the fact that a single test seed can make a 90%-accurate program look 100%, confidence intervals would be cheap to add and would materially strengthen the claim.\n\nOn citations, the related work is appropriate and doesn't oversell novelty. The paper does overstate the TSP result slightly in the RQ1 discussion, but the limitations section is honest about prompt engineering and the need for future work.\n\nThis paper deserves a serious referee. It's a legitimate extension of the evaluation literature with a clear educational audience. I'd recommend send it to review with a request for a revised TSP analysis and seed variance. My own verdict would be a conditional accept.","headline":"Useful extension of LLM assignment-benchmarking to advanced CS, but the TSP verifier overstates two of the three 'correct' advanced results; central gap claim still holds.","tokens_in":11219,"tokens_out":2355,"would_cite":true,"duration_ms":20105,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims LLMs are highly effective at introductory programming assignments but rarely solve advanced CS4/CS5 assignments, and that they tend to identify the underlying algorithm while ignoring constraints.","keywords":["large language models","code generation","programming assignments","computer science education","advanced CS courses","LLM evaluation","constraint adherence","introductory programming"],"falsifier":"Run the paper's nine advanced prompts through a newer LLM and grade against the original courses' hidden test suites. If most generated programs pass those suites, the central claim that advanced CS assignments are rarely solved would be weakened.","tokens_in":10327,"feed_emoji":"🤖","tokens_out":7536,"duration_ms":66763,"temperature":0.7,"pith_summary":"This paper attempts to establish that large language models are reliable at generating working code for introductory programming assignments but rarely produce fully correct solutions for advanced, second- and third-year computer science problems. Four LLM tools were each asked to solve twelve handpicked assignments in Python, Java, and C, and every generated program was graded against 1000 randomly generated test cases using the authors' reference verifier. On the advanced problems, only three of 108 tool-language-problem combinations achieved a perfect score: two for the traveling-salesman problem and one for a dynamic-programming board-cutting problem. The models usually recognized the underlying algorithm, but they ignored task-specific constraints or substituted greedy heuristics for the required method. If the results hold, teachers in advanced courses can make assignments harder to solve by adding variations and constraints to well-known algorithmic problems.","feed_headline":"Three of 108 LLM code attempts solve advanced CS problems","feed_subtitle":"Testing four LLM tools on CS4/CS5 problems shows they find the algorithm but miss the constraints.","key_machinery":"The evaluation rests on a three-part prompt (problem description, input format, output format) fed to each tool, followed by automated verification: a Verifier computes the reference answer for each of 1000 random instances, and accuracy is the fraction of instances the generated program answers correctly. A problem is considered solved only when accuracy is 100%; for problems with multiple valid answers, such as TSP, the verifier checks structural properties of the answer instead of a single number. This machinery matters because every conclusion about LLM capability is defined by it, and it turns vague impressions of coding skill into a countable score across languages and tools.","core_discovery":"The paper's central discovery is a sharp drop in correctness exactly where courses stop teaching basic syntax and start teaching algorithm design. Using accuracy defined as the share of correctly verified test cases, with a problem counted as solved only at 100%, the models solved the CS1 baseline problems almost everywhere, including perfect scores for the shortest-path baseline, but almost never solved the nine CS4/CS5 problems. In the advanced set, Github Copilot produced the only perfect solutions, while the other tools mostly generated heuristics, solutions that solved a simplified version of the prompt, or code that did not run. The authors conclude that LLMs are good at naming and sketching the textbook algorithm but weak at conforming to the constraints that make an assignment a genuine test of understanding.","pith_inferences":["The perfect scores on TSP and Cut Boards should be read cautiously: for NP-hard problems the random test generator must keep instances small enough to verify exhaustively, so larger or adversarial instances could reveal failures that the paper's 1000-case suites do not.","Because the dominant failure pattern is ignoring constraints, a cheap robustness experiment would be to move a constraint sentence earlier or later in the prompt and measure accuracy shifts; if placement matters, students could game tools by rewording assignments.","A likely next step is repair: since many generated solutions are close, feeding failing test outputs back to the model could convert some partial solutions into correct ones, which would raise the effective usefulness of the tools beyond the paper's one-shot accuracy.","The paper's setup amounts to a constraint-adherence benchmark; if the pattern generalizes, advanced CS assignments could serve as a practical testbed for measuring how well LLMs follow multi-part instructions."],"forward_implications":["Instructors can no longer assume that textbook algorithm assignments are LLM-proof: the tools recognize the underlying problem and return plausible code, so assessment must probe constraints and edge cases.","Adding a single problem-specific constraint, as in the TV-Zapping problem, drove accuracy down to 96.4 percent at best and near zero for most tools, indicating that constraint variation is an effective way to make assignments harder for LLMs.","GitHub Copilot was the only tool to produce any perfect advanced solutions, while the other tools more often produced non-functional code or heuristics.","C code was the most fragile, with twelve compilation or runtime errors across the advanced problems, so language choice materially changes what a student gets from a tool.","Students using LLMs on advanced work will frequently receive partially correct code that passes easy cases, which makes hidden-constraint testing important for fair grading."],"supporting_citations":[{"why":"Establishes the CS1 baseline claim that Copilot solves most introductory exercises, which this paper extends to advanced courses.","marker":"[6]"},{"why":"Shows Copilot outperforms most students on an introductory exam, another baseline for the intro-is-easy result.","marker":"[7]"},{"why":"Evaluates Codex on CS2 data-structures exercises, the closest prior advanced evaluation this paper goes beyond.","marker":"[8]"},{"why":"Shows LLMs can turn textual descriptions into constraint-programming specifications, background for the claim that models identify underlying problems.","marker":"[13]"},{"why":"Provides the Rainfall problem that the P1 baseline is a variant of.","marker":"[15]"},{"why":"Evaluates Copilot on programming-course assessments and shows it improves with autograder feedback, another intro-level performance baseline.","marker":"[16]"}],"fun_headline_variants":["LLMs ace intro CS, flop on advanced algorithms","LLM code fails 105 of 108 advanced CS tests","Advanced CS: LLMs miss constraints, not algorithms","LLMs solve intro, but advanced? 3/108 correct","LLMs spot the algorithm, then break on constraints"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole comparison assumes that the authors' reference solutions are error-free and that 1000 random test cases are representative enough of what an advanced course would actually test.","fun_headline_variants_meta":{"raw":{"variants":["LLMs ace intro CS, flop on advanced algorithms","LLM code fails 105 of 108 advanced CS tests","Advanced CS: LLMs miss constraints, not algorithms","LLMs solve intro, but advanced? 3/108 correct","LLMs spot the algorithm, then break on constraints"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000515,"raw_usage":{"total_tokens":2488,"prompt_tokens":922,"completion_tokens":1566,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":538,"completion_tokens_details":{"reasoning_tokens":1484}},"tokens_in":538,"tokens_out":1566,"duration_ms":11018,"temperature":1.0,"reasoning_tokens":1484,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:35:27.919436+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the paper's nine advanced prompts through a newer LLM and grade against the original courses' hidden test suites. If most generated programs pass those suites, the central claim that advanced CS assignments are rarely solved would be weakened.","supporting_citations":[{"cited_title":"In: Proceedings of the 54th ACM Technical Symposium on Computer Science Education V","cited_arxiv_id":null,"evidence_quote":"Establishes the CS1 baseline claim that Copilot solves most introductory exercises, which this paper extends to advanced courses."},{"cited_title":"In: Proceedings of the 24th Australasian Computing Education Conference, pp","cited_arxiv_id":null,"evidence_quote":"Shows Copilot outperforms most students on an introductory exam, another baseline for the intro-is-easy result."},{"cited_title":"In: Proceedings of the 25th Australasian Computing Education Conference, pp","cited_arxiv_id":null,"evidence_quote":"Evaluates Codex on CS2 data-structures exercises, the closest prior advanced evaluation this paper goes beyond."},{"cited_title":"Communications of the ACM29(9), 850–858 (1986)","cited_arxiv_id":null,"evidence_quote":"Provides the Rainfall problem that the P1 baseline is a variant of."},{"cited_title":"1 (2023) A Changed solutions We made some small alterations to the LLM-generated solutions before the tests","cited_arxiv_id":null,"evidence_quote":"Evaluates Copilot on programming-course assessments and shows it improves with autograder feedback, another intro-level performance baseline."}],"review_version":1}