Evaluating Code Generation of LLMs in Advanced Computer Science Problems

Emir Catir; Robin Claesson; Rodothea Myrsini Tsoupidi

REVIEW 3 major objections 5 minor 17 references

Evaluating Code Generation of LLMs in Advanced Computer Science Problems

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims LLMs are highly effective at introductory programming assignments but rarely solve advanced CS4/CS5 assignments, and that they tend to identify the underlying algorithm while ignoring constraints.

desk verdict Useful extension of LLM assignment-benchmarking to advanced CS, but the TSP verifier overstates two of the three 'correct' advanced results; central gap claim still holds. read the letter →

arxiv 2504.14964 v1 pith:TUZODANP submitted 2025-04-21 cs.AI cs.CY

Emir Catir , Robin Claesson , Rodothea Myrsini Tsoupidi This is my paper · ORCID

classification cs.AIcs.CY

keywords largelanguagemodelscodegenerationprogrammingassignmentscomputerscienceeducationadvancedCScoursesLLMevaluationconstraintadherenceintroductory

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper attempts to establish that large language models are reliable at generating working code for introductory programming assignments but rarely produce fully correct solutions for advanced, second- and third-year computer science problems. Four LLM tools were each asked to solve twelve handpicked assignments in Python, Java, and C, and every generated program was graded against 1000 randomly generated test cases using the authors' reference verifier. On the advanced problems, only three of 108 tool-language-problem combinations achieved a perfect score: two for the traveling-salesman problem and one for a dynamic-programming board-cutting problem. The models usually recognized the underlying algorithm, but they ignored task-specific constraints or substituted greedy heuristics for the required method. If the results hold, teachers in advanced courses can make assignments harder to solve by adding variations and constraints to well-known algorithmic problems.

What carries the argument

The evaluation rests on a three-part prompt (problem description, input format, output format) fed to each tool, followed by automated verification: a Verifier computes the reference answer for each of 1000 random instances, and accuracy is the fraction of instances the generated program answers correctly. A problem is considered solved only when accuracy is 100%; for problems with multiple valid answers, such as TSP, the verifier checks structural properties of the answer instead of a single number. This machinery matters because every conclusion about LLM capability is defined by it, and it turns vague impressions of coding skill into a countable score across languages and tools.

What would settle it

Run the paper's nine advanced prompts through a newer LLM and grade against the original courses' hidden test suites. If most generated programs pass those suites, the central claim that advanced CS assignments are rarely solved would be weakened.

Watch

Extended reading notes

Core claim

The paper's central discovery is a sharp drop in correctness exactly where courses stop teaching basic syntax and start teaching algorithm design. Using accuracy defined as the share of correctly verified test cases, with a problem counted as solved only at 100%, the models solved the CS1 baseline problems almost everywhere, including perfect scores for the shortest-path baseline, but almost never solved the nine CS4/CS5 problems. In the advanced set, Github Copilot produced the only perfect solutions, while the other tools mostly generated heuristics, solutions that solved a simplified version of the prompt, or code that did not run. The authors conclude that LLMs are good at naming and sketching the textbook algorithm but weak at conforming to the constraints that make an assignment a genuine test of understanding.

Load-bearing premise

The whole comparison assumes that the authors' reference solutions are error-free and that 1000 random test cases are representative enough of what an advanced course would actually test.

Editorial extensions

If this is right

Instructors can no longer assume that textbook algorithm assignments are LLM-proof: the tools recognize the underlying problem and return plausible code, so assessment must probe constraints and edge cases.
Adding a single problem-specific constraint, as in the TV-Zapping problem, drove accuracy down to 96.4 percent at best and near zero for most tools, indicating that constraint variation is an effective way to make assignments harder for LLMs.
GitHub Copilot was the only tool to produce any perfect advanced solutions, while the other tools more often produced non-functional code or heuristics.
C code was the most fragile, with twelve compilation or runtime errors across the advanced problems, so language choice materially changes what a student gets from a tool.
Students using LLMs on advanced work will frequently receive partially correct code that passes easy cases, which makes hidden-constraint testing important for fair grading.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The perfect scores on TSP and Cut Boards should be read cautiously: for NP-hard problems the random test generator must keep instances small enough to verify exhaustively, so larger or adversarial instances could reveal failures that the paper's 1000-case suites do not.
Because the dominant failure pattern is ignoring constraints, a cheap robustness experiment would be to move a constraint sentence earlier or later in the prompt and measure accuracy shifts; if placement matters, students could game tools by rewording assignments.
A likely next step is repair: since many generated solutions are close, feeding failing test outputs back to the model could convert some partial solutions into correct ones, which would raise the effective usefulness of the tools beyond the paper's one-shot accuracy.
The paper's setup amounts to a constraint-adherence benchmark; if the pattern generalizes, advanced CS assignments could serve as a practical testbed for measuring how well LLMs follow multi-part instructions.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

The paper does something straightforward and useful: it benchmarks four LLM tools (Copilot, CodePal, Llama-3-70B, Mistral-Nemo) on 12 programming assignments, from CS1 to CS4/CS5, in Python, Java, and C. The setup is honest: 1000 random tests per problem-tool-language, with a shared seed, and the data is on GitHub. The main finding—LLMs nail most intro problems but almost never fully solve advanced ones—is clear in Table 2 and robust to the issues I mention below. That is a real contribution, because prior work mostly stopped at CS2. The paper also gives teachers something to take away: LLMs often identify the right algorithm but miss constraints, so adding a twist or constraint to a known problem is a cheap way to keep assignments LLM-resistant.

The soft spots are, in order of seriousness. First, the TSP verifier. Section 4.4 says that for problems with many valid answers, 'such as TSP,' verification checks that nodes exist in a path and bounds are not exceeded—but not that the tour is optimal. The two TSP cells at 100% (Copilot Python and Java) therefore only prove feasibility, not correctness for the assignment's actual request (shortest route). The stress-test note is right; this is a concrete flaw. It shrinks the advanced 100% count from 3 to 1 (Cut Boards). The qualitative conclusion doesn't change, but the RQ1 numbers should be corrected and the verifier either upgraded to exact optimality or those cells relabeled.

Second, the 'minor changes' to generated code (Appendix A) are documented, which is good, but their effect isn't discussed. Lowering constants on P7 and P9 could affect results in non-obvious ways; changing output format is benign, but a reader can't tell how many wrong answers were actually fixed by these edits. A sensitivity check or at least a comment on which changes could flip a result would address this.

Third, there are no error bars or seed variance. The paper uses one seed and reports point accuracies. Given the small number of problems and the fact that a single test seed can make a 90%-accurate program look 100%, confidence intervals would be cheap to add and would materially strengthen the claim.

On citations, the related work is appropriate and doesn't oversell novelty. The paper does overstate the TSP result slightly in the RQ1 discussion, but the limitations section is honest about prompt engineering and the need for future work.

This paper deserves a serious referee. It's a legitimate extension of the evaluation literature with a clear educational audience. I'd recommend send it to review with a request for a revised TSP analysis and seed variance. My own verdict would be a conditional accept.

Referee Report

3 major / 5 minor

Summary. This paper evaluates the code-generation performance of four LLM tools (GitHub Copilot, CodePal, Llama3-70b, and Mistral-Nemo) on twelve programming problems: three introductory CS1 problems as a baseline and nine advanced CS4/CS5 problems, each in Java, Python, and C. For each problem, the authors generate 1000 random test cases, run the generated programs against those tests, and compare the outputs with their own reference solutions, defining accuracy as the fraction of correctly solved test cases and correctness as 100% accuracy. The main empirical finding is that LLMs solve most introductory problems but rarely solve advanced assignments correctly; however, they often produce partial solutions or recognize the underlying algorithm. The paper also reports tool- and language-level differences, concluding that GitHub Copilot outperforms the other tools and that C causes the most errors.

Significance. If the results hold, this is a useful empirical contribution to the growing literature on LLMs in computing education, complementing earlier work that focused on introductory programming. The study's strengths include a handpicked problem set spanning multiple institutions and levels, a handwritten baseline problem to reduce memorization risk, a public repository with prompts and data, and clear operational definitions of accuracy and correctness. The central qualitative claim—that LLMs are far more successful on introductory than advanced assignments—is visible in Table 2 and is robust to the specific verifier weaknesses discussed below. However, the quantitative strength of the claims is limited by unresolved verification and statistical issues, so the paper needs revision before its specific numbers can be taken at face value.

major comments (3)

[4.4, Table 2, P5 (TSP)] The Traveling Salesman Problem verifier checks feasibility only, not optimality. Section 4.4 states that for problems with many valid answers, 'such as the Traveling Salesman Problem (TSP) problem', the verification step 'checks each aspect of the given answer, such as verifying all nodes exist in a path and that bounds are not exceeded.' The TSP assignment asks for the shortest route, so a feasible Hamiltonian tour is not a correct solution. Consequently, the two 100% accuracy cells for TSP (Copilot in Python and Copilot in Java) do not establish that those programs solve the assignment. Since the paper reports only three advanced-problem combinations with 100% accuracy, and two of them are TSP, the specific RQ1 count is overstated. The paper should either verify optimality against an exact solver (e.g., exhaustive search on the same small instances) or re-classify these cells.
[4.4, Definition 1, Table 2] Accuracy is reported as a point estimate with t=1000 test cases and no variance across test-generation seeds. The test cases for all solutions use the same random seed per problem, so the reported differences between tools and languages (e.g., P5 Python: Copilot 100 vs CodePal 0; P7 Java: Copilot 0 vs Llama 77.4) could change substantially under a different seed, especially for NP-hard problems where instance-size distributions matter. This lack of error bars or confidence intervals weakens the quantitative comparisons in RQ3 and the partial-solution thresholds used in Section 5.3 (e.g., accuracy > 5%). The authors should run multiple test seeds and report variances, or at minimum justify why a single seed is representative.
[4.4 and 6] The test-suite generation process is not described in enough detail to assess whether the 1000 random cases are representative of real assignments. For NP-hard problems like Boxes and TSP, the instances must be small enough for exhaustive verification, so the difficulty distribution may differ from typical course assignments. Moreover, the reference solutions are assumed correct without supporting evidence; a bug in any reference solution would shift every accuracy in Table 2. The manuscript should document the instance-generation parameters, the input distributions, and the verification logic per problem, and should make the reference solutions and generators available for inspection alongside the existing prompt/data repository.

minor comments (5)

[5.1] The sentence 'Among P4 to P12, only P4,TSP and P11, Cut Boards, have two and one correct results, respectively' mislabels the TSP problem as P4; TSP is P5 according to Table 1.
[5.3] The text 'it seems that CP in Python generates the same heuristic as CG in C' should read 'GC' instead of 'CG' for GitHub Copilot.
[Listing 1.1] Listing 1.1 contains stray semicolons in Python code (e.g., 'return b o x e s _ n e e d e d ;'), which are likely transcription artifacts; the code should be cleaned up to match what was actually executed.
[2 Related Work] The sentence 'Their results show that the ACG performs better than students in this course Finnie-Ansley et al..' is grammatically incomplete and has a duplicated period; it should be rewritten and properly cited.
[Table 2 header] The table header is visually ambiguous: the HF column is split into LL and MS subcolumns, making it look like five tools rather than four. Use a clearer multi-row header that groups Llama and Mistral under a single Hugging Face tool category.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper reports a direct empirical measurement against independently written reference solutions.

full rationale

This paper is an empirical evaluation, not a derivation. The central claim—that LLMs solve CS1 assignments better than CS4/CS5 assignments—is supported by running generated programs against 1000 random test cases per problem and comparing outputs with the authors' reference solutions, as described in Section 4.4. No parameter is fitted to a subset of the data and then used to predict a closely related quantity; correctness is defined directly in Definitions 1 and 2, and the accuracy values in Table 2 are measured results rather than consequences of the definition. The bibliography contains no work by Catir, Claesson, or Tsoupidi, so no load-bearing premise is justified only by a self-citation, and no uniqueness theorem or ansatz is imported from the authors' own prior papers. The one notable weakness is that, for problems with many valid answers such as TSP, the verifier checks feasibility ('all nodes exist in a path and that bounds are not exceeded') rather than optimality, which threatens the validity of individual 100% TSP cells; however, this is an evaluation-validity concern, not circularity, and it cuts against the paper's advanced-problem difficulty claim only by potentially over-crediting LLMs. Even discounting both TSP cells, the qualitative gap between CS1 and CS4/CS5 remains, so the central claim does not reduce to the paper's own inputs by construction.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central measurement rests on three assumptions: correct reference solutions, representative random tests, and algorithm-preserving modifications. The paper provides no formal verification or independent benchmarking of these foundations.

free parameters (3)

Test suite size t = 1000
The accuracy metric divides number of correct cases by t=1000 (Definition 1). The chosen sample size affects the precision of every reported percentage.
Floating-point tolerance = 0.1
The verifier accepts any answer up to one decimal point from the correct answer as correct (Section 4.4), which can inflate accuracy scores for numerical problems.
Partial-solution accuracy threshold = >5%
In RQ3, solutions with accuracy greater than 5% are counted as partially correct. Changing this threshold would change the qualitative summary of which tools and languages produce useful partial solutions.

assumptions (3)

domain assumption The reference solutions are correct implementations of the problems.
Section 4.4 states that outputs are verified using 'our reference solution, the Verifier'. No independent validation, tests, or proofs are provided for these reference solutions.
domain assumption Randomly generated test cases are representative of the problem space.
Section 4.4 generates 1000 random test cases per problem with a single seed. No coverage analysis or distribution description is given, so the difficulty of the sampled instances relative to real course assignments is unknown.
domain assumption Manual modifications to generated code do not alter the algorithm.
Appendix A lists 19 solutions that were changed to fix output format, imports, main() calls, floating-point types, or runtime constants. The claim that none of these changes alters the algorithm is asserted but not verified by tests such as differential testing on original versus modified code.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Evaluating Code Generation of LLMs in Advanced Computer Science Problems." pith.science (2026). https://pith.science/paper/TUZODANP

@misc{pith2026250414964,
  author       = {Pith},
  title        = {Pith review of: Evaluating Code Generation of LLMs in Advanced Computer Science Problems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TUZODANP}},
  note         = {Machine review of arXiv:2504.14964}
}

read the original abstract

Large Language Models (LLMs), such as GitHub Copilot and ChatGPT have become popular among programming students. Students use LLMs to assist them in programming courses, including generating source code. Previous work has evaluated the ability of LLMs in solving introductory-course programming assignments. The results have shown that LLMs are highly effective in generating code for introductory Computer Science (CS) courses. However, there is a gap in research on evaluating LLMs' ability to generate code that solves advanced programming assignments. In this work, we evaluate the ability of four LLM tools to solve programming assignments from advanced CS courses in three popular programming languages, Java, Python, and C. We manually select 12 problems, three problems from introductory courses as the baseline and nine programming assignments from second- and third-year CS courses. To evaluate the LLM-generated code, we generate a test suite of 1000 test cases per problem and analyze the program output. Our evaluation shows that although LLMs are highly effective in generating source code for introductory programming courses, solving advanced programming assignments is more challenging. Nonetheless, in many cases, LLMs identify the base problem and provide partial solutions that may be useful to CS students. Furthermore, our results may provide useful guidance for teachers of advanced programming courses on how to design programming assignments.

Figures

Figures reproduced from arXiv: 2504.14964 by the authors.

**Figure 2.** shows the overview of the methodology, which consists of two main parts 1) code generation and 2) code verification part. The code generation part includes selecting the LLM tools (Section 4.1) and the problems (Section 4.2) to evaluate. For each of the problems, we create a prompt and provide this prompt as input to each LLM tool to generate source code (Section 4.3). The code verification part includes evaluating … view at source ↗

**Figure 3.** Cow Game: Generated code by Github Copilot in C and corrected code. Summary The ability of LLMs to generate correct solutions in advanced CS problems compared to CS1 problems is low. Among all LLMs Github Copilot is the only tool that is able to generate correct results. 5.2 RQ2: What is the capability of LLM tools at identifying the problem to solve? This research question investigates the ability of LLM tools to i… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 14 canonical work pages

[1]

arXiv preprint arXiv:2404.04603 (2024)

Arora, C., Venaik, U., Singh, P., Goyal, S., Tyagi, J., Goel, S., Singhal, U., Kumar, D.: Analyzing llm usage in an advanced computing class in india. arXiv preprint arXiv:2404.04603 (2024)

arXiv 2024
[2]

In: Proceedings of the 54th ACM Technical Symposium on Computer Science Education V

Becker, B.A., Denny, P., Finnie-Ansley, J., Luxton-Reilly, A., Prather, J., Santos, E.A.: Programming is hard-or at least it used to be: Educational opportunities and challenges of ai code generation. In: Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1, pp. 500–506 (2023)

work page 2023
[3]

arXiv preprint arXiv:2107.03374 (2021)

Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H.P., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., Bavarian, M., Winter, C., Tillet, P., Such, F.P., Cummings, D., Plappert, M., Chan...

arXiv 2021
[4]

Insights from the Frontline: GenAI Utilization Among Software Engineering Students

Choudhuri, R., Ramakrishnan, A., Chatterjee, A., Trinkenreich, B., Stein- macher, I., Gerosa, M., Sarma, A.: Insights from the frontline: Genai utiliza- tion among software engineering students. arXiv preprint arXiv:2412.15624 (2024)

work page Pith review arXiv 2024
[5]

"ChatGPT Is Here to Help, Not to Replace Anybody" -- An Evaluation of Students' Opinions On Integrating ChatGPT In CS Courses

Cipriano, B.P., Alves, P.: "chatgpt is here to help, not to replace anybody"– an evaluation of students’ opinions on integrating chatgpt in cs courses. arXiv preprint arXiv:2404.17443 (2024)

work page Pith review arXiv 2024
[6]

In: Proceedings of the 54th ACM Technical Symposium on Computer Science Education V

Denny, P., Kumar, V., Giacaman, N.: Conversing with copilot: Exploring prompt engineering for solving cs1 problems using natural language. In: Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1, pp. 1136–1142 (2023)

work page 2023
[7]

In: Proceedings of the 24th Australasian Computing Education Conference, pp

Finnie-Ansley, J., Denny, P., Becker, B.A., Luxton-Reilly, A., Prather, J.: The robots are coming: Exploring the implications of openai codex on intro- ductory programming. In: Proceedings of the 24th Australasian Computing Education Conference, pp. 10–19 (2022)

work page 2022
[8]

In: Proceedings of the 25th Australasian Computing Education Conference, pp

Finnie-Ansley, J., Denny, P., Luxton-Reilly, A., Santos, E.A., Prather, J., Becker, B.A.: My ai wants to know if this will be on the exam: Testing openai’s codex on cs2 programming exercises. In: Proceedings of the 25th Australasian Computing Education Conference, pp. 97–104 (2023)

work page 2023

Show all 17 references

[9]

Learning and individual differences103, 102274 (2023) 14 E

Kasneci, E., Seßler, K., Küchemann, S., Bannert, M., Dementieva, D., Fis- cher, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., et al.: Chat- gpt for good? on opportunities and challenges of large language models for education. Learning and individual differences103,...

2023
[10]

In: Proceedings of the 24th Koli Calling International Conference on Computing Education Research, pp

Keuning, H., Alpizar-Chacon, I., Lykourentzou, I., Beehler, L., Köppe, C., de Jong, I., Sosnovsky, S.: Students’ perceptions and use of generative ai tools for programming across different computing courses. In: Proceedings of the 24th Koli Calling International Conference on ...

2024
[11]

In: Proceedings of the 24th Koli Calling International Conference on Computing Education Research, pp

Korpimies, K., Laaksonen, A., Luukkainen, M.: Unrestricted use of llms in a software project course: Student perceptions on learning and impact on course performance. In: Proceedings of the 24th Koli Calling International Conference on Computing Education Research, pp. 1–7 (2024)

2024
[12]

In: Proceedings of the 2024 on Innovation and Technology in Com- puter Science Education V

Margulieux, L.E., Prather, J., Reeves, B.N., Becker, B.A., Cetin Uzun, G., Loksa, D., Leinonen, J., Denny, P.: Self-regulation, self-efficacy, and fear of failure interactions with how novices use llms to solve programming prob- lems. In: Proceedings of the 2024 on Innovation ...

2024
[13]

In: Shaw, P

Michailidis, K., Tsouros, D., Guns, T.: Constraint Modelling with LLMs Using In-Context Learning. In: Shaw, P. (ed.) 30th International Confer- ence on Principles and Practice of Constraint Programming (CP 2024), Leibniz International Proceedings in Informatics (LIPIcs), vol. ...

2024 doi
[14]

In: Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V

Reeves, B., Sarsa, S., Prather, J., Denny, P., Becker, B.A., Hellas, A., Kim- mel, B., Powell, G., Leinonen, J.: Evaluating the performance of code gen- eration models for solving parsons problems with small prompt variations. In: Proceedings of the 2023 Conference on Innovati...

2023
[15]

Communications of the ACM29(9), 850–858 (1986)

Soloway, E.: Learning to program= learning to construct mechanisms and explanations. Communications of the ACM29(9), 850–858 (1986)

1986
[16]

1 (2023) A Changed solutions We made some small alterations to the LLM-generated solutions before the tests

Šavelka, J., Agarwal, A., Bogart, C., Song, Y., Sakr, M.F.: Can generative pre-trained transformers (gpt) pass assessments in higher education pro- gramming courses? Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 (2023) A Cha...

2023
[17]

No changes were made to the algorithms of the solutions

to lower the runtime of the programs. No changes were made to the algorithms of the solutions. Table 3 list all solutions that had some changes made before running the tests. There were four different types of changes made. The most common change Evaluating Advanced Course Ass...

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.

[1] [1]

arXiv preprint arXiv:2404.04603 (2024)

Arora, C., Venaik, U., Singh, P., Goyal, S., Tyagi, J., Goel, S., Singhal, U., Kumar, D.: Analyzing llm usage in an advanced computing class in india. arXiv preprint arXiv:2404.04603 (2024)

arXiv 2024

[2] [2]

In: Proceedings of the 54th ACM Technical Symposium on Computer Science Education V

Becker, B.A., Denny, P., Finnie-Ansley, J., Luxton-Reilly, A., Prather, J., Santos, E.A.: Programming is hard-or at least it used to be: Educational opportunities and challenges of ai code generation. In: Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1, pp. 500–506 (2023)

work page 2023

[3] [3]

arXiv preprint arXiv:2107.03374 (2021)

Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H.P., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., Bavarian, M., Winter, C., Tillet, P., Such, F.P., Cummings, D., Plappert, M., Chan...

arXiv 2021

[4] [4]

Insights from the Frontline: GenAI Utilization Among Software Engineering Students

Choudhuri, R., Ramakrishnan, A., Chatterjee, A., Trinkenreich, B., Stein- macher, I., Gerosa, M., Sarma, A.: Insights from the frontline: Genai utiliza- tion among software engineering students. arXiv preprint arXiv:2412.15624 (2024)

work page Pith review arXiv 2024

[5] [5]

"ChatGPT Is Here to Help, Not to Replace Anybody" -- An Evaluation of Students' Opinions On Integrating ChatGPT In CS Courses

Cipriano, B.P., Alves, P.: "chatgpt is here to help, not to replace anybody"– an evaluation of students’ opinions on integrating chatgpt in cs courses. arXiv preprint arXiv:2404.17443 (2024)

work page Pith review arXiv 2024

[6] [6]

In: Proceedings of the 54th ACM Technical Symposium on Computer Science Education V

Denny, P., Kumar, V., Giacaman, N.: Conversing with copilot: Exploring prompt engineering for solving cs1 problems using natural language. In: Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1, pp. 1136–1142 (2023)

work page 2023

[7] [7]

In: Proceedings of the 24th Australasian Computing Education Conference, pp

Finnie-Ansley, J., Denny, P., Becker, B.A., Luxton-Reilly, A., Prather, J.: The robots are coming: Exploring the implications of openai codex on intro- ductory programming. In: Proceedings of the 24th Australasian Computing Education Conference, pp. 10–19 (2022)

work page 2022

[8] [8]

In: Proceedings of the 25th Australasian Computing Education Conference, pp

Finnie-Ansley, J., Denny, P., Luxton-Reilly, A., Santos, E.A., Prather, J., Becker, B.A.: My ai wants to know if this will be on the exam: Testing openai’s codex on cs2 programming exercises. In: Proceedings of the 25th Australasian Computing Education Conference, pp. 97–104 (2023)

work page 2023