Pith. sign in

REVIEW 3 cited by

Large Language Models in Introductory Programming Education: ChatGPT's Performance and Implications for Assessments

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2308.08572 v1 pith:VJPIUD4K submitted 2023-08-15 cs.SE cs.AIcs.HC

Natalie Kiesler , Daniel Schiffner This is my paper · ORCID
classification cs.SEcs.AIcs.HC
keywords llmsperformanceprogrammingwereassessmentavailabilitycodecodingbat
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

This paper investigates the performance of the Large Language Models (LLMs) ChatGPT-3.5 and GPT-4 in solving introductory programming tasks. Based on the performance, implications for didactic scenarios and assessment formats utilizing LLMs are derived. For the analysis, 72 Python tasks for novice programmers were selected from the free site CodingBat. Full task descriptions were used as input to the LLMs, while the generated replies were evaluated using CodingBat's unit tests. In addition, the general availability of textual explanations and program code was analyzed. The results show high scores of 94.4 to 95.8% correct responses and reliable availability of textual explanations and program code, which opens new ways to incorporate LLMs into programming education and assessment.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. When LLM Tutoring Responses Work: Evidence from Student Programming Conversations

    cs.HC 2026-07 accept novelty 6.0 of 10

    LLM response styles show small but significant associations with productive student continuation in programming dialogues, with larger differences under high cognitive load and debugging.

  2. Students' Feedback Requests and Interactions with the SCRIPT Chatbot: Do They Get What They Ask For?

    cs.AI 2025-07 conditional novelty 6.0 of 10

    In a 136-student trial, novice programmers' feedback requests to a purpose-built ChatGPT tutor followed a consistent sequence, and the tutor's responses aligned with requested feedback types in 75% of exchanges.

  3. That's Not the Feedback I Need! -- Student Engagement with GenAI Feedback in the Tutor Kai

    cs.CY 2025-06 conditional novelty 5.0 of 10

    In an eye-tracking study of 11 students, novices fixated twice as long on AI-generated feedback as experienced peers, relied on it instead of compiler output, and could not comprehend about 20% of the AI feedback they...

Pith tools