REVIEW 4 cited by
Evaluating Language Models for Generating and Judging Programming Feedback
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The emergence of large language models (LLMs) has transformed research and practice across a wide range of domains. Within the computing education research (CER) domain, LLMs have garnered significant attention, particularly in the context of learning programming. Much of the work on LLMs in CER, however, has focused on applying and evaluating proprietary models. In this article, we evaluate the efficiency of open-source LLMs in generating high-quality feedback for programming assignments and judging the quality of programming feedback, contrasting the results with proprietary models. Our evaluations on a dataset of students' submissions to introductory Python programming exercises suggest that state-of-the-art open-source LLMs are nearly on par with proprietary models in both generating and assessing programming feedback. Additionally, we demonstrate the efficiency of smaller LLMs in these tasks and highlight the wide range of LLMs accessible, even for free, to educators and practitioners.
Forward citations
Cited by 4 Pith papers
-
Wisdom of the Crowd, Without the Crowd: A Socratic LLM for Asynchronous Deliberation on Perspectivist Data
A Socratic LLM that questions annotators during labeling improved post-deliberation accuracy and confidence compared to a prior synchronous human-deliberation benchmark.
-
On the Effectiveness of LLM-as-a-judge for Code Generation and Summarization
Even the best tested LLM judge, GPT-4-turbo, frequently misjudges code correctness (e.g., 50% of wrong Java functions judged correct) but agrees moderately with humans when judging code summary quality.
-
That's Not the Feedback I Need! -- Student Engagement with GenAI Feedback in the Tutor Kai
In an eye-tracking study of 11 students, novices fixated twice as long on AI-generated feedback as experienced peers, relied on it instead of compiler output, and could not comprehend about 20% of the AI feedback they...
-
FEAT: A Preference Feedback Dataset through a Cost-Effective Auto-Generation and Labeling Framework for English AI Tutoring
A cost-effective dataset framework for English AI tutoring finds that mixing 5-10% human-annotated feedback with LLM-generated feedback outperforms using only human-annotated feedback.
Discussion (0). Continue with ORCID to comment.