{"paper":{"title":"The Effects of Structured LLM-Generated Feedback on Programming Assignment Performance","license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","headline":"LLM-generated feedback is associated with faster time to solution for programming assignments than compiler messages alone.","cross_cats":[],"primary_cat":"cs.HC","authors_text":"Arto Hellas, Bita Akram, Evanfiya Logacheva, Francisco Castro, Jing Fan, Juho Leinonen, Narges Norouzi, Peter Brusilovsky, Tsvetomila Mihaylova","submitted_at":"2026-05-16T11:02:33Z","abstract_excerpt":"When programming students encounter errors in their code, compiler messages or static analysis output often provide limited guidance, particularly for novice programmers. Personalized feedback from instructors can be effective but does not scale well. Recent advances in large language models (LLMs) enable automated feedback generation at scale. This study examines whether LLM-generated feedback with different levels of guidance is associated with differences in students' problem-solving behavior. We analyze effects on time to solution and number of attempts, and examine whether these effects d"},"claims":{"count":4,"items":[{"kind":"strongest_claim","text":"Results from an online programming course show that LLM-generated feedback is associated with faster time to solution compared to the no-feedback baseline, with less guided feedback showing slightly stronger effects.","source":"verdict.strongest_claim","status":"machine_extracted","claim_id":"C1","attestation":"unclaimed"},{"kind":"weakest_assumption","text":"The study assumes that the three feedback types were implemented with meaningfully different levels of guidance and that students actually read and used the feedback rather than ignoring it or treating all conditions the same.","source":"verdict.weakest_assumption","status":"machine_extracted","claim_id":"C2","attestation":"unclaimed"},{"kind":"one_line_summary","text":"LLM-generated feedback was associated with faster time to solution for programming students than compiler messages alone, with less-guided versions showing slightly stronger effects.","source":"verdict.one_line_summary","status":"machine_extracted","claim_id":"C3","attestation":"unclaimed"},{"kind":"headline","text":"LLM-generated feedback is associated with faster time to solution for programming assignments than compiler messages alone.","source":"verdict.pith_extraction.headline","status":"machine_extracted","claim_id":"C4","attestation":"unclaimed"}],"snapshot_sha256":"7053297b16f946c3fb5949a5f7603b06e40b5ff9d2d2298ae79723943f34957f"},"source":{"id":"2605.16933","kind":"arxiv","version":1},"verdict":{"id":"8f8c2993-3353-4647-990f-991b6ffa14a9","model_set":{"reader":"grok-4.3"},"created_at":"2026-05-19T20:36:28.362267Z","strongest_claim":"Results from an online programming course show that LLM-generated feedback is associated with faster time to solution compared to the no-feedback baseline, with less guided feedback showing slightly stronger effects.","one_line_summary":"LLM-generated feedback was associated with faster time to solution for programming students than compiler messages alone, with less-guided versions showing slightly stronger effects.","pipeline_version":"pith-pipeline@v0.9.0","weakest_assumption":"The study assumes that the three feedback types were implemented with meaningfully different levels of guidance and that students actually read and used the feedback rather than ignoring it or treating all conditions the same.","pith_extraction_headline":"LLM-generated feedback is associated with faster time to solution for programming assignments than compiler messages alone."},"integrity":{"clean":true,"summary":{"advisory":0,"critical":0,"by_detector":{},"informational":0},"endpoint":"/pith/2605.16933/integrity.json","findings":[],"available":true,"detectors_run":[{"name":"doi_title_agreement","ran_at":"2026-05-19T21:01:19.139529Z","status":"completed","version":"1.0.0","findings_count":0},{"name":"doi_compliance","ran_at":"2026-05-19T20:51:10.857366Z","status":"completed","version":"1.0.0","findings_count":0},{"name":"cited_work_retraction","ran_at":"2026-05-19T20:22:00.190460Z","status":"completed","version":"1.0.0","findings_count":0},{"name":"claim_evidence","ran_at":"2026-05-19T18:41:56.252557Z","status":"completed","version":"1.0.0","findings_count":0},{"name":"ai_meta_artifact","ran_at":"2026-05-19T18:33:26.334217Z","status":"skipped","version":"1.0.0","findings_count":0}],"snapshot_sha256":"e39151dd09c094f8068ddc4eeb25c1e5d79199ac511e655777d77b5735481918"},"references":{"count":69,"sample":[{"doi":"10.1007/978-3-031-98417-4_30","year":2025,"title":"Moraes, Fernanda Oliveira, and Carla A","work_id":"1a694fe2-47ef-4ef5-868b-4beb528c9022","ref_index":1,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":1994,"title":"Craig Boyle and Antonio O. Encarnacion. 1994. Metadoc: an adaptive hypertext reading system. User Modeling and User-Adapted Interaction, 4(1):1--19","work_id":"7ffc307e-5904-4de0-9223-461a11c9175b","ref_index":3,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"10.2307/2290687","year":1993,"title":"Breslow and David G","work_id":"ad8db19d-1205-4683-9614-1d17092de5cd","ref_index":4,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2025,"title":"Neil CC Brown, Pierre Weill-Tessier, Juho Leinonen, Paul Denny, and Michael K \\\"o lling. 2025. Howzat? appealing to expert judgement for evaluating human and ai next-step hints for novice programmers.","work_id":"10362f55-7a23-4e2e-bada-64972f6771a9","ref_index":5,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2018,"title":"Marc Brysbaert and Micha \\\"e l Stevens. 2018. Power analysis and effect size in mixed effects models: A tutorial. Journal of cognition, 1(1):9","work_id":"093cb6ab-68b6-4d82-aeef-1d24811f9fab","ref_index":6,"cited_arxiv_id":"","is_internal_anchor":false}],"resolved_work":69,"snapshot_sha256":"f3126f4539c3f43060fd7b4a0b793a6f439371d98ed37f90d1aecf98b15cf312","internal_anchors":0},"formal_canon":{"evidence_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"}