{"id":"4c366f9e-9751-4cde-8794-b8d9a4b52f0e","arxiv_id":"2412.03516","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"GPT-4 generated the requested feedback type in 63 of 66 test cases, but evaluation overlap with prompt development and misleading outputs limit the strength of the result.","lead":"Researchers tested whether GPT-4 can be prompted to give a specific type of programming feedback, such as only pointing out mistakes or only giving next-step hints. The model delivered the requested feedback type in 63 of 66 cases, but the evaluation included data used to design the prompt and some feedback was misleading.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Prompt-development data is included in the 66-message evaluation, so the 63/66 match rate reflects prompt tuning, not held-out performance.","rationale":"The reader's weakest assumption concerns coder bias in the expert consensus coding. That is plausible, but the paper already attempts to mitigate it by concealing the requested type and randomizing order. The inclusion of prompt-development data is not mitigated at all: the prompt was deliberately tuned on 18 of the 66 evaluation messages, and no held-out analysis is reported. Since RQ1's answer is a single match count, this overlap directly inflates the numerator. The issue is not internal inconsistency in the coding; it is that the evaluation protocol violates the basic train/test separation required to claim that a prompt can generate a specified feedback type for arbitrary submissions. The qualitative examples and the characteristics analysis remain useful, and the study's conclusion about feasibility might survive a held-out reanalysis, but the headline quantitative claim cannot be accepted at face value without that reanalysis. Therefore I do not propose REJECT; the paper can be repaired by re-reporting the statistics separately for the development and held-out subsets. The conditional verdict already fits, and this concern makes the condition explicit.","tokens_in":25329,"tokens_out":3632,"duration_ms":34615,"concrete_test":"Recompute the DFT-AFT match rate from Table 5 separately for the three prompt-development submissions (10_TEOS, 13_NEGF, 01_TTBSA; 18 messages) and the eight added submissions (48 messages). If the held-out match rate is substantially below 63/66, the central claim should be revised to exclude or separately report development data. This single reanalysis settles whether the headline number is an artifact of prompt fitting.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.3.1 describes using one submission per exercise (Physics, Triangles, NegaFib) for prompt development, and Section 3.4 details five iterative prompt revisions based on those submissions. Table 4 then labels exactly those three submissions (10_TEOS, 13_NEGF, 01_TTBSA) as the 'Test set', and Table 5 reports DFT-AFT matches for all 66 messages, including the 18 generated from those same three submissions. Because the prompt was revised until it produced the intended types on those specific submissions, those 18 cases are not an independent test of prompt generality. Excluding them leaves only 48 held-out messages, and the paper never reports the match rate on that subset. The headline '63 out of 66' therefore overstates the controllability claim; the true capability on unseen submissions is unknown. Section 5 acknowledges single-model and task-domain limits but does not flag this overlap.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper investigates whether GPT-4 can be prompted to generate feedback for introductory programming submissions in six predefined types from Keuning et al.'s feedback taxonomy. The authors iteratively refined a single prompt over five iterations using three student submissions, then applied the final prompt to 11 submissions (including the same three) across six exercises and four programming languages, producing 66 feedback messages. Two expert coders coded the feedback types and 13 characteristics, resolving disagreements by consensus. The paper reports that in 63 of 66 cases the desired feedback type matched the actual feedback type, notes that 19 messages contained additional types, and presents a qualitative analysis of characteristics such as misleading feedback, performance percentages, and concept explanations. The authors conclude that controllable generation of specific feedback types is promising, while acknowledging single-model and single-domain limitations.","tokens_in":25470,"tokens_out":8818,"duration_ms":88845,"significance":"If the core result were robust, this paper would be a useful contribution: it would show that a single prompt template can steer an LLM toward qualitatively distinct feedback types, and it provides a concrete prompt (Listing 1), a transparent iterative design process, and a taxonomy-based analysis that could inform follow-up work on feedback effects and educational tool design. The study covers authentic student submissions in multiple languages, and the authors are commendably transparent about misleading outputs and about cases where additional feedback types appear. However, the central quantitative claim is currently not established at the level the title and abstract suggest: the evaluation includes the prompt-development submissions, the coding reliability is not quantified, and the reported match rate is inclusive rather than strict. These issues need to be resolved before the claim of type-controllability can be accepted as a generalizable result.","major_comments":[{"comment":"The three submissions labeled as the test set in Table 4 (10_TEOS, 13_NEGF, 01_TTBSA) are exactly the submissions used for prompt development in Section 3.3.1, and Section 3.4 states that the prompt was refined by generating all six feedback types for those three submissions across five iterations. Therefore, 18 of the 66 messages in Table 5 are not independent evaluations of the final prompt; they reflect prompt fitting rather than generalization. Please report the DFT-AFT match rate for the eight held-out submissions separately from the three development submissions, and state the match rate excluding those 18 development messages. The headline '63 out of 66' currently conflates prompt development with evaluation.","section":"Sections 3.3.1, 3.4, and Tables 4-5"},{"comment":"The feedback analysis relies on two expert coders coding independently and resolving disagreements by consensus, but no measure of inter-rater reliability is reported for the feedback-type coding or for the 13 characteristic categories. The central 63/66 match count is produced entirely by this coding, so coding reliability is load-bearing for the paper's main claim. Please report initial agreement, both raw and chance-corrected (e.g., Cohen's kappa), for the AFT codes, and clarify whether the reported AFT values reflect the initial independent codes or the consensus outcome. Without this information, the main result cannot be distinguished from coder interpretation.","section":"Section 3.5 and Table 5"},{"comment":"The paper reports 63/66 matches and, in the same paragraph, notes that 19 messages contained additional feedback types beyond the requested one. The reported 'match' rate therefore appears to mean 'the desired type is among the actual types,' not 'the generated feedback consists only of the desired type.' The final prompt (Listing 1, line 23) explicitly instructs the model not to provide additional information, so the strict-match rate (DFT equals AFT, with no other types) is the more relevant measure of type control. Please report the strict-match rate separately and discuss the 19 mixed-type messages, such as KM appearing with KH and KTC, as partial deviations from the prompt instruction.","section":"Section 4.1 and Table 5"},{"comment":"The study reports a single generation per task-type pair (66 total) using the ChatGPT interface, but it does not state the exact GPT-4 model version, sampling parameters, or number of runs. Because GPT-4 is stochastic and the paper's abstract itself emphasizes randomness as a relevant property of generated feedback, the 63/66 rate is a point estimate from one set of outputs. Please specify the model and interface settings, including temperature if available, and either report the stability of the match rate across repeated generations or state clearly that the result comes from a single run and is subject to sampling variation.","section":"Sections 3.4 and 3.5"}],"minor_comments":[{"comment":"The word 'usefull' in the prompt text should be corrected to 'useful.'","section":"Listing 1, line 19"},{"comment":"The sentence stating that the 'COMP' category was obsolete because no complete code examples were generated appears to refer to the wrong category: Table 2 defines COMPL as 'it matches the task description,' while examples are coded under EXA and code under CODE. Please correct the category label or the explanation.","section":"Section 4.2.4"},{"comment":"The legend uses nonstandard symbols (CIRCLE, LEFTCIRCLE) that are difficult to read in the PDF; using conventional check/cross marks or explicit Y/N values would improve readability.","section":"Table 5"},{"comment":"The exact task description used for the modified Triangles exercise is not provided in the paper. Since the prompt includes the task text, please include the full prompt inputs (task descriptions, class skeletons, and model solutions) in an appendix or repository to support replication.","section":"Section 3.3.1 and Table 3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within the journal's scope and makes a useful, transparent empirical contribution. The principal concern is methodological: the evaluation set contains the prompt-development submissions, and the coding reliability is not quantified. These issues are fixable by reanalyzing the existing data and reporting additional summary statistics, so major revision seems appropriate. I see no integrity concerns; the authors are candid about the limitations and about misleading examples in the output."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the paper does something genuinely new: it asks whether a single prompt can steer GPT-4 to produce feedback matching one of six taxonomy-defined types (KR, KP, KTC, KC, KM, KH) for introductory programming submissions, and it publishes the full final prompt. Second, the headline numbers are not as clean as they look: the 63/66 match rate includes the three submissions used to iteratively develop that prompt, so it partly measures prompt fitting. The paper never reports the match rate on the eight held-out submissions (48 messages).\n\nWhat is good: the qualitative analysis is careful and candid. The authors document misleading feedback, inconsistent KP percentages for correct solutions, and the tendency for KM elements to leak into KH and KTC requests. The final prompt in Listing 1 is a concrete, reusable artifact. The task design is sensible: 11 solutions, 6 exercises, 4 languages, varied error types. The coding procedure (two coders, consensus, concealed requested type) is standard, though no inter-rater reliability statistic is reported.\n\nThe soft spots, in order: (1) The development/test overlap is real. Section 3.3.1 selects three submissions for prompt design; Section 3.4 says the prompt was revised until output was adequate on those very submissions; Table 4 then labels those three as 'Test set' and Table 5 counts them. The paper's Section 5 acknowledges single-model and task-domain limits but not this. (2) No kappa or similar for the feedback-type coding; the 63/66 count rests entirely on the two authors' consensus judgment. (3) The evaluation is a single snapshot: one model (GPT-4, March 2024), one prompt, no temperature or randomness checks. (4) The authors do not ship the generated outputs, so the qualitative claims cannot be checked against raw data.\n\nNone of this kills the paper. The central claim—that prompt engineering can substantially steer feedback type—is plausible and supported by the examples. But the quantitative framing overstates it. A revision that reports held-out match rates, adds inter-rater reliability, and discusses the overlap would make this a clean contribution.\n\nFor whom: computing-education researchers studying feedback generation, and tool developers who want a starting prompt for selectable feedback modes. It is worth a serious referee; I would ask for the held-out re-analysis before acceptance.","headline":"A useful, mostly honest study of prompting GPT-4 for taxonomy-defined feedback, but the headline 63/66 match rate is inflated by including the prompt-development submissions in the evaluation set.","tokens_in":26010,"tokens_out":3247,"would_cite":true,"duration_ms":30876,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A carefully engineered prompt can make GPT-4 produce a requested feedback type for novice programming submissions in 63 of 66 cases.","keywords":["large language models","GPT-4","feedback types","introductory programming","prompt engineering","formative feedback","feedback classification","computing education"],"falsifier":"Have a new rater team, blind to the requested feedback type, classify the same 66 raw messages; if the requested type is coded as present in fewer than about 50 of the 66, the paper's claim of controllable feedback types is not robust.","tokens_in":25099,"feed_emoji":"🎓","tokens_out":9970,"duration_ms":95730,"temperature":0.7,"pith_summary":"This paper asks whether a large language model can be made to produce one specific kind of feedback for introductory programming exercises, instead of an uncontrolled mixture of corrections, hints, and explanations. Using six feedback categories from an established taxonomy and a prompt refined over five iterations, the authors report that GPT-4 delivers the requested type in 63 of 66 generated responses. The finding matters because programming learning tools have mostly offered simple correct/incorrect or test-based feedback, while richer formative feedback, such as explanations of concepts, descriptions of mistakes, and hints for next steps, has been hard to automate. The paper also shows that such controlled output is not error-free: misleading statements still appear, especially when the submitted program is already correct.","feed_headline":"Crafted prompt controls GPT-4 feedback type in 63 of 66 responses","feed_subtitle":"A five-iteration prompt keeps GPT-4 to the requested teaching-feedback category in almost every output.","key_machinery":"The load-bearing mechanism is the final prompt (Listing 1), built through five iterations. It bundles the task description, class skeleton, student solution, and a model solution, then tells the model which of six feedback types to emit, with per-type instructions (KR should contain only correct/incorrect; KM should describe errors without fixes; KH should give concrete next-step hints; KP should divide the task into subgoals and give a percentage; KTC and KC should raise constraints or concepts only where the student's code actually has an issue). Global rules ask for brief wording, no mention of the model solution, and no additional information beyond the requested type. The taxonomy supplies the shared labels used both for generation and for the subsequent coding of the output.","core_discovery":"The central claim is that feedback content can be steered by prompt design: given the task description, the starter code, the student's submission, a model solution, and a label naming one of six feedback types, GPT-4 produces feedback whose actual type matches the requested type in 63 of 66 cases. The six types are knowledge of result (correct/incorrect), knowledge about task constraints, knowledge about concepts, knowledge about mistakes, knowledge on how to proceed, and knowledge of performance. In 19 cases the output contained an additional type as well, most often knowledge about mistakes appearing alongside how-to-proceed or task-constraint feedback. The authors read this as evidence that LLMs can be controlled well enough to support richer automated feedback, while acknowledging that the match is not exact and that some statements are misleading.","pith_inferences":["The 63-of-66 figure is evidence about GPT-4 at the time of the study, not about LLMs in general; re-running the same prompt on newer models would be needed to see whether the control improves or degrades over time.","The frequent leakage of KM into KH and KTC feedback may reflect a genuine conceptual boundary: a useful next-step hint usually has to name the mistake it addresses, so strict single-type generation may be fighting the nature of helpful feedback.","The model's habit of labelling style advice as 'mistakes' suggests it follows the everyday English label more than the taxonomy's intended definitions, so renaming the categories inside the prompt could reduce a source of misleading feedback."],"forward_implications":["Learners could be given a menu of feedback types, such as 'what is wrong' versus 'what do I do next', that sends a preset prompt so they do not need to phrase their own request to the model.","A hybrid tool that first runs test cases and inserts their results into the prompt should make simple feedback types like KR and KP more reliable, because the model would not have to judge correctness on its own.","Educators should treat generated feedback as needing a guardrail: the model still claims problems that are not present in the student's code, so learners need guidance in evaluating such feedback critically.","The prompt and the six-type classification give researchers a way to isolate feedback type as a variable in studies of feedback effects and learners' informational needs."],"supporting_citations":[{"why":"Supplies the feedback-type taxonomy whose six types define both the generation targets (KR, KTC, KC, KM, KH, KP) and the coding scheme.","marker":"[15]"},{"why":"Provides the original student submissions used for prompt development and the deductive coding categories that the analysis extends with PERS and COMPL.","marker":"[14]"},{"why":"Identifies GPT-4 as the model whose outputs the whole experiment evaluates.","marker":"[62]"},{"why":"Contributes the readability guidelines (economy of words, simple vocabulary, complete sentences) that shaped the prompt's quality conditions.","marker":"[59]"},{"why":"Provides a contemporary GPT-4 feedback-generation study whose characteristics and accuracy findings are compared with this paper's results.","marker":"[8]"},{"why":"Supplies the PERS and COMPL characteristics and the GPT-3.5 feedback baseline showing why controlled prompting is needed.","marker":"[12]"}],"fun_headline_variants":["Prompt steers GPT-4 feedback type 63 of 66 times","LLM feedback type pinned down by prompt: 63 of 66","GPT-4 keeps requested feedback type in 63 of 66 outputs","Specific feedback via prompt: 63 of 66 match type","Can LLMs give type-specific feedback? 63 of 66 yes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the two authors' consensus coding of the generated feedback types is a valid, unbiased measure of what GPT-4 actually produced; if other coders would not place the messages in the same categories, the 63-out-of-66 match rate could be a measurement artifact.","fun_headline_variants_meta":{"raw":{"variants":["Prompt steers GPT-4 feedback type 63 of 66 times","LLM feedback type pinned down by prompt: 63 of 66","GPT-4 keeps requested feedback type in 63 of 66 outputs","Specific feedback via prompt: 63 of 66 match type","Can LLMs give type-specific feedback? 63 of 66 yes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000172,"raw_usage":{"total_tokens":1284,"prompt_tokens":964,"completion_tokens":320,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":580,"completion_tokens_details":{"reasoning_tokens":226}},"tokens_in":580,"tokens_out":320,"duration_ms":3695,"temperature":1.0,"reasoning_tokens":226,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T22:17:36.970177+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have a new rater team, blind to the requested feedback type, classify the same 66 raw messages; if the requested type is coded as present in fewer than about 50 of the 66, the paper's claim of controllable feedback types is not robust.","supporting_citations":[{"cited_title":"Exploring the Potential of Large Language Models to Generate Formative Programming Feedback","cited_arxiv_id":null,"evidence_quote":"Provides the original student submissions used for prompt development and the deductive coding categories that the analysis extends with PERS and COMPL."},{"cited_title":"AI-enhanced Auto-Cor rection of Programming Exercises: How Effective is GPT-3.5? International Journal of Engineering Pedagogy (iJEP) 2023 Dec;13(8):67–83","cited_arxiv_id":null,"evidence_quote":"Supplies the PERS and COMPL characteristics and the GPT-3.5 feedback baseline showing why controlled prompting is needed."}],"review_version":1}