{"id":"70181cad-6e23-46ce-953e-c1bb9e8a8016","arxiv_id":"2501.09210","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Personalized Parsons puzzles as help scaffolding led to significantly longer practice time than showing complete LLM solutions in a randomized Python classroom study.","lead":"In an 80-minute randomized classroom study, students who received personalized Parsons puzzles when asking for help spent significantly more time practicing Python than students who were shown a complete solution. The result suggests AI tutors should scaffold struggling learners with active puzzles instead of simply handing over finished code.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The engagement measure is confounded by the help format: completing a Parsons puzzle takes time by design, so the observed time advantage may not reflect more practice engagement.","rationale":"The paper is a small, honest classroom study with a randomized between-subjects design and a statistically significant main result (U = 2368.0, p < .001, CLES = .69). I agree with the reader's conditional verdict. The most load-bearing assumption is not statistical but construct-validity: 'practice time' is used as a proxy for 'practice engagement' and 'deeper cognitive engagement.' Because completing a Parsons puzzle is itself a time-consuming activity and the control help is a static solution, the measured difference is confounded with the format of the help. This concern is not resolved by the paper's own data; in fact, Section 4 identifies the absence of a posttest, and the plan to analyze additional behavioral metrics ('practice attempts') admits that time alone is insufficient. Re-analyzing logs by segment would settle whether the extra minutes are spent actively engaging with the problem or merely completing the scaffold. Secondary concerns include underreporting of exclusions and unequal final group sizes, but these are less central than the validity of the dependent variable. With the proposed segmentation, the verdict could move from CONDITIONAL to ACCEPT if PC students also show more pre/post-help effort; until then, the existing CONDITIONAL verdict is appropriate.","tokens_in":3229,"tokens_out":4248,"duration_ms":47048,"concrete_test":"Re-analyze the existing behavior logs by segmenting each student's total time into: (1) time before first Help click, (2) time while Help is open (puzzle manipulation vs. solution viewing), and (3) time after Help closes before final submission. Compare PC vs. CC on segments (1) and (3) separately, and also compare code-edit or attempt counts as an independent engagement measure. If PC and CC differ only in segment (2), the headline should be weakened to 'scaffolding occupies more time'; if PC also shows longer pre- and post-Help writing time and more attempts, the engagement interpretation is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing premise is that total practice time (Section 3) measures engagement with the write-code problem. This premise is insecure because the two help formats are not time-equivalent. In the PC condition, clicking Help launches a drag-and-drop Parsons puzzle with immediate feedback, an activity that necessarily takes minutes; in the CC condition, the same click displays a complete solution that can be read and copied in seconds. The paper even notes in Section 4 that students could request help before writing any code, so a PC student who clicks Help immediately contributes puzzle-completion time to 'practice time,' whereas a CC student who does the same contributes only copy-paste time. The roughly 7-minute mean gap is therefore plausibly a mechanical property of the scaffold, not evidence of deeper cognitive engagement. The manuscript's own qualitative findings are mixed (some PC students wanted lighter support, some CC students preferred the quick solution), and the Conclusion's 'deeper cognitive engagement' inference is unsupported because no posttest or independent engagement metric was reported. The claim is not internally inconsistent, but the dependent variable's construct validity is the central weak point.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports an 80-minute randomized between-subjects classroom experiment comparing two help formats for write-code practice: personalized Parsons puzzles (PC) versus complete LLM-generated solutions (CC). The research question is whether students in the PC condition show greater practice engagement, operationalized as total practice time. The reported results show significantly longer practice time in the PC condition (M = 22.7 vs 15.8 minutes, U = 2368.0, p < .001, CLES = .69), with comparable pretest and self-efficacy scores. The authors interpret this as evidence that active puzzle scaffolding supports engagement, while explicitly acknowledging the absence of a posttest and some technical difficulties.","tokens_in":1006,"tokens_out":925,"duration_ms":49914,"significance":"If the result is taken at face value, it is a useful empirical data point for the SIGCSE community about an alternative to freely available AI solutions that can encourage students to spend more time with practice problems. The study's strengths include the randomized assignment, identical practice problems across conditions, the use of a nonparametric test with an effect size, and the authors' transparent acknowledgment of the missing posttest. However, the paper's central dependent variable, time on task, is confounded with the format of the help itself, so the current analysis does not by itself establish deeper cognitive engagement. The manuscript is clear and readable but needs additional behavioral analyses or a more cautious interpretation to support the broader claim.","major_comments":[{"comment":"The reported time-on-task difference is mechanically influenced by the help format. In the PC condition, clicking Help opens a drag-and-drop Parsons puzzle that must be completed and provides immediate feedback, an activity that inherently takes minutes; in the CC condition, clicking Help displays a complete solution that can be read and copied in seconds. The paper itself notes that some CC students finished in under two minutes and that help could be requested before writing any code. Consequently, the 7-minute mean gap likely reflects format-induced activity time rather than a pure increase in practice engagement. Please report time spent in the write-code editor separately from time spent completing the puzzle, report the number and timing of Help requests, and consider excluding or separately analyzing students who requested help before writing code. The Conclusion's phrase 'deeper cognitive engagement' should be removed or supported by additional evidence, given that no posttest is available.","section":"Section 3"},{"comment":"The exclusion criterion ('worked on all four practice questions as instructed and did not encounter any technical difficulties') is stated only for the analyzed sample. Please report the number of students initially randomized, the number excluded for each reason, and whether exclusions were balanced across conditions. Without this participant flow, the reported baseline equivalence cannot be assumed to hold for the analyzed subsample.","section":"Section 3"}],"minor_comments":[{"comment":"The Limitations paragraph says 'students in the PS condition spent more time practicing'; the abbreviation should be PC, which was introduced in Section 2.","section":"Section 4"},{"comment":"The sentence 'The personalized Parsons puzzle applied as programming scaffolding applied two levels of personalization' contains a repeated 'applied' and should be reworded.","section":"Section 2"},{"comment":"The phrase 'On the contrary' is misapplied before the example of six CC students finishing in under two minutes; this is an illustration rather than a contrast with the preceding statistical result.","section":"Section 3"},{"comment":"The self-efficacy measure is mentioned in Section 3 but its instrument or administration is not described; one sentence clarifying how and when it was measured would improve reproducibility.","section":"Section 2"}],"recommendation":"major_revision","confidential_remarks":"The paper is within SIGCSE scope and the empirical design is appropriate for a short paper. The main risk is overinterpretation of time-on-task as engagement; the authors likely have the log data to address this with additional analyses. I recommend major revision rather than rejection because the core comparison is sound and the requested changes are feasible within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's what I take away from arXiv:2501.09210. It's a short, real classroom experiment from the CodeTailor group. The new thing is the randomized comparison: when a student clicks Help, the puzzle-scaffolding condition gets a personalized Parsons puzzle and the control gets a complete GPT-4 solution. Both see the same practice problems. The headline result is that PC students spent about seven more minutes practicing (median 22.8 vs 11.7, U=2368, p<.001, CLES=.69), and the pretest and self-efficacy scores were comparable across conditions. For a two-page SIGCSE paper, that's a clean, honest setup.\n\nWhat I like: the design is randomized, the analysis uses a nonparametric test with an effect size, and the authors are upfront about the big limitation—no posttest because the lecture ran out of time. They also report that some students in the control copied the solution immediately, which is the behavior the field worries about.\n\nThe soft spot, as you'd expect, is the dependent variable. Time-on-task includes the time it takes to drag and drop blocks, so the puzzle condition has a built-in time cost. The control can read a solution and paste it in under two minutes, and indeed six students did. That means the observed gap is partly a mechanical property of the help format, not necessarily longer thinking. The paper's own qualitative data show mixed preferences: some PC students wanted lighter hints, some wanted more guidance. So the conclusion that the puzzle leads to 'deeper cognitive engagement' is more than the data can support. The authors hedge it with 'potentially,' which is honest, but the claim needs either a posttest or a time-independent engagement metric (like attempts or code edits) to fly.\n\nThere are also some underreported exclusion criteria—the analysis only includes students who attempted all four questions and had no technical issues, so the 118 is a subset of the class. That's not fatal for a feasibility study, but it should be spelled out.\n\nNet: this is a useful proof-of-concept. It moves the CodeTailor line forward by adding a randomized control against the most typical LLM output. The primary result (more time on task) is credible; the interpretation (deeper engagement) is not yet. I'd send it to peer review, but the reviewers should push on the outcome measure and the inference. The paper is short, so it might suit a workshop or a work-in-progress track, but it deserves serious engagement.","headline":"A genuinely randomized classroom comparison showing the puzzle format yields more practice time, but the time-on-task measure is partly built into the intervention, so the deeper-engagement claim outruns the data.","tokens_in":3898,"tokens_out":2552,"would_cite":true,"duration_ms":27999,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A randomized classroom study shows that students who receive personalized Parsons puzzles as scaffolding practice significantly longer than students shown complete LLM-generated solutions.","keywords":["Parsons puzzles","LLM scaffolding","practice engagement","computing education","GPT-4","active learning","randomized classroom study"],"falsifier":"A randomized three-arm study that adds a read-only condition showing the same scrambled blocks arranged for the student, or that measures posttest coding performance after equalizing practice time, would determine whether the longer clock-time is actual engagement or simply the overhead of a drag-and-drop interface.","tokens_in":3040,"feed_emoji":"🧩","tokens_out":4520,"duration_ms":41605,"temperature":0.7,"pith_summary":"This paper tests how to help struggling students in a programming classroom without letting generative AI do the thinking for them. The authors claim that when students ask for help on a write-code problem, giving them a personalized Parsons puzzle—a drag-and-drop arrangement of code blocks that they must reconstruct—keeps them practicing significantly longer than showing them a complete LLM-generated solution. In an 80-minute randomized between-subjects study with 118 students, the puzzle condition averaged 22.7 minutes of practice versus 15.8 minutes for the solution condition, a difference the authors attribute to active versus passive engagement. The finding matters because AI code generators threaten to turn homework into copy-paste, and this offers a way to keep help-giving compatible with learning. The paper also reports that some students found puzzles too supportive or too vague, suggesting the design needs tuning for different learners.","feed_headline":"Puzzles beat copied AI answers at keeping students practicing","feed_subtitle":"Students given drag-and-drop code puzzles practiced about 7 more minutes on average than those shown full solutions.","key_machinery":"The central mechanism is the personalized Parsons puzzle, an active learning exercise in which the student assembles a scrambled set of drag-and-drop code blocks into a correct program, optionally with distractor blocks that do not belong. Here the puzzle is generated by an LLM (GPT-4) and personalized at two levels: the solution itself is tailored to the student's current code, and the blocks are chosen to match that code so the student works within their own partial solution. Immediate feedback on the arrangement and the ability to regenerate a puzzle give the student repeated, low-stakes attempts. This mechanism carries the argument by replacing a one-time passive reading of a full answer with a series of deliberate micro-decisions, which is what the authors claim drives the increased practice time.","core_discovery":"The central discovery is that personalized Parsons puzzles, deployed as on-demand scaffolding inside real write-code practice, produce longer practice engagement than complete AI-generated solutions. In the experiment, both conditions did the same four Python problems on nested dictionaries and could press a Help button at any time; the only difference was whether Help returned an interactive puzzle or a full solution. Students in the puzzle condition spent on average 22.7 minutes (median 22.8) practicing, compared with 15.8 minutes (median 11.7) for the solution condition, with a Mann–Whitney U = 2368.0, p < .001, and a common-language effect size of .69. Six solution-condition students finished in under two minutes, often copying the provided answer straight into the submission box. The authors interpret the longer time as evidence that active reconstruction of code blocks leads to deeper cognitive engagement than reading a passive answer.","pith_inferences":["The time-on-task gap may overstate the learning benefit, since a puzzle demands more physical and mechanical interaction than reading; a follow-up should compare puzzle engagement against a read-only list of the same blocks or against non-drag-and-drop hint sequences.","The study's own design hints that help should be adaptive: some students wanted lighter hints and some wanted more placement guidance, so a future system could personalize not just the code but the amount of scaffolding.","Because the classroom setting lacks a posttest, the real test of the scaffolding hypothesis is whether puzzle-scaffolded practice improves end-of-class or delayed coding performance, not just time spent.","Logging attempt counts and hint reuse would separate 'spent time' from 'persisted through errors,' giving a finer-grained measure of engagement than clock time alone."],"forward_implications":["If the central claim holds, LLM-based help in programming classrooms should be designed as scaffolded, interactive tasks rather than as ready-made code output.","Practice time becomes a measurable outcome for comparing help formats in classrooms where formal posttests are too time-consuming.","The personalized puzzle approach can be scaled automatically: the same LLM that produces a solution can also produce its scrambled-block form.","The design points to a trade-off: puzzle support is not one-size-fits-all, and students want adjustable levels of hinting.","Longer engagement alone does not guarantee better learning; the next step is linking engagement to posttest performance."],"supporting_citations":[{"why":"Supplies the adaptive Parsons mechanism that the personalization builds on.","marker":"[1]"},{"why":"Defines the two-level personalization (solution and block) used to generate the puzzles.","marker":"[2]"},{"why":"Documents the copy-paste behavior with LLM code generators that motivates the comparison.","marker":"[3]"}],"fun_headline_variants":["Puzzles beat full AI solutions at keeping students practicing","Drag-and-drop puzzles extend practice more than AI answers","Interactive code puzzles outlast plain AI solutions in practice","Puzzle scaffolding keeps learners coding longer than AI shortcuts","Active puzzle help yields longer practice than passive AI answers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison hinges on treating the time students spend on the practice as a valid proxy for engagement; since dragging blocks inherently takes longer than reading an answer, the measured gap may reflect the puzzle's mechanics rather than extra motivation or thought.","fun_headline_variants_meta":{"raw":{"variants":["Puzzles beat full AI solutions at keeping students practicing","Drag-and-drop puzzles extend practice more than AI answers","Interactive code puzzles outlast plain AI solutions in practice","Puzzle scaffolding keeps learners coding longer than AI shortcuts","Active puzzle help yields longer practice than passive AI answers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000494,"raw_usage":{"total_tokens":2376,"prompt_tokens":847,"completion_tokens":1529,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":463,"completion_tokens_details":{"reasoning_tokens":1453}},"tokens_in":463,"tokens_out":1529,"duration_ms":11687,"temperature":1.0,"reasoning_tokens":1453,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:08:57.396645+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A randomized three-arm study that adds a read-only condition showing the same scrambled blocks arranged for the student, or that measures posttest coding performance after equalizing practice time, would determine whether the longer clock-time is actual engagement or simply the overhead of a drag-and-drop interface.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the adaptive Parsons mechanism that the personalization builds on."}],"review_version":1}