{"id":"0a5a4648-93ea-4205-a40a-cf685a36e540","arxiv_id":"2509.11065","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A multimodal LLM system that diagnoses Scratch bugs from code plus gameplay video reported 100% repair success on ten curated tasks, far above text-only ChatGPT baselines.","lead":"ViScratch is a new automated debugging assistant for Scratch that looks at both the block code and a video of the program running to find and fix bugs. In tests on ten selected projects it reports perfect results, faster and cheaper than human tutors, though the small hand-picked dataset leaves open questions.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing video-removed ViScratch ablation: the 100% result conflates video with the full repair/verification pipeline, so the central claim that video is a crucial debugging signal is not established.","rationale":"The reader's verdict (CONDITIONAL) is appropriate: the core idea is plausible and partially supported by the human video benefit and the ChatGPT comparison, but the strongest claim—100% success and video as the decisive factor—rests on a confounded comparison. The most load-bearing concern is the absence of a video-free ViScratch ablation, which the reader mentioned in the rationale but did not place at the center of the weakest_assumption. The reader focused more on dataset representativeness and the manual oracle. I agree those are real threats, but the ablation is the single most decisive gap for the causal claim about video. The proposed concrete test directly settles the attribution question. Since the reader already asked for a video-free ablation, the verdict remains CONDITIONAL: the claim may be true, but current evidence is insufficient to accept the 100% video-attribution claim at face value.","tokens_in":172,"tokens_out":2407,"duration_ms":39507,"concrete_test":"Run ViScratch on the same 10 projects (30 rounds) in two variants: (a) with the real gameplay video, and (b) with the video removed or replaced by a black/blank video of the same duration, while keeping all prompts, LLM, AST edit, and verification modules identical. If variant (b) also achieves near-100% success, the video is not load-bearing for the reported results; if success drops substantially, the video claim is supported. Additionally, independently re-implement the §4.5 verification oracle as a formal behavioral assertion (e.g., 'score must increase by exactly 1 after a single click') and re-run the evaluations to confirm that the 'defect resolved' determination is not made by the diagnosing LLM or by subjective human judgement.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that gameplay video is a first-class debugging signal, evidenced by ViScratch's 100% success on 10 projects. But the evaluation contrasts ViScratch (video + code + JSON + media, plus AST-based repair and VM verification) against ChatGPT variants that lack video but also lack ViScratch's repair/verification loop. This confounds the video signal with the entire pipeline: the 100% could come from the Gemini backbone, the curated prompt taxonomy, the minimal-edit AST repair, or the iterative verification process, rather than from video. The human experiment shows video helps humans (41%→69%), but it does not validate ViScratch's specific video integration. A video-free ViScratch ablation is absent; without it, the contribution of video to ViScratch's success cannot be isolated. Furthermore, the verification module (§4.5) is under-specified: if it relies on the same LLM or on manual visual inspection, it may leak the expected diagnosis or fix, and the manual assessment of 'bug correctly identified' and 'fix valid' on projects deliberately curated to match the prompt taxonomy risks circularity. Thus the headline result overstates the causal role of video.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ViScratch, a multimodal debugging system for Scratch that combines the project's block code, JSON representation, media assets, and a generated gameplay video. A vision-language model (Gemini 2.5 Pro) diagnoses one critical bug, proposes minimal AST-level repairs, and an iterative repair-and-verify loop checks patches in a Scratch VM. The central empirical claim is that gameplay video is a crucial debugging signal: on 10 curated projects with exactly one bug, ViScratch achieves a reported 100% success rate in bug identification and repair, outperforming human participants and four ChatGPT/GPT-4o prompt variants. The paper also reports a human study showing video improves human debugging success (41.2% to 68.6%) and an ablation varying the LLM backbone (Gemini, Qwen3-Max, GLM-4V).","tokens_in":17987,"tokens_out":5901,"duration_ms":70674,"significance":"If the central claim held, this would be a useful advance: it proposes a practical way to treat visual runtime behavior as a first-class debugging signal in block-based programming, where correctness is inherently perceptual. The human study is a valuable positive control, and the design principle of minimal, verified repairs is pedagogically sound. The system is described with enough detail to be reimplemented, and the cost/runtime figures (52.2 s, $0.1185/project) are concrete. However, the evidence as presented does not isolate the contribution of video to ViScratch's success, and the verification and evaluation protocols are under-specified. The headline 100% figure is therefore not yet established as a measure of the video signal.","major_comments":[{"comment":"The abstract and Section 5.2 claim that 'gameplay video is a crucial debugging signal' for ViScratch, but the comparison confounds video with the whole pipeline. ViScratch differs from the ChatGPT baselines not only by having video, but also by using Gemini 2.5 Pro, a taxonomy-primed prompt, AST-based repair, and a verification loop. The ablation in Section 5.3 varies only the LLM while keeping video in all conditions. A ViScratch condition with video removed (and ideally with/without project descriptions) is necessary to attribute the 100% success to video. Without it, the result may largely reflect the other components.","section":"§5.1, §5.2, Table 3"},{"comment":"The verification module is under-specified. The text says it 'loads the repaired project and checks whether the defect is resolved' and gives a score-update example, but does not state whether the check is a scripted VM assertion, an LLM call, or a human visual judgment. If the same LLM or a human who knows the expected fix performs verification, the reported '100% verified' repairs are not an independent automated outcome. Section 5.1's statement that 'we manually assessed the results' makes this ambiguity concrete. Please clarify what is automated in verification and how the verifier avoids leaking the diagnosis.","section":"§4.5, Algorithm 1 line 14"},{"comment":"The evaluation risks circularity. The ten projects were selected for 'the presence of exactly one bug which falls into the patterns outlined in Section 3', while the reasoning prompt (Figure 6) is primed with a bug taxonomy derived from the same forum study that motivates the system. Consequently, the 100% success rate may reflect the model's ability to match a known, supplied taxonomy rather than a general debugging capability. Additionally, the manual assessment of 'bug correctly identified' and 'fix valid' is not described as blinded or independently annotated. Please report inter-rater agreement, use a less curated or randomly sampled bug set, or ablate the taxonomy prompt.","section":"§3.2, §4.3, §5.1"},{"comment":"The comparison in Section 5.2 refers to 'the strongest human configuration (Blocks + Project Description + Video)', but the human procedure described in Section 5.1 only provides Blocks first and then Blocks + Video, with no project description. This is an inconsistency in reporting the headline risk ratio (1.46). If humans did receive descriptions, the procedure must be corrected; if they did not, the configuration label must be fixed. As written, it is unclear which human configuration produced the reported comparison.","section":"§5.2 vs §5.1"}],"minor_comments":[{"comment":"Finding 1 says there are 17 clone-related cases, but the rows 'Missing clone operation' (10) and 'Recursive cloning' (8) sum to 18. Please reconcile the count.","section":"§3.3 / Table 1"},{"comment":"References [19] and [20] appear to be duplicates of the same ICSE paper ('On the Applicability of Language Models to Block-Based Programs') with different author lists. Please verify.","section":"References"},{"comment":"The 100% success rate is reported over '30 rounds of trials' (10 projects × 3 rounds). The paper should state clearly how a 'round' is defined for ViScratch (e.g., whether a round is a single diagnosis attempt or allows iteration within the repair loop), since this affects the interpretation of the success rate.","section":"§5.2"},{"comment":"Some human success percentages are not multiples of 20% despite '5 participants each' (e.g., Tester 8 shows 33% in the Blocks+Video column). Clarify the per-participant denominators.","section":"Table 3b"}],"recommendation":"major_revision","confidential_remarks":"The central idea is interesting and the human video-vs-no-video result is a solid contribution, but the main empirical claim is currently overclaimed. The missing video-removed ViScratch ablation and the under-specified verification oracle are fixable with additional experiments and reporting. If the authors cannot run such ablations, the paper should be reframed as a feasibility study rather than as evidence that video is 'crucial' for ViScratch. The 10-project, manually scored evaluation would then be a pilot rather than a definitive result."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear [Colleague],\n\nThe thing to know: ViScratch is worth reading for the idea, not for the numbers. Using gameplay video as a first-class debugging spec for Scratch is genuinely new—no prior system I know does that—and the paper actually ships a working pipeline with diagnosis, AST-level repair, and VM verification. The bug taxonomy from 50 forum posts (70% video-required) is a useful empirical contribution, even if the methodology is ad hoc.\n\nWhat's good: the human study (41% to 69% when shown video, p=0.009) is a clean, interpretable result. The LLM-backbone ablation (Gemini 10/10, Qwen 2/10, GLM 7/10) is a nice sanity check. And the cost (52s, about $0.12 per project) matters for classroom reality. So the system is real and the direction is plausible.\n\nBut the central claim—that video is the crucial signal—is not established. The 100% success rate compares ViScratch (video + code + JSON + media, plus AST repair and VM verification) against ChatGPT variants that lack video but also lack the repair/verification loop. A video-free ViScratch ablation is absent. The human result shows video helps people, but it doesn't isolate video's contribution to ViScratch. On its own, the 100% could be Gemini 2.5 Pro, the hand-primed prompt taxonomy, or the repair loop.\n\nThe other soft spots are real but not fatal. Ten curated projects, each with exactly one bug matching the same taxonomy the prompts are primed with, is a narrow test. Correctness was manually assessed by the authors, with no inter-rater reliability or released artifacts. The verification module (Section 4.5) is under-specified: if it uses the same LLM or manual visual inspection, it may leak the expected answer. The paper's own threats-to-validity section acknowledges several of these—construct validity, external validity—which I read as honest rather than damning.\n\nMy verdict: the idea deserves a serious referee, but the empirical support for the headline causal claim is conditional. If they release code/data, add a video-free ViScratch ablation, specify and ideally automate the verifier, and test on an uncurated set, this could be a solid contribution. As it stands, treat the 100% figure as an upper bound, not a fact.\n\nI'd bring it to a reading group as a case study in evaluation design, and I'd cite it if I were working on multimodal debugging. Recommend: don't desk reject; send to review, but expect the authors to address the ablation and oracle issues honestly.\n\nCheers,\n[Name]","headline":"Good idea, weak isolation: the video-as-specification claim is plausible but the evaluation never ablates video away from the rest of the pipeline.","tokens_in":18530,"tokens_out":3801,"would_cite":true,"duration_ms":38182,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Watching gameplay video alongside block code lets an LLM diagnose and repair Scratch bugs perfectly on a ten-project test set, without any project description.","keywords":["Scratch debugging","gameplay video","multimodal LLM","vision-language model","automated program repair","visual programming","block-based programming","AST-level repair"],"falsifier":"Take a random sample of 100 Scratch projects with real learner bugs (not filtered by 'exactly one bug' or taxonomy fit), run ViScratch's exact pipeline, and have an independent group of Scratch teachers label whether the identified bug is the actual root cause and whether the fix resolves it without introducing new problems. If the success rate falls well below 100%, the perfection claim fails.","tokens_in":17601,"feed_emoji":"🎬","tokens_out":5274,"duration_ms":58984,"temperature":0.7,"pith_summary":"The paper argues that in Scratch, correctness is perceptual: many bugs—flickering sprites, missed collisions, race-conditioned score skips—only show up in rendered gameplay, not in block inspection. To exploit this, it builds ViScratch, a two-stage pipeline that feeds a vision-language model both the project's code and its gameplay video, aligns observed behavior with code structure, proposes a single critical bug, and repairs it via minimal AST edits verified by execution in the Scratch virtual machine. In a study of 50 real forum bugs, 70% required video for diagnosis. On ten curated projects (30 trials), ViScratch achieved a 100% success rate, outperforming human debuggers given code plus descriptions plus video, and outperforming LLM baselines given code and descriptions, all without ever seeing a project description. If the results hold, video is a first-class specification for visual programming, and automated feedback for open-ended learner projects becomes feasible without expert-authored tests or rules.","feed_headline":"Gameplay video pushes LLM Scratch fixing to 100%","feed_subtitle":"Watching the rendered game alongside block code lets an AI debugger fix every curated bug — no project description needed.","key_machinery":"The carrying mechanism is the diagnose–repair–verify loop. A vision-language model is prompted with the gameplay video and the AST of the buggy project, guided by a curated taxonomy of Scratch bug patterns (broadcast mismatch, clone lifecycle, layering/visibility, collision/bounds, scene transitions, audio–visual timing). It must output exactly one critical bug and two to three fix options. The repair module then translates the chosen fix into atomic JSON/AST edits (insert, replace, delete) under a minimal-edit policy that forbids new features or wholesale rewrites. The patched project is repackaged and executed in the Scratch virtual machine; if the defect persists, the failure is logged an","core_discovery":"The paper's central claim is that gameplay video is a necessary, first-class debugging signal in Scratch, not a convenient extra. A survey of 50 forum-reported bugs shows 70% (35/50) are only diagnosable from visual symptoms, and all 17 clone-lifecycle cases require video. On this basis, the paper builds ViScratch, which takes the project's block code, JSON, media assets, and a normalized gameplay video; a vision-language model aligns observed sprite behavior with the AST, isolates one critical bug, and proposes two to three minimal fixes. The repair module turns the chosen fix into atomic AST edits, and the Scratch virtual machine re-executes the patched project to verify the defect is gone","pith_inferences":["A natural extension is to apply the video-plus-code approach to other block-based or visual environments (App Inventor, GDevelop, game engines) where behavior is perceptual; the taxonomy would need to be rebuilt, but the diagnose-repair-verify loop is language agnostic.","The 100% success on ten clean, single-bug projects is likely to degrade on real projects with multiple interacting bugs or bugs outside the taxonomy; the useful claim to test is not perfect accuracy but whether video consistently beats code-only input.","Video could also be used for hint generation beyond fixes: explaining to a learner why the observed behavior is wrong, by pointing to the exact frame where the symptom appears.","The paper leaves the verification module's internals unspecified; if it were shown to rely on the same LLM that diagnosed the bug, the measured success rate would conflate diagnosis with patch validation, so a blind check by independent execution would be a stronger test."],"forward_implications":["Gameplay video can serve as a first-class debugging signal for visual programming, enabling automated feedback without needing project descriptions or instructor-authored tests.","Automated repair in block-based languages can be made reliable by constraining LLMs to minimal AST edits and verifying each patch by execution in the virtual machine.","The passive, on-demand design means learners can keep their normal Scratch workflow; the tool intervenes only when asked.","The same pipeline generalizes, in principle, to other visual programming environments where correctness is judged by what the learner sees on screen.","The reported 100% success rate, if it holds beyond the curated set, would give classroom deployment a low-cost per-project fix (about 52 seconds and $0.12 per project)."],"fun_headline_variants":["Gameplay video key to LLM debugging in Scratch","ViScratch uses video and code to fix Scratch bugs","AI debugger for Scratch relies on watching gameplay","For Scratch, visual bugs need video, not just code","LLM + video improves Scratch bug repairs"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The 100% figure rests on the assumption that the ten curated projects—each with exactly one bug fitting the paper's taxonomy—represent the variety of real learner bugs, and that the authors' manual judgment of 'bug correctly identified' and 'fix valid' is an unbiased oracle; the verification module's logic is not specified, so it could be checking the same thing the diagnosing LLM already decided.","fun_headline_variants_meta":{"raw":{"variants":["Gameplay video key to LLM debugging in Scratch","ViScratch uses video and code to fix Scratch bugs","AI debugger for Scratch relies on watching gameplay","For Scratch, visual bugs need video, not just code","LLM + video improves Scratch bug repairs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000883,"raw_usage":{"total_tokens":3657,"prompt_tokens":757,"completion_tokens":2900,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":501,"completion_tokens_details":{"reasoning_tokens":2824}},"tokens_in":501,"tokens_out":2900,"duration_ms":21265,"temperature":1.0,"reasoning_tokens":2824,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T17:08:44.897781+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of 100 Scratch projects with real learner bugs (not filtered by 'exactly one bug' or taxonomy fit), run ViScratch's exact pipeline, and have an independent group of Scratch teachers label whether the identified bug is the actual root cause and whether the fix resolves it without introducing new problems. If the success rate falls well below 100%, the perfection claim fails.","supporting_citations":[],"review_version":1}