{"id":"37e7647e-dcd1-4ce3-aa12-203653d77e1b","arxiv_id":"2501.09236","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Providing a VLM with a README, bug type descriptions, and a bug-free screenshot enables up to 100% per-application visual bug detection in HTML5 canvas apps, but average accuracy across all screenshots is only 39%.","lead":"The paper tests whether GPT-4o can spot visual bugs in HTML5 canvas apps by giving it screenshots plus context like the app's README and a bug-free screenshot. It finds the approach can reach 100% accuracy for some apps, but average detection is much lower and varies widely across apps.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Bug-injection realism is the load-bearing risk: all four shaders alter opacity/position globally, and human-verifiability of injected bugs is attested only by a single author.","rationale":"The reader's weakest_assumption already identified the same load-bearing risk: the injected bugs' realism and the single-author verification of the ground truth are the least secure premises. My stress-test pass confirms this is the central vulnerability and sharpens it technically: the generation procedure uses exactly one global shader mutation per bug type, so the 80 bug-injected screenshots contain only four canonical pixel-level perturbation signatures. The paper's threats-to-validity section (Section 8, Construct Validity) explicitly concedes that the causes of injected bugs may differ from real bugs and that a single author verified all 100 screenshots. Those are manuscript-internal admissions, not artifacts of the review pipeline. The reader's strongest_claim matches the paper's abstract and conclusion, and the reported 39% overall accuracy with 95-100% bug-free accuracy is plausible evidence for a feasibility claim, but the per-application 'up to 100%' figure is not representative. I do not see an internally inconsistent argument or a methodology error that would justify REJECT; the authors are appropriately cautious in their conclusions and release artifacts. The correct disposition is to keep CONDITIONAL, with the condition being an independent or alternative injection study, or at minimum a second-annotator ground-truth check. The concern is empirical, not logical, so it does not overturn the study; it caps how strongly the headline can be stated.","tokens_in":24823,"tokens_out":2052,"duration_ms":18819,"concrete_test":"Construct a second dataset of naturally-occurring or independently-injected bugs whose visual signatures are not generated by the four shared shaders (e.g., hand-edit application logic, or use a second independent injection implementation that perturbs COR properties rather than the WebGL shader). Run the same AllContextExceptAssets protocol on that dataset. If per-application accuracy drops substantially relative to the original dataset, the reported accuracy is an artifact of the specific injection protocol; if accuracy is comparable, the concern is resolved. A smaller, immediately feasible check: have a second annotator, blind to the injection protocol, independently label a random subset of the 100 screenshots and measure inter-annotator agreement on both bug-presence and bug-freeness.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central feasibility claim is that a VLM with sufficient context can detect 'visual bugs' that resemble those a human tester would encounter. The dataset is generated by exactly four hard-coded WebGL shader mutations, one per bug type (Section 5.3.2, Table 4): a global opacity multiplier of 0.001 for State, a per-band fragment scaling factor for Rendering, a uniform scale/position adjustment for Layout, and a global opacity multiplier of 0.5 for Appearance. Because every injected instance of a type shares the same generic pixel-level transformation, the evaluation set has effectively one canonical visual signature per bug type, applied regardless of application semantics. This makes the task easier (or at least different) than detecting naturally occurring bugs, which arise from application logic and are not uniform across apps. The paper's own threat-to-validity section admits that 'the causes of our injected visual bugs may not necessarily be the same as real visual bugs,' and that a single author manually verified both bug-freeness of the 20 baseline screenshots and bug visibility in all 80 injected screenshots. Two consequences follow. First, the measured per-application accuracy (up to 100% for react-photo-studio, but as low as 20% on some repetitions for several apps) is an upper-bound estimate for this specific injection protocol, not a general statement about VLM bug detection. Second, the manual verification is a plausible point of failure: if some 'bug-free' screenshots contain incidental visual differences due to nondeterminism (which the paper itself notes can occur between executions), the bug-free accuracy numbers are unknowingly contaminated. The 'up to 100%' headline is additionally cherry-picked from the best application, as the reader noted.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies whether a pre-trained vision-language model (GPT-4o) can detect visual bugs in HTML5 <canvas> applications by prompting it with context about the application's expected functionality. The authors construct a dataset of 100 screenshots from 20 free/open-source PixiJS applications: 20 bug-free screenshots plus 80 screenshots created by injecting four bug types (State, Rendering, Layout, Appearance) through custom WebGL shaders. Five prompting strategies are compared, ranging from no context to combinations of README text, bug-type descriptions, a bug-free screenshot, and image assets. The headline result is that the strategy AllContextExceptAssets (README + bug descriptions + bug-free screenshot) yields up to 100% per-application accuracy, with 95–100% bug-free accuracy, while overall accuracy across all 100 screenshots is 39%. State bugs are detected most reliably and Appearance bugs least reliably. A small ablation study examines which parts of the README help or hurt recall.","tokens_in":25102,"tokens_out":7484,"duration_ms":76379,"significance":"The paper addresses a real and under-studied testing problem, and the core idea—prompting a VLM with a bug-free screenshot instead of maintaining a pixel-level oracle—is plausible and potentially useful for regression testing of <canvas> applications. Strengths include a publicly released dataset and injection framework, the use of real FOSS applications, a transparent comparison of prompting strategies, and honest reporting of per-application variability and threats to validity. If the findings generalize beyond the specific injection protocol, the approach would offer a low-cost complement to manual testing. However, because the bugs are produced by only four global shader transforms and are labeled by a single author, the quantitative results should be read as an upper-bound estimate for this protocol rather than as a general accuracy estimate for naturally occurring canvas bugs.","major_comments":[{"comment":"Bug-injection realism is load-bearing for the paper's central feasibility claim. Table 4 shows that each of the four bug types is represented by exactly one hand-written WebGL shader applied uniformly to all 20 applications: opacity multiplied by 0.001 for State, a fixed band-scaling factor for Rendering, a scale/position change for Layout, and opacity multiplied by 0.5 for Appearance. Consequently, every injected instance of a type has the same canonical visual signature, independent of application semantics. The paper's own threat-to-validity paragraph acknowledges that 'the causes of our injected visual bugs may not necessarily be the same as real visual bugs,' but the abstract and conclusion state that VLMs can detect visual bugs with relatively high accuracy. Since a naturally occurring canvas bug is not a global shader transformation, the measured per-application and per-type accuracies are protocol-specific upper bounds. Please either add multiple distinct bug instances per type (e.g., several shader variants and semantic-level injections) and show that the accuracy holds, or explicitly restrict the central claim throughout the title, abstract, and conclusion to bugs of the visual character studied here.","section":"Section 5.3.2 / Table 4; Section 8"},{"comment":"Ground-truth verification rests on a single author. Section 5.4.1 states that the author manually analyzed all 20 bug-free screenshots to ensure no bugs were present and manually verified that all 80 injected bugs were visible and human-detectable; Section 6.2 adds that the same author manually judged whether the VLM descriptions matched the observed bugs. No second annotator, inter-rater agreement statistic, or objective labeling criterion is reported. If any bug-free screenshot contains an unintended artifact (e.g., from timing-dependent rendering) or any injected bug is not clearly visible, all accuracy, precision, and recall figures move, and the ranking of prompting strategies could change. This is a load-bearing threat to the dataset's ground truth. Please add a second independent annotator with agreement statistics, or a pre-defined objective protocol (e.g., pixel-difference verification for the shader effects), and re-report the affected metrics.","section":"Sections 5.4.1 and 6.2; Section 8"},{"comment":"The best strategy is selected on the same data used to report its accuracy. The experiments run all five prompting strategies on the same 100 screenshots, and AllContextExceptAssets is then presented as the best strategy, with its per-application accuracy (up to 100%) and per-bug-type recall reported prominently. There is no held-out split, pre-registration, or correction for multiple comparisons, so the reported advantage of AllContextExceptAssets over the other strategies is subject to selection bias. The 'up to 100%' figure is for a single application chosen post hoc, while Figure 6 shows per-application accuracies as low as 20%. For a feasibility study this is not fatal, but the strategy comparison should be framed as exploratory, and the chosen strategy should be validated on a hold-out set (new applications or new injections) before the paper claims that this prompting strategy is the recommended one.","section":"Section 7 (RQ1), Table 6, Figures 4–6"},{"comment":"The headline wording overstates the aggregate results. The abstract says VLMs can be leveraged to detect visual bugs 'with relatively high accuracy' and highlights 'up to 100% per-application accuracy,' but Table 6 shows that the best overall strategy, AllContextExceptAssets, achieves only 39% mean accuracy across the 100 screenshots, with pass@1 at 39% and large standard deviations. Section 7 itself describes accuracy as low as 20% on some repetitions for several applications. The contribution should be reframed as a feasibility result: VLMs can detect some visual bugs in some canvas applications with high bug-free accuracy when a bug-free screenshot is provided, but the overall detection rate is modest. Please either report the aggregate 39% in the abstract or temper the 'relatively high accuracy' language.","section":"Abstract and Section 7 (RQ1)"}],"minor_comments":[{"comment":"The caption says per-application accuracy is 'computed over the 100 screenshots,' but each application contributes only five screenshots (one bug-free plus four bug-injected). The caption should say 'over the five screenshots for each application.'","section":"Figure 6 caption"},{"comment":"For p5aholic-playground, README(Good) has 0 tokens. The paper should state how the empty README block was handled in the prompt (e.g., whether the 'README:' section was omitted or left blank), because an empty template block could itself be a confound in the ablation.","section":"Table 5 / Section 6.1.1"},{"comment":"The bug-free screenshot is introduced with a human-written, pre-determined response ('This screenshot is free of any visual bugs as defined in the provided set of categories') rather than letting the VLM analyze it. This is a reasonable way to provide context, but it may anchor the model; please report a sensitivity check in which the bug-free screenshot is supplied without the pre-written assertion.","section":"Appendix (AllContextExceptAssets / AllContext prompts)"},{"comment":"AllContext is evaluated only on the 11 applications with image assets (55 screenshots), while AllContextExceptAssets is evaluated on all 20 applications (100 screenshots). The distributions in Figure 4 therefore mix different test sets; this should be stated next to the figure so readers do not compare the two strategies directly.","section":"Section 6 / Figure 4"},{"comment":"With n=4 repetitions, pass@4 is 1 whenever at least one of the four responses is correct, so pass@4 conveys little information beyond 'did any response succeed.' Consider reporting the proportion of screenshots with at least one correct response separately, or using a larger number of repetitions if pass@k is a central metric.","section":"Section 6.3 (pass@k)"},{"comment":"The title refers to 'Vision-Language Models' in the plural, but the experiments use a single model (GPT-4o, snapshot gpt-4o-2024-08-06). Consider making the title or abstract explicit that this is a single-model feasibility study.","section":"Title"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a software-engineering venue and the dataset/framework release is a useful contribution. My main concern is that the synthetic injection protocol and single-author labeling make the quantitative claims fragile; the issues are addressable by reframing the claims as protocol-specific and/or adding independent validation, so I recommend major revision rather than rejection. Please ensure the authors do not rely on the 'up to 100%' phrasing in the abstract without the aggregate 39% accuracy being equally visible."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Ben,\n\nQuick read on Macklon and Bezemer's canvas/VLM paper. The honest headline: this is a useful first dataset and framework for a real testing gap, but the 'up to 100% accuracy' in the abstract is a cherry-picked per-application number, and the evidence supports a much more modest feasibility claim.\n\nWhat's genuinely new: they built the first VLM-based evaluation for procedural-graphics canvas apps, where prior work (including their own) only covered asset-based graphics or open-world games. The dataset of 100 screenshots across 20 PixiJS apps and the shader-injection framework are concrete, reusable artifacts, and they're on Zenodo. The paper is also commendably honest in its threats-to-validity section: it explicitly acknowledges that injected bugs may differ from real ones, that a single author verified everything, and that one bug instance per type limits generalization.\n\nThe soft spots are exactly the ones the stress-test flags, and they're load-bearing for the accuracy numbers. Each bug type is injected via a single global shader transform (opacity drop, band scaling, scale/position shift, half-opacity). That means every 'Layout bug' in the dataset looks roughly the same—a uniform scale/position shift—rather than the diverse, application-logic-driven bugs a tester would hit. The evaluation also selects the best prompting strategy on the same 100 screenshots, with no held-out split, and the bug-free accuracy for AllContextExceptAssets is inflated by feeding the same screenshot as both context and test. Those issues don't kill the central claim—yes, a VLM given a README, bug descriptions, and a bug-free screenshot can catch some visual bugs—but they cap what the numbers mean. The overall 39% accuracy and 14% recall on Appearance bugs are more representative than the 100% figure.\n\nI think the reader's conditional verdict is about right. The paper is worth peer review: the dataset and framework are a contribution, and the negative-ish results are useful for the community. But I'd want the revision to reframe the claim as an upper-bound feasibility result, report per-strategy selection properly, and ideally add a second bug-injection instance per type to show the results aren't artifacts of one shader. That last one may be too much to ask for a first study, but at minimum the limitations need to be in the abstract.","headline":"Useful first dataset and feasibility study for VLM-based visual bug detection in canvas apps, but the headline 100% accuracy overstates what the evidence supports.","tokens_in":25667,"tokens_out":2679,"would_cite":false,"duration_ms":26247,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GPT-4o catches canvas app bugs up to 100% of the time with context","keywords":["HTML5 canvas","visual bug detection","vision-language model","GPT-4o","WebGL shader bug injection","test oracle","PixiJS","software testing"],"falsifier":"Collect N real visual bugs from canvas application issue trackers that include a bug-free 'before' screenshot, run the AllContextExceptAssets prompt on each, and compare recall to the 33–100% range reported here; a chance-level result would show the injected shader bugs were easier than real ones.","tokens_in":24626,"feed_emoji":"🐛","tokens_out":6157,"duration_ms":53034,"temperature":0.7,"pith_summary":"HTML5 <canvas> applications draw everything to a bitmap, so standard DOM-based web testing tools cannot see what is on screen, and visual bugs must often be found by hand. This paper asks whether a vision-language model (VLM) can act as an automatic visual test oracle by looking at a screenshot and deciding whether it matches the application's expected behavior. The authors built a dataset of 100 screenshots from 20 PixiJS canvas applications, injected four types of visual bugs with custom WebGL shaders, and prompted GPT-4o with different combinations of context. They report that when the prompt includes the application README, a description of possible visual bug types, and a bug-free screenshot, the VLM detects visual bugs with up to 100% per-application accuracy, and aggregate accuracy rises to 39% from a 26% baseline. The same prompting strategy keeps bug-free accuracy between 95% and 100%, suggesting the approach is usable as a low-cost complement to manual testing.","feed_headline":"GPT-4o catches canvas app bugs up to 100% of the time with context","feed_subtitle":"Canvas apps escape DOM-based testing; a VLM given a README and one clean screenshot becomes a visual test oracle.","key_machinery":"The central mechanism is a prompting strategy, called AllContextExceptAssets, that gives the VLM a text description of the application (its README), a taxonomy of what visual bugs look like (Layout, Rendering, Appearance, State), and one bug-free screenshot as a visual reference, then asks whether a second screenshot from the same application contains a bug. The bug-free screenshot is the load-bearing piece: without it, median precision sits around 34–50%, and with it, median precision jumps to 100%. To build the evaluation data, the authors created a just-in-time bug-injection framework that overrides PixiJS' WebGL shaders with four custom shader programs, altering only the rendered bitmap and leaving the canvas object representation untouched. This setup allows controlled, repeatable injection of one bug instance per bug type into each of 20 applications.","core_discovery":"The paper's central claim is that a pre-trained vision-language model can detect visual bugs in HTML5 <canvas> applications when it is given enough context about what the screen should look like, and that the most important piece of context is a bug-free screenshot from the same application. With the best prompting strategy, AllContextExceptAssets, which adds the README and bug-type descriptions to the bug-free screenshot, GPT-4o achieves up to 100% accuracy on a per-application basis, raises overall accuracy across 100 screenshots to 39% from 26% with no context, and keeps bug-free accuracy between 95% and 100%. State bugs (objects missing or in the wrong state) are detected most reliably, with 33% average recall, while Appearance bugs are the hardest, at 14% average recall. The authors conclude that VLMs are not yet ready to replace manual testing, but they can automatically catch some of the most obvious visual bugs, including in procedurally generated graphics that previous asset-based oracles could not handle.","pith_inferences":["A natural extension is to run the same prompting strategy on real bug reports from canvas application issue trackers that include before-and-after screenshots; if accuracy drops, the shader-injected dataset is the reason.","Since the README ablation shows functional descriptions help and setup instructions hurt, automatically extracting only functionality-oriented sentences from a README could raise accuracy without any model training.","Providing canvas object representation (COR) property data, such as positions, sizes, and tint values, alongside the screenshot might specifically improve Appearance-bug detection, because those bugs are hard to see but easy to compute from the object representation.","The shader-injection method alters only the bitmap, so bugs caused by logic errors in the canvas object graph are underrepresented, and real deployment bugs may be harder for a VLM that cannot inspect application state."],"forward_implications":["Developers of canvas applications can use a VLM plus a known-good screenshot as a lightweight regression-test oracle, catching a useful share of visual bugs without maintaining pixel-perfect snapshot baselines.","State bugs are the most promising target for VLM-based detection, while Appearance bugs will still need manual inspection.","Generating several VLM responses per screenshot and treating it as buggy if any response flags it (pass@k) improves detection rates, so sampling multiple outputs is a practical mitigation for the model's stochasticity.","The approach extends to procedurally generated graphics, which prior asset-based test oracles could not cover, because the VLM compares screenshots to textual and visual descriptions rather than to image assets.","Per-application accuracy varies widely, so teams should evaluate VLM usefulness case-by-case before adopting it."],"supporting_citations":[{"why":"Supplies the taxonomy of four visual bug types that the dataset is built around and the prior evidence that visual bugs matter in canvas applications.","marker":"[26]"},{"why":"Prior asset-based canvas testing approach that this paper extends to procedural graphics; provides the baseline showing why visual test oracles are hard to maintain.","marker":"[25]"},{"why":"Shows VLMs can detect glitches in open-world video game screenshots, supporting the hypothesis that VLMs can act as visual bug detectors.","marker":"[42]"},{"why":"Defines vision-language models and explains how pre-trained VLMs connect image and text inputs without a hand-built oracle.","marker":"[48]"},{"why":"Existing canvas testing via visual inference, which is limited to State bugs and five applications, providing the comparison point for the VLM approach.","marker":"[6]"},{"why":"Provides the pass@k metric formulation used to evaluate how often repeated VLM outputs catch a bug.","marker":"[12]"}],"fun_headline_variants":["GPT-4o catches canvas bugs with a clean screenshot as clue","Vision-language model spots canvas bugs using one bug-free screenshot","Canvas bugs: GPT-4o hits up to 100% accuracy with context","One clean screenshot helps GPT-4o find canvas app visual bugs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results transfer to real canvas development only if the shader-injected visual bugs look like the visual bugs developers actually encounter, and if the single author's manual labels on all 100 screenshots are correct.","fun_headline_variants_meta":{"raw":{"variants":["GPT-4o catches canvas bugs with a clean screenshot as clue","Vision-language model spots canvas bugs using one bug-free screenshot","Canvas bugs: GPT-4o hits up to 100% accuracy with context","One clean screenshot helps GPT-4o find canvas app visual bugs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000583,"raw_usage":{"total_tokens":2811,"prompt_tokens":1082,"completion_tokens":1729,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":698,"completion_tokens_details":{"reasoning_tokens":1653}},"tokens_in":698,"tokens_out":1729,"duration_ms":15148,"temperature":1.0,"reasoning_tokens":1653,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:08:31.062221+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect N real visual bugs from canvas application issue trackers that include a bug-free 'before' screenshot, run the AllContextExceptAssets prompt on each, and compare recall to the 33–100% range reported here; a chance-level result would show the injected shader bugs were easier than real ones.","supporting_citations":[{"cited_title":"A Taxonomy of Testable HTML5 Canvas Issues,","cited_arxiv_id":null,"evidence_quote":"Supplies the taxonomy of four visual bug types that the dataset is built around and the prior evidence that visual bugs matter in canvas applications."},{"cited_title":"Automatically Detect- ing Visual Bugs in HTML5 Canvas Games,","cited_arxiv_id":null,"evidence_quote":"Prior asset-based canvas testing approach that this paper extends to procedural graphics; provides the baseline showing why visual test oracles are hard to maintain."},{"cited_title":"GlitchBench: Can Large Multimodal Models Detect Video Game Glitches?","cited_arxiv_id":null,"evidence_quote":"Shows VLMs can detect glitches in open-world video game screenshots, supporting the hypothesis that VLMs can act as visual bug detectors."},{"cited_title":"Vision-language models for vision tasks: A survey,","cited_arxiv_id":null,"evidence_quote":"Defines vision-language models and explains how pre-trained VLMs connect image and text inputs without a hand-built oracle."},{"cited_title":"Web Canvas Testing Through Visual Inference,","cited_arxiv_id":null,"evidence_quote":"Existing canvas testing via visual inference, which is limited to State bugs and five applications, providing the comparison point for the VLM approach."}],"review_version":1}