{"id":"f0e26262-a253-476a-bb67-9044dfed3226","arxiv_id":"2504.17282","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A VLM-generated template-matching script masks the action space of a DQN agent, yielding large sample-efficiency gains on MiniWob++ in the low-data regime.","lead":"CoGA uses a vision-language model to write code that finds the clickable elements in a webpage screenshot, then uses that code to block out most actions an RL agent is allowed to try. On the MiniWob++ web benchmark the method reports large early-training speedups, though the evidence rests on tiny hand-labeled validation sets and a closed-source model.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'orders of magnitude' sample-efficiency claim is measured as success rate at a single 1000-step checkpoint, conflating a success-rate ratio with a steps-to-threshold measure.","rationale":"The reader's verdict is CONDITIONAL, and our analysis supports that verdict: the method is plausible and the empirical results are suggestive, but the headline claim is not yet rigorously established. The reader identified the single early checkpoint as a weakness in the rationale, which matches our primary concern. However, the reader's formal 'weakest_assumption' was the representativeness of the 5 template/test observations under hard masking. That is a real threat to the method's general applicability, but it is not the most direct threat to the central sample-efficiency claim. The most direct threat is that the reported metric (success rate at 1000 steps) is not a sample-efficiency metric at all; it is a performance-at-a-fixed-budget metric. A 10x success-rate ratio at 1000 steps can arise even when both agents would reach the same asymptotic performance, and the ratio becomes meaningless when the RL baseline is near zero. The per-task hyperparameter selection and the absence of error bars on the headline figure further weaken the quantitative claim. Our concrete test would settle this by converting the comparison to steps-to-threshold, which is the standard way to measure sample efficiency. If the steps-to-threshold ratios are consistently large (e.g., median >10) with non-overlapping CIs, the central claim would be substantially supported; if not, the claim should be downgraded. We therefore recommend keeping the CONDITIONAL verdict, with the condition that the sample-efficiency analysis be redone on a proper metric before the 'orders of magnitude' phrasing is retained.","tokens_in":19458,"tokens_out":3912,"duration_ms":38552,"concrete_test":"Recompute the comparison as steps-to-threshold: for each of the 23 tasks, from the logged evaluation curves, record the first step at which CoGA and RL reach 30%, 50%, and 80% success for each seed; compute the median ratio of steps-to-threshold and a bootstrapped 95% confidence interval across seeds. If the median ratio is below 10, or if the CI crosses 1, the 'orders of magnitude' claim is not supported. As a secondary check, rerun the analysis with 'best' replaced by 'median over the hyperparameter search' to test sensitivity to the selection procedure.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim ('CoGA is orders of magnitude more sample efficient') is operationalized in Sec. 4.3 as a ratio of evaluation success rates at a single checkpoint: 'we report the best evaluation success rates at 1000 steps for the RL agent and CoGA... CoGA enables over 10x sample efficiency gains over the RL agent early in training at only 1000 steps.' A 10x ratio in success rate at 1000 steps is not a 10x gain in sample efficiency; sample efficiency should be measured as steps required to reach a fixed performance level or as area under the learning curve. On tasks where the RL agent's success rate is near 0 at 1000 steps, any positive CoGA performance yields an arbitrarily large ratio, so the headline is not robust. Moreover, the reported values are the 'best' over a per-task hyperparameter search (Appendix C) with only 3 seeds, and Figure 3 shows no confidence intervals; Table 1 shows within-task stds as large as 28.79 percentage points. A secondary issue is that tasks were partly selected by F1 score, so the 'wide range of tasks' overstates coverage. The representativeness concern flagged by the reader (5 template/test observations, hard masking) is real but secondary: it affects robustness and generalization, whereas the checkpoint-based metric directly undermines the headline even on the tasks tested.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CoGA (Code as Generative Affordances), a method that uses a pre-trained vision-language model (VLM) to generate Python code that computes, from a pixel observation, a set of affordable actions (action types and pixel coordinates) for web navigation tasks in MiniWob++. The generated affordances are used as a hard mask on the action space of a DQN agent during training and inference. The paper claims three results: (1) CoGA is orders of magnitude more sample efficient than its RL baseline, (2) generated affordance scripts generalize within task families, and (3) CoGA performs better than or on par with behavior cloning (BC) in the low-demonstration regime. The evidence consists of per-task success rates at 1000 steps for 23 tasks, F1 scores on 5 manually annotated observations per task, a three-task transfer experiment, and BC comparisons on a subset of tasks.","tokens_in":19635,"tokens_out":3539,"duration_ms":33596,"significance":"If the central claim were fully supported, the contribution would be practically meaningful: distilling VLM knowledge into cheap, executable action-mask code could avoid expensive VLM calls during RL and substantially improve sample efficiency in pixel-based web navigation. The method is creative, the pipeline is described in sufficient detail to be reproducible, and the authors include example generated scripts and templates in the appendix. The potential weakness is not the idea but the evidence: the headline 'orders of magnitude' claim rests on a single-checkpoint success-rate ratio, the F1 estimates rely on only five manual observations per task, and the task selection is influenced by script quality. These issues make the current version's quantitative claims stronger than the data warrant, though the approach itself is defensible and worth revising.","major_comments":[{"comment":"The central claim 'CoGA is orders of magnitude more sample efficient' is operationalized as a ratio of best evaluation success rates at a single 1000-step checkpoint, not as a step-to-threshold or area-under-the-curve comparison. On tasks where the RL baseline's success rate is near zero at 1000 steps (e.g., click-dialog-2, use-spinner, click-checkboxes-large), any positive CoGA success yields an arbitrarily large ratio, so the reported 'over 10x' figure is not a robust measure of sample efficiency. The authors should re-measure sample efficiency as the number of steps needed to reach a fixed success threshold, or report the area under the learning curve, and present confidence intervals over seeds.","section":"§4.3, Figure 3"},{"comment":"The F1 scores that justify the choice of tasks are computed from only 5 manually annotated observations per task, and the template images are extracted from 5 observations. Because the generated code applies a hard mask that removes any undetected action (Section 3.3), a single template-matching failure in a new observation can make a task unsolvable. With 5 test observations, the recall estimate has very high variance and cannot establish the generalizable recall needed to support the sample-efficiency claim. The authors should evaluate script quality on a larger held-out set of observations (or a held-out set of tasks) and relate measured recall to downstream RL performance; the limitation acknowledged in Section 6.1 (template matching handles only isomorphic objects, VLM pixel mapping is unreliable) makes this concern concrete.","section":"§3.2, §3.3, §4.3 (Figure 2)"},{"comment":"The claim that generated scripts generalize within a task family rests on only three tasks, one of which (click-checkboxes-large) has near-zero success with both the original and transfer scripts (0.33% and 0.67%), while another (focus-text-2) has an RL baseline with a standard deviation of 28.79 percentage points. This evidence is too thin to support the general statement 'CoGA's generated affordance scripts can generalize within the same family of tasks' as formulated in the abstract and Section 4.3. Additional transfer pairs and more seeds are needed before this claim can be considered established.","section":"§4.3, Table 1"}],"minor_comments":[{"comment":"The text says 'over 10 times more sample efficient' and 'orders of magnitude more sample efficient' interchangeably; these are not the same, and the abstract's plural 'orders of magnitude' overstates what the data show even if the measurement issue is fixed.","section":"§4.3"},{"comment":"The text references 'Figure 4 (right)' for a statement about average performance, but the caption labels the right panel as per-task success rates and the left panel as the mean. Please make the references consistent with the panel labels.","section":"§4.3 and Figure 4 caption"},{"comment":"There are several typos: 'Qualitatevely' and 'Quantitavely' in Section 4.3, 'wihtout' in the Appendix A prompt, 'Ony return' in Appendix B, and 'inthetemplate' in Appendix A. These should be corrected before publication.","section":"Throughout"},{"comment":"The template matching threshold is listed in Appendix A as 0.5 in one script and 0.7 in the example scripts in Appendix E; the paper should state whether the threshold is a fixed hyperparameter or is tuned per task and how the final threshold is selected.","section":"§3.1"}],"recommendation":"major_revision","confidential_remarks":"The paper's core idea is promising and the pipeline is clearly presented, but the headline sample-efficiency claim is currently based on a metric (success rate at a single 1000-step checkpoint) that conflates success-rate ratios with true sample efficiency. The revision should re-analyze the existing data using steps-to-threshold or AUC, add confidence intervals, and expand the F1 evaluation beyond 5 observations per task. I would also recommend asking the authors to clarify the relation to the Code as Reward pipeline (Venuto et al.) in the experimental setup, since the prompting structure is shared and the novelty of CoGA lies primarily in the affordance-as-code framing. The paper fits the journal's scope, provided the quantitative claims are made commensurate with the evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a genuine extension of Code as Reward — generate action-space affordances as template-matching code with a VLM, verify it, mask a DQN's action space. The pipeline is described clearly, the limitations section is honest, and the within-family transfer result is a nice touch. I'd send it to a serious referee. But the central 'orders of magnitude' claim needs to be re-measured before I'd believe it.\n\nWhat's actually new: prior VLM-code work targeted reward functions (Venuto et al.) or robotic waypoints (VoxPoser, MOKA, KAGI). CoGA targets low-level pixel-click affordances for GUI RL, and the template-image extraction plus critique-verification loop is a plausible way to get reusable, cheap-at-runtime action priors. The F1 evaluation on the generated scripts is a reasonable quality gate, even if thin. The BC comparison in the low-data regime is fair and gives a useful calibration: CoGA beats BC up to about 200 expert trajectories, then BC takes over.\n\nSoft spots, in order of severity:\n\n1. The sample-efficiency headline. They report 'best evaluation success rates at 1000 steps' and call a >10x ratio in success rate 'orders of magnitude more sample efficient.' That conflates a success-rate gap at one early checkpoint with a steps-to-threshold measure. On tasks where the RL baseline is near zero at 1000 steps, the ratio is arbitrary. The right metric is area under the learning curve or steps to a fixed success level. This is the load-bearing claim and it needs re-analysis, not just a footnote.\n\n2. Task selection and external validity. They say 'wide range of tasks' but admit template matching only handles isomorphic objects and they excluded most text-varying tasks. The 23 tasks are the ones where the method already works. That is fine for a proof of concept, but the abstract overstates coverage.\n\n3. The evaluation of the affordance scripts rests on 5 manually annotated test observations per task. That is genuinely thin, especially because hard masking means a single missed object can make the task unsolvable. The paper acknowledges this, but the consequence is that the F1 numbers in Figure 2 carry large uncertainty.\n\n4. No code release, and the VLM (GPT-4o) is closed and may have seen MiniWob++ in pretraining. Neither is disqualifying by itself, but combined they make reproduction harder.\n\nThe citation pattern is honest — Code as Reward gets clear credit, and the related work is accurate. The math is standard DQN; no circularity. Who is this for? People working on sample-efficient web agents, and anyone building on the 'LLM-generated code as inductive bias' line. It deserves peer review, but with a request to re-derive the efficiency claim properly and ideally release code.","headline":"A real extension of Code as Reward with a clear pipeline and honest limitations, but the 'orders of magnitude' claim is measured at a single 1000-step checkpoint and needs re-analysis before it can be taken at face value.","tokens_in":20245,"tokens_out":1841,"would_cite":false,"duration_ms":17209,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A vision-language model can write code that computes the affordable actions in any screenshot, and hard-masking a deep RL agent with that code makes web navigation orders of magnitude more sample-efficient.","keywords":["affordances","reinforcement learning","sample efficiency","vision-language models","code generation","action masking","web navigation","MiniWob++"],"falsifier":"Run a task with varying text content (e.g., use-spinner or a click-button variant with unseen labels), extract templates from five observations drawn from only one part of the distribution, and evaluate the generated script's recall on held-out observations from another part; if recall falls below the level needed for task success and the hard-masked agent's success rate collapses to zero while the unmasked DQN still learns, then the sample-efficiency claim holds only when the five-observation templates happen to cover the states the agent encounters.","tokens_in":19207,"feed_emoji":"🤖","tokens_out":18049,"duration_ms":148594,"temperature":0.7,"pith_summary":"Reinforcement learning for GUI-based web navigation is sample-hungry because the action space is huge and rewards are sparse: at any state, only a few of the thousands of possible clicks matter. This paper proposes Code as Generative Affordances (CoGA), a pipeline in which a pre-trained vision-language model infers the task's intents, extracts image templates of the relevant objects from five example screenshots, and writes Python code that returns the set of affordable actions (action type plus pixel coordinates) for any observation. A critique VLM reviews and regenerates the code, and precision and recall against five manually annotated test cases select the final program, which then runs inside the training loop as a hard mask on a DQN agent's action space. The authors claim that on the MiniWob++ benchmark CoGA is orders of magnitude more sample-efficient than its unmasked RL agent, that the scripts transfer within families of tasks sharing a GUI, and that CoGA with no expert demonstrations outperforms behavior cloning trained on up to 200 expert trajectories.","feed_headline":"10× faster web RL from VLM-written action masks","feed_subtitle":"A VLM writes code that prunes each screenshot's action space, letting the agent learn with far fewer steps","key_machinery":"The load-bearing object is `determine_affordable_actions(observation)`, a generated Python function that maps a pixel screenshot to a set of affordable actions, each an action type (e.g., CLICK) with pixel bounding-box coordinates, detected by matching VLM-extracted object templates in grayscale. The machinery has three stages: the VLM infers task-level intents and object coordinates using a superimposed grid on the screenshot; a pre-written script crops the objects as grayscale templates from five example observations; and a critique VLM performs up to three rounds of review and regeneration while ground-truth precision and recall over five manual test cases pick the final program. In RL, the returned affordances hard-mask the agent's action space of $4$ action types by $1024$ discretized pixel bins, with the mask applied both to action selection and to bootstrapping the next observation's Q-target.","core_discovery":"The central claim is that intent-based affordances can be specified as executable code generated offline by a vision-language model, and that using this code to prune the RL agent's action space converts an unwieldy exploration problem into a tractable one. Concretely, CoGA has the VLM name the relevant intents (e.g., 'click a tab'), localize the corresponding objects as bounding boxes on a coordinate-gridded screenshot, crop their templates from five sampled observations, and write a script that uses template matching to detect those objects in any new observation and return the affordable actions. The script is improved by a critique VLM for up to three rounds and selected by precision and recall against five manually annotated test observations. The paper demonstrates on 23 MiniWob++ tasks that the masked DQN reaches over ten times the success rate of the unmasked baseline within the first 1000 training steps, that a script generated for one task transfers to a sibling task with the same GUI, and that CoGA with zero expert demonstrations beats behavior cloning trained on up to 200 expert trajectories, with BC overtaking it only at the 1000-demonstration regime.","pith_inferences":["The paper does not ablate its VLM-written scripts against hand-written affordance code built from the same templates, so the relative contribution of the VLM's code-generation versus the masking mechanism itself remains untested; replacing the generated script with a human-written one derived from the same five templates would isolate where the gains come from.","The hard-mask failure mode the authors identify is not addressed: a low-recall script permanently removes useful actions, so any state in which the template matcher misses an object becomes unsolvable; a soft-mask variant or an empty-set fallback that reopens the action space would test whether CoGA's gains survive the imperfect-recall regime.","Because the pipeline spends the expensive multimodal inference once, offline, and ships deterministic matching code, CoGA points toward a general recipe for cutting per-deployment foundation-model costs in embodied RL; the paper states the qualitative advantage but does not quantify the cost trade-off, which a simple token-cost accounting of the pipeline versus in-the-loop VLM querying would settl"],"forward_implications":["On the 23 MiniWob++ tasks tested, agents using CoGA's masks reach higher evaluation success rates at 1000 steps than the unmasked DQN baseline on most tasks when the generated script has high F1, implying that affordance pruning can substitute for large amounts of environment interaction.","Affordance scripts transfer across tasks that share a GUI and affordance set but demand different optimal policies (e.g., click-test-2 and click-button-sequence), so one script can serve several related tasks without regeneration.","In the low-data regime, CoGA with no expert demonstrations outperforms behavior cloning trained on up to 200 expert trajectories, which suggests that action-space masking uses prior knowledge more efficiently than a small amount of imitation data.","Because the VLM is queried only during the offline pipeline and the deployed script is ordinary template matching, the RL loop pays no per-step VLM inference cost, keeping the sample-efficiency gains cheap at deployment."],"supporting_citations":[{"why":"Supplies the formal definition of intents and affordances as state-action pairs that complete intended consequences, the conceptual basis that CoGA operationalizes as code.","marker":"Khetarpal et al. (2020)"},{"why":"Contributes the code-generation and verification pipeline that CoGA extends from reward functions to low-level affordance specification.","marker":"Venuto et al. (2024)"},{"why":"Defines the MiniWob++ pixel action space used throughout and supplies the expert rollouts from which the behavior-cloning baselines are trained.","marker":"Shaw et al. (2023)"},{"why":"Introduced MiniWoB, the web-benchmark family whose tasks CoGA is evaluated on.","marker":"Shi et al. (2017)"},{"why":"Extended MiniWoB to MiniWob++ and proposed workflow-guided exploration, a related action-space-constraining approach that CoGA contrasts with.","marker":"Liu et al. (2018)"},{"why":"Chain-of-thought prompting is used in the pipeline to elicit the step-by-step strategies that the generated affordance scripts implement.","marker":"Wei et al. (2023)"},{"why":"Provides the error-review-and-regenerate loop that the critique VLM follows when improving the generated scripts.","marker":"Wang et al. (2023)"}],"fun_headline_variants":["VLM-written code prunes actions, boosting RL sample efficiency","CoGA: code-generated affordances slash web RL data needs","Automated affordance code enables data-efficient web RL","Generative affordances via code outperform behavior cloning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the five sampled observations used to extract object templates and the five manually annotated test observations are representative of the full distribution of states the RL agent will visit; because the mask is hard, any object the template matcher fails to detect is permanently removed from the action set, so a single low-recall state makes the task impossible.","fun_headline_variants_meta":{"raw":{"variants":["VLM-written code prunes actions, boosting RL sample efficiency","CoGA: code-generated affordances slash web RL data needs","Automated affordance code enables data-efficient web RL","Generative affordances via code outperform behavior cloning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000263,"raw_usage":{"total_tokens":1673,"prompt_tokens":1093,"completion_tokens":580,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":709,"completion_tokens_details":{"reasoning_tokens":514}},"tokens_in":709,"tokens_out":580,"duration_ms":5578,"temperature":1.0,"reasoning_tokens":514,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:44:18.769469+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a task with varying text content (e.g., use-spinner or a click-button variant with unseen labels), extract templates from five observations drawn from only one part of the distribution, and evaluate the generated script's recall on held-out observations from another part; if recall falls below the level needed for task success and the hard-masked agent's success rate collapses to zero while the unmasked DQN still learns, then the sample-efficiency claim holds only when the five-observation templates happen to cover the states the agent encounters.","supporting_citations":[{"cited_title":"Code as reward: Empowering reinforcement learning with vlms, 2024","cited_arxiv_id":null,"evidence_quote":"Contributes the code-generation and verification pipeline that CoGA extends from reward functions to low-level affordance specification."}],"review_version":1}