Pith. sign in

REVIEW 3 major objections 4 minor 36 references

Cracking the Code of Action: a Generative Approach to Affordances for Reinforcement Learning

T0 review · 3 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A vision-language model can write code that computes the affordable actions in any screenshot, and hard-masking a deep RL agent with that code makes web navigation orders of magnitude more sample-efficient.

desk verdict A real extension of Code as Reward with a clear pipeline and honest limitations, but the 'orders of magnitude' claim is measured at a single 1000-step checkpoint and needs re-analysis before it can be taken at face value. read the letter →

arxiv 2504.17282 v1 pith:BT53RUBK submitted 2025-04-24 cs.AI

Lynn Cherif , Flemming Kondrup , David Venuto , Ankit Anand , Doina Precup , Khimya Khetarpal This is my paper · ORCID
classification cs.AI
keywords affordancesreinforcementlearningsampleefficiencyvision-languagemodelscodegenerationactionmaskingwebnavigationMiniWob++
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Reinforcement learning for GUI-based web navigation is sample-hungry because the action space is huge and rewards are sparse: at any state, only a few of the thousands of possible clicks matter. This paper proposes Code as Generative Affordances (CoGA), a pipeline in which a pre-trained vision-language model infers the task's intents, extracts image templates of the relevant objects from five example screenshots, and writes Python code that returns the set of affordable actions (action type plus pixel coordinates) for any observation. A critique VLM reviews and regenerates the code, and precision and recall against five manually annotated test cases select the final program, which then runs inside the training loop as a hard mask on a DQN agent's action space. The authors claim that on the MiniWob++ benchmark CoGA is orders of magnitude more sample-efficient than its unmasked RL agent, that the scripts transfer within families of tasks sharing a GUI, and that CoGA with no expert demonstrations outperforms behavior cloning trained on up to 200 expert trajectories.

What carries the argument

The load-bearing object is `determine_affordable_actions(observation)`, a generated Python function that maps a pixel screenshot to a set of affordable actions, each an action type (e.g., CLICK) with pixel bounding-box coordinates, detected by matching VLM-extracted object templates in grayscale. The machinery has three stages: the VLM infers task-level intents and object coordinates using a superimposed grid on the screenshot; a pre-written script crops the objects as grayscale templates from five example observations; and a critique VLM performs up to three rounds of review and regeneration while ground-truth precision and recall over five manual test cases pick the final program. In RL, the returned affordances hard-mask the agent's action space of $4$ action types by $1024$ discretized pixel bins, with the mask applied both to action selection and to bootstrapping the next observation's Q-target.

What would settle it

Run a task with varying text content (e.g., use-spinner or a click-button variant with unseen labels), extract templates from five observations drawn from only one part of the distribution, and evaluate the generated script's recall on held-out observations from another part; if recall falls below the level needed for task success and the hard-masked agent's success rate collapses to zero while the unmasked DQN still learns, then the sample-efficiency claim holds only when the five-observation templates happen to cover the states the agent encounters.

Watch

Extended reading notes

Core claim

The central claim is that intent-based affordances can be specified as executable code generated offline by a vision-language model, and that using this code to prune the RL agent's action space converts an unwieldy exploration problem into a tractable one. Concretely, CoGA has the VLM name the relevant intents (e.g., 'click a tab'), localize the corresponding objects as bounding boxes on a coordinate-gridded screenshot, crop their templates from five sampled observations, and write a script that uses template matching to detect those objects in any new observation and return the affordable actions. The script is improved by a critique VLM for up to three rounds and selected by precision and recall against five manually annotated test observations. The paper demonstrates on 23 MiniWob++ tasks that the masked DQN reaches over ten times the success rate of the unmasked baseline within the first 1000 training steps, that a script generated for one task transfers to a sibling task with the same GUI, and that CoGA with zero expert demonstrations beats behavior cloning trained on up to 200 expert trajectories, with BC overtaking it only at the 1000-demonstration regime.

Load-bearing premise

The load-bearing premise is that the five sampled observations used to extract object templates and the five manually annotated test observations are representative of the full distribution of states the RL agent will visit; because the mask is hard, any object the template matcher fails to detect is permanently removed from the action set, so a single low-recall state makes the task impossible.

Editorial extensions

If this is right

  • On the 23 MiniWob++ tasks tested, agents using CoGA's masks reach higher evaluation success rates at 1000 steps than the unmasked DQN baseline on most tasks when the generated script has high F1, implying that affordance pruning can substitute for large amounts of environment interaction.
  • Affordance scripts transfer across tasks that share a GUI and affordance set but demand different optimal policies (e.g., click-test-2 and click-button-sequence), so one script can serve several related tasks without regeneration.
  • In the low-data regime, CoGA with no expert demonstrations outperforms behavior cloning trained on up to 200 expert trajectories, which suggests that action-space masking uses prior knowledge more efficiently than a small amount of imitation data.
  • Because the VLM is queried only during the offline pipeline and the deployed script is ordinary template matching, the RL loop pays no per-step VLM inference cost, keeping the sample-efficiency gains cheap at deployment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not ablate its VLM-written scripts against hand-written affordance code built from the same templates, so the relative contribution of the VLM's code-generation versus the masking mechanism itself remains untested; replacing the generated script with a human-written one derived from the same five templates would isolate where the gains come from.
  • The hard-mask failure mode the authors identify is not addressed: a low-recall script permanently removes useful actions, so any state in which the template matcher misses an object becomes unsolvable; a soft-mask variant or an empty-set fallback that reopens the action space would test whether CoGA's gains survive the imperfect-recall regime.
  • Because the pipeline spends the expensive multimodal inference once, offline, and ships deterministic matching code, CoGA points toward a general recipe for cutting per-deployment foundation-model costs in embodied RL; the paper states the qualitative advantage but does not quantify the cost trade-off, which a simple token-cost accounting of the pipeline versus in-the-loop VLM querying would settl
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes CoGA (Code as Generative Affordances), a method that uses a pre-trained vision-language model (VLM) to generate Python code that computes, from a pixel observation, a set of affordable actions (action types and pixel coordinates) for web navigation tasks in MiniWob++. The generated affordances are used as a hard mask on the action space of a DQN agent during training and inference. The paper claims three results: (1) CoGA is orders of magnitude more sample efficient than its RL baseline, (2) generated affordance scripts generalize within task families, and (3) CoGA performs better than or on par with behavior cloning (BC) in the low-demonstration regime. The evidence consists of per-task success rates at 1000 steps for 23 tasks, F1 scores on 5 manually annotated observations per task, a three-task transfer experiment, and BC comparisons on a subset of tasks.

Significance. If the central claim were fully supported, the contribution would be practically meaningful: distilling VLM knowledge into cheap, executable action-mask code could avoid expensive VLM calls during RL and substantially improve sample efficiency in pixel-based web navigation. The method is creative, the pipeline is described in sufficient detail to be reproducible, and the authors include example generated scripts and templates in the appendix. The potential weakness is not the idea but the evidence: the headline 'orders of magnitude' claim rests on a single-checkpoint success-rate ratio, the F1 estimates rely on only five manual observations per task, and the task selection is influenced by script quality. These issues make the current version's quantitative claims stronger than the data warrant, though the approach itself is defensible and worth revising.

major comments (3)
  1. [§4.3, Figure 3] The central claim 'CoGA is orders of magnitude more sample efficient' is operationalized as a ratio of best evaluation success rates at a single 1000-step checkpoint, not as a step-to-threshold or area-under-the-curve comparison. On tasks where the RL baseline's success rate is near zero at 1000 steps (e.g., click-dialog-2, use-spinner, click-checkboxes-large), any positive CoGA success yields an arbitrarily large ratio, so the reported 'over 10x' figure is not a robust measure of sample efficiency. The authors should re-measure sample efficiency as the number of steps needed to reach a fixed success threshold, or report the area under the learning curve, and present confidence intervals over seeds.
  2. [§3.2, §3.3, §4.3 (Figure 2)] The F1 scores that justify the choice of tasks are computed from only 5 manually annotated observations per task, and the template images are extracted from 5 observations. Because the generated code applies a hard mask that removes any undetected action (Section 3.3), a single template-matching failure in a new observation can make a task unsolvable. With 5 test observations, the recall estimate has very high variance and cannot establish the generalizable recall needed to support the sample-efficiency claim. The authors should evaluate script quality on a larger held-out set of observations (or a held-out set of tasks) and relate measured recall to downstream RL performance; the limitation acknowledged in Section 6.1 (template matching handles only isomorphic objects, VLM pixel mapping is unreliable) makes this concern concrete.
  3. [§4.3, Table 1] The claim that generated scripts generalize within a task family rests on only three tasks, one of which (click-checkboxes-large) has near-zero success with both the original and transfer scripts (0.33% and 0.67%), while another (focus-text-2) has an RL baseline with a standard deviation of 28.79 percentage points. This evidence is too thin to support the general statement 'CoGA's generated affordance scripts can generalize within the same family of tasks' as formulated in the abstract and Section 4.3. Additional transfer pairs and more seeds are needed before this claim can be considered established.
minor comments (4)
  1. [§4.3] The text says 'over 10 times more sample efficient' and 'orders of magnitude more sample efficient' interchangeably; these are not the same, and the abstract's plural 'orders of magnitude' overstates what the data show even if the measurement issue is fixed.
  2. [§4.3 and Figure 4 caption] The text references 'Figure 4 (right)' for a statement about average performance, but the caption labels the right panel as per-task success rates and the left panel as the mean. Please make the references consistent with the panel labels.
  3. [Throughout] There are several typos: 'Qualitatevely' and 'Quantitavely' in Section 4.3, 'wihtout' in the Appendix A prompt, 'Ony return' in Appendix B, and 'inthetemplate' in Appendix A. These should be corrected before publication.
  4. [§3.1] The template matching threshold is listed in Appendix A as 0.5 in one script and 0.7 in the example scripts in Appendix E; the paper should state whether the threshold is a fixed hyperparameter or is tuned per task and how the final threshold is selected.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central sample-efficiency claim is an empirical comparison against an external RL baseline, not a quantity forced by construction or by a self-citation chain.

full rationale

The paper's three claims are experimental. CoGA's affordance scripts are generated by a VLM, verified against five manually annotated test cases, and then used as a hard mask during RL training and evaluation. The 'orders of magnitude' sample-efficiency claim is operationalized as best evaluation success rates at 1000 steps compared with a DQN baseline on the same tasks and seeds; this is an external comparison rather than an algebraic consequence of the mask. The F1-based task selection and the 'best over 1000 steps' reporting are evaluation-design concerns (selection bias and checkpoint sensitivity), not circular derivations. The self-citation to Venuto et al. (2024)'s Code as Reward pipeline is used only as a prompting-pipeline starting point and is not the evidence for any of the paper's claims. The Khetarpal et al. (2020) affordance definition is a conceptual framing, not a theorem imported to force a conclusion. Possible VLM pretraining contamination on MiniWob++ is a data-contamination worry, not a circularity in the derivation chain. No equation or fitted parameter is shown to define the target result into existence; each claim (sample efficiency, within-family transfer, BC parity) is measured against external baselines or held-out task instances. Therefore no specific reduction can be exhibited, and the honest finding is no significant circularity.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central claims rest on a few assumptions about the representativeness of the small observation samples, the reliability of template matching, the competence of the VLM, and the correctness of manual annotations. The only fitted scalar that directly controls the core mechanism is the template matching threshold, which the critique VLM tunes on 5 test cases per task.

free parameters (1)
  • Template matching threshold per task = Varies by task (e.g., 0.5 in click-test and count-sides; 0.7 in click-tab)
    Set inside the VLM-generated script and adjusted by the critique VLM against 5 manually annotated test observations; it directly controls precision and recall of the affordance detector that the sample-efficiency result depends on.
assumptions (4)
  • domain assumption The 5 randomly sampled observations used for template extraction are representative of the full observation distribution for each task.
    The VLM extracts object templates from 5 observations, and the generated code is expected to detect objects in all states encountered by the RL agent. Section 3.1 states that templates are derived from 5 randomly sampled observations to maximize generalization, but no evidence is given that 5 is sufficient.
  • domain assumption OpenCV template matching with normalized cross-correlation reliably identifies the relevant GUI objects across the observation distribution.
    The entire affordance detection rests on template matching between grayscale images and a fixed threshold. Section 6.1 concedes this only detects isomorphic objects and fails on varying text-based objects and tasks like bisect-angle.
  • domain assumption The VLM (GPT-4o) is able to infer correct intents, object bounding boxes, and executable code from a handful of observations and the prompting template.
    The method depends on the VLM's ability to map pixel coordinates correctly and write robust code. Section 6.1 acknowledges the VLM struggles with exact pixel coordinates and that token limits and reasoning errors occur.
  • domain assumption The 5 manually annotated ground-truth test cases are correct and sufficient to estimate script precision and recall.
    Section 3.2 uses 5 manual test cases to compute F1 and to select the best script; the critique VLM does not see the ground truth. This small annotated set is the only quantitative validation of the affordance scripts.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Cracking the Code of Action: a Generative Approach to Affordances for Reinforcement Learning." pith.science (2026). https://pith.science/paper/BT53RUBK

@misc{pith2026250417282,
  author       = {Pith},
  title        = {Pith review of: Cracking the Code of Action: a Generative Approach to Affordances for Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BT53RUBK}},
  note         = {Machine review of arXiv:2504.17282}
}
abstract

Agents that can autonomously navigate the web through a graphical user interface (GUI) using a unified action space (e.g., mouse and keyboard actions) can require very large amounts of domain-specific expert demonstrations to achieve good performance. Low sample efficiency is often exacerbated in sparse-reward and large-action-space environments, such as a web GUI, where only a few actions are relevant in any given situation. In this work, we consider the low-data regime, with limited or no access to expert behavior. To enable sample-efficient learning, we explore the effect of constraining the action space through $\textit{intent-based affordances}$ -- i.e., considering in any situation only the subset of actions that achieve a desired outcome. We propose $\textbf{Code as Generative Affordances}$ $(\textbf{$\texttt{CoGA}$})$, a method that leverages pre-trained vision-language models (VLMs) to generate code that determines affordable actions through implicit intent-completion functions and using a fully-automated program generation and verification pipeline. These programs are then used in-the-loop of a reinforcement learning agent to return a set of affordances given a pixel observation. By greatly reducing the number of actions that an agent must consider, we demonstrate on a wide range of tasks in the MiniWob++ benchmark that: $\textbf{1)}$ $\texttt{CoGA}$ is orders of magnitude more sample efficient than its RL agent, $\textbf{2)}$ $\texttt{CoGA}$'s programs can generalize within a family of tasks, and $\textbf{3)}$ $\texttt{CoGA}$ performs better or on par compared with behavior cloning when a small number of expert demonstrations is available.

Figures

Figures reproduced from arXiv: 2504.17282 by the authors.

Figure 1
Figure 1. Left: Overview of our method, CoGA. The VLM processes available task descriptions and example observations to extract relevant intents (e.g., “click on a tab”) and object template images (e.g., every tab), which are then used to generate code that returns a set of affordable actions given an observation. The code is validated and improved by a critique VLM. The set of affordances are then used to mask the action spa… view at source ↗
Figure 2
Figure 2. Left: Examples of returned affordances for three tasks (left to right): click-test, click-test-2, click-tab Right: F1-score across tested tasks. We observe that most gener￾ated affordance scripts have a high F1-score, implying wide and precise coverage of ground truth affordances. Quantitavely, we measure the quality of the generated affordance scripts using precision and recall, and aggregate them through an F1-sco… view at source ↗
Figure 3
Figure 3. Left: Evaluation success rates at 1000 steps for the RL agent and CoGA across tasks. We observe that CoGA is over 10 times more sample efficient than the RL agent early in training at only 1000 steps. Right: Evaluation success rate curves for the RL agent and CoGA on count-sides (left) and click-test-2 (right) CoGA’s Affordance Scripts can Generalize within the Same Family of Tasks. We define a family of tasks as ta… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Right: Evaluation success rates across tasks and expert data regimes of the BC agent, the RL agent, and CoGA (mean and standard deviation over 3 seeds). Left: Mean of evaluation success rates across tasks for the RL agent, CoGA, and increasing expert data regimes of th…
Figure 5
Figure 5. Figure 5: Example scripts across 2 successful tasks (click-tab, count-sides) and 2 unsuccessful tasks [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]
Figure 6
Figure 6. Figure 6: Example template images across 2 successful tasks (click-tab, count-sides) and 2 unsuc [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

36 extracted references · 7 canonical work pages

  1. [1]

    Developing a computer use model

    Anthropic. Developing a computer use model. 2024. URL https://www.anthropic.com/news/developing-computer-use

    work page 2024

  2. [2]

    Uniter: Universal image-text representation learning

    Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. 2020

    work page 2020

  3. [3]

    Seeclick: Harnessing gui grounding for advanced visual gui agents, 2024

    Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. Seeclick: Harnessing gui grounding for advanced visual gui agents, 2024. URL https://arxiv.org/abs/2401.10935

    arXiv 2024

  4. [4]

    The theory of affordances

    James J Gibson. The theory of affordances. Hilldale, USA, 1 0 (2), 1977

    work page 1977

  5. [5]

    Voxposer: Composable 3d value maps for robotic manipulation with language models, 2023

    Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. Voxposer: Composable 3d value maps for robotic manipulation with language models, 2023. URL https://arxiv.org/abs/2307.05973

    arXiv 2023

  6. [6]

    A data-driven approach for learning to control computers, 2022

    Peter C Humphreys, David Raposo, Toby Pohlen, Gregory Thornton, Rachita Chhaparia, Alistair Muldal, Josh Abramson, Petko Georgiev, Alex Goldin, Adam Santoro, and Timothy Lillicrap. A data-driven approach for learning to control computers, 2022. URL https://arxiv.org/abs/2202.08137

    arXiv 2022

  7. [7]

    What can i do here? a theory of affordances in reinforcement learning

    Khimya Khetarpal, Zafarali Ahmed, Gheorghe Comanici, David Abel, and Doina Precup. What can i do here? a theory of affordances in reinforcement learning. In International Conference on Machine Learning, pp.\ 5243--5253. PMLR, 2020

    2020

  8. [8]

    Language models can solve computer tasks, 2023

    Geunwoo Kim, Pierre Baldi, and Stephen McAleer. Language models can solve computer tasks, 2023. URL https://arxiv.org/abs/2303.17491

    arXiv 2023

Show all 36 references
  1. [9]

    Lee, Annie Xie, Kuan Fang, Karl Pertsch, and Chelsea Finn

    Olivia Y. Lee, Annie Xie, Kuan Fang, Karl Pertsch, and Chelsea Finn. Affordance-guided reinforcement learning via visual prompting, 2024. URL https://arxiv.org/abs/2407.10341

    2024 arXiv

  2. [10]

    Code as policies: Language model programs for embodied control, 2023

    Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control, 2023

    2023

  3. [11]

    Reinforcement learning on web interfaces using workflow-guided exploration, 2018

    Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tianlin Shi, and Percy Liang. Reinforcement learning on web interfaces using workflow-guided exploration, 2018. URL https://arxiv.org/abs/1802.08802

    2018 arXiv

  4. [12]

    Moka: Open-world robotic manipulation through mark-based visual prompting, 2024

    Fangchen Liu, Kuan Fang, Pieter Abbeel, and Sergey Levine. Moka: Open-world robotic manipulation through mark-based visual prompting, 2024. URL https://arxiv.org/abs/2403.03174

    2024 arXiv

  5. [13]

    Eureka: Human-level reward design via coding large language models, 2024

    Yecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Eureka: Human-level reward design via coding large language models, 2024. URL https://arxiv.org/abs/2310.12931

    2024 arXiv

  6. [14]

    Playing atari with deep reinforcement learning, 2013

    Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning, 2013. URL https://arxiv.org/abs/1312.5602

    2013 arXiv

  7. [15]

    Computer-using agent: Introducing a universal interface for ai to interact with the digital world

    OpenAI. Computer-using agent: Introducing a universal interface for ai to interact with the digital world. 2025. URL https://openai.com/index/computer-using-agent

    2025

  8. [16]

    Gpt-4 technical report, 2023

    OpenAI, :, Josh Achiam, and et al. Gpt-4 technical report, 2023

    2023

  9. [17]

    Affordancellm: Grounding affordance from vision language models, 2024

    Shengyi Qian, Weifeng Chen, Min Bai, Xiong Zhou, Zhuowen Tu, and Li Erran Li. Affordancellm: Grounding affordance from vision language models, 2024

    2024

  10. [18]

    Learning transferable visual models from natural language supervision, 2021

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021

    2021

  11. [19]

    Sentence-bert: Sentence embeddings using siamese bert-networks, 2019

    Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks, 2019. URL https://arxiv.org/abs/1908.10084

    2019 arXiv

  12. [20]

    Efficient reductions for imitation learning

    Stephane Ross and Drew Bagnell. Efficient reductions for imitation learning. In Yee Whye Teh and Mike Titterington (eds.), Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, volume 9 of Proceedings of Machine Learning Research, pp...

    2010

  13. [21]

    Prioritized experience replay, 2016

    Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay, 2016. URL https://arxiv.org/abs/1511.05952

    2016 arXiv

  14. [22]

    From pixels to ui actions: Learning to follow instructions via graphical user interfaces

    Peter Shaw, Mandar Joshi, James Cohan, Jonathan Berant, Panupong Pasupat, Hexiang Hu, Urvashi Khandelwal, Kenton Lee, and Kristina N Toutanova. From pixels to ui actions: Learning to follow instructions via graphical user interfaces. Advances in Neural Information Processing S...

    2023

  15. [23]

    World of bits: An open-domain platform for web-based agents

    Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hernandez, and Percy Liang. World of bits: An open-domain platform for web-based agents. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of...

    2017

  16. [24]

    Mastering the game of go with deep neural networks and tree search

    David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529 0 (7587): 0 484-...

    2016

  17. [25]

    Mastering chess and shogi by self-play with a general reinforcement learning algorithm, 2017

    David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Simonyan, and Demis Hassabis. Mastering chess and shogi by self-play with a general reinforce...

    2017 arXiv

  18. [26]

    Reinforcement learning: An introduction

    Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018

    2018

  19. [27]

    Deep reinforcement learning with double q-learning, 2015

    Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning, 2015. URL https://arxiv.org/abs/1509.06461

    2015 arXiv

  20. [28]

    Code as reward: Empowering reinforcement learning with vlms, 2024

    David Venuto, Sami Nur Islam, Martin Klissarov, Doina Precup, Sherry Yang, and Ankit Anand. Code as reward: Empowering reinforcement learning with vlms, 2024

    2024

  21. [29]

    Voyager: An open-ended embodied agent with large language models, 2023

    Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models, 2023

    2023

  22. [30]

    Q-learning

    Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8: 0 279--292, 1992

    1992

  23. [31]

    Chain-of-thought prompting elicits reasoning in large language models, 2023

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023. URL https://arxiv.org/abs/2201.11903

    2023 arXiv

  24. [32]

    Octopus: Embodied vision-language programmer from environmental feedback, 2024

    Jingkang Yang, Yuhao Dong, Shuai Liu, Bo Li, Ziyue Wang, Chencheng Jiang, Haoran Tan, Jiamu Kang, Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. Octopus: Embodied vision-language programmer from environmental feedback, 2024. URL https://arxiv.org/abs/2310.08588

    2024 arXiv

  25. [33]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  26. [34]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  27. [35]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  28. [36]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.