Pith. sign in

REVIEW 3 major objections 3 minor 3 cited by

Digi-Q: Learning Q-Value Functions for Training Device-Control Agents

T0 review · 3 major / 3 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The central claim is that offline TD-learning on frozen VLM features, preceded by actionability fine-tuning and followed by Best-of-N reranking, is an effective recipe for training Android device-control agents without environment…

desk verdict Genuinely new offline RL recipe with strong internal ablations, but the headline gain over DigiRL is not a matched comparison; still deserves review with a request for controlled baselines. read the letter →

arxiv 2502.15760 v1 pith:HUAP2UUM submitted 2025-02-13 cs.LG

Hao Bai , Yifei Zhou , Li Erran Li , Sergey Levine , Aviral Kumar This is my paper
classification cs.LG
keywords devicecontrolvision-languagemodelQ-functionofflinereinforcementlearningtemporal-differenceBest-of-NpolicyextractionAndroid-in-the-Wildrepresentationfine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Digi-Q claims that a capable Android control agent can be trained entirely from a static interaction log, without any live rollouts, by learning an action-value function on top of a frozen vision-language model. The method fine-tunes the VLM's representations on a binary question — does this action change the screen? — so that the features carry actionable information, then runs temporal-difference learning to train a Q-head, and finally extracts a policy by sampling 16 candidate actions per state and imitating whichever one the Q-function likes best. The practical appeal is that on-policy reinforcement learning, the current default for agentic foundation models, needs expensive and sometimes privacy-sensitive environment interaction; an offline Q-function promises the same kind of policy improvement while reusing suboptimal historical data. On Android-in-the-Wild, the paper reports a 21.2% relative improvement over the best prior offline method and, in some cases, success rates that roughly match an online RL agent that is allowed to interact.

What carries the argument

The load-bearing object is a turn-level action-value function $Q_\theta(s,a)$ over a screenshot-plus-history state and a token-level action, realized as an MLP head on frozen intermediate VLM features and trained with TD bootstrapping against a state-only value function $V_\psi(s)$. Three design choices do the work: representation fine-tuning, which trains the VLM to answer 'Will this action lead to a different state?' from $\ell^2$ pixel-change labels and supplies features that do not ignore the action; TD learning on the frozen features with delayed target networks, which stabilizes the critic; and Best-of-N policy extraction, which samples $N=16$ actions from a behavior-cloned policy, keeps the unique top action with positive advantage $Q(s,a)-V(s)$, and imitates it with a supervised log-loss. The Best-of-N step is what lets the policy move away from the data-collection policy without the unstable negative-gradient term in policy-gradient methods.

What would settle it

On the same AitW training buffers, compute the binary label $y_t = \mathbf{1}\{d(s_t,s_{t+1}) > \epsilon\}$ for a sweep of $\epsilon$ values and measure its agreement with the reward signal or with human judgments of whether the transition advanced the task; if no threshold gives better-than-chance agreement, or if the best threshold changes across the General and Web Shopping subsets, the representation fine-tuning label is not carrying actionable information and the Digi-Q pipeline should degrade toward a state-only Q-function.

Watch

Extended reading notes

Core claim

The central claim is that Q-learning, rather than filtered imitation or policy gradients, is the right offline objective for training device-control agents with VLMs. Digi-Q instantiates the critic as a small MLP head on frozen intermediate features of a VLM, with a state-only value function used in the TD target; before freezing, the VLM is fine-tuned with a binary cross-entropy loss that labels a transition $(s_t,a_t,s_{t+1})$ positive when the $\ell^2$ pixel distance between $s_t$ and $s_{t+1}$ exceeds a threshold $\epsilon$. This representation fine-tuning is what prevents the Q-function from degenerating into a state-only value function. Policy extraction is a Best-of-N operator: the model samples $N=16$ actions from a behavior-cloned policy $\pi_\beta$, ranks them by advantage $Q(s,a)-V(s)$, and maximizes the log-likelihood of the single best action with positive advantage. The paper reports 61.5% train and 71.2% test success on AitW General, and 53.1% train and 58.0% test on AitW Web Shopping, a 21.2% relative gain over the best preceding offline method and rough parity with the online DigiRL method in some settings.

Load-bearing premise

The load-bearing premise is that a big visual change from one screen to the next means the action was useful; the paper never reports the value of the threshold $\epsilon$ used to make those labels, and if that visual-change proxy does not track task progress, the representation fine-tuning stage cannot produce features that support TD-learning.

Editorial extensions

If this is right

  • Offline TD learning on frozen VLM features is sufficient to train a competitive device-control agent, so costly or privacy-sensitive environments can still yield strong policies from static historical logs.
  • The paper's ablations show success rising monotonically as $N$ goes from 1 to 16, so extra action-sampling compute converts directly into policy quality without any environment interaction.
  • The qualitative stitching example shows a Q-function combining a successful-but-lengthy trajectory with a failed-but-short trajectory into a successful short policy, which imitation-only methods cannot do.
  • Without the representation fine-tuning stage, the Q-function ignores the action and approximates a state-only value function, so any offline Q-learning recipe on VLM features must address action coverage explicitly.
  • In some AitW settings Digi-Q matches an online RL agent, meaning on-policy interaction can be replaced by more offline data or more sampled actions without losing performance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • We infer that the fixed pixel-distance threshold is the least portable part of the recipe: useful actions with delayed or subtle visual effects, such as typing or waiting, may produce small pixel differences, so a learned progress detector or transition-level preference labels could replace $\epsilon$ and likely transfer across apps.
  • We infer that the Best-of-N operator is structurally similar to verifier-guided search in reasoning: if the Q-function is accurate, the same frozen VLM could score thousands of candidate actions at deployment, scaling test-time compute for device control without any weight updates.
  • We infer that because only the Q-function needs the fine-tuned VLM while the V-function uses off-the-shelf visual features, the marginal cost over behavior cloning is small, and the fine-tuned representations may transfer to other GUI domains, which the paper leaves untested.
  • We infer that the critic could serve as a rollout filter in an online loop the paper lists as future work: accept only newly collected actions with positive advantage, which would raise the signal-to-noise ratio of the next offline dataset.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper introduces Digi-Q, an offline reinforcement learning method for Android device control. The method trains a Q-function on top of frozen VLM features, after a representation fine-tuning phase that uses a binary pixel-change label to emphasize actionable information. The trained Q-function is then used with a Best-of-N policy extraction objective, which samples several candidate actions from a behavior policy, ranks them by Q-value, and trains the policy to imitate the highest-scoring action with positive advantage. The authors report a 21.2% relative improvement over the best prior offline method (DigiRL) on Android-in-the-Wild and performance roughly matching online DigiRL in some cases. Internal ablations show that representation fine-tuning and Best-of-N extraction are beneficial when the critic and training data are held fixed.

Significance. If the headline results were established in a controlled comparison, this would be a meaningful advance: it demonstrates a scalable and relatively stable way to apply value-based offline RL to vision-language device-control agents, with compute savings from frozen representations and a stable supervised policy-extraction loss. The paper is transparent about known limitations (footnote 1, Appendix C.1) and provides open-source code. The internal ablations, which use the same critic and the same data, provide credible evidence for the value of representation fine-tuning and Best-of-N extraction over AWR, REINFORCE, and Monte-Carlo critic training. However, the main performance claim is not currently supported by a matched comparison, and a load-bearing hyperparameter is unreported, so the empirical contribution is only partially established.

major comments (3)
  1. [Section 5.1, Table 1, Appendix E, footnote 1] The headline 21.2% relative improvement over DigiRL is not a controlled comparison. Appendix E states that the results for Filtered BC and DigiRL are kept the same as in Bai et al. (2024), while footnote 1 states that the authors 'also collected trajectory dataset with higher success rate for offline training' and used it for Digi-Q. Since the baselines were not retrained on the same offline dataset, the reported gain conflates two differences: the Digi-Q algorithm itself and a different, higher-quality training distribution. The argument that the test environment has become harder since March 2024 does not remove this confound, because the training-distribution advantage remains. To support the central claim that Digi-Q outperforms prior offline methods, the authors must either retrain the baselines on the identical offline trajectory dataset and report the resulting numbers, or clearly separate the algorithmic contribution from the dataset contribution in the headline claims.
  2. [Section 4.1, Eq. (3)] The binary label used for representation fine-tuning is defined by an unspecified threshold epsilon on the l2 image distance: y_t = 0 if d(s_t, s_{t+1}) < epsilon and 1 otherwise. The value of epsilon is never reported anywhere in the paper, and no sensitivity analysis is given. This is not a cosmetic omission: Table 2 shows that the representation fine-tuning stage is critical for the method (removing it drops the Web-Shopping test success rate from 58.0% to 31.9%). Without epsilon, the procedure cannot be reproduced, and it is unclear whether the state-change proxy is robust to the choice of threshold. Additionally, the paper asserts that 'a useful action should lead to a substantial visual change' but never validates this proxy against actual task progress or human success labels, even though the method's first stage rests entirely on this assumption. The authors should report the exact epsilon, provide a sensitivity study, and ideally validate the proxy against the success signal used for evaluation.
  3. [Section 5.1, Table 1] The statement that Digi-Q 'roughly matches' online DigiRL is not supported by a matched comparison. Digi-Q is trained offline on 1296 (Web Shopping) and 1008 (General) trajectories, while online DigiRL uses a different setup that begins with 512 initial trajectories and is allowed additional environment interaction. The two settings have different information available to the algorithm, and the reported success rates are close but not directly comparable. The claim should be qualified to note that Digi-Q matches a specific online RL approach under a different data budget, or the comparison should be made under matched conditions (e.g., same number of trajectories and no online interaction for either method).
minor comments (3)
  1. [Throughout] The manuscript contains several typos and formatting errors: 'cirtic' in Section 3.3, 'prrcess' in Appendix B.1, 'magnitide' in Appendix B.4, 'expriments' in Section 4, and 'thebolded' in Table 4. These should be corrected before publication.
  2. [Appendix B.3] The text says the Best-of-N values are drawn from {1, 2, 4, 8, 16}, but Figure 3 (Left) reports results for N in {1, 4, 8, 16}. Clarify whether N=2 was evaluated and why it is omitted from the plot.
  3. [Table 2] The row label 'Digi-Q w/ BLIP-2 + BERT' is confusing because Appendix B.2 states that the final Digi-Q architecture also uses BLIP-2 and BERT features in addition to the VLM intermediate-layer representations. Please clarify what exactly differs between this ablation and the full Digi-Q.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Digi-Q's TD-learning and Best-of-N extraction are self-contained; the unmatched-baseline comparison and unreported epsilon are correctness concerns, not circular reductions.

full rationale

Digi-Q's derivation chain is self-contained: the Q-function is trained by TD-learning (Eq. 4) against environment rewards plus bootstrapped state values, the V-function is trained by Eq. 5, and the actor is extracted by Best-of-N imitation of actions ranked by the learned Q (Eq. 6). No reported success-rate metric is defined in terms of the method's own parameters, and the state-change pretext label in Section 4.1 is an auxiliary representation task, not the evaluation objective. The ablation in Table 2 shows the pretext is empirically necessary, but that does not make the final claim circular. The main comparison in Table 1 is not fully controlled—Appendix E states 'Results for all other methods (Filtered BC and DigiRL) are kept the same as discussed in the original paper (Bai et al., 2024),' while Section 5.1 notes Digi-Q used a 'trajectory dataset with higher success rate for offline training'—but this is a benchmarking confound and a correctness risk, not an equation-level reduction: the baseline numbers are externally published results, and the final evaluation is an external environment verdict. The unreported threshold epsilon in Section 4.1 is a reproducibility gap, not a self-definition. Hence no circular step is identifiable from the paper's equations or citations.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on a small set of offline-RL assumptions plus one ad hoc proxy (state-change as action usefulness). The proxy is not backed by an independent falsifiable handle, and the threshold epsilon is undisclosed. No new physical or conceptual entities such as particles, forces, or dimensions are introduced.

free parameters (4)
  • epsilon (l2 image distance threshold for state-change label)
    Defines the binary label y_t in Equation 3 used for representation fine-tuning. The value is never reported, so the most load-bearing preprocessing choice is not specified.
  • discount factor gamma
    Appears in the TD target in Equation 4 but is never assigned a value. With horizons H=10 and H=20, the choice of discount affects credit assignment in the Q-function.
  • Best-of-N sample count N = 16
    Chosen by hand after experiments; Figure 3 shows performance improves as N increases, so the reported result depends on this hand-selected value.
  • advantage threshold A = 0.05 (webshop)
    Used in Appendix B.4 to decide which state-action pairs the actor should learn from; this threshold shapes the filtered imitation objective.
assumptions (4)
  • domain assumption The state is fully captured by the interaction history plus current screenshot, and the Gemini evaluator's 0/1 verdict is a valid reward signal.
    Section 3.1 defines state and reward this way. If the evaluator is noisy or the history is insufficient, all Q and V targets are corrupted.
  • ad hoc to paper An action is useful exactly when it produces a large pixel change in the next screenshot, d(s_t, s_{t+1}) > epsilon.
    Equation 3 and the surrounding paragraph assert this proxy, and the paper shows that without this fine-tuning stage the Q-function degenerates.
  • domain assumption The state-only bootstrap target r + gamma V(s') is a valid target for Q(s,a) under the current policy.
    Equations 4 and 5 use state-only bootstrapping instead of a max over actions. This implicitly assumes the V-function correctly tracks E_{a~pi} Q(s,a) even as the actor is updated later.
  • domain assumption The behavior-cloned policy pi_beta covers actions sufficient to improve over the data-collecting policy.
    Equation 6 samples actions from pi_beta only; if the behavior policy has poor coverage of good actions, Best-of-N cannot select actions outside that support.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Digi-Q: Learning Q-Value Functions for Training Device-Control Agents." pith.science (2026). https://pith.science/paper/HUAP2UUM

@misc{pith2026250215760,
  author       = {Pith},
  title        = {Pith review of: Digi-Q: Learning Q-Value Functions for Training Device-Control Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HUAP2UUM}},
  note         = {Machine review of arXiv:2502.15760}
}
read the original abstract

While a number of existing approaches for building foundation model agents rely on prompting or fine-tuning with human demonstrations, it is not sufficient in dynamic environments (e.g., mobile device control). On-policy reinforcement learning (RL) should address these limitations, but collecting actual rollouts in an environment is often undesirable in truly open-ended agentic problems such as mobile device control or interacting with humans, where each unit of interaction is associated with a cost. In such scenarios, a method for policy learning that can utilize off-policy experience by learning a trained action-value function is much more effective. In this paper, we develop an approach, called Digi-Q, to train VLM-based action-value Q-functions which are then used to extract the agent policy. We study our approach in the mobile device control setting. Digi-Q trains the Q-function using offline temporal-difference (TD) learning, on top of frozen, intermediate-layer features of a VLM. Compared to fine-tuning the whole VLM, this approach saves us compute and enhances scalability. To make the VLM features amenable for representing the Q-function, we need to employ an initial phase of fine-tuning to amplify coverage over actionable information needed for value function. Once trained, we use this Q-function via a Best-of-N policy extraction operator that imitates the best action out of multiple candidate actions from the current policy as ranked by the value function, enabling policy improvement without environment interaction. Digi-Q outperforms several prior methods on user-scale device control tasks in Android-in-the-Wild, attaining 21.2% improvement over prior best-performing method. In some cases, our Digi-Q approach already matches state-of-the-art RL methods that require interaction. The project is open-sourced at https://github.com/DigiRL-agent/digiq

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Thinking vs. Doing: Agents that Reason by Scaling Test-Time Interaction

    cs.LG 2025-06 conditional novelty 6.0 of 10

    Scaling the number of interaction steps, trained via a curriculum over rollout horizon, improves web-agent task success and outperforms scaling per-step reasoning under fixed token budgets.

  2. EvoCUA-1.5: Online Reinforcement Learning for Multi-turn Computer-Use Agents

    cs.AI 2026-07 conditional novelty 5.0 of 10

    Online RL with STEPO, policy-aware task filtering, DTAC curriculum, and async rollout infrastructure lifts a 32B computer-use agent to 63.2% on OSWorld-Verified.

  3. GUI-Reflection: Empowering Multimodal GUI Models with Self-Reflection Behavior

    cs.AI 2025-06 conditional novelty 5.0 of 10

    GUI-Reflection trains an 8B multimodal GUI agent to recognize mistakes, undo incorrect actions, and retry, improving AndroidWorld success rate from 14.58% (filtered BC baseline) to 34.72% with reflection data and onli...

Reference graph

Works this paper leans on

43 extracted references · 5 canonical work pages · cited by 3 Pith papers

  1. [1]

    Then the VLM parameters will be kept frozen and we train the Q- and V- functions using TD-learning on top of frozen VLM representations

    After initializing the parameters, we perform the representation fine-tuning procedure on top of VLM to obtain actionable features for later TD-learning. Then the VLM parameters will be kept frozen and we train the Q- and V- functions using TD-learning on top of frozen VLM representations. After both value functions are trained, we perform gradient update...

    work page 2023

  2. [3]

    URLhttps://arxiv.org/abs/2407.07726. Yevgen Chebotar, Quan Vuong, Alex Irpan, Karol Hausman, Fei Xia, Yao Lu, Aviral Kumar, Tianhe Yu, Alexander Herzog, Karl Pertsch, Keerthana Gopalakrishnan, Julian Ibarz, Ofir Nachum, Sumedh Sontakke, Grecia Salazar, Huong T Tran, Jodilyn Peralta, Clayton Tan, Deeksha Manjunath, Jaspiar Singht, Brianna Zitkovich, Tomas ...

    arXiv

  3. [4]

    Baian Chen, Chang Shu, Ehsan Shareghi, Nigel Collier, Karthik Narasimhan, and Shunyu Yao

    URL https://arxiv.org/abs/2309.10150. Baian Chen, Chang Shu, Ehsan Shareghi, Nigel Collier, Karthik Narasimhan, and Shunyu Yao. Fireact: Toward language agent fine-tuning,

    arXiv

  4. [5]

    William Chen, Oier Mees, Aviral Kumar, and Sergey Levine

    URLhttps://arxiv.org/abs/2310.05915. William Chen, Oier Mees, Aviral Kumar, and Sergey Levine. Vision-language models provide promptable representations for reinforcement learning,

    arXiv

  5. [6]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al

    URLhttps://arxiv.org/abs/2402.02651. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168,

    arXiv

  6. [7]

    usb-c to usb-a

    These problems were not observed in Bai et al. (2024). This is the main reason why some scores on the AitW Webshop subset in this paper falls a little behind Bai et al. (2024). ... costco.com does not respond to <ENTER> ... newegg.com banned Internet agents Go to costco.com, search for 'razer blade' Go to newegg,com, search for “usb-c to usb-a” Figure 7: ...

    work page 2024

  7. [9]

    org/abs/2312.11805

    URLhttps://arxiv. org/abs/2312.11805. Izzeddin Gur, Ofir Nachum, Yingjie Miao, Mustafa Safdari, Austin Huang, Aakanksha Chowdhery, Sharan Narang, Noah Fiedel, and Aleksandra Faust. Understanding html with large language models,

    arXiv

  8. [10]

    Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine

    URL https://arxiv.org/abs/2210.03945. Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor,

    arXiv

Show all 43 references
  1. [11]

    Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu

    URLhttps://arxiv.org/abs/ 1801.01290. Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. Webvoyager: Building an end-to-end web agent with large multimodal models,

    arXiv

  2. [12]

    Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver

    URL https://arxiv.org/abs/2401.13919. Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. InProceedings of the AAAI c...

    arXiv

  3. [13]

    Training compute-optimal large language models.arXiv preprint arXiv:2203.15556,

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models.arXiv preprint arXiv:2203.15556,

    arXiv

  4. [14]

    Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, and Sergey Levine

    URLhttps://arxiv.org/abs/2312.08914. Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, and Sergey Levine. Qt-opt: Scalable deep reinforcement learning for vision-based ...

    arXiv

  5. [15]

    Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried

    URLhttps://arxiv.org/ abs/1806.10293. Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks, 2024a. URLht...

    arXiv

  6. [17]

    Offline q-learning on diverse multi-task data both scales and generalizes.arXiv preprint arXiv:2211.15144,

    Aviral Kumar, Rishabh Agarwal, Xinyang Geng, George Tucker, and Sergey Levine. Offline q-learning on diverse multi-task data both scales and generalizes.arXiv preprint arXiv:2211.15144,

    arXiv

  7. [19]

    URLhttps://arxiv.org/abs/2308.03688. Xiao Liu, Tianjie Zhang, Yu Gu, Iat Long Iong, Yifan Xu, Xixuan Song, Shudan Zhang, Hanyu Lai, Xinyi Liu, Hanlin Zhao, Jiadai Sun, Xinyue Yang, Yu Yang, Zehan Qi, Shuntian Yao, Xueqiao Sun, Siyi Cheng, Qinkai Zheng, Hao Yu, Hanchen Zhang, W...

    arXiv

  8. [20]

    Max Sobol Mark, Tian Gao, Georgia Gabriela Sampaio, Mohan Kumar Srirama, Archit Sharma, Chelsea Finn, and Aviral Kumar

    URL https://arxiv.org/abs/2408.02544. Max Sobol Mark, Tian Gao, Georgia Gabriela Sampaio, Mohan Kumar Srirama, Archit Sharma, Chelsea Finn, and Aviral Kumar. Policy agnostic rl: Offline rl and online rl fine-tuning of any class and backbone. arXiv,

    arXiv

  9. [22]

    Seohong Park, Kevin Frans, Sergey Levine, and Aviral Kumar

    URLhttps://arxiv.org/abs/2404.06474. Seohong Park, Kevin Frans, Sergey Levine, and Aviral Kumar. Is value learning really the main bottleneck in offline rl?,

    arXiv

  10. [23]

    Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine

    URLhttps://arxiv.org/abs/2406.09329. Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning,

    arXiv

  11. [25]

    URLhttps: //arxiv.org/abs/2408.07199. 16 Digi-Q: Learning VLM Q-Value Functions for Training Device-Control Agents Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueg...

    arXiv

  12. [26]

    Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap

    URLhttps://arxiv.org/ abs/2103.00020. Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. Android in the wild: A large-scale dataset for android device control,

    arXiv

  13. [27]

    URLhttps://arxiv.org/abs/2307.10088. Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, Daniel Toyama, Robert Berry, Divya Tyama- gundlu, Timothy Lillicrap, and Oriana ...

    arXiv

  14. [28]

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov

    URLhttps://arxiv.org/abs/2405.14573. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms,

    arXiv

  15. [30]

    YifanSong,DaYin,XiangYue,JieHuang,SujianLi,andBillYuchenLin

    URLhttps://arxiv.org/abs/2206.11871. YifanSong,DaYin,XiangYue,JieHuang,SujianLi,andBillYuchenLin. Trialanderror: Exploration-based trajectory optimization for llm agents,

    arXiv

  16. [31]

    Fahim Tajwar, Anikait Singh, Archit Sharma, Rafael Rafailov, Jeff Schneider, Tengyang Xie, Stefano Ermon, Chelsea Finn, and Aviral Kumar

    URLhttps://arxiv.org/abs/2403.02502. Fahim Tajwar, Anikait Singh, Archit Sharma, Rafael Rafailov, Jeff Schneider, Tengyang Xie, Stefano Ermon, Chelsea Finn, and Aviral Kumar. Preference fine-tuning of llms should leverage suboptimal, on-policy data,

    arXiv

  17. [32]

    Siddharth Verma, Justin Fu, Mengjiao Yang, and Sergey Levine

    URLhttps://arxiv.org/abs/2404.14367. Siddharth Verma, Justin Fu, Mengjiao Yang, and Sergey Levine. Chai: A chatbot ai for task-oriented dialogue with offline reinforcement learning,

    arXiv

  18. [33]

    Ronald J Williams

    URLhttps://arxiv.org/abs/2204.08426. Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8:229–256,

    arXiv

  19. [34]

    Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao

    URLhttps://arxiv.org/abs/ 2311.07562. Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v,

    arXiv

  20. [35]

    URLhttps://arxiv.org/abs/2310.12823. 17 Digi-Q: Learning VLM Q-Value Functions for Training Device-Control Agents Yuexiang Zhai, Hao Bai, Zipeng Lin, Jiayi Pan, Shengbang Tong, Yifei Zhou, Alane Suhr, Saining Xie, Yann LeCun, Yi Ma, and Sergey Levine. Fine-tuning large vision-...

    arXiv

  21. [36]

    Chi Zhang, Zhao Yang, Jiaxuan Liu, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu

    URLhttps://arxiv.org/abs/2405.10292. Chi Zhang, Zhao Yang, Jiaxuan Liu, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. Appagent: Multimodal agents as smartphone users,

    arXiv

  22. [38]

    Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su

    URL https://arxiv.org/abs/2309.11436. Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. Gpt-4v(ision) is a generalist web agent, if grounded,

    arXiv

  23. [39]

    Shuyan Zhou, Frank F

    URLhttps://arxiv.org/abs/2401.01614. Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. Webarena: A realistic web environment for building autonomous agents, 2024a. URLht...

    arXiv

  24. [41]

    We found it a bit hard to fine-tune an entire VLM with TD-learning, which required iteration on hyperparameters such as learning rate and soft update rates for target networks. Due to the compute-intensive nature, we use a 3B VLM (PaLiGemma (Beyer et al., 2024)) for these runs...

    2022

  25. [42]

    transitions

    B.3. Training Dataset Construction We use the pre-trained AutoUI checkpoint to collect offline trajectories. Specifically, to collect each trajectory, starting from the home screen, the agent generates an action, and then the environment takes the action and transitions to the...

    2024

  26. [44]

    yes” or “no

    Results for all other methods (Filtered BC and DigiRL) are kept the same as discussed in the original paper (Bai et al., 2024). 24 Digi-Q: Learning VLM Q-Value Functions for Training Device-Control Agents Input: You're given a user interface. There is a cursor in the screen. R...

    2024

  27. [2013]

    URLhttps://arxiv.org/ abs/1312.5602. OpenAI. Gpt-4 technical report, 2024a. URLhttps://arxiv.org/abs/2303.08774. OpenAI. Gpt-4v(ision) technical work and authors, 2024b. URL https://openai.com/ contributions/gpt-4v/. Jiayi Pan, Yichi Zhang, Nicholas Tomlin, Yifei Zhou, Sergey ...

    arXiv

  28. [2017]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al

    URLhttps://arxiv.org/abs/1707.06347. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,

    arXiv

  29. [2018]

    GeminiTeam

    URLhttps://arxiv.org/abs/1802.09477. GeminiTeam. Gemini: A family of highly capable multimodal models,

    arXiv

  30. [2019]

    Pranav Putta, Edmund Mills, Naman Garg, Sumeet Motwani, Chelsea Finn, Divyansh Garg, and Rafael Rafailov

    URLhttps://arxiv.org/abs/1910.00177. Pranav Putta, Edmund Mills, Naman Garg, Sumeet Motwani, Chelsea Finn, Divyansh Garg, and Rafael Rafailov. Agent q: Advanced reasoning and learning for autonomous ai agents,

    1910 arXiv

  31. [2020]

    Aviral Kumar, Rishabh Agarwal, Tengyu Ma, Aaron Courville, George Tucker, and Sergey Levine

    URLhttps://arxiv.org/abs/2006.04779. Aviral Kumar, Rishabh Agarwal, Tengyu Ma, Aaron Courville, George Tucker, and Sergey Levine. Dr3: Value-based deep reinforcement learning requires explicit regularization. arXiv preprint arXiv:2112.04716,

    2006 arXiv

  32. [2021]

    14 Digi-Q: Learning VLM Q-Value Functions for Training Device-Control Agents Scott Fujimoto, Herke van Hoof, and David Meger

    URLhttps://arxiv.org/abs/2004.07219. 14 Digi-Q: Learning VLM Q-Value Functions for Training Device-Control Agents Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods,

    2004 arXiv

  33. [2022]

    Improved baselines with visual instruction tuning, 2024a

    15 Digi-Q: Learning VLM Q-Value Functions for Training Device-Control Agents Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2024a. URLhttps://arxiv.org/abs/2310.03744. Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanha...

    arXiv

  34. [2023]

    Hao Bai, Yifei Zhou, Mert Cemri, Jiayi Pan, Alane Suhr, Sergey Levine, and Aviral Kumar

    URL https://arxiv.org/abs/2311.18232. Hao Bai, Yifei Zhou, Mert Cemri, Jiayi Pan, Alane Suhr, Sergey Levine, and Aviral Kumar. Digirl: Training in-the-wild device-control agents with autonomous reinforcement learning,

    arXiv

  35. [2024]

    URL https://arxiv.org/abs/2406.11896. Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, Thomas Unterthiner, Daniel Keysers, Skanda Koppula, Fangyu Liu, ...

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.