Pith. sign in

REVIEW 4 major objections 7 minor 48 references

FocusMem: Factorizing Content, Readout, and Trust in Latent GUI Memory

T0 review · 4 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read FocusMem claims that separating what latent GUI memory retains, how it is read at each decision, and whether it is trusted adds 10.8–18.0 points of task success over a fixed latent-memory baseline on five benchmarks, with the GUI policy…

desk verdict Solid factorization framing and careful diagnostics, but the component-level gains rest on single-rollout numbers that are within noise. read the letter →

arxiv 2608.04530 v1 pith:B4NTLP2L submitted 2026-08-05 cs.CV

Zhuoran Zhang , Bowen Li , Jingcheng Ju , Yang Shi , Qixun Wang , Haotian Wang , Wei Chen , Tengjiao Wang This is my paper
classification cs.CV
keywords latentmemoryGUIagentsepisodicworkingstate-conditionedreadouttrustgatefrozenpolicyQ-Former
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

FocusMem claims that trajectory-based latent memory for GUI agents fails not because the latent bottleneck is too small, but because one fixed memory block is asked to serve three different jobs at once: keeping the right content, exposing the part that matters at the current decision step, and deciding whether the exposed evidence should influence the action at all. The paper separates those jobs inside a compact latent interface, training a role-aware content basis, a state-conditioned readout, and a block-level trust gate while the GUI policy itself stays frozen. On five web benchmarks, FocusMem reports +10.8, +13.5, +18.0, +17.0, and +13.9 points over a fully matched action-only fixed-memory baseline, reaching 48.3 overall versus 33.7. A cumulative ablation attributes the gain to all three components, with the content basis, dynamic readout, and trust gate each adding roughly 4–5 points. The point is that effective latent memory is about what is retained, what is exposed, and what is allowed, not just how much is compressed.

What carries the argument

The central object is a factorized K-token latent memory block built on a Q-Former-style compressor shared by episodic and working memory. Stage A establishes role-specific base queries for each role and trains them with a semantic KL term, which transfers broadly verbalizable trajectory content from the frozen policy, plus role-specific functional supervision, which emphasizes reusable decisions for episodic items and state changes and progress for working items. Stage B adds a zero-initialized state-item adapter that generates a residual over the base queries from the pooled current decision state and the encoded item, so the same evidence yields a decision-specific block, and a block-level trust gate that scores each dynamic block and prunes it at inference using a validation-selected threshold, trained with straight-through HardConcrete masks and injected irrelevant negatives. This machinery does the work of separating what is retained, what is exposed, and what is allowed into the policy without ever updating the policy parameters.

What would settle it

Run each configuration in the cumulative ablation (No Memory, +Fixed, +Content, +Readout, +Gate) on a fixed benchmark with at least ten independent rollouts per configuration under varied seeds, and check whether the reported +5.0, +4.0, and +4.5 point increments survive with non-overlapping confidence intervals; alternatively, re-judge the same logged trajectories with a different judge prompt and verify that the ranking of variants is preserved.

Watch

Extended reading notes

Core claim

FocusMem's central claim is that latent GUI memory should factorize into three learned responsibilities instead of compressing each trajectory into one state-independent block. Each episodic or working item first learns a role-specific content basis: shared queries that are trained, through semantic KL distillation from the frozen policy and role-specific functional targets, to keep reusable experience or current progress recoverable after compression. At decision time, a lightweight state-item adapter writes a residual over those base queries, so the same stored evidence produces different policy-facing blocks for different decision states. A separate block-level trust gate scores each dynamic block and drops those below a learned threshold before they enter the frozen policy's attention. With the policy held frozen, FocusMem outperforms the matched Action-only Fixed baseline on every benchmark by 10.8, 13.5, 18.0, 17.0, and 13.9 points, and the cumulative ablation of content basis, dynamic readout, and gate yields successive gains of +5.0, +4.0, and +4.5 points. Diagnostics show that semantic and functional supervision preserve complementary information, that the dynamic readout degrades less than a fixed readout when surrounding trajectory context grows (6–12 point drops versus 22–25 under oracle-core expansion), and that the trust gate suppresses injected irrelevant episodic evidence, with episodic passage falling from 86% to 35% under maximum contamination.

Load-bearing premise

Every headline number comes from a single interactive run per configuration, judged by an automated LLM, with no repeated seeds or confidence intervals, so the component-level gains could partly reflect run-to-run or judge noise.

Editorial extensions

If this is right

  • Latent memory can be improved substantially without retraining the GUI policy: the memory pathway alone adds nearly 20 points over the no-memory policy and about 15 points over inserting full raw trajectories, while using at most 48 latent tokens before gating.
  • State-conditioned readout specifically pays off when relevant evidence must compete with surrounding trajectory context: under oracle-core expansion the dynamic readout loses 6–12 accuracy points versus 22–25 for the fixed readout across latent budgets of 4, 8, and 16 tokens.
  • A trust gate that drops apparently irrelevant blocks protects success under contaminated retrieval, keeping full FocusMem 9–11 points stronger than its no-gate variant at maximum episodic contamination.
  • Semantic and functional content supervision are complementary: combining them gives the best recoverability on every content-probe metric for both episodic and working memory, while action-only training leaves the weakest recoverable basis.
  • Each factorized component contributes independently: removing the content basis, the dynamic readout, or the gate from the full model removes a separate 4–5 point increment on the MMInA-Shopping ablation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The factorization is generic beyond web GUI agents: any agent that conditions on retrieved experience could separate retention targets, state-dependent readout, and admission control without retraining its policy, but the paper's web-only evaluation leaves that transfer untested.
  • The evidence-expansion result implies the dynamic readout's advantage should grow with horizon or context pressure, which is directly testable by scaling episode length and measuring the fixed-versus-dynamic gap.
  • Because every headline number comes from a single interactive rollout per configuration, an inexpensive decisive check is to re-run the five-step ablation with multiple seeds and judge instances and verify that the monotonic +4 to +5 point increments survive.
  • A trust gate trained only against injected, obviously irrelevant negatives may behave differently on subtler mismatches or stale working memory, which the paper itself lists as an open limitation; a harder test would inject plausible but outdated evidence.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper proposes FocusMem, a latent-memory interface for frozen GUI policies that factorizes memory into three trainable components: a role-aware content basis that separates episodic and working-memory content, a state-conditioned readout that generates decision-specific latent blocks, and a block-level trust gate that can suppress irrelevant evidence. The method is evaluated on five GUI benchmarks with a frozen Qwen3-VL-8B policy, reporting consistent SR improvements over a matched action-only fixed-memory baseline and prior latent-memory adaptations, together with targeted diagnostics on content recoverability, evidence expansion, and contamination. The writing is clear, and the experimental protocol is unusually careful in its matching and disclosure.

Significance. If the reported results hold, the paper offers a valuable and intuitive decomposition of latent GUI memory, and the diagnostic methodology (oracle-centered expansion, evidence-dependence interventions, contamination analysis) is a useful template for the field. The manuscript deserves credit for matched variants, three-level leakage prevention, method-blind judges, explicit disclosure of the single-run protocol, and a public code repository. The central empirical claim, however—that each of the three factorized components adds nontrivial value—rests on unreplicated increments that are comparable to the expected noise, and the cumulative ablation starts from a baseline that is weaker than a published prior in the same class.

major comments (4)
  1. [§D.4; Tables 1–3; Fig. 4] The run protocol in Supplementary §D.4 states that every per-benchmark SR and every diagnostic score in Tables 1–3 and Figs. 3–4 comes from a single interactive rollout, with no seed variation or confidence intervals. In the cumulative ablation of Table 2 (N=200 per row), the successive +5.0, +4.0, and +4.5 point increments are within roughly 1.2–1.5 binomial standard errors, so the claim that Content Basis, Dynamic Readout, and Evidence Gate each add non-trivial value is not established at the reported magnitude without replication or a paired-error analysis. This issue is load-bearing because the paper's central contribution is the factorization itself.
  2. [§4.3, Table 2] The cumulative ablation starts from Action-only Fixed at 45.5 SR on MMInA-Shopping, which is 9 points below the Mem-W-style dual-fixed-latent baseline (54.5) on the same benchmark. Because the factorized components are added to this unusually weak starting point, the reported +13.5 point total gain over Action-only Fixed may overstate the improvement attributable to the factorization relative to a competitive fixed-latent memory. The paper should report an ablation that also builds on the stronger baseline, or at least discuss why the matched baseline is so much weaker than Mem-W-style despite being in the same dual-fixed-latent class.
  3. [§4.4, Table 3; Eq. (7)] The Aligned content-recovery metric in Table 3 is partially circular: the Aligned probe templates (reusable procedure and anchor decision for episodic; latest state and observed change for working) are the same functional categories that the role-specific functional loss L_func in Eq. (7) is explicitly trained to preserve. Higher Aligned scores under the 'Both' condition are therefore partly a direct check on the training target rather than an independent measure of recoverability. The Transfer metric is a genuine held-out check, and the paper should rely on Transfer for the recoverability claim and reframe Aligned as a training-target sanity check.
  4. [§4.6, Fig. 4(b)–(c)] In the contamination diagnostic, the gate passage rate in Fig. 4(c) is pooled over unreplaced and injected episodic candidates, so the drop from 86% to 35% does not separate the gate's rejection of injected negatives from its behavior on ordinary retrieved blocks; reporting passage separately for the two candidate classes would isolate the trust mechanism. Additionally, the 9–11 point SR gap at maximum contamination in Fig. 4(b) is from a single rollout on 100 tasks, so it inherits the replication concern of the first major comment.
minor comments (7)
  1. [Abstract] The abstract contains a missing space: 'We introduceFocusMem' should read 'We introduce FocusMem'.
  2. [§4.1 / §D.4] The single-rollout protocol should be stated in the main text's experimental setup, not only in the supplementary material; readers of Table 2 need to know immediately that the ablation rows are unreplicated.
  3. [Table 2 caption] The caption 'Hit-Max is the 15-step-limit rate' is ambiguous; it should say 'fraction of tasks that reach the 15-step cap (lower is better), matching the definition in Supplementary Table 7.'
  4. [Throughout] The benchmark name 'WebVoyager' appears as 'WebV oyager' in many places (Table 1, §4.6, Fig. 4); this appears to be a rendering artifact and should be fixed.
  5. [Table 1] The labels 'MMInA-Shop' and 'MMInA-Shopping' are used inconsistently; one consistent benchmark name should be chosen.
  6. [Figure 2] The figure labels the trust gate 'Stage B: Evidence Trust' while the readout block is also 'Stage B: Conditional Readout'; the stage labels should be synchronized with the text in §3.3 and §3.4.
  7. [Eq. (6)] The semantic-distillation branch shares the same frozen policy as teacher and student-conditioned decoder; the paper should note that this is a self-distillation setup and discuss any risk that the teacher distribution is uninformative.

Circularity Check

1 steps flagged · score 3.0 of 10

No load-bearing circularity: the main SR comparisons are matched empirical evaluations, and the only localized overlap is the disclosed Aligned content-probe metric, which mirrors the functional-supervision training categories by construction.

  1. self definitional [Supplementary Material Table 8 caption; Section 3.3, Eq. (7) L_func; Section 4.4 Content-Basis Probes.]
    "Episodic targets emphasize reusable decisions, procedures, and constraints, whereas working targets emphasize state changes, progress, and remaining constraints. ... Aligned probes target the same functional categories as training but use held-out wording and response formats."

    Eq. (7) trains the content basis to maximize the frozen policy's likelihood of reconstructing role-specific targets (reusable procedure and anchor decision for episodic; latest state and observed change for working). The Aligned probe metric then asks the same frozen policy to recover exactly those categories from the latent block. The Functional Only gain on Aligned therefore measures, by construction, whether the model can output the categories it was explicitly trained to output, rather than demonstrating an independent property of the factorization. The paper discloses this overlap, and the Transfer probes (subgoals, harmful repetition) are not training targets and improve as well, so the complementarity claim retains independent support.

full rationale

The central results are direct, matched empirical comparisons: FocusMem versus Action-only Fixed shares policy, data, retrieved candidates, budgets, grounding, decoding, and evaluator, so the +10.8-to-+18.0 SR gains are not forced by construction. The component ablation adds real mechanisms (content basis, dynamic readout, gate) and reports monotonic SR gains; this is an empirical claim, not a fitted input renamed as a prediction. The trust threshold gamma=0.3 is selected on a disjoint validation split, so evaluation is not circular. No load-bearing uniqueness theorem or self-citation chain appears; prior latent-memory baselines are adapted from external work and the paper does not rely on its own cited results to justify its architecture. The one localized circular element is the Aligned content-probe metric in Table 3: its categories coincide with the L_func training targets, so the Functional Only improvement on Aligned is partly self-definitional. The paper explicitly acknowledges this ('Aligned probes target the same functional categories as training'), and the Transfer column plus the raw-evidence-grounded judge give non-circular support. The single-rollout protocol is a statistical robustness concern, not a circularity concern, and is honestly disclosed in Section D.4. Overall, the factorization claim does not reduce to its inputs; only a small diagnostic is partially self-referential.

Assumptions & free parameters 6 free parameters · 5 assumptions · 3 invented entities

The central claims rest on one trained interface (role queries, semantic and functional distillation, readout adapter, trust gate) whose main free choices are the latent budget K, retrieval count M, chunk size W, loss weights, trust threshold, and the two-stage schedule. The paper provides matched ablations and diagnostics for each learned component, so the invented entities carry falsifiable handles. The evaluation nevertheless rests on LLM judging and an in-domain episodic bank, which are domain assumptions rather than standard math.

free parameters (6)
  • K latent tokens per memory block = 8 (4, 8, 16 ablated in Fig. 3)
    The memory budget per item, set by hand. The evidence-expansion diagnostic shows Dynamic Readout benefits from larger K, so this choice interacts with the central claims.
  • Top-M retrieved episodic candidates = 3
    Number of retrieved trajectories per step, chosen by hand. The contamination diagnostic replaces 0 to 3 of these slots, so M bounds the gate's exposure.
  • W working-chunk size = 4 events
    Expired history is split into consecutive chunks of 4 events; chunking determines how many working blocks exist and what each block must compress.
  • Trust threshold gamma = 0.3
    Selected on a held-out validation split. Gate passage rate and the contamination results depend on this choice.
  • Loss weights lambda_sem, lambda_func, lambda_basis, lambda_m = 0.5, 0.5, 0.25, 0.05
    Chosen on validation. Content-basis probes and the cumulative ablation are trained under these weights, so they shape the reported component gains.
  • Learning rates and batch size = 1e-5 (Stage A), 5e-6 (Stage B), batch 32
    Standard hyperparameters tuned on validation. Stage B initializes from Stage A, so the two-stage schedule is part of the method.
assumptions (5)
  • domain assumption The frozen Qwen3-VL-8B policy's own next-token distributions and functional QA outputs serve as faithful teacher targets for what a GUI latent memory should retain.
    Invoked in Section 3.3 and A.2 where the raw path and latent path use the same frozen policy head for semantic KL and functional targets. If the policy's self-descriptions are biased, recoverability probes measure self-consistency rather than true recoverability.
  • domain assumption The trajectory retriever (Qwen3-VL-Embedding-2B) surfaces useful experience, and its failures are not corrected by the memory pathway.
    Section 3.2 states the retriever is frozen and not optimized with the memory pathway; the trust gate can only drop what was retrieved, so retrieval quality bounds the whole episodic memory contribution.
  • domain assumption Three-level exclusion (task ID, annotated instance, slot-normalized instruction template) fully prevents overlap between memory sources and evaluation tasks.
    Section C.5 defines the leakage prevention. Residual task-adjacent leakage cannot be ruled out by template matching, and the episodic bank is explicitly in-domain for the evaluation websites.
  • domain assumption LLM judges (Gemini-3.1-Pro) with method-blind, fixed prompts provide valid task-success, probe, and action-validity scores, and the accessibility-screen auditor correctly separates site-imposed obstruction from agent failure.
    Sections D.2 and C.1. All success rates, probe scores, and diagnostic accuracies rest on these judgments, with no reported human sample check or inter-judge agreement.
  • standard math Standard deep-learning machinery (Q-Former cross-attention, LoRA, straight-through HardConcrete) behaves as documented.
    Used throughout Sections A.1 to A.3 for compression, adapters, and the gating estimator. No novel mathematical apparatus is introduced.
invented entities (3)
  • Role-specific latent content basis (role base queries, role embeddings, functional pseudo-targets) independent evidence
    purpose: Controls which information survives compression separately for episodic and working memory.
    Assessed by the content-basis probes in Table 3 and the matched Content ablation step, which serve as falsifiable handles for the design.
  • State-conditioned readout adapter (residual query generator) independent evidence
    purpose: Recomputes each memory block from the current decision state so the same evidence serves different interaction stages.
    Falsifiable via the oracle-centered evidence-expansion experiment in Fig. 3 and the evidence-dependence intervention in Fig. 4(a).
  • Block-level trust gate with validation-selected threshold independent evidence
    purpose: Suppresses memory blocks judged irrelevant to the current decision before they enter the policy.
    Falsifiable via the irrelevant-trajectory contamination diagnostic in Fig. 4(b,c), where gate passage falls from 86% to 35% as contamination rises.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FocusMem: Factorizing Content, Readout, and Trust in Latent GUI Memory." pith.science (2026). https://pith.science/paper/B4NTLP2L

@misc{pith2026260804530,
  author       = {Pith},
  title        = {Pith review of: FocusMem: Factorizing Content, Readout, and Trust in Latent GUI Memory},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/B4NTLP2L}},
  note         = {Machine review of arXiv:2608.04530}
}
read the original abstract

GUI agents must remember both useful experience from earlier tasks and unfinished progress in the current interaction. Latent memory offers a compact solution by compressing multimodal trajectories into a few continuous tokens. Existing methods, however, usually map each trajectory to one fixed memory block and train it mainly through next-action supervision. This creates three practical problems: important details may be lost during compression, the same memory block must serve different decision stages, and irrelevant retrieved trajectories may still mislead the agent. We introduce FocusMem, which separates these responsibilities within a compact latent-memory interface. A role-aware content basis encourages episodic memory to retain reusable experience and working memory to retain task progress. A state-conditioned readout generates a decision-specific view of the same stored evidence, while a lightweight trust gate can suppress memory blocks that appear irrelevant to the current step. All components are trained while the GUI policy remains frozen. Across five GUI-agent benchmarks, FocusMem consistently outperforms a fully matched action-only fixed-memory baseline and prior latent memory adaptations. Further analysis shows that semantic and functional supervision preserve complementary information, state-conditioned readout is more robust as surrounding trajectory context grows, and the trust gate reduces the harm caused by injected irrelevant episodic evidence. These results show that effective latent memory depends not only on compressing past interaction, but also on what is retained, what is exposed, and what is allowed.

Figures

Figures reproduced from arXiv: 2608.04530 by the authors.

Figure 1
Figure 1. Three latent-memory failure modes in one shopping trajectory. Compact memory must preserve useful evidence, expose the [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. FocusMem retrieves complete episodic trajectories, segments expired working history, and encodes each evidence item once. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Oracle-centered evidence expansion on 100 WebVoy [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (19 more)
Figure 4
Figure 4. Figure 4: Evidence-dependence and episodic-contamination diag [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Prompt assembly for functional supervision. The raw teacher produces the supervision target from the original trajectory, and the [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]
Figure 6
Figure 6. Figure 6: Prompt assembly for action imitation. Continuous episodic and working memory blocks precede the ordinary GUI-agent [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: Task-accessibility screening prompt (§C.1), part 1: the auditor’s task and the seven site-imposed BLOCK categories. The not-obstruction rules and output format continue in [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 8
Figure 8. Figure 8: Task-accessibility screening prompt (§C.1), part 2: the not-obstruction (OK) rules and output format. The auditor receives TASK: <intent> followed by each step as the action taken and the resulting screenshot (>12 steps are evenly subsampled, always keeping the first a…
Figure 10
Figure 10. Figure 10: MMInA-Shopping task-seed generation prompt (product-grounded). The user message supplies the sampled real products. [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]
Figure 11
Figure 11. Figure 11: MMInA-Wikipedia task-seed generation prompt: self-contained single-article factual or visual questions. [PITH_FULL_IMAGE:figures/full_fig_p019_11.png]
Figure 12
Figure 12. Figure 12: High-quality scoring rubric. {MODE NOTE} and {IMAGE NOTE} are instantiated for the process pass (actions plus reasoning only) and the full pass (screenshots included for final verification). 20 [PITH_FULL_IMAGE:figures/full_fig_p020_12.png]
Figure 13
Figure 13. Figure 13: Action serialization schema. Only the arguments relevant to the action type are present in any given step. [PITH_FULL_IMAGE:figures/full_fig_p021_13.png]
Figure 14
Figure 14. Figure 14: MMInA fuzzy-match success judge, used for MMInA-Wikipedia and MMInA-Shopping. Text only (no screenshots); the gold [PITH_FULL_IMAGE:figures/full_fig_p023_14.png]
Figure 15
Figure 15. Figure 15: WebVoyager / DeepShop success judge (shared). The final answer is assessed together with the last five trajectory screenshots. [PITH_FULL_IMAGE:figures/full_fig_p024_15.png]
Figure 16
Figure 16. Figure 16: Online-Mind2Web official WebJudge, stages 1–2 (key-point extraction and per-screenshot scoring), reproduced verbatim from [PITH_FULL_IMAGE:figures/full_fig_p025_16.png]
Figure 17
Figure 17. Figure 17: Online-Mind2Web official WebJudge, stage 3 (final verdict), reproduced verbatim from OSU-NLP-Group/Online-Mind2Web [PITH_FULL_IMAGE:figures/full_fig_p025_17.png]
Figure 18
Figure 18. Figure 18: Prompt used by the frozen policy to answer content-basis probes from a fixed latent block. [PITH_FULL_IMAGE:figures/full_fig_p026_18.png]
Figure 19
Figure 19. Figure 19: Method-blind judge used for Semantic, Aligned, and Transfer content-basis probes. [PITH_FULL_IMAGE:figures/full_fig_p027_19.png]
Figure 20
Figure 20. Figure 20: Method-blind one-step action-validity judge used in the oracle-centered expansion and evidence-dependence intervention. [PITH_FULL_IMAGE:figures/full_fig_p028_20.png]
Figure 21
Figure 21. Figure 21: Case 1. Same frozen Qwen3-VL-8B policy under No Memory (top) and FocusMem (bottom). Steps 3–14 of the No-Memory [PITH_FULL_IMAGE:figures/full_fig_p034_21.png]
Figure 22
Figure 22. Figure 22: Case 2. A two-product comparison under No Memory (top) and FocusMem (bottom). Green badges trace how FocusMem [PITH_FULL_IMAGE:figures/full_fig_p035_22.png]
Figure 23
Figure 23. Figure 23: Case 3. Matched checkpoints trained under identical settings except for the readout design. At step 4 both stand on the same [PITH_FULL_IMAGE:figures/full_fig_p036_23.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

48 extracted references · 27 canonical work pages

  1. [1]

    Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025

    Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025. 5

    arXiv 2025

  2. [2]

    Mga: Memory-driven gui agent for observation- centric interaction.arXiv preprint arXiv:2510.24168, 2025

    Weihua Cheng, Ersheng Ni, Wenlong Wang, Yifei Sun, Jun- ming Liu, Wangyu Shen, Yirong Chen, Botian Shi, and Ding Wang. Mga: Memory-driven gui agent for observation- centric interaction.arXiv preprint arXiv:2510.24168, 2025. 2

    arXiv 2025

  3. [3]

    Adapting language models to compress con- texts

    Alexis Chevalier, Alexander Wettig, Anirudh Ajith, and Danqi Chen. Adapting language models to compress con- texts. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023. 3

    work page 2023

  4. [4]

    Naive visual memory is not enough: A failure-mode study of gui agents, 2026

    Seoyoung Choi, Minseok Ko, Hyunseok Lee, Kunwoong Kim, Woomin Song, Chanseok Jeon, and Jinwoo Shin. Naive visual memory is not enough: A failure-mode study of gui agents, 2026. 3

    work page 2026

  5. [5]

    InstructBLIP: Towards general- purpose vision-language models with instruction tuning

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstructBLIP: Towards general- purpose vision-language models with instruction tuning. arXiv preprint arXiv:2305.06500, 2023. 3

    arXiv 2023

  6. [6]

    Memp: Exploring agent procedural memory

    Runnan Fang, Yuan Liang, Xiaobin Wang, Jialong Wu, Shuofei Qiao, Pengjun Xie, Fei Huang, Huajun Chen, and Ningyu Zhang. Memp: Exploring agent procedural memory. InFindings of the Association for Computational Linguis- tics: ACL 2026, pages 17490–17502, 2026. 1

    work page 2026

  7. [7]

    In-context autoencoder for context compression in a large language model

    Tao Ge, Jing Hu, Haixun Wang, Si-Qing Chen, and Furu Wei. In-context autoencoder for context compression in a large language model. InICLR 2024, 2023. 3

    work page 2024

  8. [8]

    Navigating the digital world as humans do: Universal visual grounding for gui agents.arXiv preprint arXiv:2410.05243, 2024

    Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. Navigating the digital world as humans do: Universal visual grounding for gui agents.arXiv preprint arXiv:2410.05243, 2024. 5

    arXiv 2024

Show all 48 references
  1. [9]

    Molmoweb: Open visual web agent and open data for the open web.arXiv preprint arXiv:2604.08516, 2026

    Tanmay Gupta, Piper Wolters, Zixian Ma, Peter Sushko, Rock Yuren Pang, Diego Llanes, Yue Yang, Taira Ander- son, Boyuan Zheng, Zhongzheng Ren, Harsh Trivedi, Tay- lor Blanton, Caleb Ouellette, Winson Han, Ali Farhadi, and Ranjay Krishna. Molmoweb: Open visual web agent and ope...

    2026 arXiv

  2. [10]

    We- bvoyager: Building an end-to-end web agent with large mul- timodal models.arXiv preprint arXiv:2401.13919, 2024

    Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. We- bvoyager: Building an end-to-end web agent with large mul- timodal models.arXiv preprint arXiv:2401.13919, 2024. 1

    2024 arXiv

  3. [11]

    Hiagent: Hierarchical working memory management for solving long-horizon agent tasks with large language model

    Mengkang Hu, Tianxing Chen, Qiguang Chen, Yao Mu, Wenqi Shao, and Ping Luo. Hiagent: Hierarchical working memory management for solving long-horizon agent tasks with large language model. InProceedings of the 63rd An- nual Meeting of the Association for Computational Linguis- ...

    2025

  4. [12]

    Memory in the age of ai agents

    Yuyang Hu, Shichun Liu, Yanwei Yue, Guibin Zhang, Boyang Liu, Fangyi Zhu, Jiahang Lin, Honglin Guo, Shi- han Dou, Zhiheng Xi, et al. Memory in the age of ai agents. arXiv preprint arXiv:2512.13564, 2025. 1

    2025 arXiv

  5. [13]

    Echotrail-gui: Build- ing actionable memory for gui agents via critic-guided self- exploration.arXiv preprint arXiv:2512.19396, 2025

    Runze Li, Yuwen Zhai, Bo Xu, LiWu Xu, Nian Shi, Wei Zhang, Ran Lin, and Liang Wang. Echotrail-gui: Build- ing actionable memory for gui agents via critic-guided self- exploration.arXiv preprint arXiv:2512.19396, 2025. 1

    2025 arXiv

  6. [14]

    What memory do gui agents really need? from passive records to active task-driving states, 2026

    Chen Liu, Ling Chen, Hanzhang Zhou, Xu Zhang, Quyu Kong, Panrong Tong, Wenhao Wang, Xin Yu, Steven Hoi, and Yue Wang. What memory do gui agents really need? from passive records to active task-driving states, 2026. 3

    2026

  7. [15]

    Lost in the middle: How language models use long contexts.arXiv preprint arXiv:2307.03172, 2023

    Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts.arXiv preprint arXiv:2307.03172, 2023. 1

    2023 arXiv

  8. [16]

    Deepshop: A bench- mark for deep research shopping agents.arXiv preprint arXiv:2506.02839, 2025

    Yougang Lyu, Xiaoyu Zhang, Lingyong Yan, Maarten de Ri- jke, Zhaochun Ren, and Xiuying Chen. Deepshop: A bench- mark for deep research shopping agents.arXiv preprint arXiv:2506.02839, 2025. 5

    2025 arXiv

  9. [17]

    Learning to compress prompts with gist tokens.Advances in Neural Information Processing Systems, 36, 2023

    Jesse Mu, Xiang Lisa Li, and Noah Goodman. Learning to compress prompts with gist tokens.Advances in Neural Information Processing Systems, 36, 2023. 3

    2023

  10. [18]

    Gui agents: A survey

    Dang Nguyen, Jian Chen, Yu Wang, Gang Wu, Namyong Park, Zhengmian Hu, Hanjia Lyu, Junda Wu, Ryan Aponte, Yu Xia, et al. Gui agents: A survey. InFindings of the As- sociation for Computational Linguistics: ACL 2025, pages 22522–22538, 2025. 1

    2025

  11. [19]

    Le, Samira Daruki, Xi- angru Tang, Vishy Tirumalashetty, George Lee, Mahsan Ro- fouei, Hangfei Lin, Jiawei Han, Chen-Yu Lee, and Tomas Pfister

    Siru Ouyang, Jun Yan, I-Hung Hsu, Yanfei Chen, Ke Jiang, Zifeng Wang, Rujun Han, Long T. Le, Samira Daruki, Xi- angru Tang, Vishy Tirumalashetty, George Lee, Mahsan Ro- fouei, Hangfei Lin, Jiawei Han, Chen-Yu Lee, and Tomas Pfister. Reasoningbank: Scaling agent self-evolving w...

    2025 arXiv

  12. [20]

    Executable agentic memory for gui agent, 2026

    Zerui Qin, Sheng Yue, Xingyuan Hua, Yongjian Fu, and Ju Ren. Executable agentic memory for gui agent, 2026. 2

    2026

  13. [21]

    Dual latent memory in vision-language- action models for robotic manipulation.arXiv preprint arXiv:2607.07608, 2026

    Hongyu Qu, Jianzhe Gao, Xiaobin Hu, Shaohuan Yang, Xinlei Yu, Rui Yan, Wenguan Wang, Xiangbo Shu, and Shuicheng Yan. Dual latent memory in vision-language- action models for robotic manipulation.arXiv preprint arXiv:2607.07608, 2026. 3

    2026 arXiv

  14. [22]

    Memoryvla: Perceptual-cognitive memory in vision-language-action models for robotic manipulation

    Hao Shi, Bin Xie, Yingfei Liu, Lin Sun, Fengrong Liu, Tiancai Wang, Erjin Zhou, Haoqiang Fan, Xiangyu Zhang, and Gao Huang. Memoryvla: Perceptual-cognitive memory in vision-language-action models for robotic manipulation. arXiv preprint arXiv:2508.19236, 2025. 2

    2025 arXiv

  15. [23]

    Androtmem: From interaction trajecto- ries to anchored memory in long-horizon gui agents.arXiv preprint arXiv:2603.18429, 2026

    Yibo Shi, Jungang Li, Linghao Zhang, Zihao Dongfang, Biao Wu, Sicheng Tao, Yibo Yan, Chenxi Qin, Weiting Liu, Zhixin Lin, et al. Androtmem: From interaction trajecto- ries to anchored memory in long-horizon gui agents.arXiv preprint arXiv:2603.18429, 2026. 1

    2026

  16. [24]

    Mmina: Benchmarking multihop multimodal internet agents

    Shulin Tian, Ziniu Zhang, Liang-Yu Chen, and Ziwei Liu. Mmina: Benchmarking multihop multimodal internet agents. InFindings of the Association for Computational Linguistics: ACL 2025, pages 13682–13697, 2025. 1

    2025

  17. [25]

    Stamp: Training explicit memory for mobile gui agents in controllable and scalable virtual environments, 2026

    Junyang Wang, Haiyang Xu, Xi Zhang, Zhaoqing Zhu, Ming Yan, Jieping Ye, and Jitao Sang. Stamp: Training explicit memory for mobile gui agents in controllable and scalable virtual environments, 2026. 3 9

    2026

  18. [26]

    History-aware rea- soning for gui agents, 2025

    Ziwei Wang, Leyang Yang, Xiaoxuan Tang, Sheng Zhou, Dajun Chen, Wei Jiang, and Yong Li. History-aware rea- soning for gui agents, 2025. 2

    2025

  19. [27]

    Agent workflow memory.arXiv preprint arXiv:2409.07429, 2024

    Zora Zhiruo Wang, Jiayuan Mao, Daniel Fried, and Gra- ham Neubig. Agent workflow memory.arXiv preprint arXiv:2409.07429, 2024. 1, 2

    2024 arXiv

  20. [28]

    Auto-scaling continu- ous memory for gui agent.arXiv preprint arXiv:2510.09038,

    Wenyi Wu, Kun Zhou, Ruoxin Yuan, Vivian Yu, Stephen Wang, Zhiting Hu, and Biwei Huang. Auto-scaling continu- ous memory for gui agent.arXiv preprint arXiv:2510.09038,

  21. [29]

    From human memory to ai memory: A survey on memory mechanisms in the era of llms.arXiv preprint arXiv:2504.15965, 2025

    Yaxiong Wu, Sheng Liang, Chen Zhang, Yichao Wang, Yongyue Zhang, Huifeng Guo, Ruiming Tang, and Yong Liu. From human memory to ai memory: A survey on memory mechanisms in the era of llms.arXiv preprint arXiv:2504.15965, 2025. 1

    2025 arXiv

  22. [30]

    An illusion of progress? assessing the current state of web agents.arXiv preprint arXiv:2504.01382, 2025

    Tianci Xue, Weijian Qi, Tianneng Shi, Chan Hee Song, Boyu Gou, Dawn Song, Huan Sun, and Yu Su. An illusion of progress? assessing the current state of web agents.arXiv preprint arXiv:2504.01382, 2025. 5

    2025

  23. [31]

    Vismem: Latent vision memory unlocks potential of vision-language models

    Xinlei Yu, Chengming Xu, Guibin Zhang, Zhangquan Chen, Yudong Zhang, Yongbo He, Peng-Tao Jiang, Jiangning Zhang, Xiaobin Hu, and Shuicheng Yan. Vismem: Latent vision memory unlocks potential of vision-language models. arXiv preprint arXiv:2511.11007, 2025. 3

    2025

  24. [32]

    Mementogui: Learning agentic mul- timodal memory control for long-horizon gui agents.arXiv preprint arXiv:2605.18652, 2026

    Ziyun Zeng, Hang Hua, Bocheng Zou, Mu Cai, Rogerio Feris, and Jiebo Luo. Mementogui: Learning agentic mul- timodal memory control for long-horizon gui agents.arXiv preprint arXiv:2605.18652, 2026. 1, 2

    2026 arXiv

  25. [33]

    Memgen: Weaving generative latent memory for self-evolving agents

    Guibin Zhang, Muxin Fu, and Shuicheng Yan. Memgen: Weaving generative latent memory for self-evolving agents. arXiv preprint arXiv:2509.24704, 2025. 3

    2025

  26. [34]

    Mem-w: Latent memory-native gui agents

    Guibin Zhang, Yaohui Ling, Fanci Meng, Kun Wang, and Shuicheng Yan. Mem-w: Latent memory-native gui agents. arXiv preprint arXiv:2605.09317, 2026. 1, 2, 6

    2026 arXiv

  27. [35]

    A survey on the memory mechanism of large language model- based agents.ACM Transactions on Information Systems, 43 (6):1–47, 2025

    Zeyu Zhang, Quanyu Dai, Xiaohe Bo, Chen Ma, Rui Li, Xu Chen, Jieming Zhu, Zhenhua Dong, and Ji-Rong Wen. A survey on the memory mechanism of large language model- based agents.ACM Transactions on Information Systems, 43 (6):1–47, 2025. 1

    2025

  28. [36]

    Expel: Llm agents are ex- periential learners

    Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. Expel: Llm agents are ex- periential learners. InProceedings of the AAAI Conference on Artificial Intelligence, pages 19632–19642, 2024. 1

    2024

  29. [37]

    progress summary

    Sibo Zhu, Wenyi Wu, Kun Zhou, Stephen Wang, and Bi- wei Huang. Hybrid self-evolving structured memory for gui agents.arXiv preprint arXiv:2603.10291, 2026. 1, 2, 6 10 FocusMem: Factorizing Content, Readout, and Trust in Latent GUI Memory Supplementary Material Contents A Detai...

    2026

  30. [38]

    Semantic similarity to the reference answer

  31. [39]

    Key information overlap

  32. [40]

    I'm sorry, I can't answer that question

    Factual accuracy Note: If the predicted answer is like "I'm sorry, I can't answer that question.", you should,→ return "no". Respond with only "yes" or "no": - "yes": if the predicted answer is correct or equivalent to the reference answer - "no": if the predicted answer is in...

  33. [41]

    Web Task Instruction: a clear, specific directive in natural language detailing the online,→ activity (search, verify, compare, etc.)

  34. [42]

    Result Screenshots: the last few screens showing the result or intermediate state of the,→ web task

  35. [43]

    Result Response: a textual answer produced at the end of browsing. -- You SHOULD NOT make assumptions based on information not present in the screenshots.,→ -- Assess the task instruction against the outcome in the screenshots and the response.,→ -- If the instruction has mult...

  36. [44]

    Read the task description carefully

  37. [45]

    best," "highest,

    Identify and extract **key points** directly stated in the task description. - A **key point** is a critical element, condition, or step explicitly mentioned in the,→ task description. - Do not infer or add any unstated elements. - Words such as "best," "highest," "cheapest," ...

  38. [46]

    Provide a detailed description of the image, including its contents, visible elements, text,→ (if any), and any notable features

  39. [47]

    Carefully examine the image and evaluate whether it contains necessary steps or evidence,→ crucial to task completion

  40. [48]

    best/cheapest/latest

    Provide your response in the format: - **Reasoning**: [explanation, citing specific elements in the image] - **Score**: 1 (no relevant info) ... 5 (clearly displays necessary steps/evidence). 24 Figure 16. Online-Mind2Web official WebJudge, stages 1–2 (key-point extraction and...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.