Pith. sign in

REVIEW 3 major objections 5 minor 76 references

SynWeaver: Website-Prior Task and Trajectory Co-Synthesis for Web Agents

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Training web agents on 822 co-synthesized task–trajectory pairs from a website map beats exploration-based baselines that use 1,000 trajectories, on WebArena and WebVoyager alike.

desk verdict SynWeaver is a solid, clearly-written synthesis pipeline with a genuinely new collaborative refinement step and an honest scaling study; the main caveats are evaluation noise and an unvalidated state-equivalence assumption in the crawler. read the letter →

arxiv 2608.12429 v1 pith:IZYIDN3K submitted 2026-08-12 cs.SE cs.AI

Ruitao Wang , Yuwen Hao , Menglin Yang This is my paper · ORCID
classification cs.SEcs.AI
keywords webagentssyntheticsupervisionwebsitemapUIpriorstask-trajectoryco-synthesiscollaborativerefinementArenaVoyager
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the bottleneck in synthetic training data for web agents is grounding, not quantity. Tasks proposed without website-specific knowledge tend to be hallucinated or infeasible, and refining a task separately from its execution trajectory drifts the supervision away from natural user intent. SynWeaver addresses both problems by crawling a website into a map of functionally distinct states and executable transitions, fine-tuning a UI-aware model on supervision derived from that map, and then co-refining the task and trajectory together whenever execution stalls or contradicts the stated intent. With 822 validated task–trajectory pairs, fewer than the 1,000 trajectories used for each baseline, the trained Qwen3-VL-8B-Instruct agent reaches 19.91 percent success on WebArena and 27.06 percent on WebVoyager; ablations attribute the gains to the map, the website-prior proposer, collaborative refinement, and post-verification. If the claim holds, structured exploration and joint refinement can substitute for brute-force data collection in web agent training.

What carries the argument

The load-bearing object is the website map $G=(S,T)$: nodes are page states judged functionally distinct, and edges are executable interactions with recorded actions. A depth-first crawler builds the map using a state processor whose progressive comparison checks URL equality, then row-wise accessibility-tree similarity after stripping digits, with an optional vision-model fallback, and whose duplicate-trigger detection prunes elements, lists, and forms whose functional signatures or XPaths were already explored. Two assets come out of the map: five formats of UI supervision (page description, page QA, element description, forward transition description, inverse transition description) used to LoRA-tune a UI-aware task proposer, and transition triplets from which tasks are reverse-synthesized. The second mechanism is collaborative refinement: when execution stalls or the page contradicts the task, a stronger teacher either rewrites the task alone or jointly edits task and trajectory prefix by deleting steps, swapping commutable steps, and updating affected reasoning, preserving natural intent while restoring executability. Post-verification then filters or repairs the collected pairs.

What would settle it

Crawl a site whose pages share a URL and near-identical accessibility trees but render different content or actions depending on session state, and check whether progressive state comparison merges them into one node; if it does, tasks synthesized from that node are infeasible on one of the two states, and an agent fine-tuned on SynWeaver's data should fail exactly those tasks while passing single-state variants.

Watch

Extended reading notes

Core claim

The paper's central claim is that high-quality web agent supervision couples task intent with executable trajectories, grounded in an explicit map of the target website, and that this yields stronger in-domain and out-of-domain generalization than exploration-based synthesis. On its own terms, SynWeaver 'consistently outperforms strong synthesis baselines and yields more effective supervision.' The evidence: 822 validated pairs fine-tuned into Qwen3-VL-8B-Instruct give 19.91 percent WebArena success versus 16.81 for the strongest baseline SynthAgent, and 27.06 percent WebVoyager success versus 22.42; InternVL3-8B shows the same ordering (14.16 against 12.83). Ablations locate the sources of the gain: replacing the map with a random walk costs 4.42 points, replacing collaborative refinement with decoupled refinement costs 4.87 points, replacing the website-prior proposer with a general-purpose model costs 2.21 points, and dropping post-verification costs 5.31 points. The paper also claims the website prior transfers, since the model trained only on five WebArena websites posts its best score on seven of nine unseen WebVoyager sites.

Load-bearing premise

Everything downstream inherits the crawler's notion of functionally distinct state: if Algorithm 3's state comparison merges two pages that behave differently, or if duplicate-trigger detection prunes an interaction with a distinct effect, the UI prior, the proposed tasks, and the trajectory supervision all inherit that blind spot.

Editorial extensions

If this is right

  • Data efficiency: 822 validated task–trajectory pairs beat 1,000-trajectory baselines, and the advantage persists when the baselines are trained on their full released datasets (NNetNav 17.26 and SynthAgent 18.58 at full scale, versus SynWeaver 19.91).
  • Out-of-domain transfer: UI priors and trajectories learned on five self-hosted WebArena websites carry over to unseen live websites, with the best score on seven of nine WebVoyager sites.
  • Scaling inside a map: resampling transitions from the same website map continues to raise success rates (14.16 to 19.91 to 20.80 to 23.89 as the trajectory budget grows from 0.5× to 3.12×), so the map is a reusable synthesis asset rather than a one-shot crawl.
  • Task quality: the map-grounded UI-aware proposer yields tasks that need fewer refinements and convert into higher downstream success than tasks from a stronger general-purpose proposer, and the UI prior lifts a held-out UI-knowledge test by 14.5 points.
  • Synthesis cost: collaborative refinement retains 99.52 percent of collected trajectories instead of discarding them, lowering the estimated per-trajectory cost below both the decoupled variant and the SynthAgent reference.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication the paper leaves implicit: the map-then-co-refine pipeline is not web-specific, since any GUI with a stable DOM (desktop applications, mobile interfaces, IDEs) could be crawled into a functional map, and the five UI-supervision formats would transfer with little change.
  • Testable extension: the map could serve as a coverage certificate, reporting what fraction of map transitions a synthesized dataset or an agent's rollouts actually exercise; the paper reports state and transition counts but never checks coverage of the generated trajectories against the map.
  • The hierarchical teacher design (a cheap teacher for routine synthesis, a strong teacher only for repairs) is reported as a cost figure but not analyzed; a natural experiment is to lower the strong teacher's capability until the co-refinement loop begins producing repairs that preserve the original errors.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces SynWeaver, a three-stage framework for synthesizing web-agent training data. Stage 1 performs DFS-based website exploration to construct a 'website map' of functionally distinct states and transitions, using progressive state comparison and duplicate-trigger detection to prune redundant exploration. Stage 2 derives page-level and transition-level UI supervision from this map and fine-tunes the target backbone with LoRA to obtain a UI-aware model. Stage 3 performs collaborative task-trajectory synthesis: reverse task synthesis from map transitions, online execution with joint refinement of task and trajectory when they become inconsistent, and post-verification with reconstruction of failing trajectories. On a filtered WebArena set (226 tasks) and a filtered WebVoyager set (388 tasks), SynWeaver with Qwen3-VL-8B-Instruct reaches 19.91% and 27.06% success rate, outperforming NNetNav, OS-Genesis, and SynthAgent. Ablations show contributions from the website map, website prior, collaborative refinement, and post-verification, and a scaling study shows monotonic aggregate gains with more synthesized trajectories.

Significance. If the claims hold, SynWeaver is a valuable contribution to web-agent data synthesis: it places explicit website-specific knowledge before task synthesis, introduces a joint refinement mechanism for task-trajectory consistency, and provides a relatively complete pipeline with public code. The paper's strengths include clearly described algorithms, consistent component-level ablations, a held-out UI knowledge test, and a data-scaling study. The central empirical claim, however, rests on the validity of the website map and on evaluations that lack statistical reliability measures. The most serious risk is that the state-equivalence rule may silently omit subtrees reachable only through menus, popups, or hover interactions, which would propagate to all downstream supervision. The paper is publishable in principle, but the coverage concern and the absence of error bars or significance tests need to be addressed before the performance claims can be accepted at face value.

major comments (3)
  1. [§3.1, Algorithm 3, Prompt 3] The state-equivalence rule used to build the website map treats dropdown-menu open/closed states, hover/focus states, and popup hints or tooltips as acceptable differences (Prompt 3), and Algorithm 3 returns 'same' whenever the URL is identical and the accessibility-tree similarity passes the tolerance. Since Algorithm 1 recurses only when StateEquiv returns false (lines 31-35), any subtree reachable only after expanding a collapsed menu or hovering over a submenu is never added to G. The UI prior, reverse task synthesis, and collaborative trajectory synthesis are all derived from G, so this coverage blind spot propagates to the entire training set. The manuscript reports map sizes and a scaling study but provides no precision/recall audit of StateEquiv decisions and no check that G contains states reachable via menus, popups, or hover interactions. I ask for such an audit (e.g., compare G against a manually annotated interaction inventory for at least one WebArena site, or measure how many evaluation tasks have their target state or a required intermediate state reachable in G) and, if the blind spot is confirmed, a revised equivalence rule that treats newly revealed actionable elements as a new state.
  2. [§4.2, Tables 2–4] All success rates are reported as point estimates with no error bars, confidence intervals, or significance tests, and the text does not indicate that evaluation was repeated over multiple runs or seeds. Given only 226 WebArena evaluation tasks and per-site counts of roughly 34–61 tasks, several headline differences (e.g., 19.91 vs 16.81 overall in Table 2, or CMS 14.04 vs 15.79) are within plausible sampling noise. The WebVoyager per-site cells are even smaller. Since the central claim is that SynWeaver 'consistently outperforms' strong baselines, the paper should provide repeated evaluation runs or bootstrap confidence intervals, and ideally per-site intervals, so that the reader can distinguish genuine gains from noise.
  3. [§4.1, Appendix D.2/D.3] The comparison is defined on filtered benchmark subsets and filtered maps, which makes it difficult to separate method quality from exploration coverage. WebVoyager drops 5 of 15 sites and 255 of 643 tasks; WebArena keeps only 226 of 812 tasks; and the CMS and GitLab maps are aggressively filtered (646→169 states and 606→203 transitions after capping and hop sampling). A random-walk baseline can still reach pages that the filtered map omits, so the difference between Ours and −Map in Table 4 could reflect map completeness rather than co-synthesis quality. Please report sensitivity of the main results to the filtering choices (e.g., minimally filtered or unfiltered WebArena evaluation, and unfiltered map synthesis for at least one site), or at minimum quantify how many evaluation tasks require a page or interaction that is absent from G.
minor comments (5)
  1. [Abstract] The abstract reads 'reaches 27.06 success rate'; this should be '27.06%' for consistency with the tables.
  2. [Figure 1] Figure 1 is very dense, and the labels for the website map, state processor, and collaborative refinement stages are difficult to read at normal zoom; a higher-resolution version or a simplified schematic would help.
  3. [Table 6] The cost comparison in Table 6 is not apples-to-apples because SynthAgent uses GPT-4.1 while DR and CR use Gemini-3-Flash plus Gemini-3.1-Pro; please state this caveat in the text and, if possible, report API prices using comparable model tiers.
  4. [Appendix B] The trajectory validity check refers to a predefined step budget B but does not state its value; please give the concrete budget used in the experiments.
  5. [Appendix D.3] The original map construction uses Qwen3-VL-235B-A22B-Instruct for representative-item sampling and Qwen3-VL-Plus as fallback for state equivalence, but these models are not listed in the main model description; a one-sentence clarification in §3.1 or §4.1 would improve reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the headline results are empirical evaluations against independent benchmarks, not derivations from fitted inputs or self-citations.

full rationale

SynWeaver's central claims are empirical: synthesized task-trajectory supervision is fine-tuned and evaluated on WebArena and WebVoyager official task protocols, with baselines trained under a matched two-stage LoRA setup. The website map is constructed by an independent DFS crawler; the UI-prior model is trained on teacher-generated descriptions of map states and transitions; task-trajectory pairs are produced by online execution and filtered by deterministic heuristics plus teacher reconstruction. No parameter is fitted to the evaluation tasks, and no held-out metric is constructed from the training objective; the scaling experiment explicitly declines to fit a parametric law. The paper contains no load-bearing self-citations: SynthAgent and OS-Genesis are prior baselines, not warrants for SynWeaver's own claims. The state-equivalence rule in Algorithm 3 and the possibility that proprietary teachers already know benchmark websites are real threats to coverage and external validity, but they are correctness and contamination concerns, not cases where a predicted result is equal to its input by construction. The Limitations section similarly acknowledges CAPTCHA/anti-bot fragility and teacher-bound performance; these are scope limits, not circular derivations. Accordingly, no circular step can be exhibited.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The method introduces no fitted physical constants. The hand-chosen thresholds in state equivalence and heuristic verification are listed as free parameters because they influence which states and trajectories are retained, and therefore the final results. No invented entities are postulated.

free parameters (4)
  • a11y_tolerance
    Accessibility-tree similarity threshold in Algorithm 3; hand-chosen and not reported, controls state merging in the website map.
  • loop coverage threshold
    40% coverage and repeat count >=4 in the trajectory consistency check (Appendix B); hand-chosen heuristics that determine which trajectories are retained.
  • step_budget B
    Predefined step limit in trajectory validity check (Appendix B); affects which trajectories pass verification.
  • scrolling cap and hop sampling
    One state per URL, one transition per two hops, and 25-scroll cap for CMS and GitLab (Appendix D.3); hand-chosen to balance the UI dataset.
assumptions (4)
  • domain assumption DFS with LLM-guided state equivalence produces a functionally complete website map.
    Section 3.1 and Algorithm 3 assume that URL, accessibility-tree similarity, and visual fallback correctly identify functionally distinct states; if the map misses interactions, downstream task diversity and grounding degrade.
  • domain assumption Gemini-3-Flash and Gemini-3.1-Pro provide reliable supervision for UI data, task proposals, and trajectory refinement.
    Sections 3.2 and 3.3 rely on proprietary teachers for all labels and refinements; their errors propagate into the training data.
  • domain assumption Heuristic verification rules are sufficient to guarantee executable, semantically aligned supervision.
    Section 3.3 and Appendix B assume that termination effectiveness, trajectory validity, and consistency checks filter out unusable trajectories; a false accept would degrade training data.
  • domain assumption Matched two-stage LoRA training isolates the effect of the synthesis method.
    Section 4.1 assumes that using SynWeaver's UI data to initialize all models does not interact unfairly with baseline trajectories.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SynWeaver: Website-Prior Task and Trajectory Co-Synthesis for Web Agents." pith.science (2026). https://pith.science/paper/IZYIDN3K

@misc{pith2026260812429,
  author       = {Pith},
  title        = {Pith review of: SynWeaver: Website-Prior Task and Trajectory Co-Synthesis for Web Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IZYIDN3K}},
  note         = {Machine review of arXiv:2608.12429}
}
read the original abstract

Web agents often struggle to generalize to unseen websites because they lack website-specific supervision. Recent exploration-based data synthesis methods reduce manual annotation, but they still face two key limitations: they often fail to cover the full functionality of a website, and without sufficient website prior knowledge, they tend to propose hallucinated tasks, which in turn limits the diversity and efficiency of downstream trajectory synthesis. We present \textbf{SynWeaver}, a website-prior task-trajectory co-synthesis framework designed to address these challenges. SynWeaver first performs structured website exploration and constructs a website map that covers a broad set of functionally distinct page states and executable interactions on the target website. It then derives page-level and transition-level supervision from this map to train a UI-aware model with website-specific priors, enabling more grounded task proposals. Finally, SynWeaver performs collaborative task-trajectory synthesis, jointly updating the task and execution trajectory when they become inconsistent, and then verifies and repairs the collected results to produce executable, semantically aligned supervision. Experiments on WebArena and WebVoyager demonstrate that SynWeaver consistently outperforms strong synthesis baselines and yields more effective supervision for both in-domain and out-of-domain generalization.

Figures

Figures reproduced from arXiv: 2608.12429 by the authors.

Figure 1
Figure 1. Overview of SynWeaver. The framework first constructs a website map, then learns website-specific [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Case study of collaborative trajectory synthesis. When the target review is non-existent, SynWeaver [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Effect of website-prior on task proposal qual [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Performance of SynWeaver across different [PITH_FULL_IMAGE:figures/full_fig_p017_4.png]
Figure 5
Figure 5. Figure 5: Example of page description data in the UI dataset. [PITH_FULL_IMAGE:figures/full_fig_p021_5.png]
Figure 6
Figure 6. Figure 6: Example of page QA data in the UI dataset. [PITH_FULL_IMAGE:figures/full_fig_p022_6.png]
Figure 7
Figure 7. Figure 7: Example of element description data in the UI dataset. [PITH_FULL_IMAGE:figures/full_fig_p023_7.png]
Figure 8
Figure 8. Figure 8: Example of forward transition description data in the UI dataset. [PITH_FULL_IMAGE:figures/full_fig_p024_8.png]
Figure 9
Figure 9. Figure 9: Example of inverse transition description data in the UI dataset. [PITH_FULL_IMAGE:figures/full_fig_p025_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

76 extracted references · 71 canonical work pages

  1. [1]

    A description of what changed

    work page

  2. [2]

    InProceedings of the 33rd ACM International Con- ference on Multimedia, pages 6878–6887

    Pg-agent: An agent powered by page graph. InProceedings of the 33rd ACM International Con- ference on Multimedia, pages 6878–6887. Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. 2023. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36:28091–28114. Apurva Gand...

    arXiv 2023

  3. [3]

    Ziyi Guan, Jason Chun Lok Li, Zhijian Hou, Ping- ping Zhang, Donglai Xu, Yuzhi Zhao, Mengyang Wu, Jinpeng Chen, Thanh-Toan Nguyen, Pengfei Xian, and 1 others

    Websynthesis: World-model-guided mcts for efficient webui-trajectory synthesis.arXiv preprint arXiv:2507.04370. Ziyi Guan, Jason Chun Lok Li, Zhijian Hou, Ping- ping Zhang, Donglai Xu, Yuzhi Zhao, Mengyang Wu, Jinpeng Chen, Thanh-Toan Nguyen, Pengfei Xian, and 1 others. 2025. Kg-rag: Enhancing gui agent decision-making via knowledge graph-driven retrieval...

    arXiv 2025

  4. [4]

    values" array containing the fill value for each field **in order**: ```json {

    **Contextual**: Consider the form's purpose based on URL and accessibility tree ## Output Format Return a JSON object with a "values" array containing the fill value for each field **in order**: ```json { "values": ["value_for_field_1", "value_for_field_2", " value_for_field_3", ...], "reasoning": "Brief explanation of the values chosen" } ``` **Important...

    work page

  5. [5]

    When you believe the task is complete (e.g., you have the answer), use NONE with the final answer in value

    work page

  6. [6]

    Be concise and avoid redundant or risky actions; ensure 29 each action clearly advances the task

    work page

  7. [7]

    25 H Prompts This appendix provides the main prompts used in SynWeaver

    The predicted action including the action and the node (e.g., 'Click on [576] link 'add the blog') Input Output Inverse Transition Description Figure 9: Example of inverse transition description data in the UI dataset. 25 H Prompts This appendix provides the main prompts used in SynWeaver. You are a webpage structure analysis model. Your task is to determ...

    work page

  8. [8]

    **If the list items represent repeated or symmetric functionality**: Only **one** representative item or **one group of items** should be clicked

    work page

Show all 76 references
  1. [9]

    set of options of the same type

    **If the list items represent distinct functionality**: All items must be clicked. ## Classification Criteria ### **1. Treat the items as *functionally identical* ( decision = "single") if:** All or most items exhibit these characteristics: - Text content represents members of...

  2. [10]

    **Page URL**: The current page URL to help understand the context

  3. [11]

    **Accessibility Tree**: A structured representation of the page content around the form

  4. [12]

    The values should be:

    **Form Fields**: A list of input fields in the form, each with: 26 - Index number (1, 2, 3, ...) - Tag type (input/textarea/select) - Input type (for input elements: text, email, password, etc.) - Name attribute - Aria-label - Placeholder text - Current text content ## Your Ta...

  5. [13]

    **Semantically appropriate**: Match the expected content type (email for email fields, phone for phone fields, etc .)

  6. [14]

    **Realistic**: Look like real user input, not obviously fake

  7. [15]

    **Valid**: Pass basic validation (proper email format, valid phone format, etc.)

  8. [17]

    email" placeholder=

    input[type=email] name="email" placeholder="Enter your email"

  9. [18]

    password

    input[type=password] name="password" placeholder=" Password" Output: ```json { "values": ["testuser@example.com", "TestPass123!"], "reasoning": "Standard login form with email and password fields" } ``` ### Example 2: Registration Form Input: - URL: https://shop.example.com/re...

  10. [19]

    first_name

    input[type=text] name="first_name" aria-label="First Name"

  11. [20]

    last_name

    input[type=text] name="last_name" aria-label="Last Name"

  12. [21]

    input[type=email] name="email"

  13. [22]

    phone" placeholder=

    input[type=tel] name="phone" placeholder="Phone number " Output: ```json { "values": ["John", "Smith", "john.smith@example.com", "555-123-4567"], "reasoning": "E-commerce registration form requiring personal contact information" } ``` Prompt 2: Prompt for filling the form cont...

  14. [23]

    **Core Page Structure**: - Same page layout and main content areas - Same navigation elements (header, footer, sidebar) - Same primary content sections

  15. [24]

    **Functional Elements**: - Same interactive elements (buttons, links, forms, input fields) - Same product listings, article content, or data tables - Same menu items and navigation options

  16. [25]

    **Page Identity**: - Same URL path (if visible) - Same page title or heading - Same main purpose/function # What to IGNORE (Acceptable Differences) The following differences should be **ignored** when comparing states:

  17. [26]

    **Transient Notifications**: - Toast messages (success/error/warning notifications) - Temporary alert boxes - Popup hints or tooltips - Tutorial/onboarding overlays - Cookie consent banners

  18. [27]

    2 minutes ago

    **Dynamic Content**: - Timestamps (e.g., "2 minutes ago" vs "5 minutes ago") - Real-time counters (shopping cart quantity, notification badges) - Live data feeds (stock prices, weather updates) - Advertisement content - Personalized recommendations

  19. [28]

    **Visual State Changes**: - Hover effects or focus states - Loading spinners or progress indicators - Scroll position - Animation states - Dropdown menu open/closed states

  20. [29]

    decision

    **Session-Specific Elements**: - Session IDs or tokens in URLs - CSRF tokens - Temporary promotional banners - A/B testing variations (minor UI tweaks) # Decision Criteria - **SAME**: If the core page structure, main content, and functional elements are identical (ignoring the...

  21. [30]

    A screenshot of a web page

  22. [31]

    What is the main heading?

    The page's accessibility tree Your task is to provide TWO types of analysis: ## Part 1: Page Description Provide a comprehensive description of the page that includes: - Visual layout and structure (header, navigation, main content, sidebar, footer) - All visible UI elements a...

  23. [32]

    A detailed description of this page

  24. [33]

    Prompt 4: System prompt and user prompt for page- level UI data generation

    One question-answer pair about specific information visible on the page Remember to respond in the specified JSON format. Prompt 4: System prompt and user prompt for page- level UI data generation. You are an expert UI analyst tasked with analyzing web page interactions. Your ...

  25. [34]

    A "before" screenshot showing the page with a marked element

  26. [35]

    An "after" screenshot showing the page after the action was performed

  27. [36]

    element_description

    Information about the action taken Your task is to provide TWO types of analysis: ## Part 1: Element Description Describe the functionality of the interacted element and how users would typically interact with it in the interface. ## Part 2: State Transition Description Descri...

  28. [37]

    A description of the element's functionality

  29. [38]

    Prompt 5: System prompt and user prompt for transition- level UI data generation

    A description of how the page changed (without mentioning what action was taken) Remember to respond in the specified JSON format. Prompt 5: System prompt and user prompt for transition- level UI data generation. You are a GUI (Graphical User Interface) expert capable of analy...

  30. [39]

    Click on the'Add to Cart'button next to the product to add it to your shopping cart

    Sub-Instruction: Create a natural language instruction for the current action based on the interface changes it caused. The instruction should be concise, clear, and actionable, incorporating specific details critical to the task, such as elements, file names, timestamps, or o...

  31. [40]

    Then, examine key elements in both screenshots and consider possible operations based on these elements

    Analysis: Carefully analyze the before-and-after 28 screenshots step by step, focusing on the changes caused by the action. Then, examine key elements in both screenshots and consider possible operations based on these elements. For example: "The previous screen displayed the ...

  32. [41]

    summarize the information about a product

    High-Level Instruction: Based on the before-and-after screenshots, the action, and the analysis, generate a high- level task that you believe can be completed within the current interface. There are three types of tasks: - Information seeking: The user wants to obtain certain ...

  33. [42]

    {high_level_task}

    High-Level Task (your ultimate goal to finish): "{high_level_task}"

  34. [43]

    History of Actions ({hint_for_history}): {previous_actions}

  35. [44]

    type": "CLICK

    Current Page (only current view, not full page, you may need to scroll to see more): - URL: {url} - Accessibility Tree (Page Context): {page_context} - Screenshot (only current view, not full page): {img_info} **Available Low-Level Actions (exact JSON formats)** Page Operation...

  36. [45]

    You must issue only actions that are valid given the current observation (accessibility tree and screenshot)

  37. [46]

    Only propose ONE atomic action; actions must be executable independently

  38. [47]

    Prefer actions grounded by element IDs present in the accessibility tree when using CLICK/TYPE/HOVER

  39. [48]

    You MUST provide meaningful and non-empty value if the action type is in {{TYPE, SCROLL, GOTO, NONE, STOP, TAB_FOCUS}}

  40. [51]

    Use STOP only when: - The task lacks necessary information - The target does not exist (hallucination) - The task is harmful or inappropriate - Multiple attempts (>=3) have failed to make progress

  41. [52]

    state_observation_summary

    You MUST first generate a "state_observation_summary" to observe the current environment, then take a step-by-step "reasoning" to decide the next action

  42. [53]

    Do NOT expect to finish in a single action

    The high-level task often requires MULTIPLE steps to complete. Do NOT expect to finish in a single action

  43. [54]

    NONE" or

    You MUST actively decide the next step. Do NOT choose " NONE" or "STOP" unless you are sure the task is finished or impossible

  44. [55]

    Choose element IDs from the accessibility tree and use them directly in CLICK/TYPE/HOVER actions

  45. [56]

    If the page does not change after an action, try scrolling to see more elements

  46. [57]

    MM/DD/YYYY

    When typing dates, use the format "MM/DD/YYYY". **Output Requirements** You MUST return a JSON dictionary with the following format (no extra commentary): {{ "state_observation_summary": "your 1-3 sentence summary of the current state relevant to the task", "reasoning": "your ...

  47. [58]

    **Root Cause Analysis**: Why did the agent issue a STOP action? - Task was impossible/invalid? - Task was unclear/ambiguous? - Trajectory led to wrong page/state? - Agent got stuck in a loop?

  48. [59]

    - If NO, proceed to Question 3

    **Can Task-Only Optimization Succeed?** - Does the current page state enable task completion? - Will clarifying or changing the **goal state** guide the agent to success in 2-3 steps? - Are the executed steps (before STOP) generally correct ? - If YES, use Strategy 1 (Task-Onl...

  49. [60]

    **What Trajectory Issues Exist?** - Are there redundant/unnecessary steps? - Are steps in the wrong order? - Did early mistakes prevent success? - If YES, use Strategy 2 (Co-Optimization). 30

  50. [61]

    analysis

    **Coherence Check** (for Strategy 2): - After reordering/deleting, does the trajectory tell a logical story? - Do you need to update reasoning to maintain coherence? - Will the modified trajectory naturally lead to completion in 2-3 steps? --- ## Output Requirements Return a J...

  51. [62]

    The refined task MUST be achievable within 2-3 more steps from the current page state

  52. [63]

    Find the pricing page

    **INTENT over INSTRUCTION**: The refined task must describe the **Goal State** (e.g., "Find the pricing page") , NOT a list of actions (e.g., "Click menu, then click pricing"). 3.`step_order`must only contain integers from 1 to { total_steps} (the STOP step should be removed)

  53. [64]

    Each step number can appear at most once in`step_order `

  54. [65]

    task_only

    If`strategy`is "task_only",`step_order`must be [{ original_order}]

  55. [66]

    co_optimize

    For "co_optimize" strategy, ensure modified trajectory is logically coherent

  56. [67]

    {high_level_task}

    Update`modified_reasonings`for any step whose context changed due to reordering/deletion. **Focus**: Prioritize enabling the agent to complete the task quickly (2-3 steps) over preserving all trajectory steps. Ensure the task remains a high-level goal, not a low -level script....

  57. [68]

    **Understanding the Context**: What was the agent trying to accomplish? What meaningful progress was made before the issues?

  58. [69]

    **Extracting Valid Steps**: Identify which steps represent meaningful progress and should be kept. You may: - **Delete** steps that are redundant, erroneous, or part of a loop - **Reorder** steps if needed for logical coherence - **Update reasoning** for steps whose context ch...

  59. [70]

    If the existing steps accomplish a meaningful sub-goal, rewrite the task to match what was achieved, then indicate we need to append a NONE action

    **Handling Different Issues**: - **MISSING_END**: The trajectory lacks a proper NONE ( completion) action. If the existing steps accomplish a meaningful sub-goal, rewrite the task to match what was achieved, then indicate we need to append a NONE action. - **LOOP_DETECTED**: R...

  60. [71]

    analysis

    **Ensuring Coherence**: The final trajectory must: - Have steps that logically follow each other - Match the (possibly rewritten) task description - End with a completion state (we'll add NONE if needed) --- ## Output Requirements Return a JSON object with EXACTLY this structu...

  61. [72]

    Each step number can appear at most once

  62. [73]

    You can only delete or reorder existing steps, NOT add new ones

  63. [74]

    If`can_reconstruct`is false, explain why in`analysis`

  64. [75]

    The reconstructed trajectory should represent a coherent , completable task

  65. [76]

    navigating to product page

    If the trajectory is too broken to salvage, set` can_reconstruct`to false **Example for LOOP_DETECTED**: If steps 3-4-5-6 form a loop pattern (3-4 repeated), keep only [1, 2, 3, 4] and update reasoning if needed. **Example for Failed Trajectory**: If a 10-step trajectory ended...

  66. [2012]

    list", role=

    Crawling ajax-based web applications through dynamic analysis of user interface state changes. ACM Transactions on the Web (TWEB), 6(1):1–30. Shikhar Murty, Hao Zhu, Dzmitry Bahdanau, and Christopher D Manning. 2024. Nnetnav: Unsu- pervised learning of browser agents through e...

    2024 arXiv

  67. [2025]

    Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov

    Qwen3-vl technical report.arXiv preprint arXiv:2511.21631. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information.Transactions of the associa- tion for computational linguistics, 5:135–146. Jianlyu Chen, Shitao ...

    2017 arXiv

  68. [2026]

    Ali Mesbah, Arie Van Deursen, and Stefan Lenselink

    Specializing llms to low-documented domains with rag: An analysis across models and retrieval depths.SN Computer Science, 7(4):320. Ali Mesbah, Arie Van Deursen, and Stefan Lenselink

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.