REVIEW 5 major objections 7 minor 1 cited by
AutoGUI: Scaling GUI Grounding with Automatic Functionality Annotations from LLMs
T0 review · 5 major / 7 minor · reviewed 2026-08-09 · deepseek-v4-flash
Pith's one-line read AutoGUI shows that an LLM reading a UI's before-and-after state can generate, at scale, functionality labels whose quality matches a trained human annotator, and that these labels teach vision-language models to locate GUI elements much…
desk verdict A genuinely new annotation mechanism for GUI grounding at scale, with broad empirical support; the quality-control evidence needs deepening but the contribution deserves peer review. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the state-difference inference $f = \mathrm{LLM}(p_{\mathrm{anno}}, s_t, s_{t+1})$: an LLM reads the diff between accessibility-tree snapshots taken before and after a simulated interaction and produces a contextual functionality summary. Because naive text diffing makes a stable element look both deleted and added, the pipeline adds custom 'attribute update' and 'repositioning' markers so the LLM sees structural changes accurately. Two filters keep the labels trustworthy: an LLM rejection stage scores whether the observed change is explicit, relevant, and predictable and discards the bottom 30% of samples, and a verification stage keeps a sample only when two different LLMs both score the annotation as fully supported by the evidence. The verified triplets are then templated into paired grounding and captioning tasks with coordinates normalized to $[0, 999]$.
What would settle it
Run the paper's own human-evaluation protocol on an order-of-magnitude larger sample, roughly 1,500 randomly drawn annotations spanning all 916 training domains plus the 2k-sample test split, with two independent trained annotators applying the paper's three criteria (context-specificity, appropriate detail, no hallucination). If measured correctness falls materially below the 95.5% human baseline, or if errors concentrate in one functional class such as filters, form fields, or nested menus, the claim that state-difference inference plus two-LLM verification reproduces human annotation quality would be falsified. A complementary check is to feed the two verifier LLMs a set of human-graded wrong annotations and measure how many still receive full scores from both, since the verification stage is only load-bearing if it catches what humans call wrong.
Extended reading notes
Core claim
The central discovery is that an element's contextual functionality is legible in the change it produces, so functionality labels can be manufactured at scale without human annotators and without reading HTML or other surface metadata. The pipeline, formalized as $f = \mathrm{LLM}(p_{\mathrm{anno}}, s_t, s_{t+1})$, gives an LLM the accessibility tree before ($s_t$) and after ($s_{t+1}$) a simulated click or hover and asks it to reason through the added, deleted, unchanged, attribute-updated, and repositioned lines and summarize what the element does. Quality control is a rejection stage that scores each sample on explicitness, relevance, and predictability of the observed change, discarding the worst 30%, followed by a verification stage that keeps a sample only if two different LLMs both award full marks to the claimed functionality. The authors report 96.7% annotation correctness against 95.5% for a trained human annotator, and show that fine-tuning Qwen-VL, SliME, Qwen2-VL, and SeeClick on the resulting 702k-sample training set improves grounding on FuncPred, ScreenSpot, ScreenSpot-v2, MoTIF, and VisualWebBench, with gains growing as data scales and with full contextual descriptions outperforming condensed descriptions and raw HTML-element pairs.
Load-bearing premise
Everything downstream rests on the assumption that an LLM reading a before-and-after snapshot of a screen's accessibility tree can correctly name what the clicked element does; if that inference is systematically wrong for some class of elements, the rejection and verification filters could keep confidently false labels, and the small (145-146 sample) human evaluation is the only external check.
Editorial extensions
If this is right
- General-purpose VLMs fine-tuned on AutoGUI-702k gain large grounding accuracy on all five benchmarks, with Qwen2-VL-7B reaching 65.0% on FuncPred and 80.0% on ScreenSpot, approaching or exceeding specialized UI experts on the functionality-focused benchmark.
- Grounding accuracy rises monotonically as the AutoGUI training data scales from 25k to 125k to 702k samples, indicating that the pipeline converts compute and crawling effort directly into model capability.
- Full contextual functionality descriptions outperform both raw HTML-element pairs and condensed one-line descriptions as training annotations, and the gap is widest on the FuncPred benchmark that emphasizes contextual functionality grounding.
- In a two-stage GUI-agent setup, replacing a proprietary planner's own coordinate prediction with an AutoGUI-trained grounding model raises average step accuracy on AITW from 14.01% to 18.37% with one planner and from 29.50% to 39.23% with another.
- Each quality-control component adds measurable correctness: rules alone reach 83.1%, adding the LLM rejector raises this to 94.4%, one verifier to 96.0%, and two verifiers to 96.7%, against the 95.5% human baseline.
Reading between the lines
- Because the paper's scaling curve shows no saturation through 702k samples, a direct extrapolation is that feeding the same pipeline multi-million-scale trajectories and stronger open LLMs would push open VLMs clearly past current UI experts; the released 2k-sample test set makes this measurable.
- The paper's own limitations imply a coverage gap: elements that modify internet content are excluded, so AutoGUI-trained models likely ground informational affordances (navigation, filtering, menus) better than transactional ones (submit, purchase, post); the dataset's verb-noun statistics make this per-category accuracy testable.
- Since Llama-3-70B both annotates and serves as one of the two verifiers, the verification stage could partly reflect self-preference; an independent check would be to verify with two models from non-overlapping model families and re-measure the 96.7% correctness number.
- The state-difference mechanism is not GUI-specific: any interactive surface with a serializable state, such as spreadsheet cells, IDE panels, or multi-step web forms, could receive the same automatic functionality annotation, which would let instruction-following data be generated for software beyond web and mobile interfaces.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes AutoGUI, a pipeline that automatically generates contextual functionality annotations for GUI elements by prompting an LLM (Llama-3-70B) to infer element functionality from UI state changes before and after simulated interactions. A hand-rule plus LLM rejector removes invalid samples, and two LLM verifiers (Llama-3-70B and Mistral-7B) filter annotations. The authors curate AutoGUI-704k, a web and mobile dataset of 704k element-functionality pairs, and report a human-evaluation correctness of 96.7% versus 95.5% for a trained human annotator. Fine-tuning open VLMs (Qwen-VL, SliME, Qwen2-VL, SeeClick) on AutoGUI data improves grounding on FuncPred, ScreenSpot, MoTIF, and VisualWebBench, with accuracy increasing as the data scale grows from 25k to 702k. The paper also shows a potential use of the dataset in a two-stage GUI agent planning pipeline evaluated on AITW.
Significance. If the claims hold, AutoGUI is a valuable contribution: it directly addresses the scarcity of large-scale contextual functionality annotations for GUI grounding, provides a reproducible automatic pipeline and a public dataset, and shows consistent gains on multiple external grounding benchmarks. Strengths include the dataset scale (704k samples), the multi-benchmark evaluation, the data-scaling analysis, and the use of open models and open-source codebases. The concerns lie in the strength of the evidence for annotation correctness: the human evaluation is small, the LLM verification shares evidence with the annotation step, and the fine-tuning results do not yet isolate label correctness from annotation style. These gaps are addressable and do not undermine the overall value of the dataset, but they should be resolved before the central claims are taken as established.
major comments (5)
- [Section 3.6, Table 3] The human evaluation is too small to support the precision implied by the 96.7% versus 95.5% comparison. Section 3.6 states N=145 samples (99 valid and 46 invalid), while Section B.1 says the human annotator evaluated 146 samples; this inconsistency should be fixed. With N≈145 and a correctness proportion near 0.95, the standard error is roughly 1.6 percentage points, so the reported difference between AutoGUI and the human annotator is within noise. Additionally, the correctness metric C/(N-R) subtracts rejected samples from the denominator, so high correctness can be achieved by aggressive rejection; the paper should report the raw accuracy, the rejection rate, and a precision/recall breakdown so that the 96.7% claim can be interpreted properly.
- [Section 3.4, 3.6] The two-LLM verification does not provide an independent check on annotation correctness. Both verifiers receive the same UI state differences (AXTree diffs or UI descriptions) that produced the annotation, so agreement between Llama-3-70B and Mistral-7B demonstrates consistency of inference from the same evidence, not correctness against ground truth. The human evaluation is the only external check and is too small to detect class-level systematic errors, such as elements whose only UI change is a focus or attribute update (as in Section A.5 and Figure D). This matters because the 96.7% correctness claim is used to justify the downstream fine-tuning results; the paper should either enlarge the human evaluation with a more diverse sample or provide a targeted analysis of error rates by interaction-outcome type.
- [Section 4.2, Table 5] The comparison of annotation types in Table 5 (Elem-HTML, Condensed Func., Func.) does not isolate label correctness from annotation style. The full functionality descriptions are longer and more structured than the condensed and HTML variants, so the gains could reflect the linguistic form and statistics of the annotations rather than their semantic correctness. A label-perturbation control is needed: for example, fine-tuning on functionality descriptions that are fluent and contextually styled but describe the wrong element (or shuffled element-description pairs) would show whether the grounding improvements in Table 4 and Figure 5 are driven by true functionality semantics. Without such a control, the observed gains are consistent with VLMs learning the annotation distribution rather than genuine element functionality.
- [Figure 5, Table 4] The scaling curves and the fine-tuned model accuracies are reported from single runs with no error bars or significance tests. This is a load-bearing issue because the central empirical claim includes a monotonic scaling effect from 25k to 125k to 702k samples, and some individual benchmark gains are small (e.g., Qwen2-VL on MoTIF, +1.2 in Table 4). Run-to-run variance in LoRA fine-tuning could be comparable to these differences. I request multiple seeds (at least three) for the key scaling points, or confidence intervals, for the FuncPred, ScreenSpot, and MoTIF results in Figures 5 and Table 4.
- [Section 4.1, FuncPred] FuncPred is constructed by the same annotation pipeline as the training data, so improvements on FuncPred may partly reflect in-distribution alignment with the AutoGUI annotation style rather than generalizable functionality understanding. The external benchmarks (ScreenSpot, MoTIF, VisualWebBench) are independent but use brief instructions or action intents rather than full functionality descriptions, so they do not directly validate the semantic correctness of the functionality annotations. The paper should explicitly acknowledge this asymmetry in the evaluation and discuss how much of the evidence for 'functionality grounding' rests on the in-distribution FuncPred test set. Reporting results on a held-out set of manually authored functionality descriptions would strengthen the claim considerably.
minor comments (7)
- [Appendix A.1] There is a typo: 'avoidding' should be 'avoiding'.
- [Limitations] The limitation heading 'AutoGUI is Not Indented to Record Task-Oriented Interaction Trajectories' contains a typo: 'Indented' should be 'Intended'.
- [Table 6] In Table 6, the Install and Single columns for the Qwen2-VL-7B SFT w/ AutoGUI rows show identical values (25.59/22.49 under GPT-4o-mini and 50.95/48.95 under Gemini), which looks like a copy-paste error; please verify these numbers.
- [Section 3.6] The sentence 'After experimenting with three runs, Tab. 3 shows...' is unclear: it is ambiguous whether the three runs refer to the human annotation, the AutoGUI pipeline, or the three LLM runs. Please specify what the three runs are.
- [Table 4, Section 4.1] The citation for Qwen2-VL in Table 4 and Section 4.1 is given as (Bai et al., 2023), but the reference list correctly attributes Qwen2-VL to Wang et al., 2024a; the in-text citation should be corrected.
- [Figure L, Section D.2] The caption of Figure L says 'ours-625k' while the text in Section D.2 consistently refers to 'ours-702k'; please make the model scale consistent.
- [Table 2] The 'Avg. Words' column in Table 2 would be clearer as 'Avg. words per annotation', since the 'Anno. Tokens' column already gives the total token count.
Circularity Check
Partially circular evaluation: FuncPred ground-truth labels are authored by the same LLM pipeline that writes the training annotations, and the two-LLM verifier judges each annotation against the same UI-state evidence that generated it; external benchmarks and the small human evaluation keep the central claim independently grounded.
-
fitted input called prediction
[Section 4.1 'Evaluation Benchmarks' (FuncPred); annotation generation in Section 3.2]
"We base our evaluation on the UI grounding benchmarks for various scenarios: FuncPred is the test split from our collected functionality dataset. This benchmark requires a model to locate the element specified by its functionality description."
FuncPred's ground-truth labels are outputs of the validated pipeline itself: each annotation is f = LLM(p_anno, s_t, s_t+1) by Llama-3-70B (Sec. 3.2), and Sec. 4.1 calls FuncPred 'the test split from our collected functionality dataset' (deduplicated from training only by box overlap, Sec. A.1). A VLM fine-tuned on 702k such labels is scored on 2k same-style labels from the same generator, so FuncPred gains (Tab. 4: Qwen-VL +45.7) partly measure reproduction of Llama-3-70B's annotation style, not externally verified functionality. In Tab. 5 the 'Func. (Ours full)' variant shares its generator with FuncPred's labels while 'Elem-HTML' and 'Condensed Func.' do not, so its FuncPred lead is partly forced by shared authorship of train and test labels.
-
self definitional
[Section 3.4 'LLM-Based Verification' (score definition), with annotation defined in Section 3.2]
"The scoring process is formulated as follows: score=LLM(p verify, e, f, st, st+1) where pverify denotes the verification prompt (Tab. E). Only if the two scores are both 3s do we consider the functionality annotation correct."
Annotation and verification are functions of the same evidence: Sec. 3.2 defines f = LLM(p_anno, s_t, s_t+1) and Sec. 3.4 defines score = LLM(p_verify, e, f, s_t, s_t+1). The verifier sees exactly the diffs that produced f, plus f itself, so accepting two full scores certifies that f is consistent with its own generating evidence; no independent signal about true functionality exists. If a state change admits a plausible but wrong reading (e.g., the focus-only change in the 'Cancel' rejection example, Tab. D), both Llama-3-70B — which wrote f — and Mistral-7B share that blind spot. The claim that verification 'eliminat[es] invalid annotations without human labor' and lifts correctness to 96.7% (Tab.
full rationale
The derivation chain is: (1) collect interaction trajectories (Sec. 3.1); (2) Llama-3-70B writes each annotation as f = LLM(p_anno, s_t, s_t+1) from AXTree or UI-description diffs (Sec. 3.2); (3) the same LLM rejects low-predictability samples (Sec. 3.3); (4) two LLMs, including the annotating model itself, re-score f given the same (s_t, s_t+1) evidence (Sec. 3.4); (5) surviving triplets become grounding and captioning tasks used to fine-tune VLMs (Secs. 3.5, 4.1). Two nodes are self-referential. First, the FuncPred benchmark is 'the test split from our collected functionality dataset' (Sec. 4.1): its ground-truth descriptions are outputs of the same pipeline that wrote the 702k training annotations, so gains on FuncPred (Tab. 4) and especially the Tab. 5 variant ranking partly measure how well a model reproduces that generator's annotation style; the full-functionality variant shares its author with FuncPred's labels while the condensed and HTML variants do not. Second, the verification equation conditions on the same state-change evidence that produced f, so a 'pass' certifies self-consistency rather than independently validated functionality; the 145-146-sample human evaluation (Sec. 3.6 and B.1) is the only external calibration of this loop. These reductions are partial, not total: the central claim that AutoGUI data improves UI grounding is also supported by externally authored benchmarks (ScreenSpot, ScreenSpot-v2, MoTIF, VisualWebBench in Tab. 4) and by AITW agent gains (Tab. 6), none of which is forced by construction. No load-bearing self-citation or imported uniqueness theorem appears, and the LLM-as-judge citations (Zheng et al. 2023; Lightman et al. 2023) are external. The paper itself concedes that annotator-LLM bias 'is probably reflected in the collected data' (Sec. E), consistent with the partial self-reference flagged here. Score 4 reflects in-pipeline evaluation that partly reduces by construction while the main claim retains independent content.
Assumptions & free parameters
free parameters (5)
- Rejector retention threshold =
bottom 30% discarded by predictability score
- Verifier agreement threshold =
both Llama-3-70B and Mistral-7B-Instruct-v0.2 must output score 3
- Rejection and verification score range =
0-3
- Trajectory horizon T_act =
10
- Context and truncation limits =
AXTree diff cap 250 lines; AXTree description cap 150 lines; verifier context 20 lines
assumptions (6)
- domain assumption LLM can infer element functionality from UI state changes (AXTree diffs or UI descriptions).
- domain assumption Accessibility tree and HTML or XML structure faithfully capture visible UI and interaction-relevant semantics.
- domain assumption LLM predictability scores and dual-verifier scores correlate with actual annotation correctness.
- domain assumption The human correctness criteria (context-specificity, appropriate details, no hallucination) are a valid gold standard.
- ad hoc to paper Correctness defined as C/(N-R) with rejected samples subtracted from the denominator is an appropriate quality metric.
- domain assumption No train and test contamination is established by checking bounding box overlap on the same GUIs.
Cite this review
Pith. "Pith review of AutoGUI: Scaling GUI Grounding with Automatic Functionality Annotations from LLMs." pith.science (2026). https://pith.science/paper/266KTUGC
@misc{pith2026250201977,
author = {Pith},
title = {Pith review of: AutoGUI: Scaling GUI Grounding with Automatic Functionality Annotations from LLMs},
year = {2026},
howpublished = {\url{https://pith.science/paper/266KTUGC}},
note = {Machine review of arXiv:2502.01977}
}
read the original abstract
User interface understanding with vision-language models (VLMs) has received much attention due to its potential for enhancing software automation. However, existing datasets used to build UI-VLMs either only contain large-scale context-free element annotations or contextualized functional descriptions for elements at a small scale. In this work, we propose the \textbf{AutoGUI} pipeline for automatically annotating UI elements with detailed functionality descriptions at scale. Specifically, we leverage large language models (LLMs) to infer element functionality by comparing UI state changes before and after simulated interactions. To improve annotation quality, we propose LLM-aided rejection and verification, eliminating invalid annotations without human labor. We construct a high-quality AutoGUI-704k dataset using the proposed pipeline, featuring diverse and detailed functionality annotations that are hardly provided by previous datasets. Human evaluation shows that we achieve annotation correctness comparable to a trained human annotator. Extensive experiments show that our dataset remarkably enhances VLM's UI grounding capabilities and exhibits significant scaling effects. We also show the interesting potential use of our dataset in UI agent tasks. Please view our project at https://autogui-project.github.io/.
Figures
Figures from the paper (3 more)
Forward citations
Cited by 1 Pith paper
-
ZeroGUI: Automating Online GUI Learning at Zero Human Cost
ZeroGUI uses VLM-generated tasks and VLM-estimated rewards with two-stage GRPO to improve GUI agent success rates on OSWorld and AndroidLab without human annotations.
Reference graph
Works this paper leans on
-
[1]
You will be shown line-by-line differences between the webpage content before and after interacting with the element. Here’s what each prefix indicates: Unchanged: Lines that are identical before and after the interaction. Added: New lines that appear after the interaction. Deleted: Lines that were present before the interaction but removed afterward. Ren...
-
[2]
You MUST thoroughly analyze the changes in webpage content (Added, Deleted, Unchanged lines) caused by interacting with the element, present a detailed reasoning process elucidating how the element affects the webpage, and finally summarize the element’s overall purpose based on your analysis
-
[3]
Avoid detailing every specific functionality of the webpage element. Instead, focus on describing its broader impact on the webpage experience. For example, if interacting with a “Products” button reveals a dropdown menu, do not catalog the subsequent webpage changes in exhaustive detail
-
[4]
Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large lan- guage models.Preprint, arXiv:2311.07575. Fangyu Liu, Julian Martin Eisenschlos, Francesco Pic- cinno, Syrine Krichene, Chenxi Pang, Kenton Lee, Mandar Joshi, Wenhu Chen, Nigel Collier, and Yasemin Altun. 2023a. Deplot: One-shot visual lan- guage reasoning by plot-...
arXiv 2024
-
[5]
Avoid mentioning specific elements from the webpage before interaction in the Summary. Instead, focus directly on the outcome of the interaction. For instance, rather than detailing a transition from one element to another, like “This element switches the focus from the ‘Destination’ input field to the ‘Check-in Date’ field, providing a date picker,” simp...
-
[6]
InThe Twelfth International Confer- ence on Learning Representations
Grounding multimodal large language models to the world. InThe Twelfth International Confer- ence on Learning Representations. Christopher Rawles, Alice Li, Daniel Rodriguez, Ori- ana Riva, and Timothy Lillicrap. 2023. An- droidinthewild: A large-scale dataset for android device control. InAdvances in Neural Information Processing Systems, volume 36, page...
work page 2023
-
[7]
The final summary must emphasize the distinctive functionality of the element and not be too vague. For example, “This element redirects the user to a specific website, allowing them to access its content and services” is too broad and lacks meaningful detail. Instead, it should precisely describe how the element provides users with unique value Now it’s ...
work page 2023
-
[8]
Os-atlas: A foundation action model for gener- alist gui agents.Preprint, arXiv:2410.23218. Renqiu Xia, Bo Zhang, Hancheng Ye, Xiangchao Yan, Qi Liu, Hongbin Zhou, Zijun Chen, Min Dou, Botian Shi, Junchi Yan, et al. 2024. Chartx & chartvlm: A versatile benchmark and founda- tion model for complicated chart reasoning.arXiv preprint arXiv:2402.12185. Tianba...
arXiv 2024
Show all 44 references
-
[9]
magnifier
Llamafactory: Unified efficient fine-tuning of 100+ language models. InProceedings of the 62nd Annual Meeting of the Association for Compu- tational Linguistics (Volume 3: System Demonstra- tions), Bangkok, Thailand. Association for Computa- tional Linguistics. A Appendix A De...
-
[10]
image 'Microsoft 365 Logo'
-
[11]
how to cut pineapple
StaticText 'Microsoft 365’ … After interacting with the candidate element, the webpage exhibits these changes: {UI differences} Filling in the prompt Figure H: An example of AutoGUI prompting the LLM as a self-verifier to determine whether an element supports its functionality...
-
[13]
(Provide a concise high-level description of the element’s function
Your output MUST follow this format: Reasoning: (Examine the webpage variation carefully to figure out how the interacted element changes the webpage) Summary: This element ... (Provide a concise high-level description of the element’s function. This description should contain...
-
[15]
Community submenu
For simplicity, you MUST not mention the text of the element in the summary. Now it’s your turn. Element interaction: After clicking a <link> element named “Community submenu” (an example of the interaction target), the webpage exhibits the following variations: (UI content ch...
-
[16]
Highly related modifications obtain a high score
Relevance of Changes: This criterion assesses the significance of the modifications in relation to the element’s intended function. Highly related modifications obtain a high score. No scores for irrelevant or unrelated content changes. 3) Pre- dictability of Outcome: This inv...
-
[17]
You will be given descriptions of a webpage before and after interaction with an element. Your primary task is to meticulously analyze the differences in content resulting from this interaction to understand what the functionality of the element is in the webpage context
-
[18]
You must present a detailed reasoning process before finally summarizing the element’s overall purpose based on your analysis
-
[19]
This approach will provide a more holistic view of the element’s impact on the webpage
Prioritize examining changes in the webpage’s regional content over individual element variations. This approach will provide a more holistic view of the element’s impact on the webpage
-
[20]
You should emphasize on the main content changes and pay less attention to less meaningful regions, such as headers, navigation bars, and footers
-
[21]
(Provide a high-level description of the element’s functionality
Your output MUST follow this format: Reasoning: (Examine the webpage variation carefully to figure out how the interacted element changes the webpage) Summary: This element ... (Provide a high-level description of the element’s functionality. This description should contain th...
-
[22]
This element triggers the disappearance of the header and language selector elements and the emergence of a login form
Avoid mentioning specific elements from the webpage prior to interaction in the Summary. Instead, focus directly on the outcome of the interaction. For instance, rather than detailing element changes, like “This element triggers the disappearance of the header and language sel...
-
[24]
The outcome can take one of two forms: changes to the webpage description, or line-by-line differences
You will be shown the outcome (webpage changes) resulting from interacting with the element. The outcome can take one of two forms: changes to the webpage description, or line-by-line differences. For the latter form, here’s what each prefix indicates: Unchanged: Lines that ar...
-
[25]
Analyze the provided outcome and provide detailed reasoning for whether this outcome helps to predict the element’s functionality, considering the following stringent criteria:
-
[26]
Show More
Explicitness of Changes: Rate how directly the changes suggest the element’s functionality. Score 1-3 for clear, unambiguous changes. Clearer changes obtain a higher score. No scores for vague, meaningless, or non-specific changes. Positive Example: A button labeled “Show More...
-
[27]
Contact Us
Relevance of Changes: Evaluate the significance of the modifications in relation to the element’s intended function. Score 1-3 for changes that enhance understanding of the element’s role. Highly related modifications obtain a high score. No scores for irrelevant or unrelated ...
-
[28]
Settings
Predictability of Outcome: Assess how anticipated the interaction outcome is based on the changes, considering common web conventions and user experience principles. Score 1-3 for highly predictable outcomes. Highly predictable changes obtain a high score. No scores for outcom...
-
[29]
Cancel” (interaction target) Outcome: (An example of interaction outcome) After clicking a <button> element named “Cancel
Your output MUST follow this format: Reasoning: - Explicitness of Changes: [Detailed reasoning and score] - Relevance of Changes: [Detailed reasoning and score] - Predictability of Outcome: [Detailed reasoning and score] Overall score: <score>[criterium 1 score] + [criterium 2...
-
[30]
Webpage content: This input represents the hierarchical structure of a webpage’s elements, emphasizing semantic information and relationships. Each node in the tree includes details such as the element’s role (e.g., button, link, heading), relevant attributes (e.g., expanded),...
-
[31]
It also introduces a candidate element for evaluation and then presents the webpage changes caused by interacting with this element
Task Description: This describes the action a user intends to perform (such as submitting a form, navigating to a particular section, or adjusting settings) or the information they seek (such as a specific content piece or form field). It also introduces a candidate element fo...
-
[32]
Analyze the provided webpage content to understand the structure and semantics of the webpage’s elements
-
[33]
Consider the element’s role, attributes, and position within the hierarchy
Evaluate the Candidate Element: Determine the suitability of the specified candidate element for the described action. Consider the element’s role, attributes, and position within the hierarchy. Your evaluation should be grounded in how well these aspects align with the requir...
-
[34]
This score should reflect the degree to which the candidate element meets the action’s requirements: 0: The element does not support the action in any capacity
Score the Element: Assign a score ranging from 0 to 3, enclosed within <score></score> tags. This score should reflect the degree to which the candidate element meets the action’s requirements: 0: The element does not support the action in any capacity. 1: The element provides...
-
[35]
Provide Reasoning: Before presenting your score, offer a detailed explanation of your reasoning. This should cover your analysis of the webpage content, the relationship between the candidate element and the specified action, and how these factors informed your scoring decision
-
[36]
Minivans
Format for Your Answer: Reasoning: (Provide a comprehensive analysis covering the webpage’s insights, the relationship between the specified action and the candidate element, and the rationale behind your scoring decision.) Score: <score>[Y ourScoreHere]</score> (An in-context...
2024
-
[37]
RootWebArea 'Login | Microsoft 365' focused: True
-
[38]
button 'All Microsoft expand to see list of Microsoft products and services' expanded: False
-
[39]
button 'Microsoft 365' expanded: False
-
[40]
StaticText 'Microsoft 365 '
-
[41]
link 'Sign in to your account'
-
[42]
region 'Office is now Microsoft 365' => [8] button ''
-
[178]
Springer. Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan P Foster, Pannag R Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. 2025...
2025 arXiv
-
[2022]
InThirty- sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track
LAION-5b: An open large-scale dataset for training next generation image-text models. InThirty- sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track. Bryan Wang, Gang Li, Xin Zhou, Zhourong Chen, Tovi Grossman, and Yang Li. 2021. Screen2words...
2021 arXiv
-
[2023]
InProceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 8469–8488
PaLM-e: An embodied multimodal language model. InProceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 8469–8488. PMLR. Longxi Gao, Li Zhang, Shihe Wang, Shangguang Wang, Yuanchun Li, and Mengwei Xu....
2024
-
[2024]
InAdvances in Neural Information Pro- cessing Systems, volume 37, pages 68772–68802
Llm evaluators recognize and favor their own generations. InAdvances in Neural Information Pro- cessing Systems, volume 37, pages 68772–68802. Curran Associates, Inc. Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shao- han Huang, Shuming Ma, Qixiang Ye, and Furu Wei
-
[2025]
InThe Thirteenth International Conference on Learning Representations
Navigating the digital world as humans do: Universal visual grounding for GUI agents. InThe Thirteenth International Conference on Learning Representations. Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, ...
2024
Reviewed August 9, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.