{"id":"b1520b0f-623b-4621-8cf5-ab76dc8e2f7e","arxiv_id":"2507.23779","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Phi-Ground models achieve state-of-the-art click accuracy on five GUI grounding benchmarks for models under 10B parameters using a 40M-sample training recipe with text-first inputs, random-resize augmentation, uniform point resampling, and multi-round DPO.","lead":"Microsoft researchers trained small AI models, Phi-Ground, to find click targets on computer screens from natural-language instructions, reporting top accuracy on five GUI-clicking benchmarks for models under 10 billion parameters. The report is a detailed recipe of data cleaning, augmentation, and post-training steps for anyone building computer-use agents.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported SOTA scores are selected as the best checkpoint on the test benchmarks themselves (§5.5), inflating Phi-Ground's numbers relative to fixed baselines; re-evaluation with a single fixed checkpoint is needed.","rationale":"The reader's conditional verdict is on the right track: the evaluation protocol is the soft spot. However, the reader's weakest assumption focuses on reproducibility (shared harness, planner outputs, Gold dataset bias). The more specific and more damaging issue is the paper's own admission in §5.5 that the reported scores are selected as the best checkpoint across the test benchmarks. This is a test-set selection bias that directly inflates every headline number and breaks comparability with fixed baselines. It is concrete, internal, and checkable, whereas the reader's reproducibility concern is broader but less decisive for accepting the current numbers. I considered whether the proprietary Gold dataset's in-domain overlap is more load-bearing; while it makes 'all five benchmarks' weaker, the model is also SOTA on the four public benchmarks in the agent setting, so the Gold dataset is not essential to the core claim. The checkpoint-selection issue affects all five and is thus more fundamental. The paper otherwise has real strengths: transparent negative results (Appendix C), multi-benchmark ablations, and careful reporting of data pipelines. Those do not offset the selection bias, but they mean the correct remedy is to re-report with a fixed checkpoint, not to discard the work. Hence the verdict stays CONDITIONAL, pending the fixed-checkpoint re-evaluation.","tokens_in":52032,"tokens_out":8184,"duration_ms":78426,"concrete_test":"Recompute the agent-setting rows of Table 10 for each Phi-Ground model using the final training checkpoint (or a checkpoint selected on a disjoint held-out validation split) instead of the best-test checkpoint, and compare with the runner-up scores in the same table. If any claimed SOTA margin inverts (e.g., ScreenSpot-pro drops below UI-TARS-1.5-7B's 48.8, or UI-Vision drops below 29.4), the central SOTA claim loses support. The per-checkpoint evaluation curves from Figure 12 should also be released so the gap between best-test and fixed-checkpoint accuracy can be audited.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central SOTA claim in the agent setting rests on Table 10, but Section 5.5 states: 'We also observed some fluctuations in the performance across different benchmarks during training, which led us to conduct tests at regular intervals and ultimately select the checkpoint that performed best across all benchmarks.' This means the reported accuracy for each Phi-Ground model is the maximum over checkpoints evaluated on the test benchmarks, not the accuracy of a single fixed model. No held-out validation split is used for this selection, and the baselines (UI-TARS, UGround, SeeClick, OS-Atlas) are fixed released checkpoints that cannot receive this benefit. The comparison is therefore asymmetric: Phi-Ground is allowed to peek at the test set and pick its best epoch, which can inflate scores by several points given the visible fluctuations in Figure 12. Since the SOTA margins on ScreenSpot-pro (55.0 vs 48.8), UI-Vision (36.2 vs 29.4), and ShowDown (73.87 vs 71.58) are all in this range, the headline 'SOTA on all five benchmarks' may be an artifact of checkpoint selection rather than a statement about a deployable model. This concern is internal to the paper and does not depend on external reproduction.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents Phi-Ground, a family of GUI grounding models under 10 billion parameters, and an empirical study of the full training pipeline from data collection to post-training. The authors introduce a two-stage agent setting in which a large MLLM planner expands short instructions into long reference expressions, and a fine-tuned grounding model produces click coordinates in text form. Using up to 40M training samples drawn from open-source, CommonCrawl, Bing search, and human-labeled sources, they study input modality order, output coordinate formats, data augmentation, center-point resampling, in-domain post-training with SFT/DPO/curriculum/reject-sampling, and scaling with respect to parameter count and image tokens. The paper claims state-of-the-art results on all five benchmarks in the agent setting (e.g., 55.0 on ScreenSpot-Pro and 36.2 on UI-Vision) and on three of five benchmarks in the end-to-end setting (43.2 and 27.2 on the same two benchmarks).","tokens_in":52290,"tokens_out":3819,"duration_ms":38556,"significance":"If the results hold, the paper provides a detailed and useful recipe for GUI grounding, with several valuable negative results (tokenized coordinates, label smoothing, loss reweighting) and a credible scaling analysis that accounts for image tokens as a computational axis. The authors deserve credit for evaluating on multiple benchmarks, for disclosing failed approaches, and for the careful error decomposition in Section 6.2. However, the central SOTA claim currently rests on an evaluation protocol that allows the authors to select the best checkpoint using the test benchmarks themselves, and on a proprietary evaluation set and self-run baseline comparisons that cannot be independently checked. These issues directly affect the validity of the headline numbers, so the paper needs substantial revision before the claims can be accepted.","major_comments":[{"comment":"Section 5.5 states: 'We also observed some fluctuations in the performance across different benchmarks during training, which led us to conduct tests at regular intervals and ultimately select the checkpoint that performed best across all benchmarks.' This means the scores reported for Phi-Ground in Table 10 are maxima over checkpoints evaluated on the same test benchmarks that are used for final comparison, not the scores of a single fixed model. The baselines (UI-TARS, UGround, OS-Atlas, SeeClick) are fixed released checkpoints that receive no such benefit. The comparison is therefore asymmetric, and because several SOTA margins in Table 10 are small (55.0 vs 48.8 on ScreenSpot-Pro, 36.2 vs 29.4 on UI-Vision, 73.87 vs 71.58 on ShowDown), the reported 'SOTA on all five benchmarks' may be an artifact of checkpoint selection rather than a property of a deployable checkpoint. The authors should re-evaluate all results with a single fixed checkpoint chosen using a held-out validation split, or explicitly report the variance over training checkpoints and demonstrate that the SOTA claims survive without test-set-based selection.","section":"§5.5"},{"comment":"The evaluation of baselines and of the proposed models is not provided as a shared, reproducible harness. All baselines are re-run by the authors using their own planner prompts and their own pipeline for generating long reference expressions, and the Gold dataset is proprietary. Since the training data also uses long reference expressions generated by the same planner style (Section 4.2), there is a risk that the comparison is biased toward Phi-Ground's training distribution. The paper should release the evaluation code, the exact prompts and planner outputs, or at least a detailed protocol that would let others reproduce the baseline numbers; otherwise the relative improvements over UGround-v1 and UI-TARS-1.5 cannot be verified.","section":"§4.1, §6.1, Table 10"},{"comment":"The paper reports only single accuracy values without confidence intervals, error bars, or statistical significance tests. Given the small differences that underlie several SOTA claims, and the checkpoint-selection issue above, the reported margins (e.g., 55.0 vs 48.8 on ScreenSpot-Pro) may be within run-to-run or checkpoint-to-checkpoint variability. The authors should report confidence intervals (e.g., Wilson intervals for binary accuracy, or bootstrap over samples) and, where possible, multiple training runs or at least a fixed final checkpoint selected on a validation split.","section":"Tables 2-7, 10"}],"minor_comments":[{"comment":"Several tables and figures contain garbled sequences such as '/uni00000013/uni00000011/uni00000014...' in the rendered text. As presented, these parts of Table 8 and the captions of Figures 12-14 are unreadable and need to be regenerated with correct encoding.","section":"Table 8, Figures 12-14"},{"comment":"The evaluation dataset is referred to as 'Gold' in most places but as 'Golden' in Figure 7 and in the text near Figure 7; please unify the naming.","section":"Figure 1 and Section 4.1"},{"comment":"The notation in Algorithms 1 and 2 is used before the variables are fully explained. In particular, the box representation (x1, y1, x2, y2) should be stated explicitly at the start of Section 5.2 so the augmentation pseudocode is self-contained.","section":"Section 5.2, Algorithms 1-2"},{"comment":"The text says 'F LOP s= 6N D' in Figure 13, but the surrounding discussion in the main text uses 'N D' as the product of image tokens and parameters; please clarify the exact formula and whether the constant 6 is intended.","section":"Section 5.5, Figure 13"},{"comment":"In the paragraph after Table 10, the authors discuss ScreenSpot V1 and V2 and state that they did not include mobile data or balance icon/text types; this is an interesting statement, but the numbers for ScreenSpot V1 are not reported in any table. Please add the V1 results or remove the explicit reference to 'V1' in the discussion.","section":"Section 6.1"}],"recommendation":"major_revision","confidential_remarks":"In my view, the checkpoint-selection issue in Section 5.5 is the single most important problem: it turns the headline SOTA claim into a best-of-checkpoints comparison against fixed baselines. The authors should be required to re-evaluate with a single fixed checkpoint and a held-out validation split, and to explicitly state whether the SOTA margins persist. I also think the evaluation of baselines needs to be made reproducible or at least much more transparent. The paper is otherwise informative and worth publishing after these issues are addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Training recipe is the real contribution here, and it's a good one; the headline SOTA numbers are not supported by the protocol as written. Section 5.5 says they periodically tested on the benchmarks and picked the best checkpoint. That means the reported Phi-Ground scores are maxima over training checkpoints evaluated on the test sets, while the baselines are fixed released checkpoints. The margins on ScreenSpot-pro, UI-Vision, and ShowDown are all in the range where this selection could matter, so I wouldn't rely on the SOTA claim without a re-evaluation using a single checkpoint chosen on a held-out split.\n\nWhat is actually valuable: the paper documents a set of non-obvious, transferable findings. Text-first input order, random-resize augmentation for high-resolution screens, center-point uniform resampling, and multi-round DPO on a saturated perceptual model are all concrete and plausible. The negative results — tokenized coordinates, label smoothing, loss reweighting, PPO/GRPO failures — are reported honestly and will save people time. The ablations are extensive and internally consistent. No error bars, but for this kind of tech report that's common.\n\nThe soft spots: the checkpoint selection is the load-bearing one. The paper acknowledges fluctuation, and Figure 12 shows it, but then uses the best checkpoint without a validation split. That's a genuine asymmetry with the baselines, and the SOTA margins are not large enough to absorb it. The Gold dataset is proprietary, so the in-domain results can't be checked. The baselines are self-run with their own planner prompts, which is fine, but without released code or checkpoints the comparison can't be independently reconstructed. The paper is a tech report, not a rigorous benchmark evaluation, so these things matter less for the recipe than for the SOTA claim.\n\nWho is this for: practitioners building GUI grounding models. They'll get a useful recipe and a set of failure modes to avoid. The paper deserves a serious referee, but the editor should send it to someone who will push on the evaluation protocol. If the authors fix the checkpoint selection and release the checkpoints, the empirical claims would be much stronger.","headline":"Useful training recipe for GUI grounding, but the reported SOTA is inflated by selecting the best checkpoint on the test benchmarks.","tokens_in":52839,"tokens_out":3813,"would_cite":true,"duration_ms":33810,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Phi-Ground, a sub-10B model family trained on 40M+ samples, achieves state-of-the-art click accuracy on all five GUI grounding benchmarks when paired with a planner, and on three in the end-to-end setting.","keywords":["GUI grounding","computer use agents","multimodal perception","click accuracy","reference expressions","data scaling","direct preference optimization","data augmentation"],"falsifier":"Release the planner outputs and rerun every sub-10B baseline in a shared harness using identical long reference expressions; if a prior model such as UGround-v1 or UI-TARS-1.5 then scores near 55.0 on ScreenSpot-pro, the gains attributed to Phi-Ground's recipe would vanish. Independently, re-annotate a sample of the proprietary Gold set from public screenshots and check whether its element positions and reference styles resemble Phi-Ground's training data more closely than do ScreenSpot-pro or UI-Vision; strong resemblance would undercut the claimed generalization.","tokens_in":51845,"feed_emoji":"🖱️","tokens_out":7402,"duration_ms":73641,"temperature":0.7,"pith_summary":"GUI grounding is the perception step that tells a computer-use agent where to click, and inaccurate clicks make such agents unusable. This paper tries to show that a small, purpose-trained model can do that step very well: the Phi-Ground family, built from 4B and 7B multimodal backbones and trained on more than 40 million samples, reaches state-of-the-art click accuracy on all five grounding benchmarks studied when a larger planner first expands each short instruction into a detailed description, and on three of the five when used end-to-end. The practical point is that the expensive reasoning part of a GUI agent can be separated from the cheap spatial part, so accurate clicking does not require a giant end-to-end agent. Along the way the paper offers a recipe of choices that survive at scale, and reports which common tricks do not.","feed_headline":"Phi-Ground: sub-10B models beat bigger GUI agents on clicks","feed_subtitle":"A small clicker model plus a planner tops five grounding benchmarks, including ScreenSpot-pro and UI-Vision.","key_machinery":"The mechanism that carries the argument is the two-stage grounding pipeline with reference-expression expansion. A planner, run with a fixed prompt, rewrites a short UI instruction into a long description covering function, position, and appearance; the trained Phi-Ground model then reads that description together with the screenshot and emits a click point directly as ordinary text coordinates, with positions represented relative to the image and scaled by 1000. Training data are produced by a 'Long-gold' variant in which the annotating multimodal model receives the ground-truth box and a crop, so the generated descriptions are precise and separable. The paper also identifies the scaling-friendly ingredients: plain next-token prediction on text coordinates outperforms tokenized coordinate tokens, label smoothing, and loss reweighting once training data exceed about a million samples; random resize of screenshots onto a white canvas is the augmentation that most helps high-resolution benchmarks; and resampling web-crawl data so element center points are spread uniformly over a 50x50 grid prevents distribution bias from common web layouts. Model variants share the same backbone but vary the number of image crops, which lets the paper treat image-token count and parameter count together as the compute axis of a scaling law.","core_discovery":"The paper's central claim is that a grounding-focused multimodal model under 10B parameters, trained with a specific data-first recipe, can outperform both the listed open-source GUI agents and commercial computer-use systems on click localization. In the two-stage 'agent' setting, an advanced multimodal planner turns a short instruction into a combined functional, positional, and appearance reference expression, and the small Phi-Ground model maps that expression to a click coordinate; there the paper reports state-of-the-art results on all five benchmarks, including 55.0 on ScreenSpot-pro and 36.2 on UI-Vision. In the single-model 'end-to-end' setting, where the model receives only the short instruction and screenshot, it reports state-of-the-art results on three benchmarks, including 43.2 on ScreenSpot-pro and 27.2 on UI-Vision. The paper attributes the gains to scale combined with several non-obvious choices: text-before-image input order, random resize augmentation, resampling training elements so their centers cover the canvas uniformly, and multi-round DPO after supervised training has saturated.","pith_inferences":["A testable extension follows from the two-stage decomposition: benchmark grounding models with a fixed planner and with released planner outputs, so the perception component can be measured separately from planning quality; the paper's error analysis already suggests that a large share of ScreenSpot-pro failures is caused by the planner rather than the model.","The random-resize result is likely transferable to other high-resolution perception tasks where targets are tiny, such as satellite imagery or medical imaging: presenting the model with randomly scaled canvases may be cheaper than simply adding more resolution.","The uniform-center resampling result implies that benchmark scores can be sensitive to where interactive elements sit in a screenshot; future datasets may want to report center-position distributions alongside accuracy.","If multi-round DPO on the model's own rollouts improves a saturated perception model, preference-style objectives may work as a general clean-label regularizer in perception, a hypothesis the paper leaves untested outside GUI grounding."],"forward_implications":["Computer-use agents can be built as a small clicker model plus a general planner, lowering deployment cost and latency while keeping high click accuracy.","Training data quantity in the tens of millions makes simple text-coordinate prediction sufficient, so future work can drop tokenized-coordinate schemes, label smoothing, and loss reweighting.","Random resize augmentation and uniform element-position resampling should become standard in GUI grounding data pipelines, especially for high-resolution or small-target benchmarks.","Preference optimization such as multi-round DPO can push a perception model past its supervised-training plateau by improving robustness to data distribution and noisy labels, not by adding reasoning.","Evaluation across five benchmarks rather than one or two changes conclusions: a model can top ScreenSpot-V2 while losing on harder high-resolution and Windows-native sets, so multi-benchmark reporting is necessary."],"supporting_citations":[{"why":"Supplies the O4-mini planner used to generate the long reference expressions in the agent setting.","marker":"[8]"},{"why":"Provides the UI-Vision benchmark whose Element Grounding task sets one of the central SOTA targets.","marker":"[24]"},{"why":"Provides ScreenSpot-pro, the high-resolution benchmark where the paper reports its largest margins.","marker":"[25]"},{"why":"Provides the SeeClick baseline and the original ScreenSpot benchmark and training data.","marker":"[26]"},{"why":"Provides the UI-TARS baseline that the paper must beat in both agent and end-to-end settings.","marker":"[28]"},{"why":"Provides the UI-TARS-1.5 baseline, the strongest sub-10B comparison on most benchmarks.","marker":"[29]"},{"why":"Provides the UGround baselines and the end-to-end visual grounding approach and agent setting.","marker":"[30]"},{"why":"Supplies the OS-Atlas open-source training data, the ScreenSpot-V2 corrections, and the two-stage agent protocol.","marker":"[31]"},{"why":"Provides the Showdown-click-dev human-click benchmark used to compare against commercial computer-use agents.","marker":"[44]"},{"why":"Supplies the DPO algorithm whose multi-round application produces the final Phi-Ground-DPO checkpoints.","marker":"[61]"}],"fun_headline_variants":["Phi-Ground sub-10B model tops five GUI grounding benchmarks","Small Phi-Ground beats bigger GUI agents on click accuracy","Phi-Ground: under-10B model achieves SOTA GUI grounding","Phi-Ground's tiny model wins GUI grounding with 10B params"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison stands or falls on whether the evaluation protocol treats all models alike: baselines are re-run with the paper's own planner prompts and measured on the paper's proprietary Gold dataset, with no shared harness, released planner outputs, or independent replication, so if those choices happen to favor Phi-Ground's training distribution the SOTA claim would weaken.","fun_headline_variants_meta":{"raw":{"variants":["Phi-Ground sub-10B model tops five GUI grounding benchmarks","Small Phi-Ground beats bigger GUI agents on click accuracy","Phi-Ground: under-10B model achieves SOTA GUI grounding","Phi-Ground's tiny model wins GUI grounding with 10B params"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000542,"raw_usage":{"total_tokens":2650,"prompt_tokens":1051,"completion_tokens":1599,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":667,"completion_tokens_details":{"reasoning_tokens":1524}},"tokens_in":667,"tokens_out":1599,"duration_ms":13023,"temperature":1.0,"reasoning_tokens":1524,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T10:23:52.753803+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Release the planner outputs and rerun every sub-10B baseline in a shared harness using identical long reference expressions; if a prior model such as UGround-v1 or UI-TARS-1.5 then scores near 55.0 on ScreenSpot-pro, the gains attributed to Phi-Ground's recipe would vanish. Independently, re-annotate a sample of the proprietary Gold set from public screenshots and check whether its element positions and reference styles resemble Phi-Ground's training data more closely than do ScreenSpot-pro or UI-Vision; strong resemblance would undercut the claimed generalization.","supporting_citations":[{"cited_title":"Seeclick: Harnessing gui grounding for advanced visual gui agents","cited_arxiv_id":null,"evidence_quote":"Provides the SeeClick baseline and the original ScreenSpot benchmark and training data."},{"cited_title":"Navigating the digital world as humans do: Universal visual grounding for gui agents.ICLR, 2025","cited_arxiv_id":null,"evidence_quote":"Provides the UGround baselines and the end-to-end visual grounding approach and agent setting."},{"cited_title":"Os-atlas: A foundation action model for generalist gui agents.ICLR, 2025","cited_arxiv_id":null,"evidence_quote":"Supplies the OS-Atlas open-source training data, the ScreenSpot-V2 corrections, and the two-stage agent protocol."},{"cited_title":"The showdown computer control evaluation suite, 2025","cited_arxiv_id":null,"evidence_quote":"Provides the Showdown-click-dev human-click benchmark used to compare against commercial computer-use agents."},{"cited_title":"Direct preference optimization: Your language model is secretly a reward model.NIPS, 2023","cited_arxiv_id":null,"evidence_quote":"Supplies the DPO algorithm whose multi-round application produces the final Phi-Ground-DPO checkpoints."}],"review_version":1}