{"id":"2cd17aea-7f7d-448d-a587-a91634baa93a","arxiv_id":"2501.02863","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Sphinx, a multi-dimensional UI-navigation benchmark, shows current foundation models score low on goal understanding, planning, grounding, instruction following, and end-to-end success, with zero successes on industrial WeChat testing tasks.","lead":"The paper introduces Sphinx, a benchmark that scores foundation models on five separate skills needed to navigate mobile apps by following high-level goals, plus end-to-end success. It finds all tested models fail most tasks and fail every industrial WeChat test case, and that text-based inputs beat screenshots.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The zero-success result on WeChat testing tasks rests on manually crafted trajectory evaluators that are never validated, so the headline finding may reflect evaluator strictness rather than FM inability; an independent QA audit of sampled trajectories would settle this.","rationale":"I read the paper as a benchmark contribution whose central claim is empirical: state-of-the-art FMs cannot solve industrial goal-based UI testing tasks, and the bottleneck is UI-specific capabilities. The strongest claim depends on the trajectory-based evaluators being correct. The paper explicitly says these evaluators are 'manually craft[ed]' (Section 3.2.1) and provides no validation statistics. This is not an internal inconsistency, but it is a genuine correctness risk: the zero-success number is not externally anchored. The reader identified the same assumption, and I agree. The proposed audit is feasible because the benchmark and task artifacts are released; it directly tests whether false negatives in the evaluators change the headline. If the audit passes, the paper's main conclusion is well supported, so this concern warrants keeping the existing CONDITIONAL verdict rather than rejecting the paper. Secondary concerns, such as the text/vision observation-format confound and the absence of repeated-run statistics, are real but less directly threatening to the industrial-testing-unsolved claim; they should be addressed in revision without changing the verdict. I credit the paper for releasing its benchmark and for grounding tasks in real industrial test cases, which makes this an empirical question that can be settled by independent validation rather than by assumption.","tokens_in":20591,"tokens_out":4286,"duration_ms":85857,"concrete_test":"Stratified audit: sample 30 testing-task trajectories (10 from near-miss ACP at least 5%, 10 from mid-range ACP, 10 from ACP near 0) across models; have two WeChat QA engineers who did not author the evaluators independently label each trajectory as success or failure against the original test-case requirement documents, blind to Sphinx's evaluator outputs. Compute agreement between Sphinx evaluators and the QA majority label (Cohen's kappa) and between the two QA raters. If kappa is below 0.8, or if any QA-labeled success is scored as failure by Sphinx, re-run Table 4 testing-task SR/ACP with corrected evaluators and update the zero-success and bottleneck conclusions accordingly. If agreement is high, the central empirical claim is robust to this concern.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's strongest empirical claim is that no benchmarked FM solves any of the 214 WeChat testing tasks (Table 4: SR 0.0%, ACP at most 6.8%) and that UI-specific capability gaps are the cause. This claim is defined by Sphinx's trajectory-based evaluator (Section 3.2.1): evaluators are 'manually craft[ed]' boolean functions over assertions such as StopPage, LastAction, FindAction, FindElement, and FindElementByAction, with orderings (sequential, consecutive, presence), and each task is assigned 'one or more evaluators.' No validation of these evaluators is reported: no inter-evaluator agreement, no comparison with an independent oracle, no audit of false negatives, and no check that alternative-path handling is complete. For testing tasks the paper additionally says success requires entering the functionality via 'the specified path,' which increases the chance that a legitimate completion is judged a failure. Because SR, ACP, and the zero-success headline are computed directly from these unvalidated evaluators, evaluator strictness is a load-bearing threat: if even a small fraction of testing-task evaluators are too strict, the 'none solved' result and the model ranking could change. The threats-to-validity section addresses model representativeness and task representativeness but not evaluator correctness.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Sphinx, a multi-dimensional benchmark for evaluating foundation models on goal-based mobile UI navigation. Sphinx contains 244 user tasks collected from 100 popular industrial apps and 214 testing tasks drawn from WeChat's internal QA process, and it evaluates five capabilities (goal understanding, app knowledge, planning, grounding, and instruction following) in addition to end-to-end success rate (SR) and average completion proportion (ACP). The authors benchmark 8 foundation models across 20 configurations, including text-only, vision, and multimodal models, and also evaluate the AppAgent UI navigation agent. Their main findings are that all models perform poorly on the benchmark, that no model solves any WeChat testing task, that text-based models outperform vision-based models, and that the primary bottlenecks are UI-specific capabilities such as grounding, planning, and instruction following. The paper also reports seven lessons learned. The benchmark and scripts are publicly available.","tokens_in":20806,"tokens_out":3989,"duration_ms":40865,"significance":"If its claims hold, Sphinx would be a valuable community resource: it is one of the first benchmark efforts to combine industrial-scale testing tasks with fine-grained capability evaluation, and the public release of tasks and interfaces would facilitate reproducible comparisons. The paper's multi-dimensional diagnosis of failure modes (grounding, planning, instruction following) is a useful step beyond binary pass/fail metrics, and the inclusion of WeChat QA test cases gives the benchmark genuine industrial relevance. The authors also credibly demonstrate that sophisticated agent designs such as AppAgent inherit the limitations of their underlying FMs, which is a practically important observation. However, the significance of the headline results is contingent on the validity of the paper's trajectory evaluators and on the comparability of its text and vision conditions, both of which are questionable as presented.","major_comments":[{"comment":"The central claim that no benchmarked FM solves any of the 214 WeChat testing tasks (SR 0.0% in Table 4) is entirely defined by the manually crafted trajectory-based evaluators described in Section 3.2.1. The paper reports no validation of these evaluators: no inter-evaluator agreement, no independent oracle, no audit of false negatives, and no evidence that the handling of alternative navigation paths is complete. This concern is compounded by Section 5.1, which states that testing tasks require entering the functionality via a specified path, so a legitimate completion that uses an alternative path would be judged a failure. Because SR, ACP, and the zero-success headline are all computed from these unvalidated evaluators, evaluator strictness is a load-bearing threat: even a small rate of overly strict evaluators could change the 'none solved' result and the model ranking. I request a concrete validation study: sample a subset of trajectories (including near-misses and failures), have QA engineers independently judge task success, and report agreement rates and false-negative rates, or otherwise demonstrate that the evaluators are neither too strict nor too lenient.","section":"§3.2.1, Table 4"},{"comment":"The paper's conclusion that 'vision modality lags significantly behind text modality' is not supported by a controlled comparison. Text-based models receive the full accessibility tree with element IDs, and the action space (Table 2) allows these models to act directly on element IDs. Vision-based models, in contrast, receive raw screenshots or SoM-labeled screenshots and must output screen coordinates. This means the text and vision conditions differ in both the information content of the observation and the difficulty of the action interface. The lower performance of vision models could be due to the lack of element IDs, the need for coordinate grounding, or the absence of semantic text, rather than to inherent visual perception limitations. I request a matched comparison, for example by providing the same element IDs to vision models as visual overlays (or by removing element IDs from the text condition), or by ablating the action space so that both modalities use equivalent output formats.","section":"§3.4.1, §5.1, Table 5"},{"comment":"All end-to-end results are reported as single point estimates with no confidence intervals and no repeated runs. Given the stochastic nature of FM decoding, the differences that support claims such as 'larger models achieve much higher SRs and ACPs' (Section 5.1) or the ordering of GPT-4-Turbo (31.1%) versus GPT-4o (28.7%) on user tasks could be within run-to-run noise. I request that the authors report either multiple runs with variance, or statistical significance tests for the main comparisons, or at least explicitly acknowledge the absence of repeated runs as a limitation and avoid strong claims about model ordering.","section":"§5.1, Table 4"},{"comment":"Parts of the benchmark content are generated or repaired by the same families of models that are benchmarked. Specifically, grounding instructions are generated by GPT-4o and then cleaned by human annotators (Section 3.3.4), and knowledge-probing and grounding outputs are 'repaired' by DeepSeek-V2 before scoring (Section 5.2.1, Section 5.2.4). Since GPT-4o and DeepSeek-V2 are among the evaluated models, their measured performance on these tasks could reflect familiarity with their own generated text or repair conventions rather than a general UI capability. This does not invalidate the benchmark, but it does threaten the comparability of the multi-dimensional results across models. I request that the paper either report results separately for human-authored versus model-generated task subsets, or provide evidence that the generation/repair process does not differentially benefit the generating models.","section":"§3.3.2, §3.3.4, §5.2"}],"minor_comments":[{"comment":"Figure 2 appears garbled in the manuscript, with series of 'uni' escape sequences instead of readable axis labels and legend text; the figure should be regenerated and the underlying rendering issue fixed.","section":"General"},{"comment":"References [62] and [63] point to the same paper ('Understanding the weakness of large language model agents within a complex android environment') with the same authors and venue; if they are indeed meant to be the same, one should be removed, and the in-text citations to 'AndroidArena' should be made consistent.","section":"§2.1, References"},{"comment":"The text says 'an BQ' where it should read 'a BQ', and the sentence 'A BQ is a question with only two possible answers' is slightly redundant; a quick copyedit would improve readability.","section":"§3.3.2"},{"comment":"The threats-to-validity section addresses model representativeness and task representativeness but omits the most immediate threat, the correctness of the trajectory evaluators; this should be discussed even if no additional validation is performed.","section":"§6, Threats to Validity"}],"recommendation":"major_revision","confidential_remarks":"The manuscript comes from an experienced group and the Sphinx benchmark is likely to attract attention in the software engineering and FM-agent communities. The major issues are fixable in principle: a focused evaluator-validation study and a controlled text-vs-vision comparison would substantially strengthen the paper. I see no evidence of fabrication, but the reliance on unvalidated evaluators makes me unwilling to recommend acceptance in the current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague—\n\nThe short version: Sphinx is a solid, useful benchmark and the paper is worth your time if you work on FM-based UI navigation or automated testing. What's genuinely new is the isolated capability tasks—knowledge probing, completion judgment, grounding, focused context—and the WeChat QA test set. The authors evaluate 8 models with 20 configurations, which is real work, and they ship the code and data. The finding that current FMs fail badly on goal-based navigation, and that the bottlenecks are grounding, planning, and instruction-following rather than goal understanding or app knowledge, is coherent and consistent with what others see in web and desktop agents.\n\nNow the soft spots, in order of importance. The headline result—zero success on all 214 WeChat testing tasks—is defined by the trajectory-based evaluators described in Section 3.2.1. Those evaluators are manually crafted, and the paper reports no validation of them: no inter-evaluator agreement, no independent audit, no check on whether the specified-path requirement is too strict. The threats-to-validity section covers model and task representativeness but is silent on evaluator correctness. If even a fraction of the testing evaluators are over-strict, the 0% SR and the ACP numbers change; the exact model ranking could shift too. This is a load-bearing gap, not a cosmetic one. It doesn't kill the qualitative conclusion—FMs are poor at industrial UI testing—but it means the precise numbers are provisional.\n\nSecond, all results are point estimates. No repeated runs, no confidence intervals. Some reported differences—for instance GPT-4-Turbo vs GPT-4o on user tasks—are within the range where you'd want to see variance before trusting them.\n\nThird, the text-versus-vision comparison is confounded. Text models get the full accessibility tree with element IDs, which effectively solves part of the grounding problem for them; vision models get raw screenshots or SoM markings. So the \"vision lags text\" lesson is true in this setup, but it's not a clean statement about the modalities themselves.\n\nOn novelty: the paper claims to be the first multi-dimensional benchmark, but it cites AndroidArena as evaluating five distinct capabilities. The new thing here is isolating those capabilities into dedicated tasks rather than only measuring them inside end-to-end trajectories. That distinction should be stated more carefully.\n\nBottom line: this paper deserves a serious referee. I'd send it to review with two required revisions—validate the evaluators on a sample of trajectories (QA engineers auditing false negatives would do) and report confidence intervals or repeated runs. If the authors can't validate the evaluators, they should soften the zero-success claim. For a reader in this area, it's a useful benchmark to know about, and the seven lessons are a fair summary of where the field stands.","headline":"Sphinx is a genuinely useful multi-dimensional benchmark with real industrial tasks, but the zero-success testing result rests on manually crafted evaluators that are never validated, so treat the exact numbers as provisional.","tokens_in":21398,"tokens_out":3710,"would_cite":true,"duration_ms":30967,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper builds Sphinx, a multi-dimensional benchmark that decomposes mobile UI navigation into five capabilities, and finds that today's foundation models solve at most 16.6% of tasks and zero of 214 industrial testing tasks, with…","keywords":["GUI testing","UI navigation","foundation models","mobile apps","Android","benchmark","multi-dimensional evaluation","goal-based navigation"],"falsifier":"Have independent human raters judge a sample of the 214 WeChat testing-task trajectories that Sphinx marks as failed, allowing any completion path; if even one trajectory is accepted as satisfying the task's intent, the 0% success-rate claim and the ranking it anchors would need revision.","tokens_in":20378,"feed_emoji":"📱","tokens_out":8300,"duration_ms":74442,"temperature":0.7,"pith_summary":"This paper argues that pass/fail scoring cannot explain why foundation models fail at mobile UI navigation, and proposes a benchmark, Sphinx, that separates navigation into five testable capabilities: goal understanding, app knowledge, planning, grounding, and instruction following. On 244 user tasks drawn from 100 popular Android apps and 214 professional test cases from WeChat, the best text-based model completes only 16.6% of all tasks, and no model completes a single testing task. The capability scores point to the cause: models are strong at goal understanding and app knowledge (most above 85%) but weak at grounding actions to UI elements, judging when a task is finished, and obeying output-format and no-repetition rules. The paper concludes that agent designs layered on current foundation models cannot by themselves reach industrial-grade UI navigation, and that improvement must come from UI-specific training or external enforcement.","feed_headline":"Zero AI agents pass a single real mobile UI test task","feed_subtitle":"Best model scores 16.6%; grounding, planning, and instruction following are the bottlenecks.","key_machinery":"The load-bearing mechanism is the five-capability decomposition, each measured by a dedicated evaluation toolkit. Goal understanding and app knowledge are probed with multiple-choice and binary questions; planning is measured by a completion-judgment task that asks the model to decide 'continue' or 'stop' at every step of a trajectory; grounding is measured by single-step tasks that map a natural-language instruction to the correct UI element; and instruction following is measured by invariant checks that detect repeated actions, malformed action formats, and format violations in a distraction-free focused-context setting. End-to-end success is judged by manually crafted trajectory evaluators, which check assertions about the final screen, last action, and encountered elements along the generated path, allowing alternative paths and producing partial credit through average completion proportion.","core_discovery":"Sphinx's central discovery is that the failure of foundation models in goal-based mobile UI navigation is concentrated in UI-specific capabilities rather than in general language or knowledge. In the end-to-end evaluation, the best model reaches 16.6% success rate and 21.5% average completion proportion across all tasks, and all models score 0% on the 214 WeChat testing tasks, which require longer, more context-sensitive, and path-specific sequences of actions. The multi-dimensional probes then localize the deficit: models answer goal-understanding and app-knowledge questions with accuracy above 85% for most strong models, but grounding accuracy tops out at 87.5% in text and 65.0% in vision, the average accuracy of stop decisions in planning is 40.3%, and every model repeats actions about half the time or more on end-to-end traces. These gaps cascade, so even a multimodal navigation agent built on a strong model achieves only 8.0-11.0% success on a subset of Sphinx tasks, roughly matching or falling below the plain text-agent baseline. The paper reads this as evidence that UI grounding, planning, and instruction following, not goal understanding or app knowledge, are the bottlenecks that must be trained or engineered around.","pith_inferences":["A testable extension of the paper's own diagnosis: if an environment-level repetition blocker and format validator are added to the same chain-of-thought agent pipeline, success rates should rise toward the level implied by the capability scores; if they do not, the attribution of failure to instruction following would need revisiting.","The lack of validation for the manually crafted trajectory evaluators means the 0% testing-task result should be stress-tested by an independent human re-judgment; until then, the zero-success figure is as much a claim about evaluator strictness as about model ability.","The benchmark's task distribution (popular public apps plus a single industrial super-app) suggests that domain transfer of the conclusions is plausible for Android but unproven for other ecosystems such as iOS or non-mobile embodied agents.","If the multi-dimensional bottleneck pattern generalizes, similar capability decompositions for web or desktop navigation would likely reveal the same grounding-versus-knowledge asymmetry, making Sphinx's toolkit adaptable beyond mobile."],"forward_implications":["A direct corollary is that adding a deterministic output-format validator and a repetition suppressor to a UI navigation agent should raise end-to-end success without any model improvement, because the paper's invariants identify these as frequent failure modes.","Because text observations from the accessibility tree strongly outperform raw or annotated screenshots, current agent designs should feed text first and treat vision as auxiliary; vision-language models are not yet competitive on this task class.","Generic high scores on language benchmarks do not imply UI navigation competence, so downstream-specific benchmarks like Sphinx are needed for model selection in this domain.","Agent architectures cannot compensate for the underlying model's grounding and planning limits, so further gains depend on UI-specific fine-tuning or external support systems rather than prompt-level agent design.","The multi-dimensional scorecard allows practitioners to choose models based on the capability their task needs most, e.g. grounding accuracy for element-heavy tasks or stop-decision accuracy for open-ended exploration."],"supporting_citations":[{"why":"Prior Android UI navigation benchmark; its low reported success rates motivate Sphinx's finer-grained capability evaluation.","marker":"[57]"},{"why":"Industrial-app UI navigation benchmark with agent designs, establishing the low end-to-end baseline that Sphinx extends.","marker":"[41]"},{"why":"Supplies the trajectory-based evaluation approach with manually crafted evaluators that Sphinx adapts.","marker":"[73]"},{"why":"A multimodal UI navigation agent used in RQ3; its failure cascades demonstrate that agent designs inherit model capability limits.","marker":"[67]"},{"why":"The reasoning-and-acting agent pipeline used uniformly across all benchmarked models in the end-to-end evaluation.","marker":"[70]"},{"why":"Supplies the annotated-image prompting technique used for vision-modality observations, informing the comparison between plain and marked screenshots.","marker":"[65]"},{"why":"Defines the Android accessibility API that produces the accessibility-tree text observations used for the strongest results.","marker":"[17]"},{"why":"Documents alternative paths in UI navigation, motivating the trajectory evaluator design that allows different valid routes.","marker":"[26]"},{"why":"Supplies the prompt-based knowledge probing technique used to measure goal understanding and app knowledge.","marker":"[52]"}],"fun_headline_variants":["All AI agents score 0% on real WeChat UI tasks","Best mobile UI agent hits just 16.6% success","UI grounding and planning are the true bottlenecks","Every AI agent repeats actions half the time","Mobile navigation fails on UI specifics, not language"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The zero-success and ranking results depend on the manually crafted trajectory evaluators correctly judging whether a generated action sequence achieves the task's intent; the paper does not validate those evaluators against an external oracle or check inter-judge agreement, so a stricter-than-intended evaluator would lower scores across the board.","fun_headline_variants_meta":{"raw":{"variants":["All AI agents score 0% on real WeChat UI tasks","Best mobile UI agent hits just 16.6% success","UI grounding and planning are the true bottlenecks","Every AI agent repeats actions half the time","Mobile navigation fails on UI specifics, not language"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000782,"raw_usage":{"total_tokens":3484,"prompt_tokens":1004,"completion_tokens":2480,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":620,"completion_tokens_details":{"reasoning_tokens":2404}},"tokens_in":620,"tokens_out":2480,"duration_ms":17077,"temperature":1.0,"reasoning_tokens":2404,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:01:00.317250+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have independent human raters judge a sample of the 214 WeChat testing-task trajectories that Sphinx marks as failed, allowing any completion path; if even one trajectory is accepted as satisfying the task's intent, the 0% success-rate claim and the ranking it anchors would need revision.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Industrial-app UI navigation benchmark with agent designs, establishing the low end-to-end baseline that Sphinx extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the Android accessibility API that produces the accessibility-tree text observations used for the strongest results."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents alternative paths in UI navigation, motivating the trajectory evaluator design that allows different valid routes."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the prompt-based knowledge probing technique used to measure goal understanding and app knowledge."}],"review_version":1}