{"id":"2af5d56d-e547-4ad7-b94f-9e9394eae40c","arxiv_id":"2507.19132","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A new 416-task desktop benchmark with a five-level automation taxonomy and a demand-based generalization hierarchy shows current computer-using agents succeed on roughly one in nine tasks, with near-zero success on high-autonomy tasks.","lead":"OS-MAP is a new benchmark for computer-using agents that organizes 416 desktop tasks into five automation levels and three generalization scopes. Early results show the best tested agents succeed on only about 11 percent of tasks, far below human performance.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The stepwise decline across automation levels in Table 3 is confounded with task horizon and instruction detail, so the claimed level-specific capability bottlenecks are not yet identifiable; small per-level counts and unreported label agreement compound the problem.","rationale":"The reader's conditional verdict already identifies the automation-level labels as the weakest assumption. My stress-test agrees and sharpens the threat: even if the labels are perfectly valid, the observed decline is expected from horizon length and instruction ambiguity alone, because success is evaluated once at the end of the full trajectory and higher levels are deliberately longer and more underspecified. The aggregate claim that current agents are far from deployment is well supported by an executable benchmark with public artifacts, and the failure analysis is informative; the issue is specifically the causal interpretation of the level-wise gradient as distinct capability bottlenecks. Since the reader already conditioned the verdict on fixing this and related gaps, no further verdict change is needed. The contradiction between Table 1 and Section 3.5 on task sources is a concrete, fixable metadata error that should be corrected in revision and adds to the need for stronger evidence before accepting the taxonomy-based conclusions.","tokens_in":25218,"tokens_out":8080,"duration_ms":82629,"concrete_test":"From the released task JSONs and agent trajectories, compute per-level gold-step counts and instruction word counts, then fit a length-only baseline predicting task success as q^N with a shared per-step reliability q per agent. Compare the predicted L1-L4 success profile to Table 3; also re-run UI-TARS-72B on L2 and L3 subsets matched for step count and instruction detail. If the length-only baseline or the matched-subset comparison reproduces the stepwise decline, the bottleneck interpretation must be withdrawn or reformulated as a horizon/instruction-detail effect.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central discovery is the Table 3 stepwise decline across L1-L4: UI-TARS-72B falls from 48% to 14% to 1% to 0%. But this factor is not identified as a capability effect. Higher automation levels are constructed to omit instruction details and include misleading cues (Appendix C.1), and receive larger step budgets (15/15/30/50, Appendix D.1). Because success is a single binary outcome after a full trajectory, even a fixed per-step reliability q yields expected end-to-end success q^N, so longer and more underspecified L3/L4 tasks are expected to show near-zero success without any level-specific capability deficit. Per-level N is also small (25 L1, 42 L4), so the 0% L4 is 0/42 and the 48% L1 is 12/25; the fine-grained cross-model ordering is within sampling noise. No inter-annotator agreement is reported for the L1-L4 assignments, and the task-source counts are internally inconsistent (Table 1: 161 authored/255 OSWorld vs Section 3.5: 138/296), further weakening confidence in the metadata. The aggregate 11.4% failure result is likely robust, but the level-wise bottleneck conclusion, which is the load-bearing claim, is under-identified as reported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces OS-MAP, a benchmark for computer-using agents that organizes 416 tasks across 15 Ubuntu applications along two dimensions: a five-level automation taxonomy (L1 execution through L5 proactivity, with L5 not yet instantiated) and a three-level generalization scope derived from a user-demand hierarchy. The benchmark is built on the OSWorld VM infrastructure, uses screenshot-only observations and atomic GUI actions, and evaluates a range of general-purpose VLMs, a GUI-specialized model, and planning-grounding agents. The headline empirical result is that the best agent achieves only 11.4% overall success, with strong performance only on the simplest level (48% at L1) and near-zero performance at L3 and L4, while human performance is reported at 71.9%. The paper also provides a failure analysis attributing breakdowns to instruction following, hallucination, grounding, planning, adaptation, and orchestration.","tokens_in":25404,"tokens_out":5583,"duration_ms":58945,"significance":"If the benchmark and its level-wise analysis are sound, OS-MAP is a valuable community resource: it provides an executable, state-based evaluation environment; a task taxonomy defined before experimentation; hand-labeled tasks; a measured human baseline rather than an assumed one; and public code, environments, baselines, and data. The aggregate finding that current agents fail most realistic desktop tasks is credible and important, and the two-dimensional evaluation matrix is a useful organizing framework. However, the paper's most distinctive claim, the stepwise capability decline across automation levels, is currently under-identified because it is confounded with task horizon and instruction detail, and several metadata inconsistencies weaken confidence in the supporting statistics. The contribution is therefore significant but conditional on additional validation and reporting.","major_comments":[{"comment":"The benchmark statistics are internally inconsistent. Table 1 reports 283 single-app and 154 multi-app tasks, which sum to 437 rather than the stated total of 416, and the reported percentages (62.3% and 37.7%) do not match either 283/416 or 154/416. Section 3.5 states that 138 tasks were authored and 296 were relabeled from OSWorld, summing to 434, whereas Table 1 and Appendix C.3 give 161 authored and 255 OSWorld-derived tasks, summing to 416. These discrepancies must be reconciled, because the per-level counts and source split are used to support the paper's descriptive claims about the benchmark's composition.","section":"Section 3.5, Table 1, Appendix C.3"},{"comment":"The stepwise decline across automation levels, which is the paper's central empirical discovery, is confounded with two design choices. Appendix C.1 states that higher-level tasks omit instruction details and include misleading cues, and Appendix D.1 assigns different maximum interaction steps of 15, 15, 30, and 50 for L1 through L4. Because success is a single binary outcome after a complete trajectory, a constant per-step reliability q already yields expected end-to-end success of approximately q^N, so longer and more underspecified L3 and L4 tasks are expected to show near-zero success without any level-specific capability deficit. The paper should report per-level task horizons, distributions of trajectory lengths, success conditioned on reaching intermediate milestones, or a regression controlling for instruction length and step budget before attributing the decline to distinct capabilities such as adaptation and orchestration.","section":"Section 4.2, Table 3, Appendix C.1, Appendix D.1"},{"comment":"No inter-annotator agreement is reported for the L1-L4 and S1-S3 labels, even though the curation process involved nine student annotators. If the automation-level labels mainly track instruction detail or trajectory length rather than the intended capabilities, the monotonic decline in Table 3 would be an artifact of the labeling scheme. The authors should report agreement statistics such as Cohen's kappa on a sample of tasks, and ideally provide an external validation showing that the levels correspond to independently measured difficulty or capability requirements.","section":"Section 3.4, Appendix C"},{"comment":"The human performance baseline of 71.9% is a load-bearing comparison for the paper's conclusion that agents remain far from practical deployment, but no protocol is provided for how it was obtained. The authors should report the number of human participants, the instructions they received, whether they used the same screenshot-only VM interface, and how edge cases or disagreements were handled. Without this information, the human baseline is not reproducible and the size of the human-agent gap cannot be assessed.","section":"Table 3 and Section 4.2"},{"comment":"The per-level sample sizes are small, and the headline cells are correspondingly noisy. With L1 containing 25 tasks and L4 containing 42 tasks, the UI-TARS-72B results of 48% at L1 and 0% at L4 correspond to 12 successes out of 25 and 0 out of 42. Confidence intervals or bootstrap estimates should be reported so that the fine-grained ordering among agents is not over-interpreted, particularly because several near-zero percentages differ by only one or two successful tasks across the evaluated systems.","section":"Table 3, Section 4.2"}],"minor_comments":[{"comment":"The system prompt examples use the action type MOUSE_MOVE, while Table 4 defines the action type MOVE_TO. If these are aliases, that should be stated; otherwise the prompt and the action-space definition need to be aligned.","section":"Appendix E.1, Table 4"},{"comment":"The paper calls the taxonomy a five-level automation taxonomy, but OS-MAP only instantiates L1 through L4 and defers L5 to future work. The main text should state this limitation more prominently when the taxonomy is introduced, not only in the L5 paragraph.","section":"Abstract and Section 3.1"},{"comment":"The average instruction length is reported as 34.3 words without any dispersion measure or per-level breakdown. Reporting standard deviations and per-level averages would help readers evaluate the confound between instruction detail and automation level.","section":"Section 3.5, Table 1"},{"comment":"The CALL_USER action is described as delegating control to a simulated human supervisor, but the appendix then says that during Google account login a script autofills the credentials. Clarify whether CALL_USER invokes a human, a script, or both, since this affects the interpretation of human-in-the-loop evaluation.","section":"Appendix B.5"},{"comment":"The benchmark name is rendered inconsistently as OS-MAP, OS-M AP, and OS-MAp across the title, abstract, tables, and references. A single consistent spelling should be used.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The core benchmark resource is plausible and likely useful to the community, and the aggregate failure result is probably robust. The main risks are the unreported label agreement, the confound between automation level and task horizon/instruction detail, and the internal count inconsistencies, all of which are fixable. The self-citations to OSWorld and OS-Copilot are infrastructure references and do not raise concerns."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about OS-MAP. First, the benchmark is a real addition to the GUI-agent space: 416 executable tasks across 15 desktop applications, organized by a five-level automation taxonomy and a demand-hierarchy generalization scope, with human performance measured rather than assumed. The headline result — best agent at 11.4% overall — is probably directionally right. Second, the paper's central discovery, the stepwise decline across L1–L4, is not yet identifiable as a capability finding. Higher levels are deliberately more underspecified, include misleading cues, and get larger step budgets (15/15/30/50). Success is a single binary outcome after a full trajectory, so even a fixed per-step reliability q gives expected success q^N. With only 25 L1 tasks and 42 L4 tasks, the 48% to 14% to 1% to 0% pattern is consistent with horizon alone. No inter-annotator agreement is reported for the level labels, which makes the attribution to perception, planning, adaptation, and orchestration hard to separate from task length and instruction detail.\n\nWhat is genuinely new is the two-dimensional framing itself: combining a SAE-style automation ladder with a user-demand hierarchy into one matrix is a useful organizing idea, and the suite is non-trivial to build. The failure analysis is concrete and informative — the theater-mode order swap and the Zotero citation case show real bottlenecks. The environment is executable, released, and built on OSWorld infrastructure, which counts for a lot.\n\nThe soft spots are fixable but real. Table 1's single-app and multi-app counts sum to 437, not 416, and Section 3.5 says 138 authored and 296 OSWorld-derived tasks while Table 1 says 161 and 255. Those mechanical inconsistencies need to be reconciled. The \"first\" claim is overstated given AndroidArena and ScienceBoard already include difficulty levels and structured evaluation. Repeated runs and confidence intervals are missing, and the human baseline procedure is undocumented. None of this sinks the benchmark; the aggregate 11.4% result is likely robust.\n\nFor the level-wise bottleneck claim to carry weight, the authors need to report label agreement, control for task horizon and step budget, and show the decline persists when those are held roughly fixed — or else reframe the levels as descriptive difficulty tiers rather than distinct capability deficits.\n\nThis paper deserves a serious referee. I would accept it for peer review and ask for major revision focused on the metadata consistency and the confound analysis. For anyone working on computer-using agents, the benchmark and its task set are worth citing; the interpretative claims need caution.","headline":"OS-MAP is a genuinely useful benchmark with a likely-robust headline result, but the level-wise capability story is confounded with task length and step budget and needs rework before it can carry the paper's conclusions.","tokens_in":792,"tokens_out":1791,"would_cite":true,"duration_ms":38765,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"State-of-the-art computer-using agents solve only 11.4 percent of a new 416-task desktop benchmark, and nearly none of the tasks that demand adaptation or orchestration.","keywords":["computer-using agents","GUI agents","benchmark","automation levels","generalization scope","evaluation matrix","desktop automation","vision-language models"],"falsifier":"Re-run the 416 tasks with the L3 and L4 instructions rewritten as explicit step-by-step click sequences while keeping the underlying operations identical; if success rates jump to L1/L2 levels, the automation ladder is measuring instruction ambiguity rather than agent capability. In parallel, have a second annotator team independently relabel all tasks and report agreement on level assignments.","tokens_in":24929,"feed_emoji":"🖥️","tokens_out":5178,"duration_ms":44916,"temperature":0.7,"pith_summary":"This paper claims that current computer-using agents fail much harder than existing benchmarks suggest, and that the reason is not only difficulty but a mismatch between task structure and the capabilities agents actually have. To make this visible, it introduces OS-MAP, a benchmark of 416 realistic desktop tasks ranked on a five-level automation scale (from simple execution to full orchestration) and sorted by how much generalization each task demands. On this benchmark the strongest available agent, UI-TARS-72B, completes only 11.4% of tasks overall, and almost none of the higher-level tasks that require adapting to surprises or coordinating several applications. The paper argues that such a structured evaluation matrix, rather than a flat collection of tasks, is what the field needs to identify which capability is missing at each level and to guide progress toward reliable deployment.","feed_headline":"Best desktop agents score just 11.4% on 416 everyday tasks","feed_subtitle":"Even the top GUI agent fails nearly all tasks that need adaptation or cross-app orchestration.","key_machinery":"The central object is the OS-MAP evaluation matrix, which organizes tasks along a five-level automation taxonomy (L1 reactive execution, L2 deterministic planning, L3 adaptive behavior, L4 global orchestration, and L5 proactive companionship, modeled on the SAE driving-automation scale) crossed with three generalization scopes (S1 narrow, S2 domain-level, S3 general) derived from a demand hierarchy over six real-world usage domains. This matrix is what carries the argument: by decoupling autonomy depth from breadth, it turns a pool of 416 tasks into a diagnostic instrument that attributes failures to specific capability gaps, and it defines an upper-right high-value region that no current agent reaches.","core_discovery":"The paper's central claim is that state-of-the-art computer-using agents remain far from practical deployment, and that this gap is best understood along two independent dimensions: how much autonomy a task requires and how broadly a task generalizes across real user demand. Evaluating ten agent configurations on 416 tasks spanning 15 desktop applications, the paper finds the strongest system reaches an 11.4% overall success rate, with a stepwise decline across its automation levels: 48.0% on level-1 execution tasks, 14.0% on level-2 planning, 1.0% on level-3 adaptation, and 0.0% on level-4 orchestration, against human performance of about 71.9% overall. The failure analysis attributes the decline to distinct bottlenecks at each level, such as grounding non-textual UI elements, avoiding similar but incorrect options, recovering with fallback strategies when the obvious action fails, and decomposing long workflows with correct dependency ordering, rather than to a single missing ingredient.","pith_inferences":["If the automation-level labels mainly track instruction vagueness and task length rather than distinct cognitive capabilities, the reported stepwise decline could shrink substantially when the same underlying steps are re-issued as concrete click-by-click instructions.","The mix of single-app versus multi-app tasks positions OS-MAP to quantify the jump from within-application competence to cross-application coordination, which may turn out to be the real threshold for everyday usefulness.","A natural extension is to reuse the taxonomy on other surfaces such as mobile, web, and terminal agents, giving the community a common vocabulary for automation levels across platforms.","The paper reports no inter-annotator agreement on level assignments, so a simple relabeling study by an independent team would determine whether the ladder measures the agent or measures the wording of the prompt."],"forward_implications":["Flat-list benchmark scores overstate agent readiness: the 11.4% ceiling on OS-MAP, with near-zero performance on adaptive and orchestration tasks, is the number that any practical deployment claim must beat.","The field gets a targeted roadmap: grounding and basic planning are partially solved, while fallback behavior under deviation (L3) and task decomposition with context switching (L4) are the binding constraints.","Open-source GUI-trained models such as UI-TARS-72B can match or beat much larger proprietary general VLMs for end-to-end computer use, so scaling general models is not the only route forward.","The new CALL_USER action makes human-in-the-loop evaluation possible, allowing benchmark tasks to include realistic permission boundaries such as login walls.","Because the matrix is extensible, new tasks and scenarios can be added to underexplored cells, and L5 proactive behavior is staked out as the next unmeasured frontier."],"supporting_citations":[{"why":"Supplies the executable VM environment, the Computer_13 action space, and the 255 tasks that OS-MAP relabels and filters.","marker":"[21]"},{"why":"The SAE driving-automation taxonomy whose human-agent division of labor inspires the five automation levels.","marker":"[24]"},{"why":"Industry usage data that grounds the six-domain demand hierarchy for daily computer use.","marker":"[33]"},{"why":"Survey data from the OECD ICT Access and Usage Database used to derive concrete scenarios within each domain.","marker":"[34]"},{"why":"The performance-generality framework that motivates the two-dimensional evaluation matrix.","marker":"[35]"},{"why":"UI-TARS-72B, the GUI-specific agent that achieves the best overall score of 11.4%.","marker":"[6]"},{"why":"GPT-4o, the proprietary planning model used in the planning-grounding baseline configuration.","marker":"[45]"}],"fun_headline_variants":["Top GUI agents fail 89% of everyday desktop tasks","Agent success drops from 48% to 0% as tasks get complex","0% on orchestration: agents can't handle complex workflows","Best agent hits 11.4% on 416 tasks; humans hit 71.9%","From 48% to 0%: the automation cliff for desktop agents"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire hierarchy rests on the assumption that the L1-L4 labels assigned by annotators measure distinct capabilities such as perception, planning, adaptation, and orchestration, rather than just longer instructions, more steps, or vaguer wording.","fun_headline_variants_meta":{"raw":{"variants":["Top GUI agents fail 89% of everyday desktop tasks","Agent success drops from 48% to 0% as tasks get complex","0% on orchestration: agents can't handle complex workflows","Best agent hits 11.4% on 416 tasks; humans hit 71.9%","From 48% to 0%: the automation cliff for desktop agents"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000645,"raw_usage":{"total_tokens":2990,"prompt_tokens":996,"completion_tokens":1994,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":612,"completion_tokens_details":{"reasoning_tokens":1905}},"tokens_in":612,"tokens_out":1994,"duration_ms":13906,"temperature":1.0,"reasoning_tokens":1905,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:00:22.277567+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the 416 tasks with the L3 and L4 instructions rewritten as explicit step-by-step click sequences while keeping the underlying operations identical; if success rates jump to L1/L2 levels, the automation ladder is measuring instruction ambiguity rather than agent capability. In parallel, have a second annotator team independently relabel all tasks and report agreement on level assignments.","supporting_citations":[],"review_version":2}