{"id":"62ad87bd-11f6-42e0-8ed4-af6b4dfae2d5","arxiv_id":"2606.20785","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Fara1.5 open-weight agents, trained on a new synthetic-plus-live data pipeline, set size-class state-of-the-art results on WebVoyager and Online-Mind2Web.","lead":"Microsoft researchers built a data-generation pipeline that trains computer-use agents on live websites plus synthetic copies of login-gated services, and used it to train three open-weight agents (4B/9B/27B) that set new size-class records on browser benchmarks. The largest model reaches 72.3% on Online-Mind2Web, competitive with much larger proprietary systems.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No decontamination analysis between FaraGen1.5 training data and benchmark tasks; potential task overlap could inflate the headline SoTA claims.","rationale":"The reader's weakest assumption was the reliability of the LLM verifiers. That is a real concern, but it primarily affects the validity of the training-data pipeline, not the direct truth of the benchmark scores: the benchmarks are scored by official judges (WebJudge, GPT-4o judge) that are independent of the training verifiers. Even if the training verifiers admit some noise, the model could still achieve high benchmark scores; the size-class SoTA claim would remain intact. By contrast, contamination between training data and evaluation tasks would directly invalidate the benchmark numbers, which are the central evidence for the SoTA claim. The paper's training mix is 60% live-web trajectories generated by an LLM task proposer on popular websites, and the evaluation runs on live versions of the same kind of sites. No decontamination analysis is reported anywhere in the manuscript or appendices. The paper does demonstrate contamination awareness for synthetic environments by deduplicating training and test tasks, but this discipline is not extended to the live benchmarks. The synthetic-to-real transfer experiment (Table 5) provides partial evidence that the model generalizes, but it covers only four WebVoyager domains and does not test Online-Mind2Web, where the largest gains are claimed. A concrete embedding-based overlap check would settle the question. Because the reader already flagged the absence of contamination analysis as a secondary concern, and the overall CONDITIONAL verdict remains appropriate, I recommend no change to the verdict; however, I would elevate contamination above verifier noise as the most load-bearing uncertainty.","tokens_in":32493,"tokens_out":7723,"duration_ms":74647,"concrete_test":"Compute semantic similarity (e.g., embedding cosine) between every FaraGen1.5 training task prompt (or a sample of the task-proposal LLM's generated tasks) and every Online-Mind2Web / WebVoyager task prompt; also check for URL overlap between training trajectories and benchmark task sites. Flag any near-duplicate pairs (e.g., cosine > 0.9 or same site+intent). Then re-evaluate Fara1.5-9B and Fara1.5-27B on the subset of benchmark tasks with no near-duplicate training example. If the scores drop by more than the reported 95% confidence intervals (Table 7), the SoTA claim is inflated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Fara1.5's headline benchmark numbers (Online-Mind2Web 63.4/72.3, WebVoyager 86.6/89.3) are the core evidence for the SoTA claim. The training data is dominated by live-web trajectories (60.0%) sourced from a large index of public websites and generated by an LLM task proposer, and the evaluation benchmarks are executed on live websites with tasks refreshed from the same kind of sites. The paper reports no decontamination analysis: no comparison of FaraGen1.5 training task prompts (or the task-proposal LLM's outputs) against the 300 Online-Mind2Web and 800 WebVoyager task definitions, no exclusion of benchmark tasks or near-duplicates from training, and no date/version tracking to rule out temporal overlap. Given that the task-proposal LLM is instructed to generate plausible tasks for specific sites, it is plausible that some training trajectories correspond to benchmark tasks or semantically equivalent variants. If so, the model could be partially memorizing the evaluation, and the size-class SoTA gap would be inflated. The synthetic-to-real transfer experiment (Table 5) mitigates this for four WebVoyager domains, but does not cover Online-Mind2Web, and the control baseline is trained on a small amount of data from unrelated domains, not a matched task set. The paper's own deduplication of synthetic training/test tasks (Section 2.1.2) shows awareness of overlap risk, but this discipline is not applied to the benchmarks.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces FaraGen1.5, a data-generation pipeline for computer-use agents, combining live-web and synthetic sandboxed environments, a GPT-5.4-based solver with a user simulator, and three LLM-based verifiers (correctness, efficiency, critical-point adherence). Using the resulting trajectories plus auxiliary grounding/VQA/safety data, the authors SFT-tune Qwen3.5 backbones at 4B/9B/27B to obtain the Fara1.5 family. They report state-of-the-art results among 7–9B open-weight models on Online-Mind2Web (63.4%) and WebVoyager (86.6%), with the 27B model reaching 72.3% on Online-Mind2Web and competitive with larger proprietary systems. They also report synthetic-environment transfer, grounding, and safety results, and release model weights under an MIT license.","tokens_in":32837,"tokens_out":6218,"duration_ms":53269,"significance":"If the headline results are reliable, this is a significant contribution: an open, MIT-licensed family of native pixel-action CUA models that closes much of the gap to proprietary systems, plus a reusable pipeline for synthetic trajectory generation. Strengths include three-run averages with 95% CIs, use of official external judges for WebVoyager and Online-Mind2Web, programmatic state verification in synthetic environments, a synthetic-to-real transfer experiment, and a detailed error taxonomy. The main uncertainties are contamination of the live-web training data with benchmark tasks, reliability of the LLM-based verifiers, modifications to WebTailBench, and comparability of leaderboard-sourced baselines.","major_comments":[{"comment":"No contamination analysis is reported between the FaraGen1.5 live-web training data and the WebVoyager/Online-Mind2Web benchmarks. Since 60% of the training mix consists of tasks generated for live sites from a large public URL index, and the benchmark tasks are also live-web tasks on overlapping sites, near-duplicates of benchmark tasks could be in the training set. The paper's own train/test deduplication for synthetic tasks (§2.1.2) shows awareness of this risk, but the same discipline is not applied to the benchmarks. Table 5 mitigates the concern for four WebVoyager domains, but not for Online-Mind2Web, and the control uses unrelated domains. Please report a task-level overlap analysis (exact and paraphrase/embedding-based) between FaraGen1.5 task proposals and the 300 Online-Mind2Web and 800 WebVoyager tasks, and re-evaluate on a contamination-free subset.","section":"§2.1.1, §3.2, §4.1.1"},{"comment":"WebTailBench v1.5 is a modified, in-house benchmark: 270 tasks are refreshed and the success criterion is changed so that stopping at a critical point counts as success when the task requires information not provided; the eval harness injects a canned reply to ask_user_question and the rubric judge is aware of this. These changes are disclosed but make scores non-comparable to the original WebTailBench and to baselines that do not have ask_user_question in their tool lists. Because the same type of LLM rubric (Rosset et al., 2026) is used both for training-data filtering and for this evaluation, the WebTailBench v1.5 results risk circularity. Please provide human-agreement or inter-judge reliability for the modified rubric and release the exact task list and judge prompts.","section":"§4.1.1, Table 3"},{"comment":"The entire data pipeline is gated by three verifiers, but no accuracy or calibration evidence for them is provided. The paper acknowledges that 'LLMs as judges may hallucinate' (Introduction), yet the correctness threshold (rubric process score ≥ 0.8) and efficiency threshold (rating ≥ 4) are asserted without validation. The critical-point classifier is itself an LLM judge based on the task prompt and screenshots (§2.3). If the verifiers have systematic blind spots, the SFT corpus inherits those errors, and WebTailBench v1.5, which uses a related rubric, inherits them as well. Please report verifier precision/recall against a human-annotated trajectory sample, including per-category breakdowns for the eight critical-point types.","section":"§2.3 and Introduction"},{"comment":"The cross-system comparison to larger proprietary agents relies on leaderboard numbers retrieved at different dates (e.g., OpenAI Operator May 2025, Gemini 2.5 CU Sep 2025, Yutori Navigator Nov 2025 in Figure 7), whereas WebVoyager and Online-Mind2Web are live benchmarks with refreshed time-sensitive tasks. Without re-running these baselines under the same protocol, the claim that Fara1.5-27B 'outperforms much larger proprietary systems' is not fully established. Either re-run at least the strongest competitor under the current protocol or soften the claim to an explicitly labeled reference comparison.","section":"Table 3, §4.1.2"}],"minor_comments":[{"comment":"Typos and grammar: 'We have ran' -> 'We have run'; 'these ratios have been determine' -> 'determined'; 'auxillary' -> 'auxiliary'; 'insturction' -> 'instruction'.","section":"§2.4, §3.2, §A.8"},{"comment":"The Fara1.5-27B row is missing spaces: '89.3 72.372.9 40.2'. Please fix formatting.","section":"Table 3"},{"comment":"The text refers to a 'FaraBench v1.5 ensemble judge', but the benchmark is called WebTailBench v1.5 elsewhere. Unify the naming.","section":"Appendix A.6"},{"comment":"The pass@k analysis is described without defining how pass@k is estimated from the three runs. Add a one-sentence definition.","section":"Appendix A.2"},{"comment":"The choice of three image context windows is stated to be best, but no ablation is shown. Either include the ablation or clearly mark this as a tuning choice with reference to appendix material.","section":"§3.1"}],"recommendation":"major_revision","confidential_remarks":"The two issues that most affect the headline claims are (1) the absence of any decontamination analysis between FaraGen1.5 live-web training tasks and the Online-Mind2Web/WebVoyager evaluation tasks, and (2) the modified, in-house WebTailBench v1.5 whose success criterion and judge are not independently validated. Both are addressable in a revision: a task-overlap study and a verifier-accuracy study would substantially de-risk the SoTA claims. I would not reject the paper if these analyses can be added, but the current version overstates the cross-system comparisons."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: this is a real advance, not a rehash. The FaraEnvs synthetic environments and the user-simulator loop are genuinely new pieces, and the three-verifier stack with the eight critical-point types is a thoughtful attempt at quality control. Releasing weights under MIT and reporting three-run averages with CIs is more than most papers at this level do.\n\nThe headline numbers are striking: the 9B model at 63.4% on Online-Mind2Web nearly doubles Fara-7B, and the 27B reaches 72.3%, beating some proprietary systems. Those two benchmarks use official external judges, so they aren't purely self-reported. The synthetic-to-real transfer experiment (Table 5) is a good check: training on replicas of four WebVoyager domains improves live performance by ~10 points, which supports the claim that synthetic environments are useful beyond the sandbox.\n\nNow the soft spots, in order of weight.\n\nFirst, the decontamination concern is real and unaddressed. Training data is 60% live-web trajectories generated by an LLM task proposer seeded from a web index. The benchmarks run on live tasks from similar sites. The paper reports no comparison between training task prompts and benchmark tasks, no exclusion of near-duplicates, and no temporal tracking. The stress-test note is right: this could inflate the size-class SoTA gap. It's not a demonstrated flaw—there's no evidence of actual overlap—but the absence of analysis is a genuine gap for a paper whose central claim is benchmark superiority.\n\nSecond, the verifiers are LLM judges. Correctness, efficiency, and critical-point adherence are all LLM-based. The paper admits in the introduction that LLMs as judges may hallucinate. For synthetic environments, backend state and sqldiff give stronger ground truth, but live-web verification rests on the Universal Verifier. If those judges systematically reward teacher-like behavior or miss subtle failures, the training data and the WebTailBench v1.5 results inherit the error.\n\nThird, WebTailBench v1.5 is a self-modified benchmark: they changed the success criterion so that stopping at a critical point counts as success when information is missing. That is defensible, but it makes comparisons to the original WebTailBench numbers less clean.\n\nThe paper is well-written, transparent about its choices, and the failure analysis in Appendix A.7 is thorough. This deserves peer review, but a referee should push for a decontamination analysis and a human audit of a sample of verifier decisions.\n\nBottom line: worth reading and discussing despite the caveats. I'd cite it for the pipeline, not yet for the absolute numbers.","headline":"Genuine advance in synthetic data generation for computer-use agents, with real contributions and released weights, but the headline SoTA claims rest on an unaddressed decontamination risk and a self-modified benchmark.","tokens_in":33419,"tokens_out":2194,"would_cite":true,"duration_ms":22010,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fara1.5 shows that small, open web agents can be trained from synthetic data and rival much larger proprietary systems on browser-use benchmarks.","keywords":["computer use agents","synthetic data generation","web browsing agents","supervised fine-tuning","LLM verifiers","critical-point handling","GUI grounding","browser automation"],"falsifier":"Take a random sample of accepted and rejected trajectories from FaraGen1.5 and have human annotators label them for task success, efficiency, and critical-point compliance. If human-verifier agreement is low, or if a model trained on the filtered set performs no better than one trained on the unfiltered set, the verifier stack is not the source of the reported gains.","tokens_in":32362,"feed_emoji":"🤖","tokens_out":2986,"duration_ms":29622,"temperature":0.7,"pith_summary":"The paper claims that a fully automated data-generation pipeline—live websites plus sandboxed synthetic replicas, solved by a frontier teacher model, and filtered by three LLM-based verifiers—can replace expensive human demonstrations as the main training signal for computer-use agents. Training on this data yields the Fara1.5 family (4B, 9B, 27B), which sets a new state of the art for its size class: Fara1.5-9B reaches 63.4% on Online-Mind2Web and 86.6% on WebVoyager, and Fara1.5-27B reaches 72.3% on Online-Mind2Web, competitive with much larger proprietary systems. If true, capable web automation no longer requires massive proprietary models or costly human data collection. The paper also reframes critical-point behavior: instead of always stopping before irreversible actions, the agent is trained to ask for user permission and then complete the action, which improves both safety and task completion.","feed_headline":"9B web agent tops its size class at 63.4% on Mind2Web","feed_subtitle":"Automated synthetic-data pipeline with LLM verifiers distills frontier-teacher skills into small open-weight agents.","key_machinery":"The core mechanism is the three-verifier filter applied to every generated trajectory. The correctness verifier uses an LLM-generated rubric (Universal Verifier) for live web tasks and backend-state comparison (sqldiff) or reference answers for synthetic environments. The efficiency verifier is an LLM judge that identifies redundant or looped actions and assigns a 1-5 rating. The critical-point verifier classifies each task into one of eight types based on whether the user granted permission, whether the task is fully specified, and whether all required PII is provided; it then checks that the solver asked the user at every critical point before proceeding. Only trajectories passing all thre","core_discovery":"The central claim is that a scalable, modular pipeline—FaraGen1.5—can generate high-quality computer-use trajectories at scale and distill them into small, native pixel-in/action-out agents that outperform prior open-weight agents of the same size. The pipeline's three phases are: (1) environments consisting of live websites and six synthetic sandboxed replicas (email, calendar, media, ML experiments, marketplace, scheduler) with full backend control and ground-truth verification; (2) a solver harness powered by GPT-5.4 that interacts with a user simulator to produce multi-turn trajectories; (3) three complementary verifiers—task correctness, efficiency, and critical-point adherence—that all","pith_inferences":["If the verifier reliability holds, the pipeline could be bootstrapped further: student-generated trajectories that pass the same verifiers could be used to improve the next generation of students, potentially reducing or eliminating the need for frontier teacher models.","The eight-type critical-point taxonomy and the associated ask-user-then-act training behavior could be adopted as a standalone safety curriculum for any computer-use agent, independent of the full FaraGen1.5 pipeline.","The WebTailBench v1.5 scoring change—counting a stop at a critical point as success when the task cannot legitimately be completed without missing information—makes its numbers not directly comparable to the original WebTailBench; readers tracking progress over time should treat the two as distinct benchmarks.","A testable extension would be applying the same synthetic-environment pipeline to desktop or terminal-based tasks, as the paper hints toward future work; the verifier design (state-based correctness, efficiency, and critical-point checks) would likely transfer with minimal changes."],"forward_implications":["Open-weight agents at 4B–27B parameters can achieve browser-task success rates that rival or exceed much larger proprietary systems, lowering deployment cost and hardware requirements.","Synthetic sandboxed environments can teach agents tasks that are impossible to collect on the open web—login flows, purchases, sending messages—and this learning transfers to real live websites, as shown by the synthetic-to-real experiments.","Training agents to pause at critical points and ask the user, rather than halting entirely, improves safety behavior (100% refusal on WebTailBench-Refusals) while also improving task completion.","The data mix recipe (85.3% from FaraGen1.5) plus targeted auxiliary grounding, VQA, and refusal data provides a repeatable template for training small computer-use agents.","The efficiency verifier and step-length analysis suggest that a substantial part of model quality is efficiency in taking fewer, more purposeful actions, which can be optimized independently of task correctness."],"fun_headline_variants":["Open-weight 9B agent tops size class on Mind2Web","Synthetic data pipeline distills frontier skills into 9B agent","9B agent matches proprietary systems on web navigation","Fara1.5: 27B agent hits 72.3% on Online-Mind2Web"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The three automated verifiers—especially the LLM-based correctness judge, efficiency judge, and critical-point classifier—reliably separate good trajectories from bad ones; if they systematically misjudge, the errors are baked into all downstream training data and benchmark results.","fun_headline_variants_meta":{"raw":{"variants":["Open-weight 9B agent tops size class on Mind2Web","Synthetic data pipeline distills frontier skills into 9B agent","9B agent matches proprietary systems on web navigation","Fara1.5: 27B agent hits 72.3% on Online-Mind2Web"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00095,"raw_usage":{"total_tokens":3962,"prompt_tokens":888,"completion_tokens":3074,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":632,"completion_tokens_details":{"reasoning_tokens":2994}},"tokens_in":632,"tokens_out":3074,"duration_ms":21253,"temperature":1.0,"reasoning_tokens":2994,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T10:43:19.317652+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of accepted and rejected trajectories from FaraGen1.5 and have human annotators label them for task success, efficiency, and critical-point compliance. If human-verifier agreement is low, or if a model trained on the filtered set performs no better than one trained on the unfiltered set, the verifier stack is not the source of the reported gains.","supporting_citations":[],"review_version":2}