{"id":"b27c60d1-8635-42eb-8c1b-ace92191f123","arxiv_id":"2511.12997","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"WebCoach, a retrieval-augmented coaching layer, improves WebVoyager task success for 32B and 38B web agents (e.g., 47% to 61%) by injecting advice from past trajectories, but not for a 7B agent.","lead":"The paper describes WebCoach, a memory layer that lets web-browsing agents store past navigation logs and receive targeted advice from a coach model to avoid repeating mistakes. It reports success-rate gains on the WebVoyager benchmark for two of three tested model backbones, with the largest jump from 47% to 61% on a 38B model.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 14-point success gain rests on self-evaluated completion labels (Sec. 3.1) with no human verification; the same agent that performs the task certifies success, so the headline improvement may reflect measurement bias rather than true task completion.","rationale":"The reader's weakest assumption was the in-benchmark memory distribution: dynamic EMS is built from other WebVoyager tasks on the same domains and templates, and Section 3.2 excludes only same task IDs. That is a real threat to the 'self-evolving' generalization claim, but the headline quantitative claim—47%→61% on WebVoyager—could survive if the measurements are valid; the memory objection mainly limits external validity. The more fundamental blocker is that the success labels themselves are produced by the agent being evaluated. If those labels are inflated in a condition-dependent way, the central result is unsupported regardless of memory composition. The paper explicitly relies on browser-use's evaluation capability and never validates it against human judgments, so a human re-verification is the decisive check. I therefore keep the CONDITIONAL verdict unchanged, but for a more fundamental reason than the reader's stated weakest assumption; hence partial agreement.","tokens_in":17888,"tokens_out":8678,"duration_ms":88250,"concrete_test":"Run a blind human re-evaluation of a stratified sample: select 100 WebVoyager tasks from each of the Skywork-38B baseline and Skywork-38B dynamic-WebCoach conditions, archive final browser states/screenshots and full trajectories, and have two independent annotators score success per the original WebVoyager rubric. Compare human-verified success rates with the reported automated labels. If the human-verified WebCoach advantage is not substantially positive (e.g., <5 points) or disagreement between automated and human labels is condition-dependent, the headline claim is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—38B success from 47% to 61%—depends on the success labels being true task completion. Section 3.1 states: 'we utilize browser-use agent's evaluation capability by checking the resulting state of their last action against the initial user query, and determine the success of the task.' This is an LLM-based self-evaluation, not an external/human judge, and the paper provides no validation against WebVoyager's human rubric. Since WebCoach changes the agent's behavior and injects task-specific advice, the same model that performs the task also certifies completion; coached agents may be more likely to emit a confident 'success: true' after following advice (e.g., the Appendix trace ends with the agent's own 'success: true' declaration). Without evidence that false-positive rates are equal across conditions, the 14-point gain could be a measurement artifact. No confidence intervals or significance tests further mean the 7B regression and model differences are within unknown noise. The success signal is the least secured condition for the headline claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces WebCoach, a model-agnostic memory-and-coaching layer for web-navigation agents. WebCoach uses a WebCondenser to compress trajectories, an external memory store (EMS) for retrieval, and a Coach LLM that injects advice into the actor's prompt. The authors evaluate on all 643 WebVoyager tasks with three open-weight base models (Qwen2.5-VL-7B, Qwen2.5-VL-32B, Skywork-r1v3-38B) and report that WebCoach improves success rates, e.g., 47.3%→61.4% for Skywork-38B, while roughly maintaining step counts. They also claim that a self-expanding memory store outperforms a frozen memory seeded with GPT-4o trajectories, and that smaller models with WebCoach approach GPT-4o-level performance.","tokens_in":18090,"tokens_out":4786,"duration_ms":47228,"significance":"If the results are robust, the paper makes a practically useful contribution: a lightweight, model-agnostic way to give web agents persistent cross-session memory without retraining. The full-benchmark evaluation on live websites, the per-subtask breakdowns, and the release of code are strengths. However, the headline claims rest on evaluation choices that need substantially more support: success labels come from the agent itself, the dynamic-memory evaluation draws on the same benchmark distribution it is tested on, and no uncertainty quantification is provided. These issues are load-bearing for the paper's central claims, so the manuscript currently overstates its evidence.","major_comments":[{"comment":"The success signal used for the headline metric is the browser-use agent's own completion check ('we utilize browser-use agent's evaluation capability by checking the resulting state of their last action against the initial user query'). No human validation or external judge is described. Since WebCoach changes the agent's behavior and injects task-specific advice, the same model that performs the task also certifies success; the Appendix trace shows the main agent returning 'success: true' directly. If coached agents are more likely to declare success, the reported gain could be partly a measurement artifact. The authors should validate a random subset against human/external labels and, ideally, report false-positive rates per condition.","section":"§3.1, §4, Appendix A"},{"comment":"The dynamic self-experience evaluation populates the EMS from WebVoyager tasks on the same 15 websites and similar task templates, and retrieval excludes only the same task ID. This means the agent is evaluated on a task distribution that is represented in its memory store; the later tasks in the sequence can retrieve memories of near-identical tasks from the same domain. The reported 14-point Skywork gain therefore conflates genuine cross-session learning with in-benchmark distribution leakage. A cleaner evaluation would hold out entire domains or task templates, or seed the memory exclusively from outside the evaluation benchmark, and would report how the gain changes as memory is withheld.","section":"§3.2, 'Memory Retrieval and Leakage Control', Tables 1 and 5"},{"comment":"The abstract says WebCoach 'consistently improves the performance of browser-use agents across three different LLM backbones,' but the results show Qwen-VL-7B at 0.328 baseline vs. 0.311 with dynamic self-experience WebCoach (Table 1). The text in §4 acknowledges this ('the 7B backbone does not benefit'), but the abstract and introductory framing are not corrected. This is a direct contradiction of a core claim and must be fixed. The framing should be scale-dependent, not 'consistent.'","section":"Abstract, §4, Table 1"},{"comment":"No error bars, confidence intervals, or significance tests are reported anywhere, despite 643-task runs. For example, Qwen-VL-32B goes from 319/643 to 367/643, a difference of 48 successes; the 7B regression is 21 successes. Without at least binomial confidence intervals or a paired test, it is impossible to tell which of these differences are beyond run-to-run noise. The paper's central quantitative claims need this reporting. Multiple seeds, or a bootstrap, would strengthen the conclusions considerably.","section":"Table 1, Tables 3–5"},{"comment":"The baseline success rate for Qwen-VL-7B is inconsistent: Table 2 shows 0.344 (221/643), while Table 1 and Figure 4 show 0.328. Since the 7B result is already a key negative case, this discrepancy must be resolved. The corrected value affects the interpretation of the 'no benefit' conclusion.","section":"Table 2 vs. Table 1 / Figure 4"}],"minor_comments":[{"comment":"The WebCondenser and Coach are said to use 'a small LLM (≤8B)' and 'an 8B LLM,' but no specific model, prompt template, or temperature is given. For reproducibility, these details should be provided.","section":"§2.1, §2.3"},{"comment":"The legend labels are confusing: 'WebCoach (GPT-4o)', 'WebCoach (Qwen3-8b)', 'WebCoach with Self-exp', and 'GPT-4o baseline' do not map clearly to the table columns. Clarify which color/hatch corresponds to each condition.","section":"Figure 4"},{"comment":"The sentence 'WebVoyager complements this by enabling large-scale crawling and evaluation' seems to misstate the benchmark's role; WebVoyager is the benchmark, not a crawling tool. Rephrase.","section":"§3.1"},{"comment":"The claim that self-generated embeddings result in 'smoother reasoning continuity' is not supported by quantitative evidence; if this is a hypothesis, label it as such, or provide an analysis (e.g., embedding distances or failure-mode comparisons).","section":"§4, 'Effect of memory source'"},{"comment":"The retrieval latency is measured for only 600 trajectories. The claim that HNSW supports 'millions of episodes' is extrapolation; this is fine as a design note, but should not be stated as an empirical result.","section":"§2.2, Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The paper is an empirical systems contribution; the architectural ideas are reasonable and the full-benchmark evaluation is a strength. However, the reliance on self-labeled success, the in-benchmark memory protocol, and the lack of uncertainty quantification make the headline 'consistent improvement' unsupported. The authors should be asked to address these issues before resubmission. I see no evidence of misconduct; the issues are methodological rather than intentional."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The 14-point success gain for Skywork-38B is the thing to know about this paper. It is a plausible result, but you should not read it as established: the success signal comes from the agent itself, not from WebVoyager's human rubric, and there are no uncertainty estimates anywhere. That combination means the headline number could be inflated.\n\nWhat the paper does well is straightforward. WebCoach is a clean, model-agnostic wrapper: a condenser normalizes logs, a vector store holds completed episode summaries, and a coach retrieves top-k memories and decides when to inject advice into the base agent's system prompt. That is a reasonable extension of reflexion-style memory work. The evaluation has real scope—all 643 WebVoyager tasks, live browser, three open-source backbones and a GPT-4o ceiling—and the per-subtask tables in the appendix let you see where the gains come from. The result that self-generated memory beats externally seeded memory is the most interesting finding. Steps per task stay flat or drop, which suggests the gains are not from extra exploration. Code is available.\n\nThe soft spots are proportionate to the claims. The self-evaluation issue is the biggest. Section 3.1 says the browser-use agent checks the final state against the query to decide success, and the appendix trace shows the agent simply outputting 'success: true.' There is no spot-check against human labels, so the coached condition and the baseline could have different false-positive rates. The 14-point gain could be partly a measurement artifact. The paper needs a human-judged subset or an independent judge applied to both conditions.\n\nSecond, the 'consistently improves' claim in the abstract is contradicted by the 7B results in Table 1, where success drops from 0.328 to 0.288–0.311 in every WebCoach condition. The paper does acknowledge this in Section 4, so the authors are not hiding it, but the abstract still overstates. With no confidence intervals or significance tests, the close differences between conditions could be run-to-run noise.\n\nThird, leakage control excludes only the same task ID. Retrieval can still surface memories from the same domain with nearly identical task templates, so the dynamic memory condition is partly memorizing the benchmark. A held-out memory split or cross-domain test would be needed to show the gains generalize.\n\nFourth, there is no comparison against existing memory baselines like Reflexion, Mem0, or A-Mem, so the contribution beyond the specific combination is hard to calibrate.\n\nWho gets value from this: people building low-cost memory layers on top of existing web agents. The architecture is easy to understand and reuse. This deserves a serious referee, but the evaluation needs substantial revision before the central claim is credible.","headline":"Plausible but not established: the 14-point gain rests on self-reported success and no error bars, so the paper merits refereeing, not citation yet.","tokens_in":18670,"tokens_out":3482,"would_cite":false,"duration_ms":31213,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"WebCoach, a memory-and-coach layer wrapped around any web agent, lifts task success for a 38B model from 47% to 61% on live web tasks without retraining, while keeping step counts flat.","keywords":["web agents","cross-session memory","self-evolving agents","episodic memory","retrieval-augmented coaching","continual learning","web navigation","external memory store"],"falsifier":"Run WebCoach in its dynamic self-experience mode on a held-out set of websites and task templates that never appear in the External Memory Store, and compare success against the no-memory baseline; if the gain over baseline shrinks to near zero, the reported improvement is largely in-benchmark leakage rather than transferable cross-session learning.","tokens_in":17720,"feed_emoji":"🧠","tokens_out":5348,"duration_ms":42950,"temperature":0.7,"pith_summary":"This paper argues that web agents do not need to be retrained to get better at browsing: they can improve by remembering and being advised by their own past episodes. It introduces WebCoach, a plug-in layer of three modules — a condenser that summarizes navigation logs, an external store that indexes those summaries, and a coach that retrieves relevant memories and injects short advice mid-task. On the live WebVoyager benchmark, pairing WebCoach with a 38B open-source model raises success from 47% to 61% while holding or reducing the number of steps, and lifts a 32B model to near-GPT-4o level. The central claim is that experience-driven memory, not scale or retraining, is what closes much of the gap between open-source and proprietary web agents.","feed_headline":"Memory layer lifts web-agent success from 47% to 61%","feed_subtitle":"WebCoach adds cross-session memory and on-the-fly coaching, closing most of the gap to proprietary agents without retraining.","key_machinery":"The central mechanism is a three-part memory loop operating outside the agent's policy: (1) WebCondenser, a small LLM that turns raw (observation, action, reward) traces into a fixed schema of summary text, an embedding, and success/failure metadata; (2) External Memory Store, a vector database with approximate nearest-neighbor search that stores only completed episodes and retrieves top-K experiences by cosine similarity and recency; (3) Coach, an 8B LLM that, given the current partial-trace summary and K=5 retrieved memories, returns a JSON decision to either stay silent or inject one-to-two sentences of advice as a system message before the agent's next action.","core_discovery":"The paper's central claim is that a lightweight, model-agnostic wrapper can give web navigation agents persistent episodic memory and selectively inject advice at runtime, producing continual self-improvement without any parameter update. WebCoach standardizes raw navigation traces into concise summaries (WebCondenser), stores completed episodes in a vector memory indexed by embedding similarity and recency (External Memory Store), and calls a small 8B LLM (Coach) that decides whether to interrupt the agent with task-specific guidance such as 'avoid this link, previous agents got stuck in a loop here.' The authors report consistent gains across three open-source backbones, with the largest j","pith_inferences":["A natural extension is to apply the same condenser-store-coach loop to other interactive agents—mobile UIs, desktop automation, or robotics—wherever raw action traces can be standardized; the paper's abstractions are domain-agnostic.","The reported 14-point gain may overstate real-world self-evolution because the memory is populated from the same benchmark's tasks, and only same-task-ID memories are excluded; trajectories from the very websites the agent is about to visit can be retrieved, so part of the gain could come from in-benchmark leakage rather than transferable learning.","Because retrieval uses summary embeddings, the framework inherits the condenser LLM's summarization quality and the embedding model's semantic coverage; failures in either would degrade advice even if the store and coach are sound.","Average completion time rises substantially (e.g., roughly 150 seconds added) due to Condenser and Coach inference, though step counts drop; a latency-aware analysis could determine whether the throughput trade-off is acceptable in production."],"forward_implications":["Any web agent, regardless of backbone, could improve over time by accumulating and consulting its own completed trajectories, meaning continual learning can be achieved without retraining.","Gains concentrate in semantically complex, multi-step sites (Apple, arXiv, BBC) while simpler click-heavy domains show little change, implying memory helps most where planning and element disambiguation matter.","Self-generated memory beats externally seeded memory, so agents learn most from trajectories produced by their own policy and representation space, not from demonstrations of a different model.","Provided the underlying agent is above a cognitive threshold (the 7B model fails, 32B/38B benefit), the coaching signal amplifies with agent scale; memory guidance pays off at the boundary of partial competence.","The 8B coach performs comparably to a much larger proprietary LLM as a coach, and a DPO fine-tune was deemed unnecessary, so the entire memory layer can run on small open-weight models."],"fun_headline_variants":["Cross-session memory boosts web agents from 47% to 61%","No retraining: memory wrapper lifts agent success to 61%","Memory-guided coaching yields 14-point web-agent gain","Web agents self-improve via shared episodic memory (47% → 61%)","Add memory, skip retraining: web agents jump to 61%"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The dynamic self-experience evaluation assumes that memories pulled from the same benchmark's other tasks—often on the same websites and task templates, since only same-task-ID examples are excluded—are a fair measure of self-evolution; if the memory is instead exploiting the benchmark's task distribution, the 14-point gain would overstate what a deployed agent would gain from its own past interactions.","fun_headline_variants_meta":{"raw":{"variants":["Cross-session memory boosts web agents from 47% to 61%","No retraining: memory wrapper lifts agent success to 61%","Memory-guided coaching yields 14-point web-agent gain","Web agents self-improve via shared episodic memory (47% → 61%)","Add memory, skip retraining: web agents jump to 61%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000698,"raw_usage":{"total_tokens":3017,"prompt_tokens":799,"completion_tokens":2218,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":543,"completion_tokens_details":{"reasoning_tokens":2138}},"tokens_in":543,"tokens_out":2218,"duration_ms":12771,"temperature":1.0,"reasoning_tokens":2138,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T21:54:42.527737+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run WebCoach in its dynamic self-experience mode on a held-out set of websites and task templates that never appear in the External Memory Store, and compare success against the no-memory baseline; if the gain over baseline shrinks to near zero, the reported improvement is largely in-benchmark leakage rather than transferable cross-session learning.","supporting_citations":[],"review_version":1}