{"id":"5c6b907b-4baf-4446-b8ff-11934b6845e0","arxiv_id":"2504.12682","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"BardeenAgent records a single extraction on a webpage, generalizes it with CSS selectors, and replays it across all matching items, reaching 66% recall on a new 200-task WebLists benchmark versus 31% for the best baseline.","lead":"An AI web agent that records its actions on the first item of a webpage list and then replays them as a program can extract job posts, blogs, and testimonials far more completely than existing web agents, at lower cost. The paper also introduces a 200-task live-website benchmark for structured data extraction.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline recall is computed by matching only one per-agent chosen column, so the 66.2% recall may not reflect schema-complete extraction and the comparison is not fixed.","rationale":"The reader's weakest assumption focused on DOM regularity, which is a real limitation of the record-and-replay design and is partly visible in the Job Categories row (64.5% recall vs. 86.8% for Jobs). But the paper already treats this as a known failure mode, and it does not directly threaten the headline comparison unless heterogeneous sites dominate the benchmark. The evaluation-protocol issue is more load-bearing because it is internal to the reported numbers: the task definition requires schema-bound rows, yet the metric counts one matching column as success and even varies that column per agent. Since the discussion explicitly states a stricter evaluation would lower recall, the 66.2% headline should be read as an upper bound under a lenient metric, not as schema-complete extraction. I do not see evidence of fraud or an internally inconsistent method; the framework is plausible and the ablation shows the selector model matters. The right outcome is the reader's CONDITIONAL verdict: the benchmark and a fixed evaluation protocol should be released before the quantitative claims are accepted. No change to the reader's verdict is needed.","tokens_in":13255,"tokens_out":7014,"duration_ms":81388,"concrete_test":"Release the WebLists benchmark and reference scripts, and recompute Table 3 and Table 4 under a fixed, pre-registered matching rule: a retrieved row is correct only if it matches a gold row on the URL column (the schema's stated identifier) and on at least one required text column, with the same rule applied to every agent. If the BardeenAgent recall advantage over Wilbur narrows materially, or the per-row cost advantage reverses, the central claim should be revised downward.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2.2 defines the WebLists schema as containing a URL column that 'acts as an identifier' plus free-text columns, so a valid extracted row should contain all schema columns. Section 4.2, however, states that precision and recall are computed by 'exact matching on a specific column' and that 'we choose different columns for different agents, in order to maximize matching between gold and extracted data.' Section 4.6 then concedes: 'we have evaluated with a forgiving metric, requiring only one column to match, a stricter evaluation would yield lower recall.' Consequently, the central result in Table 3 (66.2% recall, 72.5% precision, versus 30.5% recall for Wilbur) can count a row as correct when the URL identifier and other schema columns are missing or wrong, and it does so under a match rule that is not held constant across agents. Table 4's per-row cost advantage is computed from the same relaxed matches, so the '3x lower cost per correct row' claim inherits the same favorable denominator. The quantitative headline therefore rests on an evaluation rule that is weaker than the benchmark's own task definition.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces WebLists, a benchmark of 200 structured data-extraction tasks across 50 live websites and four use cases (jobs, job categories, blogs, testimonials), together with BardeenAgent, a two-phase LLM web agent that records actions as executable programs with CSS selectors and replays them to extract large lists. On the benchmark, BardeenAgent reports 66.2% recall and 72.5% precision, roughly doubling the recall of the strongest baseline Wilbur, and reducing cost per extracted row by about 3x. The paper also includes a question-answering evaluation on 50 questions and an ablation of the LLM-based selector model.","tokens_in":13463,"tokens_out":4990,"duration_ms":48317,"significance":"If the reported results hold, the paper makes a useful contribution: WebLists is a live, deterministic benchmark with a realistic scale (200 tasks, 50 websites) that addresses a real gap in web-agent evaluation, and the record-and-replay idea with generalizable CSS selectors is a sensible and potentially efficient way to scale extraction without per-item LLM calls. The ablation without the selector model (recall drops from 66.2% to 35.6%) shows that the proposed component is important, and the explicit discussion of limitations in Section 4.6 is commendable. However, the quantitative headline claims currently depend on an evaluation rule that is not held constant across agents and that is weaker than the benchmark's own schema definition, so the size of the reported advantage is not yet established.","major_comments":[{"comment":"The evaluation uses a different matching column for each agent: \"We choose different columns for different agents, in order to maximize matching between gold and extracted data\" (Section 4.2). Since the schema identifies rows by URL (Section 2.2) and the benchmark's task definition requires adherence to the full schema, a row that matches on only one hand-picked column may be counted as correct even if the URL identifier and other columns are wrong or missing. Section 4.6 concedes that this is a \"forgiving metric.\" The headline comparison (66.2% vs. 30.5% recall) is therefore not computed under a fixed, task-aligned match rule. Please report results under a fixed matching column (e.g., the URL column) and under full-schema matching for all agents.","section":"Section 4.2, Table 3"},{"comment":"The per-row cost advantage (\"3x lower cost per correct output row\") is computed using the same relaxed one-column matches as the recall figures, so the denominator of \"correct rows\" is inflated by the same per-agent column selection. The cost claim should be recomputed under the fixed matching rule proposed above, or the dependence of the cost ratio on the match rule should be explicitly reported.","section":"Table 4"},{"comment":"All results are single runs with no variance or significance measures. The policy model uses temperature 0.7 (Appendix C), and live websites change over time, so the numbers in Table 3 may not be stable. Please run each configuration multiple times, or at least report per-website variance and confidence intervals, and state the temporal window of the evaluation.","section":"Sections 4.2, 4.3 and Appendix C"},{"comment":"\"We tune goals per use-case and model\" and manually select few-shot examples for BardeenAgent and Wilbur. This means the comparison is not a fixed-prompt comparison; the reported improvement could be partly due to prompt engineering rather than the executable-agent mechanism. Please provide the exact prompts and few-shot examples for all agents, and ideally include an ablation in which BardeenAgent uses the same goal/schema phrasing as the baselines, to isolate the contribution of the record-and-replay mechanism.","section":"Section 4.2"}],"minor_comments":[{"comment":"The heading \"Balacing Extraction & Interaction\" contains a typo and should read \"Balancing Extraction & Interaction.\"","section":"Section 4.6"},{"comment":"The sentence \"In Fig. 3 We also show the the time for Operator to complete a request end-to-end\" contains a duplicated \"the\" and should be rewritten, e.g., \"Figure 3 also shows the time for Operator to complete a request end-to-end.\"","section":"Appendix D"},{"comment":"The phrase \"yielding an improvement of 36% in recall\" is ambiguous: it refers to a 35.7 percentage-point improvement, not a 36% relative improvement. Please state this as percentage points or as a relative improvement explicitly.","section":"Section 1.1"},{"comment":"The benchmark limits extraction to at most 5 pages of data per task; this is an important constraint for interpreting the recall values, but it is only mentioned once. Please restate it in the evaluation section and discuss how it affects comparability with agents that may paginate further.","section":"Section 2.2"},{"comment":"The reference to \"Sams, 2025\" is a blog with no formal author or venue; consider using a more standard source for web statistics, or clearly mark it as a web resource.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The benchmark and the BardeenAgent approach are potentially valuable, but the evaluation weaknesses are material and affect the core quantitative claims. The most serious issue is the variable match rule: if the authors cannot show that the recall and cost advantages persist under a fixed URL-based or full-schema match, the paper's central conclusion should be substantially downgraded. I recommend major revision to address these concerns before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a real contribution to the web-agent evaluation space, with a useful benchmark and an interesting record-and-replay method. But the headline recall figures are softer than advertised, because the evaluation matches on a different column for each agent and only requires one column to match. Treat the 66% vs 30% numbers as upper-bound estimates until the authors fix the metric.\n\nWhat's new: WebLists is a benchmark of 200 extraction tasks across 50 live websites, with deterministic script-based evaluation. That's genuinely useful; existing benchmarks either aren't interactive or use a handful of sites. BardeenAgent's two-phase design—record on one list item, generate a robust CSS selector, replay at scale—is a sensible way to avoid per-item LLM calls. The ablation without the selector model shows the model matters, and the method cuts cost per row. This is a credible engineering contribution.\n\nWhere it gets soft: Section 4.2 says they choose different matching columns per agent to maximize scores. That is selecting the most favorable comparison for each system, not a fixed evaluation rule. Combined with the admission in Section 4.6 that requiring one column is forgiving, the 66.2% recall doesn't mean schema-complete extraction. The cost per correct row inherits that denominator. No error bars or repeated runs either, and the benchmark and reference scripts aren't released, so I can't verify the gold sets. The baseline Wilbur is the authors' own prior work, which is fine, but it's the only strong baseline with comparable prompting attention. The \"first to exploit HTML structure\" claim overstates novelty; the macro-recorder idea has a long history, though applying it to LLM agents with AI-generated selectors is new.\n\nThe central method still probably helps: recall gains are consistent across all four use cases, and the Job Categories case shows the expected sensitivity to list structure—that's a plausible failure mode. So the idea holds, but the numbers as reported shouldn't be taken at face value.\n\nWho should read it: anyone building or evaluating web agents for structured extraction. It deserves a serious referee, but only with major revisions: fix the eval to a fixed column or report all columns, add repeated trials, and release code and data.\n\nI'd bring it to the reading group as a case study in how evaluation choices drive headline results.","headline":"A genuinely useful benchmark and a plausible record-and-replay method, but the headline recall numbers rest on a per-agent, single-column matching rule that the authors themselves admit is forgiving.","tokens_in":14038,"tokens_out":2226,"would_cite":false,"duration_ms":22332,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"BardeenAgent records a single extraction, converts it to a program with generalizable CSS selectors, and replays it to double the recall of the best prior web agent on 200 live data-extraction tasks.","keywords":["web agents","structured data extraction","record and replay","CSS selectors","live website benchmark","LLM agents","WebLists","data extraction at scale"],"falsifier":"Create a controlled variant of WebLists pages in which each list item is wrapped in a different parent element with no shared selector, run BardeenAgent's record-replay pipeline, and check whether recall collapses from 66% toward the 30.5% baseline; if recall stays high, the shared-selector assumption is not what carries the gain.","tokens_in":13044,"feed_emoji":"🕸️","tokens_out":8440,"duration_ms":79761,"temperature":0.7,"pith_summary":"The paper argues that structured data extraction from live websites is a distinct, under-tested capability for LLM web agents, and that the right way to scale it is not to ask the LLM to extract every row. Instead, an agent should record one successful extraction on the first item of a list, convert the recorded actions into a reusable program keyed to generalizable CSS selectors, then replay that program across all items and pages. To measure this, the paper introduces WebLists, a benchmark of 200 extraction tasks on 50 live company websites across four use cases, and reports that its BardeenAgent reaches 66.2% recall, more than double the 30.5% recall of the best baseline, with 72.5% precision and roughly one-third the cost per correctly extracted row. The central claim is that the record-and-replay design, not prompt tuning, is what produces this gain.","feed_headline":"Record-and-replay agent doubles web extraction recall","feed_subtitle":"On 200 live tasks, BardeenAgent hits 66.2% recall, 72.5% precision, and 3x lower cost per row.","key_machinery":"The load-bearing mechanism is the pair consisting of the EnterList tool and generalizable CSS selector generation. EnterList takes representative elements, a description, and pagination controls, identifies their least common ancestor, and scopes all subsequent actions to the first list item; the recorded actions are then replayed as a loop over the CSS-matched elements. Selector generation has two paths: a heuristic that samples diverse selector strategies and joins the survivors, and an LLM-based SelectorModel that takes the goal, column specification, the HTML of the least common parent, and a list description to produce a selector for irregular layouts. This machinery is what lets the agent generalize from one item to the whole dataset without additional LLM calls per row.","core_discovery":"The central discovery is that the regular structure of HTML can be turned into an executable extraction program. In the record phase, the LLM navigates to the page, uses the EnterList tool to select representative list elements, and records operations on only the first list item while all CSS selectors are scoped to that item. In the replay phase, those operations become a loop over every element matching a generalizable selector, with pagination handling, duplicate suppression, and a two-pass strategy that first collects item URLs and only then visits each one. Because LLM calls happen mainly during recording, the number of LLM calls does not grow with dataset size, which the paper credits for its recall gain and for the 3x reduction in cost per extracted row.","pith_inferences":["Editorial inference: If the record-and-replay pattern generalizes, the same trick should work for any repeating DOM structure — tables, grids, card walls, and paginated search results — though the paper only evaluates lists.","Editorial inference: Because the AnswerQuestion tool still invokes an LLM on every replayed item, the cost-scaling advantage is strongest when the schema can be satisfied by direct extraction or regex; strictly per-item QA schemas would partially erode the reported 3x cost reduction.","Editorial inference: A stress test the paper does not run is to mutate list markup between pagination pages, such as changing class names after page 2; a selector derived from the first page alone would likely miss later items and reveal how much of the gain depends on markup homogeneity."],"forward_implications":["If the record-and-replay design is correct, large-scale extraction no longer needs one LLM call per item; the LLM decides how to extract once, and the loop carries out the extraction.","WebLists can be re-run at any time on the same live websites because its evaluation scripts regenerate ground truth, making deterministic live-web evaluation possible without simulator proxies.","Agents optimized for navigation and question answering are poor at structured extraction, so benchmarks that only test those skills will miss a failure mode that matters for business use.","The gap between the Jobs use case (86.8% recall) and the Job Categories use case (64.5% recall) shows that filtering and multi-category list structure, not basic pagination, are the main remaining recall bottleneck."],"supporting_citations":[{"why":"Provides Wilbur, the strongest baseline whose 30.5% overall recall defines the comparison BardeenAgent must beat.","marker":"Lutz et al., 2024"},{"why":"Provides Agent-E, the state-of-the-art open-source web-agent baseline compared across all four use cases.","marker":"Abuelsaad et al., 2024"},{"why":"Provides WebVoyager, the benchmark whose agent designs are shown to underperform on extraction tasks.","marker":"He et al., 2024"},{"why":"Provides Mind2Web, used to argue that recorded-demonstration benchmarks go stale on live websites.","marker":"Deng et al., 2023"},{"why":"Provides WebArena, one of the small-website benchmarks that motivates WebLists' wider coverage.","marker":"Zhou et al., 2024"},{"why":"Supplies the sample-multiple-candidates-then-select action policy that BardeenAgent's record phase uses.","marker":"Yang et al., 2024"},{"why":"The Forbes Cloud 100 list is the sampling frame for the 50 websites in WebLists.","marker":"Cai & Konrad, 2024"},{"why":"Provides Sonar, the LLM-with-search baseline whose 3.3% recall motivates the need for interactive extraction.","marker":"Perplexity Team, 2025"}],"fun_headline_variants":["Agent replays HTML structure, 2x recall on web data extraction","New benchmark shows web agents fail; BardeenAgent 2x recall","Extraction at scale: agent learns CSS selectors, 66% recall","From 31% to 66% recall: BardeenAgent wins on web extraction","Data extraction agent cuts cost 3x, doubles recall"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The replay phase assumes the target data is exposed as repeated DOM nodes that share a generalizable CSS selector; if a website's list items have heterogeneous markup, or the data only appears after unique per-item interaction, the recorded selectors over-match or under-match and recall collapses.","fun_headline_variants_meta":{"raw":{"variants":["Agent replays HTML structure, 2x recall on web data extraction","New benchmark shows web agents fail; BardeenAgent 2x recall","Extraction at scale: agent learns CSS selectors, 66% recall","From 31% to 66% recall: BardeenAgent wins on web extraction","Data extraction agent cuts cost 3x, doubles recall"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000281,"raw_usage":{"total_tokens":1651,"prompt_tokens":916,"completion_tokens":735,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":532,"completion_tokens_details":{"reasoning_tokens":638}},"tokens_in":532,"tokens_out":735,"duration_ms":7702,"temperature":1.0,"reasoning_tokens":638,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:24:42.031287+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Create a controlled variant of WebLists pages in which each list item is wrapped in a different parent element with no shared selector, run BardeenAgent's record-replay pipeline, and check whether recall collapses from 66% toward the 30.5% baseline; if recall stays high, the shared-selector assumption is not what carries the gain.","supporting_citations":[{"cited_title":"Meet new sonar: A blazing fast model optimized for perplexity search","cited_arxiv_id":null,"evidence_quote":"Provides Sonar, the LLM-with-search baseline whose 3.3% recall motivates the need for interactive extraction."}],"review_version":1}