{"id":"0ebfa8a2-3cea-49ab-ae70-640b215bead6","arxiv_id":"2502.15688","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A two-stage LLM pipeline generates XPath queries from natural language and sampled web pages, but the reported efficiency gains over the baseline are not backed by any comparative numbers.","lead":"XPath Agent is a two-stage LLM pipeline that generates XPath queries for web scraping from natural language and a few sampled pages. The authors claim it matches a state-of-the-art agent with fewer tokens and less time, but the paper omits the comparative data that would support this claim.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central efficiency claim is unevidenced: no baseline numbers, no token/time data, and seed/test overlap is unresolved.","rationale":"The reader's verdict of REJECT is well supported. My independent reading of the manuscript confirms the strongest claim is the efficiency/comparability claim, and the weakest assumption is the seed/test split ambiguity plus the complete absence of any baseline or efficiency data. The load-bearing concern is not a single mathematical error but a systematic absence of evidence for the paper's quantitative claims, compounded by an explicit seed/test overlap risk. The manuscript itself acknowledges incompleteness: Section 5.3 reads 'TODO', the conclusion contains template instructions, and the comparative analysis refers to 'your XPath Agent' as if written by an outside reviewer. The SWDE citation is wrong (pointing to the Phi-3 technical report), and the baseline 'AutoCrawler' is actually AutoScraper (reference [5]). These are internal consistency failures, not mere disagreement with community norms. A concrete reproducibility check (running the repository and rerunning with seed exclusion, plus logging tokens/timing and baseline numbers) would settle the concern. Since the evidence required for the central claim is absent and the manuscript is unfinished, no adjustment to the REJECT verdict is needed.","tokens_in":8119,"tokens_out":1448,"duration_ms":12114,"concrete_test":"Access the linked repository (github.com/eavae/feilian) and run the provided evaluation script to reproduce Table 1. Then (1) rerun the pipeline with the 3 seed pages explicitly excluded from the 32-page evaluation set and compare metrics; (2) instrument the pipeline to log total prompt/response tokens and wall-clock time for both XPath Agent and the AutoCrawler baseline on the same SWDE tasks, and report baseline performance numbers. If seed exclusion changes scores materially or if token/time data are absent, the efficiency and comparability claims are unsupported.","verdict_should_be":"REJECT","load_bearing_attack":"The paper's central claim is that XPath Agent achieves comparable performance to a state-of-the-art agent (AutoCrawler) while significantly reducing token usage and improving clock-time efficiency. None of this is supported by data in the manuscript. Section 5.2 is a qualitative paragraph, not a comparison; Table 1 reports only four in-house LLM variants (DeepSeek, GPT-4o mini, GPT-4o, Claude 3.5) with no baseline row, no standard deviations, no sample sizes, and no token or timing measurements. The 'efficiency' claim in the abstract therefore has no evidentiary basis. Additionally, Section 4.1.3 states that 3 seed pages guide query generation while 32 web pages are sampled per task for evaluation, but the paper never states that the 3 seeds are excluded from the 32-page evaluation sample. If seeds are included, the generated XPath is partially fitted to the evaluation pages, which would inflate all metrics and make 'comparable performance' uninterpretable. The manuscript's own incompleteness (Section 5.3 'TODO'; conclusion contains template instructions) further prevents verification of the pipeline and results. Together, these gaps mean the central claim is not merely under-argued but currently unverifiable.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes XPath Agent, a two-stage LLM pipeline for generating XPath queries for web crawling and GUI testing: an information-extraction stage that identifies target values and cue texts from sanitized web pages, followed by an XPath-programming stage that generates and iteratively refines an XPath query using a conversational evaluator. The authors claim that XPath Agent achieves performance comparable to a state-of-the-art XPath programming agent while significantly reducing token usage and improving clock-time efficiency, and that extracted cue texts substantially improve the second stage. Experiments are reported on the SWDE dataset using four LLM variants, with precision, recall, F1, and accuracy metrics in Table 1.","tokens_in":8324,"tokens_out":1689,"duration_ms":16196,"significance":"If the central claims were substantiated, the paper would address a practical bottleneck in web-scraping pipelines: reducing the human effort and LLM cost of writing generalizable XPath queries. The proposed pipeline is clearly described, the code is made publicly available, and the sanitization algorithm is a reasonable engineering contribution that could benefit practitioners using LLM-based scraping tools. However, the manuscript as submitted does not provide the evidence needed to establish these contributions: the claimed comparison with AutoCrawler is absent, no token or timing measurements are reported, the seed/test overlap is unstated, and the claimed benefit of cue texts is not ablated. The paper's practical significance therefore currently rests on assertions rather than verifiable results.","major_comments":[{"comment":"The abstract claims comparable performance to a state-of-the-art agent with significantly reduced token usage and improved clock-time efficiency, but Section 5.2 is only a qualitative paragraph and Table 1 reports results for four in-house LLM variants with no AutoCrawler baseline row. No token counts, wall-clock times, standard deviations, or sample sizes are reported anywhere in the paper, so the central efficiency and comparability claims have no evidentiary basis.","section":"§5.2, Table 1"},{"comment":"The experimental setup states that 3 seed pages guide query generation and that 32 web pages are sampled per task for evaluation, but the paper never states that the 3 seed pages are excluded from the 32-page evaluation sample. Since the XPath queries are iteratively refined by the evaluator on the seeded pages, any overlap between seeds and the evaluation sample would make the reported scores partially circular and inflated; this must be clarified before the results can be interpreted.","section":"§4.1.3, §3.4"},{"comment":"The manuscript is incomplete: Section 5.3 is literally 'TODO' and the conclusion contains template formatting instructions ('Third level headings must be flush left...') instead of a conclusion. This prevents verification of the pipeline and results and indicates the manuscript is not in a publishable state.","section":"§5.3, §6"},{"comment":"The contribution that 'extracted cue texts from 1st stage significantly improve the performance of the 2nd stage' is asserted but never tested. No ablation comparing the pipeline with and without cue texts is presented, so this central contribution is currently unsupported.","section":"§1.2, §3.1"}],"minor_comments":[{"comment":"The formulas contain typos: 'presision', 'T ureP ositives', and 'T rueP ositives' should be corrected.","section":"§4.2"},{"comment":"The SWDE dataset is cited as reference [1], but reference [1] is the Phi-3 technical report; the dataset citation appears to be missing or mismatched.","section":"§4.1.2"},{"comment":"Figure 2 lacks axis labels, and the caption 'Token Stats Analysis with Algorithm 1' does not specify the unit or quantity plotted, making the figure difficult to interpret.","section":"Figure 2"},{"comment":"Algorithm 2 initializes 'target texts ← []' at the beginning, which conflicts with the parameter 'target texts' and would erase the input list; this appears to be a typo, likely intended to initialize only 'distances' and 'eles'.","section":"Algorithm 2"},{"comment":"The comparative analysis paragraph refers to 'your XPath Agent' in the second person; this should be 'our' for consistency with the rest of the paper.","section":"§5.2"}],"recommendation":"reject","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"XPath Agent is a two-stage LLM pipeline for generating XPath queries: an IE stage that extracts both target values and cue texts, then a condenser that keeps only relevant nodes, and a third stage that produces a static XPath with an evaluator feedback loop. That combination, plus the cue-text idea, is genuinely a bit new, and the sanitizer measurements (10-20% page size) are plausible. If the pipeline works as described, it would be a useful tool for web scraping.\n\nThe problem is the evidence. The abstract claims comparable performance to a state-of-the-art agent while cutting tokens and time, but the results section has no baseline row, no token or timing data, and no error bars. Table 1 only compares four LLM variants. Section 5.2 is a qualitative paragraph, not a comparison, and the cited 'AutoCrawler' is actually AutoScraper. The error analysis is literally a TODO, and the conclusion contains LaTeX template instructions. This is an unfinished manuscript.\n\nThe most serious issue is the seed/test split. Section 4.1.3 says 3 seeds guide query generation and 32 pages are sampled per task for evaluation, but never says the seeds are excluded. If the seeds are in the evaluation sample, the XPath queries are partially fitted to the test pages and all metrics are inflated. That alone makes the 'comparable performance' uninterpretable.\n\nThere is also a missing ablation for the cue-text claim. The paper says extracted cue texts significantly improve performance, but no experiment isolates that component. The SWDE dataset is cited to the Phi-3 technical report, which is wrong.\n\nOn the positive side, the code is available, and the prompts and algorithms are described in enough detail that the pipeline could be reproduced. The engineering is not crazy. But as it stands, the central efficiency claim is unverified, and the manuscript is internally incomplete.\n\nI would not send this to peer review in its current form. It is a desk reject. If the authors complete the paper, run a proper comparison against AutoScraper with the seed split clarified, and add an ablation for cue texts, it could be a fine workshop or short-paper contribution. As is, I would not cite it.","headline":"A well-described pipeline for LLM-based XPath generation, but the efficiency claim has no supporting data and the manuscript reads as unfinished.","tokens_in":8873,"tokens_out":2692,"would_cite":false,"duration_ms":25243,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"XPath Agent shows that XPath queries for web scraping can be generated by a two-stage LLM pipeline that first extracts cue texts from sanitized pages and then programs XPath from condensed HTML, achieving accuracy comparable to a leading…","keywords":["XPath generation","web scraping","large language models","cue texts","HTML sanitization","two-stage pipeline","token efficiency","web crawling"],"falsifier":"Re-run the experiment on the same benchmark with the three seed pages explicitly excluded from the 32-page test set, and run the state-of-the-art baseline on the same 32 pages measuring F1, token usage, and wall-clock time. If the baseline matches or beats XPath Agent on F1 while using comparable or fewer tokens, or if XPath Agent's F1 collapses when the seeds are removed, the central claim falls.","tokens_in":7887,"feed_emoji":"🕷️","tokens_out":6851,"duration_ms":56770,"temperature":0.7,"pith_summary":"This paper argues that XPath queries for web scraping can be generated cheaply by splitting the job into two stages: a weak LLM first reads a sanitized page and returns the target values plus short 'cue texts' that point to them, and a stronger LLM then writes one XPath query from three condensed seed pages. The authors claim this design matches the accuracy of a state-of-the-art single-agent approach while using far fewer tokens and less wall-clock time. They also report that the cue texts from the first stage are what make the second stage work well, allowing XPath to be anchored to nearby labels instead of to brittle text values. If the claim holds, teams that write and maintain scrapers could replace hand-tuned XPath with a natural-language prompt and a small sample of pages.","feed_headline":"LLM agent writes XPath with fewer tokens, same accuracy","feed_subtitle":"Two-stage pipeline prunes pages first, extracts cue texts, then writes XPath from condensed HTML to cut cost.","key_machinery":"The load-bearing mechanism is the two-stage pipeline built around cue texts. Stage one runs an HTML sanitizer that prunes invisible and empty nodes and strips attributes, cutting page size to roughly 10–20 percent, then prompts a small LLM to return target values and their cue texts in JSON. Stage two runs an HTML condenser that keeps only nodes near the extracted cue texts, replaces other children with '...', and feeds the condensed markup of three seed pages to a larger LLM; a static XPath-generation algorithm guides the model bottom-up from the target node to the root, adding class and id attributes, and a conversational evaluator executes the candidate XPath on the seed pages and feeds missing/surplus/invalid feedback back into the model for up to three rounds.","core_discovery":"XPath Agent's central discovery is that a two-stage division of labour lets an LLM produce generalizable XPath queries at a fraction of the usual cost. In the first stage, the agent strips a web page down to its meaningful text and tags, then asks a small language model to return both the requested data and the cue texts that sit next to that data in the HTML. In the second stage, it condenses three seed pages down to the neighbourhoods of those cues, feeds the condensed HTML to a larger model, and iterates with an XPath evaluator that reports missing, surplus, or invalid matches. The authors' evidence, as reported, is that this pipeline reaches F1 scores comparable to a leading alternative agent while sharply reducing token usage and improving clock-time efficiency, and that removing the cue texts degrades the second stage.","pith_inferences":["The paper does not report the baseline agent's scores or token counts, so 'comparable' is a claim to be verified; a direct head-to-head on identical pages with identical metrics would settle it.","If the seed pages are not excluded from the 32-page evaluation sample, the reported F1 reflects memorization rather than generalization; this is the first thing a reproducing reader should check.","The cue-text trick suggests a testable extension: deliberately hide the label text in the test pages and see whether the generated XPath still finds the value via structure alone.","The 'static XPath generation' guidance could be lifted as a standalone prompting technique for any LLM asked to write XPath, independent of the full pipeline."],"forward_implications":["If the efficiency claim holds, scraping pipelines that currently hand-write XPath can be bootstrapped from a natural-language query and three sample pages, cutting development time.","Because the first stage uses a weaker, cheaper model and the second only sees condensed HTML, the token cost per generated query drops enough to make LLM-based scraping viable at scale.","The cue-text mechanism, if it is real, gives a general recipe for XPath generation: anchor queries to labels rather than to the values themselves, which should improve robustness to value changes.","The same two-stage shape may transfer to GUI testing, where selecting interface elements by robust XPath is the same problem."],"supporting_citations":[{"why":"The baseline agent the paper compares against, providing the state-of-the-art performance target for the efficiency claim.","marker":"[5]"},{"why":"Supplies the 90-site, 9-domain web-extraction benchmark used for the reported metrics (the reference entry is mislisted in the paper).","marker":"[1]"},{"why":"The minification library used in the HTML sanitizer to further shrink page size before the first-stage LLM call.","marker":"[16]"}],"fun_headline_variants":["Two-stage LLM agent cuts XPath token cost","Cue texts let LLM write XPath cheaply","XPath agent prunes pages, saves tokens","LLM writes XPath faster with cue extraction","Smarter XPath generation via two-stage LLM"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation is valid only if the three seed pages used for query generation and iterative correction are not part of the 32-page test sample; the paper never states that split, and it reports no scores for the baseline agent, so without that missing information the efficiency claim has no visible support.","fun_headline_variants_meta":{"raw":{"variants":["Two-stage LLM agent cuts XPath token cost","Cue texts let LLM write XPath cheaply","XPath agent prunes pages, saves tokens","LLM writes XPath faster with cue extraction","Smarter XPath generation via two-stage LLM"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000193,"raw_usage":{"total_tokens":1300,"prompt_tokens":843,"completion_tokens":457,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":459,"completion_tokens_details":{"reasoning_tokens":382}},"tokens_in":459,"tokens_out":457,"duration_ms":4383,"temperature":1.0,"reasoning_tokens":382,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:40:57.290068+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the experiment on the same benchmark with the three seed pages explicitly excluded from the 32-page test set, and run the state-of-the-art baseline on the same 32 pages measuring F1, token usage, and wall-clock time. If the baseline matches or beats XPath Agent on F1 while using comparable or fewer tokens, or if XPath Agent's F1 collapses when the seeds are removed, the central claim falls.","supporting_citations":[{"cited_title":"Autoscraper: A progres- sive understanding web agent for web scraper gen- eration, 2024","cited_arxiv_id":null,"evidence_quote":"The baseline agent the paper compares against, providing the state-of-the-art performance target for the efficiency claim."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the 90-site, 9-domain web-extraction benchmark used for the reported metrics (the reference entry is mislisted in the paper)."},{"cited_title":"minify-html, 2020","cited_arxiv_id":null,"evidence_quote":"The minification library used in the HTML sanitizer to further shrink page size before the first-stage LLM call."}],"review_version":1}