Pith. sign in

REVIEW 4 major objections 5 minor 35 references

XPath Agent: An Efficient XPath Programming Agent Based on LLM for Web Crawler

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read XPath Agent shows that XPath queries for web scraping can be generated by a two-stage LLM pipeline that first extracts cue texts from sanitized pages and then programs XPath from condensed HTML, achieving accuracy comparable to a leading…

desk verdict A well-described pipeline for LLM-based XPath generation, but the efficiency claim has no supporting data and the manuscript reads as unfinished. read the letter →

arxiv 2502.15688 v1 pith:3OWLYWQZ submitted 2024-12-17 cs.IR cs.AIcs.SE

Yu Li , Bryce Wang , Xinyu Luan This is my paper · ORCID
classification cs.IRcs.AIcs.SE
keywords XPathgenerationwebscrapinglargelanguagemodelscuetextsHTMLsanitizationtwo-stagepipelinetokenefficiencycrawling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that XPath queries for web scraping can be generated cheaply by splitting the job into two stages: a weak LLM first reads a sanitized page and returns the target values plus short 'cue texts' that point to them, and a stronger LLM then writes one XPath query from three condensed seed pages. The authors claim this design matches the accuracy of a state-of-the-art single-agent approach while using far fewer tokens and less wall-clock time. They also report that the cue texts from the first stage are what make the second stage work well, allowing XPath to be anchored to nearby labels instead of to brittle text values. If the claim holds, teams that write and maintain scrapers could replace hand-tuned XPath with a natural-language prompt and a small sample of pages.

What carries the argument

The load-bearing mechanism is the two-stage pipeline built around cue texts. Stage one runs an HTML sanitizer that prunes invisible and empty nodes and strips attributes, cutting page size to roughly 10–20 percent, then prompts a small LLM to return target values and their cue texts in JSON. Stage two runs an HTML condenser that keeps only nodes near the extracted cue texts, replaces other children with '...', and feeds the condensed markup of three seed pages to a larger LLM; a static XPath-generation algorithm guides the model bottom-up from the target node to the root, adding class and id attributes, and a conversational evaluator executes the candidate XPath on the seed pages and feeds missing/surplus/invalid feedback back into the model for up to three rounds.

What would settle it

Re-run the experiment on the same benchmark with the three seed pages explicitly excluded from the 32-page test set, and run the state-of-the-art baseline on the same 32 pages measuring F1, token usage, and wall-clock time. If the baseline matches or beats XPath Agent on F1 while using comparable or fewer tokens, or if XPath Agent's F1 collapses when the seeds are removed, the central claim falls.

Watch

Extended reading notes

Core claim

XPath Agent's central discovery is that a two-stage division of labour lets an LLM produce generalizable XPath queries at a fraction of the usual cost. In the first stage, the agent strips a web page down to its meaningful text and tags, then asks a small language model to return both the requested data and the cue texts that sit next to that data in the HTML. In the second stage, it condenses three seed pages down to the neighbourhoods of those cues, feeds the condensed HTML to a larger model, and iterates with an XPath evaluator that reports missing, surplus, or invalid matches. The authors' evidence, as reported, is that this pipeline reaches F1 scores comparable to a leading alternative agent while sharply reducing token usage and improving clock-time efficiency, and that removing the cue texts degrades the second stage.

Load-bearing premise

The evaluation is valid only if the three seed pages used for query generation and iterative correction are not part of the 32-page test sample; the paper never states that split, and it reports no scores for the baseline agent, so without that missing information the efficiency claim has no visible support.

Editorial extensions

If this is right

  • If the efficiency claim holds, scraping pipelines that currently hand-write XPath can be bootstrapped from a natural-language query and three sample pages, cutting development time.
  • Because the first stage uses a weaker, cheaper model and the second only sees condensed HTML, the token cost per generated query drops enough to make LLM-based scraping viable at scale.
  • The cue-text mechanism, if it is real, gives a general recipe for XPath generation: anchor queries to labels rather than to the values themselves, which should improve robustness to value changes.
  • The same two-stage shape may transfer to GUI testing, where selecting interface elements by robust XPath is the same problem.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not report the baseline agent's scores or token counts, so 'comparable' is a claim to be verified; a direct head-to-head on identical pages with identical metrics would settle it.
  • If the seed pages are not excluded from the 32-page evaluation sample, the reported F1 reflects memorization rather than generalization; this is the first thing a reproducing reader should check.
  • The cue-text trick suggests a testable extension: deliberately hide the label text in the test pages and see whether the generated XPath still finds the value via structure alone.
  • The 'static XPath generation' guidance could be lifted as a standalone prompting technique for any LLM asked to write XPath, independent of the full pipeline.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes XPath Agent, a two-stage LLM pipeline for generating XPath queries for web crawling and GUI testing: an information-extraction stage that identifies target values and cue texts from sanitized web pages, followed by an XPath-programming stage that generates and iteratively refines an XPath query using a conversational evaluator. The authors claim that XPath Agent achieves performance comparable to a state-of-the-art XPath programming agent while significantly reducing token usage and improving clock-time efficiency, and that extracted cue texts substantially improve the second stage. Experiments are reported on the SWDE dataset using four LLM variants, with precision, recall, F1, and accuracy metrics in Table 1.

Significance. If the central claims were substantiated, the paper would address a practical bottleneck in web-scraping pipelines: reducing the human effort and LLM cost of writing generalizable XPath queries. The proposed pipeline is clearly described, the code is made publicly available, and the sanitization algorithm is a reasonable engineering contribution that could benefit practitioners using LLM-based scraping tools. However, the manuscript as submitted does not provide the evidence needed to establish these contributions: the claimed comparison with AutoCrawler is absent, no token or timing measurements are reported, the seed/test overlap is unstated, and the claimed benefit of cue texts is not ablated. The paper's practical significance therefore currently rests on assertions rather than verifiable results.

major comments (4)
  1. [§5.2, Table 1] The abstract claims comparable performance to a state-of-the-art agent with significantly reduced token usage and improved clock-time efficiency, but Section 5.2 is only a qualitative paragraph and Table 1 reports results for four in-house LLM variants with no AutoCrawler baseline row. No token counts, wall-clock times, standard deviations, or sample sizes are reported anywhere in the paper, so the central efficiency and comparability claims have no evidentiary basis.
  2. [§4.1.3, §3.4] The experimental setup states that 3 seed pages guide query generation and that 32 web pages are sampled per task for evaluation, but the paper never states that the 3 seed pages are excluded from the 32-page evaluation sample. Since the XPath queries are iteratively refined by the evaluator on the seeded pages, any overlap between seeds and the evaluation sample would make the reported scores partially circular and inflated; this must be clarified before the results can be interpreted.
  3. [§5.3, §6] The manuscript is incomplete: Section 5.3 is literally 'TODO' and the conclusion contains template formatting instructions ('Third level headings must be flush left...') instead of a conclusion. This prevents verification of the pipeline and results and indicates the manuscript is not in a publishable state.
  4. [§1.2, §3.1] The contribution that 'extracted cue texts from 1st stage significantly improve the performance of the 2nd stage' is asserted but never tested. No ablation comparing the pipeline with and without cue texts is presented, so this central contribution is currently unsupported.
minor comments (5)
  1. [§4.2] The formulas contain typos: 'presision', 'T ureP ositives', and 'T rueP ositives' should be corrected.
  2. [§4.1.2] The SWDE dataset is cited as reference [1], but reference [1] is the Phi-3 technical report; the dataset citation appears to be missing or mismatched.
  3. [Figure 2] Figure 2 lacks axis labels, and the caption 'Token Stats Analysis with Algorithm 1' does not specify the unit or quantity plotted, making the figure difficult to interpret.
  4. [Algorithm 2] Algorithm 2 initializes 'target texts ← []' at the beginning, which conflicts with the parameter 'target texts' and would erase the input list; this appears to be a typo, likely intended to initialize only 'distances' and 'eles'.
  5. [§5.2] The comparative analysis paragraph refers to 'your XPath Agent' in the second person; this should be 'our' for consistency with the rest of the paper.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation found; the paper's central claims are unevidenced and the manuscript is incomplete, but the failure is evidentiary rather than circular.

full rationale

XPath Agent does not derive its headline results from a chain of equations or from self-citations, so it does not exhibit the circularity patterns this pass is tasked to find. The claimed comparison with AutoCrawler is unsupported: Table 1 reports only internal LLM variants, Section 5.2 gives a qualitative description rather than measured baseline numbers, and the abstract's token and clock-time savings are never quantified. Section 5.3 is an explicit 'TODO' and Section 6 contains template typesetting instructions, confirming that the experimental report is unfinished. These are serious completeness and evidence problems, but they are not circularity: no parameter is fitted and then renamed as a prediction, and no prior result by these authors is imported as the load-bearing justification. The closest identifiable circularity risk is in Section 4.1.3, which says 'Number of Seeds is 3 initial seeds are provided to guide query generation with a sample size of 32 web pages are sampled per task to evaluate the model's adaptability and generalizability'; combined with Section 3.4's iterative evaluator on seeded pages, the results would be inflated if the three seeds were also part of the 32-page evaluation sample. However, the manuscript does not state that the seeds are included in that sample, and asserting overlap would be speculation about an unstated detail rather than a reduction exhibited in the text. Under the hard rule that circularity must be demonstrated by quoted text and a specific reduction, no circular step is established.

Assumptions & free parameters 4 free parameters · 4 assumptions · 1 invented entities

The method relies on several hand-chosen parameters and unverified assumptions about LLM behavior, HTML sanitization losslessness, and seed/test separation. The key missing pieces are the distance function, the absence of baseline data, and the unresolved seed/test overlap question.

free parameters (4)
  • seeds_per_task = 3
    Number of seed pages used for XPath generation; chosen by hand without sensitivity analysis.
  • feedback_loops = 3
    Maximum evaluator feedback iterations; chosen by hand.
  • eval_sample_size = 32
    Web pages sampled per task for evaluation; no justification.
  • distance_function = unspecified
    Used in HTML Condenser (Algorithm 2) to find nodes relevant to target texts; its definition is omitted, making the condensing process non-reproducible.
assumptions (4)
  • domain assumption LLM can reliably extract cue texts and target values from sanitized HTML.
    Section 3.1 assumes this without evidence.
  • domain assumption Sanitization preserves all information needed for XPath generation.
    Algorithm 1 removes invisible nodes and attributes; assumes no loss of relevant structure.
  • domain assumption XPath generated from 3 seed pages generalizes to the evaluation set.
    Section 4.1.3 sets seeds=3 with no analysis; if seeds overlap test split, results are circular.
  • domain assumption SWDE ground-truth labels are accurate.
    Used as reference for metrics; no verification.
invented entities (1)
  • cue_text
    purpose: Anchor text signaling nearby target information, used to compose XPath axes
    Introduced in Section 3.1 as a key contribution, but no ablation demonstrates it improves performance beyond anecdote.

how reviews work

0 comments
Cite this review

Pith. "Pith review of XPath Agent: An Efficient XPath Programming Agent Based on LLM for Web Crawler." pith.science (2026). https://pith.science/paper/3OWLYWQZ

@misc{pith2026250215688,
  author       = {Pith},
  title        = {Pith review of: XPath Agent: An Efficient XPath Programming Agent Based on LLM for Web Crawler},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3OWLYWQZ}},
  note         = {Machine review of arXiv:2502.15688}
}
read the original abstract

We present XPath Agent, a production-ready XPath programming agent specifically designed for web crawling and web GUI testing. A key feature of XPath Agent is its ability to automatically generate XPath queries from a set of sampled web pages using a single natural language query. To demonstrate its effectiveness, we benchmark XPath Agent against a state-of-the-art XPath programming agent across a range of web crawling tasks. Our results show that XPath Agent achieves comparable performance metrics while significantly reducing token usage and improving clock-time efficiency. The well-designed two-stage pipeline allows for seamless integration into existing web crawling or web GUI testing workflows, thereby saving time and effort in manual XPath query development. The source code for XPath Agent is available at https://github.com/eavae/feilian.

Figures

Figures reproduced from arXiv: 2502.15688 by the authors.

Figure 1
Figure 1. XPath Agent of two stages pipeline. The first stage is Information Extraction, which extracts target [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Token Stats Analysis with Algorithm 1. As [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

35 extracted references · 31 canonical work pages

  1. [1]

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, S´ ebastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dongdong Chen, Weizhu Chen, Yen- Chun Chen, Yi-Ling Chen, Hao Cheng, Parul Chopra, Xiyang Dai, M...

    work page 2024

  2. [2]

    Leveraging large language models for web scraping, 2024

    Aman Ahluwalia and Suhrud Wani. Leveraging large language models for web scraping, 2024

    work page 2024

  3. [3]

    Mind2web: Towards a generalist agent for the web, 2023

    Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web, 2023

    2023

  4. [4]

    The future of web data mining: In- sights from multimodal and code-based extrac- tion methods

    Evan Fellman, Jacob Tyo, and Zachary Lip- ton. The future of web data mining: In- sights from multimodal and code-based extrac- tion methods. In Ali H¨ urriyeto˘ glu, Hristo Tanev, Surendrabikram Thapa, and G¨ ok¸ ce Uludo˘ gan, ed- itors, Proceedings of the 7th Workshop on Chal- lenges and Applications of Automated Extraction of Socio-political Events fr...

    work page 2024

  5. [5]

    Autoscraper: A progres- sive understanding web agent for web scraper gen- eration, 2024

    Wenhao Huang, Zhouhong Gu, Chenghao Peng, Zhixu Li, Jiaqing Liang, Yanghua Xiao, Liqian Wen, and Zulong Chen. Autoscraper: A progres- sive understanding web agent for web scraper gen- eration, 2024

    work page 2024

  6. [6]

    Web scraping or web crawling: State of art, techniques, approaches and application

    Moaiad Ahmad Khder. Web scraping or web crawling: State of art, techniques, approaches and application. International Journal of Advances in Soft Computing & Its Applications, 13(3), 2021

    work page 2021

  7. [7]

    Autowebglm: A large language model-based web navigating agent, 2024

    Hanyu Lai, Xiao Liu, Iat Long Iong, Shuntian Yao, Yuxuan Chen, Pengbo Shen, Hao Yu, Hanchen Zhang, Xiaohan Zhang, Yuxiao Dong, and Jie Tang. Autowebglm: A large language model-based web navigating agent, 2024

    2024

  8. [8]

    Markuplm: Pre-training of text and markup lan- guage for visually-rich document understanding

    Junlong Li, Yiheng Xu, Lei Cui, and Furu Wei. Markuplm: Pre-training of text and markup lan- guage for visually-rich document understanding. CoRR, abs/2110.08518, 2021

    arXiv 2021

Show all 35 references
  1. [9]

    Freedom: A transferable neural architecture for structured information extraction on web documents

    Bill Yuchen Lin, Ying Sheng, Nguyen Vo, and Sandeep Tata. Freedom: A transferable neural architecture for structured information extraction on web documents. CoRR, abs/2010.10755, 2020

    2010 arXiv

  2. [10]

    Synthesis of forgiving data extractors

    Adi Omari, Sharon Shoham, and Eran Yahav. Synthesis of forgiving data extractors. In Pro- ceedings of the Tenth ACM International Confer- ence on Web Search and Data Mining, WSDM ’17, page 385–394, New York, NY, USA, 2017. Association for Computing Machinery

    2017

  3. [11]

    Intelligent testing tool: selenium web driver

    Renu Patil and Rohini Temkar. Intelligent testing tool: selenium web driver. International Research Journal of Engineering and Technology (IRJET), 4(06), 2017

    2017

  4. [12]

    Gollie: Annotation guidelines improve zero-shot information-extraction, 2024

    Oscar Sainz, Iker Garc ´ ıa-Ferrero, Rodrigo Agerri, Oier Lopez de Lacalle, German Rigau, and Eneko Agirre. Gollie: Annotation guidelines improve zero-shot information-extraction, 2024

    2024

  5. [13]

    Html5 and the evolution of html; tracing the origins of digital platforms

    Ra´ ul Tabar´ es. Html5 and the evolution of html; tracing the origins of digital platforms. Technol- ogy in Society, 65:101529, 2021

    2021

  6. [14]

    Webformer: The web-page transformer for structure informa- tion extraction, 2022

    Qifan Wang, Yi Fang, Anirudh Ravula, Fuli Feng, Xiaojun Quan, and Dongfang Liu. Webformer: The web-page transformer for structure informa- tion extraction, 2022

    2022

  7. [15]

    Chatie: Zero-shot information extrac- tion via chatting with chatgpt

    Xiang Wei, Xingyu Cui, Ning Cheng, Xi- aobin Wang, Xin Zhang, Shen Huang, Pengjun Xie, Jinan Xu, Yufeng Chen, Meishan Zhang, et al. Chatie: Zero-shot information extrac- tion via chatting with chatgpt. arXiv preprint arXiv:2302.10205, 2024. 7

    2024 arXiv

  8. [16]

    minify-html, 2020

    wilsonzlin. minify-html, 2020

    2020

  9. [17]

    Large lan- guage models for generative information extrac- tion: A survey, 2024

    Derong Xu, Wei Chen, Wenjun Peng, Chao Zhang, Tong Xu, Xiangyu Zhao, Xian Wu, Yefeng Zheng, Yang Wang, and Enhong Chen. Large lan- guage models for generative information extrac- tion: A survey, 2024

    2024

  10. [18]

    Hyper- text entity extraction in webpage, 2024

    Yifei Yang, Tianqiao Liu, Bo Shao, Hai Zhao, Lin- jun Shou, Ming Gong, and Daxin Jiang. Hyper- text entity extraction in webpage, 2024

    2024

  11. [19]

    Simplified dom trees for transferable attribute extraction from the web, 2021

    Yichao Zhou, Ying Sheng, Nguyen Vo, Nick Ed- monds, and Sandeep Tata. Simplified dom trees for transferable attribute extraction from the web, 2021. 8 Appendix A Information Extraction Prompt The prompt for the Information Extraction stage in the following format: Extract the ...

    2021

  12. [20]

    The label of the target text is more important in HTML semantics

  13. [21]

    The target text is completely within a tag, rather than within a sentence or paragraph

  14. [22]

    The target text is closer to other fields to be extracted

  15. [23]

    Cue Text (cue_text, from high to low):

    If these expressions can complement each other, please extract them all. Cue Text (cue_text, from high to low):

  16. [24]

    Cue Text: In HTML, the indicative text that signals the upcoming extraction of the target text, such as ‘Phone number‘ or ‘Address:‘

  17. [25]

    When there is no cue text, use an empty string. # Question: {{ query }} # Context: ‘‘‘html {{ context }} ‘‘‘ # Answer Format (ignore the format requirements in the ‘Question‘, strictly follow the answer format of cue_text and value): Thought: ...(Your thoughts, about fields me...

  18. [26]

    Do not output the xpath with exact value or element appears in the HTML

  19. [27]

    Reference to the ‘target value‘ and the generated the xpath (if exists) to get more context

  20. [28]

    When using text predication, always using ‘contains(., ’value’)‘ instead of ‘text() =’value’‘

  21. [29]

    If the target xpath ends with ‘text()[n]‘, where n is not 1, please do not ignore it

  22. [30]

    If cue text exist, using cue text and cue xpath to compose a new xpath might be a better idea

  23. [31]

    thought":

    String functions are allowed, such as ‘starts-with()‘, ‘ends-with()‘, ‘substring- before()‘, ‘substring-after()‘. Use it in caution, since it can only be used on ‘ text()‘ node. Please always response in the following Json format: { "thought": "", # a brief thought of how to c...

  24. [32]

    Extend the xpath to include the missing information if ‘Missing‘

  25. [33]

    Restrict the xpath to exclude the irrelevant information if ‘Surplus‘

  26. [34]

    Correct the xpath grammer if ‘Invalid‘

  27. [35]

    Response same xpath if no better solution. {% for feedback in feedbacks %} #### Evaluated on Fragment {{ feedback.id }}: Extracted (JSON encoded): ‘{{ feedback.extracted | tojson }}‘ Feedback Message: ‘{{ feedback.message }}‘ {%- endfor %} 10

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.