pith. sign in

arxiv: 2511.05297 · v2 · pith:QVIAXIJ7new · submitted 2025-11-07 · 💻 cs.SE · cs.LG

Building Specialized Software-Assistant ChatBot with Graph-Based Retrieval-Augmented Generation

Mohammed Hilel , Yannis Karmim , Jean De Bodinat , Reda Sarehane , Antoine Gillon This is my paper

Pith reviewed 2026-05-21 19:50 UTC · model grok-4.3

classification 💻 cs.SE cs.LG
keywords Graph-based Retrieval-Augmented GenerationState-action knowledge graphsEnterprise software assistanceDigital adoption platformsLLM hallucinationsSoftware interface extractionChatbot for enterprise applicationsRetrieval-augmented generation
0
0 comments X

The pith

Enterprise web applications can be automatically converted into state-action knowledge graphs that enable LLMs to generate accurate, context-aware assistance without hallucinations.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces a Graph-based Retrieval-Augmented Generation framework that automatically extracts and structures the interfaces of enterprise software such as CRM and ERP systems into state-action knowledge graphs. These graphs allow large language models to retrieve relevant information and produce grounded responses for user assistance tasks. The work addresses the high manual effort required for traditional digital adoption platforms and the unreliability of direct LLM queries on complex software. A sympathetic reader would care because the method supports production use with black-box models and integrates into existing workflows without fine-tuning.

Core claim

The framework automatically converts enterprise web applications into state-action knowledge graphs, enabling LLMs to generate grounded and context-aware assistance. An engineering pipeline extracts software interfaces and structures them as graphs, after which a graph-based retrieval process supplies relevant states and actions to the model during generation, all integrated into digital adoption platform workflows.

What carries the argument

State-action knowledge graphs extracted from enterprise web application interfaces, which represent possible user interactions and software states for retrieval-augmented generation.

If this is right

  • LLMs can deliver reliable assistance for navigating complex enterprise systems such as CRM, ERP, or HRMS without producing hallucinations.
  • Manual creation and maintenance of interactive guides in digital adoption platforms is reduced through automation.
  • The approach works with black-box production LLMs since no access to model weights or fine-tuning is needed.
  • Scalability improves for large enterprise applications via the automated graph extraction pipeline.
  • Industrial deployment lessons on robustness can guide integration into real company workflows.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The extraction pipeline might be adapted to handle mobile or desktop interfaces if similar parsing techniques are developed for those environments.
  • State-action graphs could be enriched with logged user behavior data to support more predictive forms of assistance.
  • This retrieval method may combine with other knowledge sources to cover software features that are hard to extract from the interface alone.

Load-bearing premise

The automatic extraction process produces state-action graphs that faithfully capture the relevant user interactions and interface states of the target enterprise software.

What would settle it

A direct test in which the assistant is asked for navigation steps on a specific feature of the enterprise software and the generated response includes incorrect or incomplete actions that do not match the actual interface.

Figures

Figures reproduced from arXiv: 2511.05297 by Antoine Gillon, Jean De Bodinat, Mohammed Hilel, Reda Sarehane, Yannis Karmim.

Figure 1
Figure 1. Figure 1: Our Graph-RAG overview : (1) the target software is automatically converted into a structured state–action graph G, (2) the user query is embedded using a language model (LM) to retrieve the most relevant nodes and attributes from the graph, (3) an optimized subgraph G ∗ is then constructed to represent the minimal context needed to answer the query, and (4) this subgraph is provided as context to the LLM,… view at source ↗
Figure 2
Figure 2. Figure 2: shows examples of typical user queries related to CRM software. In such cases, the expected answer is a clear, procedural explanation e.g. , - How to create a new lead? - Créer une opportunité - How to create a prospect? - Can I create a new event? - How do I create a new deal? - How to modify an account? Salesforce top questions [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Illustration of a state–action path for the query “Updating a Leads Status.” User interactions can be represented as paths in a state–action graph (fig. 3), where nodes correspond to in￾terface states and edges correspond to possible user actions. Formally, we define a knowledge graph G = (V, E), where each node u ∈ V represents an interface state, each edge e ∈ E represents an action, and xu and xe denote… view at source ↗
Figure 5
Figure 5. Figure 5: Example of a connected and textualized subgraph ex￾tracted with PCST (12 nodes, 13 edges). Prompt Construction and Model Guidance We then construct the final LLM prompt by concatenating the tex￾tualized subgraph and the user query. We show an exam￾ple of the contextualized prompt from the same question in fig. 6. Listing all the nodes existing in the software subgraph as well as all the possible actions in… view at source ↗
Figure 6
Figure 6. Figure 6: Prompt assembly combining the user query and the tex￾tualized subgraph G ∗ . This design allows any LLM, whether open or API-based, to leverage structured contextual information without addi￾tional fine-tuning or model retraining. We show qualitative example of this contextualized answer generation process in section 5. Practical Adaptations to G-Retriever While our pipeline is inspired by G-Retriever, sev… view at source ↗
read the original abstract

Digital Adoption Platforms (DAPs) have become essential tools for helping employees navigate complex enterprise software such as CRM, ERP, or HRMS systems. Companies like LemonLearning have shown how digital guidance can reduce training costs and accelerate onboarding. However, building and maintaining these interactive guides still requires extensive manual effort. Leveraging Large Language Models as virtual assistants is an appealing alternative, yet without a structured understanding of the target software, LLMs often hallucinate and produce unreliable answers. Moreover, most production-grade LLMs are black-box APIs, making fine-tuning impractical due to the lack of access to model weights. In this work, we introduce a Graph-based Retrieval-Augmented Generation framework that automatically converts enterprise web applications into state-action knowledge graphs, enabling LLMs to generate grounded and context-aware assistance. The framework was co-developed with the AI enterprise RAKAM, in collaboration with Lemon Learning. We detail the engineering pipeline that extracts and structures software interfaces, the design of the graph-based retrieval process, and the integration of our approach into production DAP workflows. Finally, we discuss scalability, robustness, and deployment lessons learned from industrial use cases.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 2 minor

Summary. The manuscript introduces a Graph-based Retrieval-Augmented Generation framework that automatically converts enterprise web applications (CRM, ERP, HRMS) into state-action knowledge graphs. These graphs are intended to ground LLM responses for digital adoption platforms, reducing manual effort in building interactive guides while preventing hallucinations. The work describes the extraction pipeline (DOM analysis, state identification, graph construction), the retrieval mechanism, integration into production DAP workflows, and lessons from industrial deployments developed with RAKAM and Lemon Learning.

Significance. If the automatic extraction produces complete and accurate state-action graphs, the framework could meaningfully lower the cost of maintaining digital guidance systems and improve the reliability of LLM assistants in complex enterprise software. The industrial collaboration and reported deployment lessons offer practical value for scalability and robustness in real-world settings.

major comments (1)
  1. [Extraction and structuring pipeline (as described in the abstract and engineering pipeline section)] The central claim that the framework enables grounded, hallucination-free assistance rests on the assumption that the automatic extraction pipeline faithfully captures relevant user interactions and interface states. The manuscript describes the pipeline (DOM analysis, state identification, graph construction) but provides no quantitative metrics on extraction fidelity, such as state coverage, action recall on multi-step workflows, or accuracy against manually annotated ground truth for the target CRM/ERP systems. Dynamic elements (JS-driven views, conditional rendering, role-based states) are acknowledged as challenges but receive no empirical evaluation.
minor comments (2)
  1. [Graph design] Clarify the exact definition of 'state' and 'action' nodes in the knowledge graph, including how conditional or role-specific views are represented.
  2. [Deployment lessons] The abstract and text refer to 'industrial use cases' and 'deployment lessons'; adding a brief table summarizing the systems tested and qualitative outcomes would improve readability.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the constructive feedback and for highlighting the importance of empirical validation for the extraction pipeline. We agree that quantitative metrics would strengthen the central claims regarding grounded assistance and hallucination reduction. Below we address the major comment in detail and outline the revisions we will make.

read point-by-point responses

  1. Referee: [Extraction and structuring pipeline (as described in the abstract and engineering pipeline section)] The central claim that the framework enables grounded, hallucination-free assistance rests on the assumption that the automatic extraction pipeline faithfully captures relevant user interactions and interface states. The manuscript describes the pipeline (DOM analysis, state identification, graph construction) but provides no quantitative metrics on extraction fidelity, such as state coverage, action recall on multi-step workflows, or accuracy against manually annotated ground truth for the target CRM/ERP systems. Dynamic elements (JS-driven views, conditional rendering, role-based states) are acknowledged as challenges but receive no empirical evaluation.

    Authors: We acknowledge that the manuscript, as currently written, does not report quantitative metrics on extraction fidelity such as state coverage, action recall, or accuracy against manually annotated ground truth. The paper's primary contributions center on the overall Graph-RAG framework, its integration into production DAP workflows, and lessons from industrial deployments with RAKAM and Lemon Learning rather than a controlled benchmark study of the extraction component. Dynamic elements are discussed qualitatively as engineering challenges. We agree that adding empirical evaluation would better support the claims. In the revised manuscript we will include a new evaluation subsection that reports state coverage and action recall on a representative subset of the CRM/ERP interfaces, using manually annotated ground truth where available, and we will discuss limitations for fully dynamic JS-driven views. revision: yes

Circularity Check

0 steps flagged

No circularity: engineering pipeline presented without derivations or self-referential reductions

full rationale

The paper describes a practical engineering framework for automatically extracting state-action knowledge graphs from enterprise web applications and integrating them into a graph-based RAG system for LLM assistants. No mathematical derivations, equations, fitted parameters, or predictions are present that could reduce outputs to inputs by construction. The central claims rest on the described extraction pipeline (DOM analysis, state identification, graph construction) and industrial deployment lessons rather than any self-citation chain, uniqueness theorem, or ansatz imported from prior work. The work is self-contained as an applied construction; external validation of extraction fidelity is a separate evidence question, not a circularity issue.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on the domain assumption that enterprise web interfaces can be reliably decomposed into discrete states and actions by automated scraping and parsing. No free parameters or invented entities with independent evidence are introduced in the abstract.

axioms (1)
  • domain assumption Enterprise web applications can be automatically parsed into a faithful state-action graph that captures all relevant user interactions.
    This premise is required for the conversion step to produce usable retrieval material; it is invoked when the abstract states that the framework 'automatically converts enterprise web applications into state-action knowledge graphs'.

pith-pipeline@v0.9.0 · 5741 in / 1319 out tokens · 74130 ms · 2026-05-21T19:50:31.882182+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

32 extracted references · 32 canonical work pages · 2 internal anchors

  1. [1]

    Asai, A., Wu, Z., Wang, Y ., Sil, A., and Hajishirzi, H

    URL https://www.anthropic.com/ news/claude-3-family. Asai, A., Wu, Z., Wang, Y ., Sil, A., and Hajishirzi, H. Self- rag: Learning to retrieve, generate, and critique through self-reflection. InThe Twelfth International Conference on Learning Representations, 2023. Batz, G. V . An optimization technique for subgraph match- ing strategies. Technical report,...

    work page 2023

  2. [2]

    BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

    URL https://proceedings.neurips. cc/paper_files/paper/2020/file/ 1457c0d6bfcb4967418bfb8ac142f64a-Paper. pdf. Cheng, D., Huang, S., Bi, J., Zhan, Y ., Liu, J., Wang, Y ., Sun, H., Wei, F., Deng, W., and Zhang, Q. Uprise: Universal prompt retrieval for improving zero-shot evaluation. In Proceedings of the 2023 Conference on Empirical Meth- ods in Natural L...

    work page internal anchor Pith review Pith/arXiv arXiv 2020

  3. [3]

    org/CorpusID:3144218

    URL https://api.semanticscholar. org/CorpusID:3144218. Latka, G. Lemon learning: Company profile and revenue data, 2024. URL https://getlatka.com/ companies/lemon-learning#:˜:text=In% 202024%2C%20Lemon%20Learning%E2%80% 99s%20revenue,increasing%20adoption% 20across%20various%20industries. Ac- cessed: 2025-10-26. Lewis, P., Perez, E., Piktus, A., Petroni, ...

    work page 2024

  4. [4]

    URL https://arxiv.org/abs/2005. 11401. Li, X. L. and Liang, P. Prefix-tuning: Optimizing continuous prompts for generation, 2021. URL https://arxiv. org/abs/2101.00190. Meta AI. Introducing meta llama 3: The most capable openly available llm to date, 2024. URL https://ai. meta.com/blog/meta-llama-3. Mortaheb, M., Khojastepour, M. A. A., Chakradhar, S. T.,...

    work page internal anchor Pith review Pith/arXiv arXiv 2005

  5. [5]

    org/CorpusID:201646309

    URL https://api.semanticscholar. org/CorpusID:201646309. Saha, S., Yadav, P., Bauer, L., and Bansal, M. Explagraphs: An explanation graph generation task for structured com- monsense reasoning.arXiv preprint arXiv:2104.07644, 2021. Sawarkar, K., Mangal, A., and Solanki, S. R. Blended rag: Improving rag (retriever-augmented generation) accuracy with semant...

    work page doi:10.1109/mipr62202.2024.00031 2021

  6. [6]

    Go to the Opportunities tab

    work page

  7. [7]

    Find and open the opportunity you want to modify

    work page

  8. [8]

    0.91 LLM+GRAG 1

    Click Edit, update the details and click Save. 0.91 LLM+GRAG 1. Go to the Opportunities section from the home page. This displays a list of recently viewed opportunities, with details such as the account name, stage, closing date, and owner alias. 2. From the list of opportunities, select the opportunity you want to edit by clicking on its name. This will...

    work page

  9. [9]

    Allez dans l’onglet Tˆaches ou dans la section Activit´es

    work page

  10. [10]

    Trouvez et ouvrez la tˆache `a modifier

    work page

  11. [11]

    0.75 LLM+G-RAG 1

    Cliquez sur Modifier, mettez `a jour les informations n´ecessaires, puis cliquez sur Enregistrer. 0.75 LLM+G-RAG 1. Depuis le tableau de bord principal, acc´edez `a la section de gestion des tˆaches via la barre de navigation. 2. Dans la liste des t ˆaches r´ecemment consult´ees, s´electionnez la tˆache `a modifier afin d’afficher ses d´etails. 3. Utilise...

    work page

  12. [12]

    Find and open the contact you want to modify

    work page

  13. [13]

    0.91 LLM+G-RAG 1

    Click Edit, update the details (e.g., Name, Email, Phone, Account), and click Save. 0.91 LLM+G-RAG 1. Go to the Contacts section from the main navigation bar

    work page

  14. [14]

    Find the contact you want to edit using the search bar or by browsing the list of recently viewed contacts

    work page

  15. [15]

    Select the desired contact from the list to access its details

    work page

  16. [16]

    Look for the option to edit the contact’s information (e.g., an edit button or icon)

    work page

  17. [17]

    Edit the necessary fields, such as full name, associated account name, phone number, or email address

    work page

  18. [18]

    2.47 Table 4

    Save your changes to update the contact’s information. 2.47 Table 4. Qualitative comparison between Graph-RAG (G-RAG) and a standalone LLM onSalesforceuser queries. Each example shows the user question in French and in English, the model-generated response, and the corresponding response time in second. Each block of three rows represents one query instan...

    work page

  19. [19]

    Locate the contract you want to delete in the list

    work page

  20. [20]

    Click on the contract name to open its details

    work page

  21. [21]

    Confirm the deletion when prompted

    Scroll to the bottom (or top, depending on your version) and click Delete. Confirm the deletion when prompted. 0.79 LLM+GRAG 1. From the home page, go to the Commerce section. 2. In the dedicated menu, select the option to display the list of contracts or subscriptions. On the contract management page, you can: - View the list of existing contracts. - Use...

    work page

  22. [22]

    Click New Intervention

    work page

  23. [23]

    Select the customer and fill in date, staff, and description

    work page

  24. [24]

    0.88 LLM+G-RAG 1

    Click Save. 0.88 LLM+G-RAG 1. From the home page, go to the Commerce section

    work page

  25. [25]

    In the menu dedicated to interventions, select the option to create a new intervention

    work page

  26. [26]

    Use the add button if necessary to manage third parties

    On the intervention creation page, choose a third party from the drop-down list provided for this purpose. Use the add button if necessary to manage third parties

    work page

  27. [27]

    Confirm the creation of the draft intervention using the appropriate action

    work page

  28. [28]

    6.78 User query How to modify a client invoice LLM 1

    If necessary, cancel the process using the cancel option. 6.78 User query How to modify a client invoice LLM 1. Log in to Dolibarr and go to Commercial→Invoices→Customer Invoices

    work page

  29. [29]

    Find and open the invoice you want to modify

    work page

  30. [30]

    Click Modify, update the details such as products, quantities, prices, or dates

    work page

  31. [31]

    0.72 LLM+G-RAG 1

    Click Save to apply the changes. 0.72 LLM+G-RAG 1. Go to the Home section from the main menu to view the general dashboard. 2. From the dashboard, go to the Customer Invoices section to view the list of customer invoices

    work page

  32. [32]

    From the list of customer invoices, select the invoice you want to modify. 4. Use the available options to edit the invoice details, subject to restrictions related to the invoice status (for example, modifying a “Paid” invoice may be limited). 5. Apply the desired changes and save them. Features available on the customer invoice management page: - Displa...

    work page