REVIEW 3 major objections 6 minor 42 references
xpSHACL: Explainable SHACL Validation using Retrieval-Augmented Generation and Large Language Models
T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read xpSHACL claims that SHACL validation reports become understandable when each violation is given a rule-based justification tree, retrieved domain context, and an LLM-generated explanation, with a Violation KG caching the result for reuse.
desk verdict Good systems idea with an honest efficiency evaluation, but the Violation KG cache key is too coarse and can replay the wrong explanation across different shapes. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Two objects carry the argument. The justification tree is a structured, rule-based trace of a violation: its root is the failure, its branches are premises from SHACL shapes, observations from the data graph, and inference steps connecting them; it is what is supposed to keep the LLM's prose grounded in validator logic. The violation signature is an MD5 hash over constraint component, property path, and violation type, deliberately ignoring the specific focus node and value. It is the key to the Violation KG: the system looks up a signature before generating anything, and if an explanation in the requested language exists, it is retrieved directly; if not, the new explanation is written back under that signature. The combination is what makes explanations both traceable through the tree and reusable through the signature, and the reported 99.48% cache-hit rate is the quantitative payoff of that design.
What would settle it
A controlled task-based study with non-technical participants: give one group raw SHACL validation reports and the other xpSHACL explanations for the same violations, and ask each participant to locate the offending data and state a repair. If the explanation group is not meaningfully more accurate or faster, the central claim is falsified; the paper plans this study but has not run it.
Extended reading notes
Core claim
xpSHACL's claim is that the normal SHACL validation report—a list of focus nodes, source shapes, constraint components, and violating values—can be turned into something a domain expert can act on without knowing SHACL. It does this by having an extended validator capture each violation's details, a Justification Tree Builder construct a trace of the rule-based reasoning that produced the violation, a Context Retriever pull in ontology fragments, shape documentation, similar cases, and domain rules, and an Explanation Generator ask an LLM to rewrite all of that as natural-language text with correction suggestions, in a requested language. The Violation KG stores each generated explanation under a signature that abstracts away the specific data instance, so a later violation of the same kind reuses the stored text instead of invoking the LLM again. The reported experiments show that this caching is what makes the pipeline practical: the first uncached run on a test dataset took about 65 seconds against about 4 seconds for the baseline validator, while cached runs stabilized near 20 seconds and explanation retrieval averaged 0.12 seconds per ontology across a large public-ontology run. On the paper's own account, explanation quality is provisionally supported by manual inspection, not by user measurement.
Load-bearing premise
The load-bearing premise is that an LLM, given the justification tree and retrieved context, will reliably turn them into correct, actionable explanations that non-technical users can trust; if that conversion is frequently wrong, the cached explanations spread the error instead of fixing it.
Editorial extensions
If this is right
- After a warm cache, recurring violations are explained by retrieval in about 0.12 seconds instead of a full LLM generation, so the cost of explainability amortizes over repeated validation.
- Because every violation with the same signature returns the same stored text, explanations are consistent across datasets, runs, and users instead of varying with each LLM call.
- The system's dependence on LLM API calls nearly disappears in steady state: across 2,301 lookups only 12 unique signatures were generated, so a shared or pre-populated Violation KG could serve explanations without API keys.
- Multilingual support becomes a cache dimension: each signature can hold one explanation per language tag, so translation is paid once and then reused.
- The justification tree gives every explanation a checkable backing, so a user or developer can trace the natural-language claim back to the specific SHACL rule and data facts that produced it.
Reading between the lines
- Beyond the paper, if an LLM ever produces a wrong explanation, the cache amplifies the error: every later user who hits the same signature receives the same wrong text. The paper acknowledges this risk when it warns about propagation of inaccuracies from non-deterministic LLMs, but it does not measure how often that occurs.
- The signature abstracts away the shape ID, so two violations caused by different shapes but with the same constraint component, property path, and violation type receive identical text. Whether that conflation helps or misleads is testable by comparing user understanding for violations that share a signature but come from semantically different shapes.
- The cached-run overhead near 20 seconds is now dominated by validation, tree construction, and context retrieval rather than generation. This suggests that a faster RDF query engine, or caching the tree and context as intermediate artifacts, would shrink the gap to the baseline; the paper lists both ideas as future work.
- A natural extension is to make the Violation KG a quality store rather than a text cache, recording user edits, ratings, or repair outcomes per signature so explanations improve over time instead of merely repeating the first generated version.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents xpSHACL, a system that augments standard SHACL validation with rule-based justification trees, retrieval-augmented generation (RAG), and large language models (LLMs) to produce multilingual, human-readable explanations of constraint violations. A key component is a Violation KG that caches previously generated explanations keyed by a 'ViolationSignature', which the implementation defines as an MD5 hash of constraint component, property path, and violation type. The authors report an evaluation on 431 Linked Open Vocabularies (LOV) ontologies, a 99.48% cache hit rate across 2301 lookups, and a timing comparison against pyshacl showing a 65-second first run dropping to about 20 seconds on cached runs. User studies and quantitative quality metrics are deferred to future work.
Significance. If the correctness issues were addressed, xpSHACL would be a practical contribution: it targets a real barrier to SHACL adoption, provides an open-source modular implementation, and its efficiency claims are backed by reproducible timed runs and a concrete cache analysis. The architecture, combining symbolic justification trees with RAG and LLM generation, is clearly presented. However, the current manuscript does not substantiate the central quality claim of producing accurate, consistent explanations, and the Violation KG signature design introduces a deterministic risk of serving incorrect cached explanations across different shapes or ontologies.
major comments (3)
- [§5.6, §6, §7.2.1] The ViolationSignature is defined in §5.6 as an MD5 hash of constraint component, property path, and violation type, and §6 explicitly makes it independent of the shape ID and focus node. Consequently, the first violation with a given signature generates one explanation from its own justification tree and retrieved context, and every later violation sharing only that coarse signature receives identical text even when the shapes target different classes or carry different documentation. The LOV evaluation shows this is not hypothetical: 12 unique signatures cover 2301 lookups (99.48% hit rate), and the paper's own example explanation hard-codes the class name 'ObservableProperty' for a missing rdfs:label violation. A cache hit for a different ontology with the same signature would return text naming the wrong class and potentially the wrong shape's rationale. This is a deterministic flaw in the cache key, independent of LLM nondeterminism, and it undermines the central claim that the Violation KG provides accurate and consistent reusable explanations. The signature should include the shape ID (and possibly the target type or class), or the system should verify that the cached explanation's context matches the new violation before reuse; the effect on the reported 99.48% hit rate should then be re-measured.
- [§7.2.3, §7.3] Explanation quality (RQ1) and user satisfaction (RQ4) are not evaluated: §7.2.3 reports only 'manual inspection of a sample' and §7.3 acknowledges the lack of a formal user study as a significant limitation. The abstract and introduction claim the system produces 'detailed, multilanguage, human-readable explanations', but the evaluation provides no evidence of correctness, clarity, or actionability for target users, and no multilingual evaluation is reported (the LOV run requests English only). Since these quality claims are the primary motivation for the system, the paper should either include a structured evaluation (e.g., expert annotation or a small user study) or be repositioned as a system/architecture description with preliminary efficiency results, with quality claims explicitly deferred.
- [§7, RQ3] Consistency (RQ3) is listed as a research question but never measured. Section 7.2.3 only describes planned future work using text similarity metrics, and the cache collision described in the first major comment directly threatens consistency. The claim in §4.6 that the KG 'ensures a greater degree of consistency' is therefore unsupported. A simple quantitative consistency check (e.g., comparing the text of explanations retrieved for the same signature across different datasets or runs) should be added, especially because the system is explicitly designed to enforce consistency through caching.
minor comments (6)
- [Figure 2] The Figure 2 caption contains a garbled hex/unicode line (the '/uni00000014/...' sequence) that appears to be a corrupted artifact; it should be removed or replaced with a proper axis legend.
- [§7.2.3] After the 'Scalability Evaluation' paragraph, the subsections 'Consistency (RQ3)' and 'User Satisfaction (RQ4)' are duplicated with nearly identical content, creating the impression of an editing error; one copy should be deleted.
- [§8] The sentence 'Future work may focus on the full implementation and evaluation of the xpSHACL system' contradicts the implemented system and the evaluation results reported in Section 7; this appears to be leftover text from an earlier draft and should be rephrased.
- [§4.3.3] The SPARQL example uses 'ex:hasname' while the surrounding text uses 'ex:hasName'; the capitalization should be made consistent.
- [§5.6] MD5 is a weak and deprecated hash; since no security property is needed here, using a standard strong hash such as SHA-256 would avoid an unnecessary criticism.
- [§4.5, §7] Multilingual support is listed as a contribution and as a design goal of the Explanation Generator, but no evaluation or even an example of a non-English output is provided; the paper should at least clarify that this capability is inherited from the underlying LLM and has not been tested.
Circularity Check
No significant circularity: xpSHACL's architecture and cache evaluation are self-contained; the coarse cache signature is a correctness risk, not a derivation loop.
full rationale
xpSHACL is a systems and architecture paper; its quantitative claims are limited to implementation performance and cache hit rate. No target quantity is fitted to data and then re-reported as a prediction, and no equation reduces explanation quality to an input. The justification tree is constructed from pyshacl validation results and SHACL rules, and the LLM explanations are generated from those trees plus retrieved context, with no derivation loop. The 99.48% cache hit rate is an empirical outcome of the chosen signature scheme and the LOV evaluation, not a quantity derived from itself. The self-citation to SHARK (reference [31]) appears as a source of SHACL quality shapes and as a future integration target; it is not load-bearing for the central claim about explanation generation or caching. Section 8's caveat about propagation of inaccuracies from nondeterministic LLMs and Section 7.3's acknowledgment that no user study has been conducted are honest limitations, not evidence of circularity. The skeptic's concern about the coarse ViolationSignature that omits shape ID and focus node is a correctness and generalization risk in the cache design, not a circularity step, because the paper does not claim to derive explanation correctness from the signature. Consequently, no circular step is exhibited.
Assumptions & free parameters
assumptions (3)
- domain assumption pyshacl correctly implements SHACL validation for the constraints used in the evaluation.
- domain assumption The LLM produces factually grounded, non-hallucinated explanations from the prompt.
- ad hoc to paper The violation signature (MD5 of constraint component, property path, violation type) is a sound equivalence criterion for reusing explanations.
invented entities (1)
-
ViolationSignature (MD5-based triple of constraint component, property path, violation type)
Cite this review
Pith. "Pith review of xpSHACL: Explainable SHACL Validation using Retrieval-Augmented Generation and Large Language Models." pith.science (2026). https://pith.science/paper/6IERTI46
@misc{pith2026250708432,
author = {Pith},
title = {Pith review of: xpSHACL: Explainable SHACL Validation using Retrieval-Augmented Generation and Large Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/6IERTI46}},
note = {Machine review of arXiv:2507.08432}
}
read the original abstract
Shapes Constraint Language (SHACL) is a powerful language for validating RDF data. Given the recent industry attention to Knowledge Graphs (KGs), more users need to validate linked data properly. However, traditional SHACL validation engines often provide terse reports in English that are difficult for non-technical users to interpret and act upon. This paper presents xpSHACL, an explainable SHACL validation system that addresses this issue by combining rule-based justification trees with retrieval-augmented generation (RAG) and large language models (LLMs) to produce detailed, multilanguage, human-readable explanations for constraint violations. A key feature of xpSHACL is its usage of a Violation KG to cache and reuse explanations, improving efficiency and consistency.
Figures
Reference graph
Works this paper leans on
-
[1]
Shqiponja Ahmetaj, Robert David, Magdalena Ortiz, Axel Polleres, Bojken Shehu, and Mantas Simkus. 2021. Reasoning about explanations for non-validation in SHACL. In Description Logics
work page 2021
-
[2]
Shqiponja Ahmetaj, Timo Camillo Merkl, and Reinhard Pichler. 2024. Consistent query answering over SHACL constraints.arXiv preprint arXiv:2406.16653 (2024)
work page Pith review arXiv 2024
-
[3]
Natanael Arndt, Norman Radtke, and Michael Martin. 2016. Distributed collabo- ration on rdf datasets using git: Towards the quit store. InProceedings of the 12th International Conference on Semantic Systems . 25–32
work page 2016
-
[4]
Tim Berners-Lee, Dan Connolly, Lalana Kagal, Yosi Scharf, and Jim Hendler. 2008. N3Logic: A logical framework for the World Wide Web. Theory and Practice of Logic Programming 8, 3 (2008), 249–269
work page 2008
-
[5]
Federico Bianchi, Gaetano Rossiello, Luca Costabello, Matteo Palmonari, and Pasquale Minervini. 2020. Knowledge graph embeddings and explainable AI. In Knowledge Graphs for Explainable Artificial Intelligence: Foundations, Applications and Challenges. IOS Press, 49–72
work page 2020
-
[6]
Tomaž Bratanic. 2024. Using a knowledge graph to implement a rag application . Technical Report. Accessed: 2025-03-17
work page 2024
-
[7]
Jiajing Chen, Runyuan Bao, Hongye Zheng, Zhen Qi, Jianjun Wei, and Jiacheng Hu. 2024. Optimizing Retrieval-Augmented Generation with Elasticsearch for Enhanced Question-Answering Systems. arXiv preprint arXiv:2410.14167 (2024)
work page Pith review arXiv 2024
-
[8]
Ruixi Chen. 2025. Retrieval-Augmented Generation with Knowledge Graphs: A Survey. In Computer Science Undergradaute Conference 2025@ XJTU
work page 2025
Show all 42 references
-
[9]
Julien Corman, Fernando Florenzano, Juan L Reutter, and Ognjen Savković. 2019. Validating SHACL constraints over a SPARQL endpoint. InThe Semantic Web– ISWC 2019: 18th International Semantic Web Conference, Auckland, New Zealand, October 26–30, 2019, Proceedings, Part I 18 . S...
2019
-
[10]
Luis Miguel Vieira da Silva, Aljosha Kocher, Felix Gehlhoff, and Alexander Fay
-
[11]
Thomas Delva, Anastasia Dimou, Maxime Jakubowski, and Jan Van den Bussche
-
[12]
Thomas Delva, Birte De Smedt, Sitt Min Oo, Dylan Van Assche, Sven Lieber, and Anastasia Dimou. 2021. RML2SHACL: RDF generation taking shape. In Proceedings of the 11th Knowledge Capture Conference . 153–160
2021
-
[13]
Ronald Denaux, Catherine Dolbear, Glen Hart, Vania Dimitrova, and Anthony G. Cohn. 2011. Supporting domain experts to construct conceptual ontologies: A holistic approach. Journal of Web semantics 9, 2 (2011), 113–127
2011
-
[14]
Alex JA Donkers and Ekaterina Petrova. 2024. Converting Fire Safety Regulations to SHACL Shapes Using Natural Language Processing. In Proceedings of the 3rd NLP4KGC: Natural Language Processing for Knowledge Graph Construction co- located with the 20th International Conference...
2024
-
[15]
Monireh Ebrahimi, Md Kamruzzaman Sarker, Federico Bianchi, Ning Xie, Derek Doran, and Pascal Hitzler. 2018. Reasoning over RDF knowledge bases using deep learning. arXiv preprint arXiv:1811.04132 (2018)
2018 arXiv
-
[16]
Giuseppe Futia and Antonio Vetrò. 2020. On the integration of knowledge graphs into deep learning models for a more comprehensible AI—Three challenges for future research. Information 11, 2 (2020), 122
2020
-
[17]
Jose Emilio Labra Gayo, Eric Prud’Hommeaux, Iovka Boneva, and Dimitris Kontokostas. 2017. Validating RDF data. Morgan & Claypool Publishers
2017
-
[18]
Julian Alexander Gercke. 2022. Supporting Explainable AI on Semantic Constraint Validation. Master’s thesis. Hannover: Gottfried Wilhelm Leibniz Universität
2022
-
[19]
Lars Herbold, Mersedeh Sadeghi, and Andreas Vogelsang. 2024. Generating context-aware contrastive explanations in rule-based systems. In Proceedings of the 2024 Workshop on Explainability Engineering . 8–14
2024
-
[20]
Dimitris Kontokostas. 2014. Test-driven evaluation of linked data quality. In Proceedings of the 23rd international conference on World Wide Web . 747–758
2014
-
[21]
Ilaria Lécué, Freddy Lécué, and Pascal Hitzler. 2020. Knowledge graphs for ex- plainable artificial intelligence: Foundations, applications and challenges. (2020)
2020
-
[22]
Yading Li, Dandan Song, Changzhi Zhou, Yuhang Tian, Hao Wang, Ziyi Yang, and Shuhao Zhang. 2024. A Framework of Knowledge Graph-Enhanced Large Language Model Based on Question Decomposition and Atomic Retrieval. In Findings of the Association for Computational Linguistics: EMN...
2024
-
[23]
Linhao Luo, Yuan-Fang Li, Gholamreza Haffari, and Shirui Pan. 2023. Reasoning on graphs: Faithful and interpretable large language model reasoning. arXiv preprint arXiv:2310.01061 (2023)
2023 arXiv
-
[24]
Nicholas Matsumoto, Jay Moran, Hyunjun Choi, Miguel E Hernandez, Mythreye Venkatesan, Paul Wang, and Jason H Moore. 2024. KRAGEN: a knowledge graph- enhanced RAG framework for biomedical problem solving using large language models. Bioinformatics 40, 6 (2024), btae353
2024
-
[25]
Ben De Meester, Pieter Heyvaert, Dörthe Arndt, Anastasia Dimou, and Ruben Verborgh. 2021. RDF graph validation using rule-based reasoning. Semantic Web 12, 1 (2021), 117–142
2021
-
[26]
Sedir Mohammed, Felix Naumann, and Hazar Harmouch. 2025. Step-by-Step Data Cleaning Recommendations to Improve ML Prediction Accuracy. In Proceedings 28th International Conference on Extending Database Technology (EDBT) 2025, Barcelona, Spain, March 25-28 . 542–554. https://do...
2025 doi
-
[27]
Brett Mullins. 2023. The Shape of Explanations: A Topological Account of Rule-Based Explanations in Machine Learning. arXiv preprint arXiv:2301.09042 (2023)
2023 arXiv
-
[28]
Cem Okulmus and Mantas Šimkus. 2024. SHACL Validation under the Well- founded Semantics. In Proceedings of the International Conference on Principles of Knowledge Representation and Reasoning , Vol. 21. 553–562
2024
-
[29]
Paolo Pareti and George Konstantinidis. 2021. A review of SHACL: from data validation to schema reasoning for RDF graphs. Reasoning Web International Summer School (2021), 115–144
2021
-
[30]
Axel Polleres, Aidan Hogan, Renaud Delbru, and Jürgen Umbrich. 2013. RDFS and OWL reasoning for linked data. In Reasoning Web International Summer School. Springer, 91–149
2013
-
[31]
Gustavo Correa Publio. 2018. SHARK: A test-driven framework for design and evolution of ontologies. In The Semantic Web: ESWC 2018 Satellite Events: ESWC 2018 Satellite Events, Heraklion, Crete, Greece, June 3-7, 2018, Revised Selected Papers 15. 314–324
2018
-
[32]
Enayat Rajabi and Kobra Etminani. 2024. Knowledge-graph-based explainable AI: A systematic review. Journal of information science 50, 4 (2024), 1019–1029
2024
-
[33]
Eclipse RDF4J. 2020. Validation With SHACL. https://rdf4j.org/documentation/programming/shacl/. Accessed: 2025-03-16
2020
-
[34]
Ashley Sommer, Nicholas Car, and Jonathan Yu. 2021. pySHACL. DOI: https://doi.org/10.5281/zenodo.4750840 (2021)
2021 doi
-
[35]
Yifan Song, Guoyin Wang, Sujian Li, and Bill Yuchen Lin. 2024. The good, the bad, and the greedy: Evaluation of LLMs should not ignore non-determinism. arXiv preprint arXiv:2407.10457 (2024)
2024 arXiv
-
[36]
XAQT Team. 2023. Mastering Rule-Based Systems: Implementation, Benefits, and Best Practices. https://www.xaqt.com/blog/mastering-rule-based-systems/. Accessed: 2025-03-17
2023
-
[37]
World Wide Web Consortium W3C. 2017. Shapes Constraint Language (SHACL). https://www.w3.org/TR/shacl/. Accessed: 2025-03-17
2017
-
[38]
Mengqi Zhang, Xiaotian Ye, Qiang Liu, Pengjie Ren, Shu Wu, and Zhumin Chen
-
[39]
Xishi Zhu, Xiaoming Guo, Shengting Cao, Shenglin Li, and Jiaqi Gong. 2024. StructuGraphRAG: Structured Document-Informed Knowledge Graphs for Retrieval-Augmented Generation. In Proceedings of the AAAI Symposium Se- ries, Vol. 4. 242–251
2024
-
[41]
arXiv preprint arXiv:2402.13593 (2024)
Knowledge graph enhanced large language model editing. arXiv preprint arXiv:2402.13593 (2024)
2024 arXiv
-
[2023]
In Proceedings 26th International Conference on Extending Database Technology (EDBT 2023)
Data provenance for SHACL. In Proceedings 26th International Conference on Extending Database Technology (EDBT 2023) . OpenProceedings. org
2023
-
[2024]
In 2024 IEEE 29th International Conference on Emerging Technologies and Factory Automation (ETFA)
On the use of large language models to generate capability ontologies. In 2024 IEEE 29th International Conference on Emerging Technologies and Factory Automation (ETFA). IEEE, 1–8
2024
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.