Pith. sign in

REVIEW 3 major objections 5 minor 69 references

CypherBench: Towards Precise Retrieval over Full-scale Modern Knowledge Graphs in the LLM Era

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Property graph views of Wikidata can be queried by LLMs with Cypher, and a new benchmark shows 61.58% execution accuracy for the best model.

desk verdict A valuable large-scale text-to-Cypher benchmark whose headline numbers are clouded by unresolved entity-name collisions in the gold queries. read the letter →

arxiv 2412.18702 v2 pith:6TSNVTLR submitted 2024-12-24 cs.CL cs.AIcs.DB

Yanlin Feng , Simone Papicchio , Sajjadur Rahman This is my paper · ORCID
classification cs.CLcs.AIcs.DB
keywords text-to-CypherknowledgegraphquestionansweringRDFtopropertyconversionLLMretrievalWikidataexecutionaccuracyprovenancesubgraphJaccardsimilarity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that modern RDF knowledge graphs like Wikidata are inefficient for LLM retrieval because their schemas are far too large, they use unwieldy resource identifiers, their relation types overlap, and they lack normalization. As a fix, it proposes turning an RDF graph into multiple domain-specific property graph views that an LLM can query in Cypher. It instantiates this idea as CypherBench, 11 property graphs containing 7.8 million entities and over 10,000 natural-language to Cypher tasks. The core claim is that this conversion makes precise, full-scale knowledge-graph retrieval practical for LLMs without needing external entity linkers, with the best tested model reaching 61.58% execution accuracy.

What carries the argument

The central machinery is a pipeline that converts RDF triples into schema-enforced property graphs: a curated per-domain schema maps Wikidata QIDs and PIDs to clean node and relation labels, and an engine issues SPARQL queries against a local Wikidata endpoint to fetch conforming triples, then applies datatype conversion, date precision filtering, unit standardization, and rank filtering. On top of this sit the task generation pipeline (templated MATCH and RETURN clause instantiation, LLM rewriting, and verification) and the two evaluation metrics: execution accuracy (EX), comparing returned tables, and provenance subgraph Jaccard similarity (PSJS), isolating how well the MATCH clause finds the right subgraph.

What would settle it

Take the CypherBench test instances and, for each question that names an entity, count how many distinct nodes in its graph share that exact name. Recompute execution accuracy on the subset where the count is greater than one, and compare it to the singleton-name subset; if the accuracy is markedly lower or the ground-truth queries change under the alternative entities, the clean-precision claim would need qualification.

Watch

Extended reading notes

Core claim

The paper claims that the obstacle to LLM retrieval from large RDF knowledge graphs is not the graphs' size but their modeling choices, and that a materialized property graph view per domain removes the bulk of the difficulty. The reported evidence is CypherBench, built from a Wikidata April 2024 dump, where the strongest model (claude3.5-sonnet) achieves 61.58% execution accuracy and 80.85% provenance subgraph Jaccard similarity, and gpt-4o achieves 60.18%. The intended consequence is that graph retrieval over full-scale encyclopedic knowledge becomes feasible through text-to-Cypher, a capability the paper says leading LLM frameworks lack for RDF graphs.

Load-bearing premise

The evaluation assumes that an entity's name is enough to identify that entity in the graph, because the pipeline keeps names and purposely avoids entity linking, and Wikidata does not enforce unique labels.

Editorial extensions

If this is right

  • If the central claim holds, LLMs can retrieve answers from Wikidata-scale graphs by writing Cypher instead of relying on embedding-based top-k retrieval or external entity linkers.
  • The benchmark separates graph-matching skill from formatting and aggregation skill via PSJS, which could become a standard diagnostic for text-to-query evaluation.
  • The result that no tested sub-10B-parameter model exceeds 20% execution accuracy suggests CypherBench can serve as a scaling signal for LLM graph reasoning ability.
  • The RDF-to-property-graph engine, if generalized, gives a reusable bridge from SPARQL-style world knowledge to the property graph interfaces used in enterprise databases.
  • Because execution accuracy requires running queries, CypherBench provides a testbed for measuring whether LLMs improve at precise retrieval as they grow, rather than just at paraphrase.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A likely stress test is entity-name ambiguity: since the pipeline intentionally preserves names and skips entity linking, questions that mention names shared by multiple entities may have ambiguous gold answers, so reported execution accuracy may partly reward guessing the right name rather than retrieving the right entity.
  • The property graph views are domain-specific, so a question spanning two views (for example, combining movie and politics facts) would fall outside the benchmark; a follow-up could create cross-view or union queries and investigate how well Cypher's composability handles them.
  • The benchmark could be adapted to other RDF sources (DBpedia, Freebase, domain-specific triplestores) by regenerating schemas and task instances, providing an empirical measure of how transferable the conversion benefit is.
  • PSJS, as a RETURN-clause-independent score, could be adopted by the broader text-to-SQL community as an auxiliary metric for diagnosing whether errors come from the JOIN structure or the projection/aggregation logic.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper argues that modern RDF knowledge graphs such as Wikidata are difficult for LLM-based retrieval because of overly large schemas, use of IRIs, overlapping relation types, and lack of normalization, and proposes converting such graphs into domain-specific property graphs queried via Cypher. It describes an RDF-to-property-graph conversion engine, a template-based pipeline that generates over 10,000 (question, Cypher) pairs across 11 Wikidata-derived graphs, and two evaluation metrics: execution accuracy (EX) and provenance subgraph Jaccard similarity (PSJS). The experimental evaluation covers 12 LLMs in a zero-shot text-to-Cypher setting, with claude3.5-sonnet reaching 61.58% EX and gpt-4o reaching 60.18%, while models under 10B parameters stay below 20%.

Significance. If the benchmark's validity holds, this is a valuable contribution: CypherBench is the first large-scale text-to-Cypher benchmark built on domain-complete Wikidata subsets, and the public dataset, code, Docker deployment, and evaluation scripts make it reproducible. The PSJS metric is a useful addition because it separates graph-matching errors from return-format errors, and the inclusion of global, temporal, union, and optional-match queries fills a gap relative to prior KBQA benchmarks. The construction pipeline is careful in several respects: schema curation, rank filtering, datatype conversion, unit standardization, multi-round LLM verification, and manual inspection of the test set. The main validity risk is the use of entity names as keys in ground-truth queries, which can make gold answers ambiguous when distinct entities share the same name; this needs to be quantified and addressed before the headline accuracies can be interpreted as clean measures of precise retrieval.

major comments (3)
  1. [§4.5, Table 13, §6.3, Table 10] The ground-truth Cypher queries identify named entities solely by their name property (e.g., Table 11, Q4–Q6), and §4.5 states that entity names are intentionally preserved to avoid entity linking. Because Wikidata labels are not unique, a MATCH predicate such as {name: '...'} can bind multiple nodes in the same graph, and the paper itself acknowledges this in Table 10 ('if multiple distinct entities share the same name, their names should be repeated as separate entries') and in its error taxonomy (Table 13, Entity Linking; §6.3, Incorrect Deduplication). When a gold query binds several same-named entities, the executed gold result can include unintended rows or aggregate over unintended entities, while a model that correctly isolates the intended entity is scored as wrong. The paper does not quantify how many test instances contain same-name collisions within the relevant graph. Please report the collision rate per graph and per split, and either filter or repair affected instances or add an entity-ID-based evaluation track; without this, the reported EX/PSJS numbers are not a clean measure of retrieval accuracy.
  2. [§2.2, §2.4, §6.2] The motivating claim is that RDF graphs are less efficient for LLMs and that property graph views are the solution, but Section 6 only evaluates text-to-Cypher on the converted property graphs. There is no text-to-SPARQL baseline over Wikidata, or over the same domain subsets, that would test the four claimed RDF difficulties (schema size, resource identifiers, overlapping relation types, and lack of normalization). The root-cause argument is therefore a plausible qualitative analysis rather than an empirically supported comparison. I recommend adding a zero-shot SPARQL baseline with comparable schema information and label access, and reporting the same or an equivalent execution-based metric, so the claimed advantage of property graph views is directly tested.
  3. [§6.1, Table 3] The evaluation section does not report the sampling temperature or the number of runs for any model, and Table 3 gives a single EX and PSJS value per model. Because LLM generation is stochastic, differences of a few points (e.g., claude3.5-sonnet vs. gpt-4o, or qwen2.5-72b vs. llama3.1-70b) may be within run-to-run noise. Please run each model multiple times, or explicitly set temperature to 0 and state that, and report means, standard deviations, or a significance test so that the model ranking and the headline accuracy figures are reproducible.
minor comments (5)
  1. [Title and Abstract] The phrase 'full-scale modern knowledge graphs' is stronger than what the benchmark actually provides: each graph is a domain-complete subset of Wikidata (7.8M entities out of roughly 114M), not the full Wikidata graph. Please qualify the wording, for example by saying 'domain-complete Wikidata subsets'.
  2. [Figures 4 and 5] The figures are referenced with pattern symbols and RETURN-template names that are hard to map to the tables in Appendix A; adding explicit legends or labeled axis categories would improve readability.
  3. [Table 6 and §7.2] The SpCQL row in Table 6 reports '480k relation types', but the footnote in §7.2 says that OwnThink is stored with a single relation type when loaded into Neo4j. This apparent inconsistency should be clarified.
  4. [§5.2, Eq. (2)] The paper should specify how the provenance subgraph and the associated `RETURN *` are defined for queries containing UNION, UNWIND, or aggregation, since a single `RETURN *` over such queries is not described.
  5. [References] Reference [58] appears to contain a typo: 'Ziije Zhong' should be 'Zijie Zhong'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the empirical benchmark results are not derived from their own inputs, and the paper's cited limitations concern measurement validity, not circular reasoning.

full rationale

CypherBench is an empirical benchmark rather than a derivation. The central claims are that property graph views over RDF can be queried by LLMs with measurable execution accuracy, and the evidence is obtained by executing LLM-generated Cypher against Neo4j and comparing it with ground-truth Cypher derived from Wikidata through an RDF-to-property-graph conversion engine. Ground-truth queries are produced by template instantiation, LLM rewriting, and human verification; they are not defined in terms of the models' outputs, and no parameter is fitted to the test set. The metrics EX and PSJS are standard execution-accuracy and Jaccard-style measures, adapted from Spider and applied to independently constructed gold queries. The only self-referential element is that the paper evaluates its own proposed representation rather than a competing external representation, but this is standard benchmark practice and does not make the reported numbers true by construction. The paper's own stated limitation about preserving entity names to avoid entity linking (Section 4.5), and the name-collision and deduplication errors catalogued in Table 13, are genuine threats to the validity of execution accuracy as a clean measure of retrieval ability; however, those are data-ambiguity concerns, not circularity, because the gold answers are derived from Wikidata, not from the predictions being scored. The self-citations present (references [22] and [40]) appear only in related-work contexts and are not load-bearing premises. No circular step is exhibited with quoted evidence, so the appropriate score is 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The benchmark relies on manual schema curation, Wikidata rank semantics, and LLM rewriting fidelity rather than fitted numerical parameters. The three listed thresholds are hand-chosen design decisions that shape the question set. No new physical or conceptual entities are postulated; the property graph views are data artifacts, not invented entities.

free parameters (3)
  • Max query result rows = 10^5 records
    Design threshold to remove queries with unwieldy outputs; affects benchmark composition.
  • Max ground-truth execution time = 30 seconds
    Design threshold to keep benchmark executable; affects which questions are included.
  • Inference timeout multiplier = 120 seconds (4x ground-truth max)
    Execution timeout for predicted Cypher queries in Table 3; chosen to balance false timeouts.
assumptions (4)
  • domain assumption Curation mappings from Wikidata QIDs/PIDs to domain schemas are complete and correct
    Section 3.1: schemas are manually curated; correctness of the property graphs depends on these mappings.
  • domain assumption Rank filtering semantics: preferred/normal ranks reflect temporal validity
    Section 3.2: the engine fetches non-deprecated ranks with time qualifiers; if rank semantics are misread, time-sensitive answers are wrong.
  • domain assumption LLM rewriting preserves question meaning after three verification rounds
    Section 4.5: if rewriting changes semantics, the (question, Cypher) pairs are misaligned.
  • standard math Spider execution accuracy implementation transfers to Cypher table comparison
    Section 5.1: row and column permutations are used; assumes Cypher result serialization is faithful.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CypherBench: Towards Precise Retrieval over Full-scale Modern Knowledge Graphs in the LLM Era." pith.science (2026). https://pith.science/paper/6TSNVTLR

@misc{pith2026241218702,
  author       = {Pith},
  title        = {Pith review of: CypherBench: Towards Precise Retrieval over Full-scale Modern Knowledge Graphs in the LLM Era},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6TSNVTLR}},
  note         = {Machine review of arXiv:2412.18702}
}
read the original abstract

Retrieval from graph data is crucial for augmenting large language models (LLM) with both open-domain knowledge and private enterprise data, and it is also a key component in the recent GraphRAG system (edge et al., 2024). Despite decades of research on knowledge graphs and knowledge base question answering, leading LLM frameworks (e.g. Langchain and LlamaIndex) have only minimal support for retrieval from modern encyclopedic knowledge graphs like Wikidata. In this paper, we analyze the root cause and suggest that modern RDF knowledge graphs (e.g. Wikidata, Freebase) are less efficient for LLMs due to overly large schemas that far exceed the typical LLM context window, use of resource identifiers, overlapping relation types and lack of normalization. As a solution, we propose property graph views on top of the underlying RDF graph that can be efficiently queried by LLMs using Cypher. We instantiated this idea on Wikidata and introduced CypherBench, the first benchmark with 11 large-scale, multi-domain property graphs with 7.8 million entities and over 10,000 questions. To achieve this, we tackled several key challenges, including developing an RDF-to-property graph conversion engine, creating a systematic pipeline for text-to-Cypher task generation, and designing new evaluation metrics.

Figures

Figures reproduced from arXiv: 2412.18702 by the authors.

Figure 1
Figure 1. An illustration of Cypher as a unified interface for retrieval over both RDF and property [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. Schema of the company graph with entity proper￾ties an relation properties. See Appendix B for other graphs. Each property is assigned a datatype. Properties that represent quantities are also given a unit, which is in￾dicated in the property label (e.g., runtime_minute) to inform LLMs during graph retrieval. The authors created all 11 property graph schemas from scratch, with an average time investment of ap￾proxim… view at source ↗
Figure 4
Figure 4. Distribution of graph matching patterns, [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: Performance across basic and special MATCH patterns, RETURN templates and domains. 6.3 Performance Across Graph Matching Patterns Next, we analyze the performance breakdown across various dimensions. For this analysis, we focus on gpt-4o, claude3.5-sonnet, qwen2.5-72b,…
Figure 6
Figure 6. Figure 6: Distribution of errors made by gpt-4o and llama3.1-8b on 50 randomly sampled incorrect predictions. Note that a model might make multiple errors on one instance. 6.6 Error Analysis We further conduct an error analysis to investigate the types of errors made by LLMs. Sp…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

69 extracted references · 51 canonical work pages

  1. [1]

    From local to global: A graph rag approach to query-focused summarization

    Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, and Jonathan Larson. From local to global: A graph rag approach to query-focused summarization. arXiv preprint arXiv:2404.16130, 2024

    arXiv 2024

  2. [2]

    Wikidata: a free collaborative knowledgebase

    Denny Vrandeˇci´c and Markus Krötzsch. Wikidata: a free collaborative knowledgebase. Com- munications of the ACM, 57(10):78–85, 2014

    work page 2014

  3. [3]

    Semantic parsing on Freebase from question-answer pairs

    Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on Freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1533–1544, Seattle, Washington, USA, October 2013. Association for Computational Linguistics

    work page 2013

  4. [4]

    OpenDialKG: Explainable conversational reasoning with attention-based walks over knowledge graphs

    Seungwhan Moon, Pararth Shah, Anuj Kumar, and Rajen Subba. OpenDialKG: Explainable conversational reasoning with attention-based walks over knowledge graphs. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 845–854, Florence, Italy, July 2019. Association for Computational Linguistics

    work page 2019

  5. [5]

    Lc-quad 2.0: A large dataset for complex question answering over wikidata and dbpedia

    Mohnish Dubey, Debayan Banerjee, Abdelrahman Abdelkawi, and Jens Lehmann. Lc-quad 2.0: A large dataset for complex question answering over wikidata and dbpedia. In The Semantic Web–ISWC 2019: 18th International Semantic Web Conference, Auckland, New Zealand, October 26–30, 2019, Proceedings, Part II 18, pages 69–78. Springer, 2019

    work page 2019

  6. [6]

    Beyond iid: three levels of generalization for question answering on knowledge bases

    Yu Gu, Sue Kase, Michelle Vanni, Brian Sadler, Percy Liang, Xifeng Yan, and Yu Su. Beyond iid: three levels of generalization for question answering on knowledge bases. In Proceedings of the Web Conference 2021, pages 3477–3488, 2021

    work page 2021

  7. [7]

    KQA pro: A dataset with explicit compositional programs for complex question answering over knowledge base

    Shulin Cao, Jiaxin Shi, Liangming Pan, Lunyiu Nie, Yutong Xiang, Lei Hou, Juanzi Li, Bin He, and Hanwang Zhang. KQA pro: A dataset with explicit compositional programs for complex question answering over knowledge base. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6101–6119, Dubl...

    work page 2022

  8. [8]

    Graph retrieval-augmented generation: A survey

    Boci Peng, Yun Zhu, Yongchao Liu, Xiaohe Bo, Haizhou Shi, Chuntao Hong, Yan Zhang, and Siliang Tang. Graph retrieval-augmented generation: A survey. arXiv preprint arXiv:2408.08921, 2024

    arXiv 2024

Show all 69 references
  1. [9]

    FactKG: Fact verification via reasoning on knowledge graphs

    Jiho Kim, Sungjin Park, Yeonsu Kwon, Yohan Jo, James Thorne, and Edward Choi. FactKG: Fact verification via reasoning on knowledge graphs. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computation...

    2023

  2. [10]

    Freebase: a collaboratively created graph database for structuring human knowledge

    Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. Freebase: a collaboratively created graph database for structuring human knowledge. In Proceedings of the 2008 ACM SIGMOD international conference on Management of data, pages 1247–1250, 2008

    2008

  3. [11]

    Yago: a core of semantic knowl- edge

    Fabian M Suchanek, Gjergji Kasneci, and Gerhard Weikum. Yago: a core of semantic knowl- edge. In Proceedings of the 16th international conference on World Wide Web, pages 697–706, 2007

    2007

  4. [12]

    Dbpedia–a large- scale, multilingual knowledge base extracted from wikipedia

    Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick Van Kleef, Sören Auer, et al. Dbpedia–a large- scale, multilingual knowledge base extracted from wikipedia. Semantic web, 6(2):167–195, 2015

    2015

  5. [13]

    Direct fact retrieval from knowledge graphs without entity linking

    Jinheon Baek, Alham Fikri Aji, Jens Lehmann, and Sung Ju Hwang. Direct fact retrieval from knowledge graphs without entity linking. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Ling...

    2023

  6. [14]

    Llm-based sparql query generation from natural language over federated knowledge graphs

    Vincent Emonet, Jerven Bolleman, Severine Duvaud, Tarcisio Mendes de Farias, and Ana Clau- dia Sima. Llm-based sparql query generation from natural language over federated knowledge graphs. arXiv preprint arXiv:2410.06062, 2024

    arXiv

  7. [15]

    GraphQ IR: Unifying the semantic parsing of graph query languages with one intermediate representation

    Lunyiu Nie, Shulin Cao, Jiaxin Shi, Jiuding Sun, Qi Tian, Lei Hou, Juanzi Li, and Jidong Zhai. GraphQ IR: Unifying the semantic parsing of graph query languages with one intermediate representation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language...

    2022

  8. [16]

    Compositional gen- eralization in semantic parsing: Pre-training vs

    Daniel Furrer, Marc van Zee, Nathan Scales, and Nathanael Schärli. Compositional gen- eralization in semantic parsing: Pre-training vs. specialized architectures. arXiv preprint arXiv:2007.08970, 2020

    2007 arXiv

  9. [17]

    Sparqlgen: One-shot prompt-based approach for sparql query generation

    Liubov Kovriguina, Roman Teucher, Daniil Radyush, Dmitry Mouromtsev, N Keshan, S Neu- maier, AL Gentile, and S Vahdati. Sparqlgen: One-shot prompt-based approach for sparql query generation. In SEMANTiCS (Posters & Demos), 2023

    2023

  10. [18]

    Modern baselines for sparql semantic parsing

    Debayan Banerjee, Pranav Ajit Nair, Jivat Neet Kaur, Ricardo Usbeck, and Chris Biemann. Modern baselines for sparql semantic parsing. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2260–2265, 2022

    2022

  11. [19]

    A knowledge graph to interpret clinical proteomics data

    Alberto Santos, Ana R Colaço, Annelaura B Nielsen, Lili Niu, Maximilian Strauss, Philipp E Geyer, Fabian Coscia, Nicolai J Wewer Albrechtsen, Filip Mundt, Lars Juhl Jensen, et al. A knowledge graph to interpret clinical proteomics data. Nature biotechnology, 40(5):692–702, 2022

    2022

  12. [20]

    Systematic integration of biomedical knowledge prioritizes drugs for repurposing

    Daniel Scott Himmelstein, Antoine Lizee, Christine Hessler, Leo Brueggeman, Sabrina L Chen, Dexter Hadley, Ari Green, Pouya Khankhanian, and Sergio E Baranzini. Systematic integration of biomedical knowledge prioritizes drugs for repurposing. Elife, 6:e26726, 2017

    2017

  13. [21]

    Falcon 2.0: An entity and relation linking tool over wikidata

    Ahmad Sakor, Kuldeep Singh, Anery Patel, and Maria-Esther Vidal. Falcon 2.0: An entity and relation linking tool over wikidata. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pages 3141–3148, 2020

    2020

  14. [22]

    Calibrated seq2seq models for efficient and generalizable ultra-fine entity typing

    Yanlin Feng, Adithya Pratapa, and David Mortensen. Calibrated seq2seq models for efficient and generalizable ultra-fine entity typing. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 15550–1556...

    2023

  15. [23]

    Reasoning on graphs: Faithful and interpretable large language model reasoning

    LINHAO LUO, Yuan-Fang Li, Reza Haf, and Shirui Pan. Reasoning on graphs: Faithful and interpretable large language model reasoning. In The Twelfth International Conference on Learning Representations, 2024

    2024

  16. [24]

    Decaf: Joint decoding of answers and logical forms for question answering over knowledge bases

    Donghan Yu, Sheng Zhang, Patrick Ng, Henghui Zhu, Alexander Hanbo Li, Jun Wang, Yiqun Hu, William Yang Wang, Zhiguo Wang, and Bing Xiang. Decaf: Joint decoding of answers and logical forms for question answering over knowledge bases. In The Eleventh International Conference on...

    2022

  17. [25]

    Unikgqa: Unified retrieval and reasoning for solving multi-hop question answering over knowledge graph

    Jinhao Jiang, Kun Zhou, Xin Zhao, and Ji-Rong Wen. Unikgqa: Unified retrieval and reasoning for solving multi-hop question answering over knowledge graph. In The Eleventh International Conference on Learning Representations, 2023

    2023

  18. [26]

    Semantic parsing via paraphrasing

    Jonathan Berant and Percy Liang. Semantic parsing via paraphrasing. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1415–1425, Baltimore, Maryland, June 2014. Association for Computational Linguistics

    2014

  19. [27]

    Semantic parsing via staged query graph generation: Question answering with knowledge base

    Wen-tau Yih, Ming-Wei Chang, Xiaodong He, and Jianfeng Gao. Semantic parsing via staged query graph generation: Question answering with knowledge base. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Co...

    2015

  20. [28]

    Few-shot in- context learning on knowledge base question answering

    Tianle Li, Xueguang Ma, Alex Zhuang, Yu Gu, Yu Su, and Wenhu Chen. Few-shot in- context learning on knowledge base question answering. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational L...

  21. [29]

    On generating characteristic-rich question sets for QA evaluation

    Yu Su, Huan Sun, Brian Sadler, Mudhakar Srivatsa, Izzeddin Gür, Zenghui Yan, and Xifeng Yan. On generating characteristic-rich question sets for QA evaluation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 562–572, Austin, Tex...

    2016

  22. [30]

    Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task

    Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task. In Proceed...

    2018

  23. [31]

    Introducing the property graph index: A powerful new way to build knowledge graphs with llms, 2023

    LlamaIndex Team. Introducing the property graph index: A powerful new way to build knowledge graphs with llms, 2023. Accessed: 2024-10-24

    2023

  24. [32]

    Question answering over Freebase with multi- column convolutional neural networks

    Li Dong, Furu Wei, Ming Zhou, and Ke Xu. Question answering over Freebase with multi- column convolutional neural networks. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language...

  25. [33]

    UniK-QA: Unified representations of structured and unstructured knowledge for open-domain question answering

    Barlas Oguz, Xilun Chen, Vladimir Karpukhin, Stan Peshterliev, Dmytro Okhonko, Michael Schlichtkrull, Sonal Gupta, Yashar Mehdad, and Scott Yih. UniK-QA: Unified representations of structured and unstructured knowledge for open-domain question answering. In Findings of the Ass...

    2022

  26. [34]

    Beyond ned: fast and effective search space reduction for complex question answering over knowledge bases

    Philipp Christmann, Rishiraj Saha Roy, and Gerhard Weikum. Beyond ned: fast and effective search space reduction for complex question answering over knowledge bases. In Proceedings of the fifteenth ACM international conference on web search and data mining, pages 172–180, 2022

    2022

  27. [35]

    Conversational question answer- ing on heterogeneous sources

    Philipp Christmann, Rishiraj Saha Roy, and Gerhard Weikum. Conversational question answer- ing on heterogeneous sources. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 144–154, 2022

    2022

  28. [36]

    Explainable conversational ques- tion answering over heterogeneous sources via iterative graph neural networks

    Philipp Christmann, Rishiraj Saha Roy, and Gerhard Weikum. Explainable conversational ques- tion answering over heterogeneous sources via iterative graph neural networks. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information R...

    2023

  29. [37]

    Multi-hop question answering under temporal knowledge editing

    Keyuan Cheng, Gang Lin, Haoyang Fei, Lu Yu, Muhammad Asif Ali, Lijie Hu, Di Wang, et al. Multi-hop question answering under temporal knowledge editing. arXiv preprint arXiv:2404.00492, 2024

    2024 arXiv

  30. [38]

    Subgraph retrieval enhanced model for multi-hop knowledge base question answering

    Jing Zhang, Xiaokang Zhang, Jifan Yu, Jian Tang, Jie Tang, Cuiping Li, and Hong Chen. Subgraph retrieval enhanced model for multi-hop knowledge base question answering. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long P...

    2022

  31. [39]

    QA- GNN: Reasoning with language models and knowledge graphs for question answering

    Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut, Percy Liang, and Jure Leskovec. QA- GNN: Reasoning with language models and knowledge graphs for question answering. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Lingu...

    2021

  32. [40]

    Scalable multi-hop relational reasoning for knowledge-aware question answering

    Yanlin Feng, Xinyue Chen, Bill Yuchen Lin, Peifeng Wang, Jun Yan, and Xiang Ren. Scalable multi-hop relational reasoning for knowledge-aware question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1295–1309, ...

    2020

  33. [41]

    Stark: Benchmarking llm retrieval on textual and relational knowledge bases

    Shirley Wu, Shiyu Zhao, Michihiro Yasunaga, Kexin Huang, Kaidi Cao, Qian Huang, Vassilis N Ioannidis, Karthik Subbian, James Zou, and Jure Leskovec. Stark: Benchmarking llm retrieval on textual and relational knowledge bases. arXiv preprint arXiv:2404.13207, 2024

    2024 arXiv

  34. [42]

    Open domain question answering with a unified knowledge interface

    Kaixin Ma, Hao Cheng, Xiaodong Liu, Eric Nyberg, and Jianfeng Gao. Open domain question answering with a unified knowledge interface. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1605–1620, Dublin, Ir...

    2022

  35. [43]

    Paths- over-graph: Knowledge graph empowered large language model reasoning

    Xingyu Tan, Xiaoyang Wang, Qing Liu, Xiwei Xu, Xin Yuan, and Wenjie Zhang. Paths- over-graph: Knowledge graph empowered large language model reasoning. arXiv preprint arXiv:2410.14211, 2024

    2024 arXiv

  36. [44]

    Karpa: A training-free method of adapting knowledge graph as references for large language model’s reasoning path aggregation

    Siyuan Fang, Kaijing Ma, Tianyu Zheng, Xinrun Du, Ningxuan Lu, Ge Zhang, and Qingkun Tang. Karpa: A training-free method of adapting knowledge graph as references for large language model’s reasoning path aggregation. arXiv preprint arXiv:2412.20995, 2024

    2024 arXiv

  37. [45]

    The web as a knowledge-base for answering complex questions

    Alon Talmor and Jonathan Berant. The web as a knowledge-base for answering complex questions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 641–65...

    2018

  38. [46]

    Formal query building with query structure prediction for complex question answering over knowledge base

    Yongrui Chen, Huiying Li, Yuncheng Hua, and Guilin Qi. Formal query building with query structure prediction for complex question answering over knowledge base. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelli...

    2021

  39. [47]

    Langchain.https://github.com/langchain-ai/langchain, October 2022

    Harrison Chase. Langchain.https://github.com/langchain-ai/langchain, October 2022. Version released on 2022-10-17

    2022

  40. [48]

    Sparkle: Enhancing sparql generation with direct kg integration in decoding

    Jaebok Lee and Hyeonjeong Shin. Sparkle: Enhancing sparql generation with direct kg integration in decoding. arXiv preprint arXiv:2407.01626, 2024

    2024 arXiv

  41. [49]

    Fine-tuned LLMs know more, hallucinate less with few-shot sequence-to- sequence semantic parsing over Wikidata

    Silei Xu, Shicheng Liu, Theo Culhane, Elizaveta Pertseva, Meng-Hsi Wu, Sina Semnani, and Monica Lam. Fine-tuned LLMs know more, hallucinate less with few-shot sequence-to- sequence semantic parsing over Wikidata. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedin...

    2023

  42. [50]

    SPINACH: SPARQL-based information navigation for challenging real-world questions

    Shicheng Liu, Sina Semnani, Harold Triedman, Jialiang Xu, Isaac Dan Zhao, and Monica Lam. SPINACH: SPARQL-based information navigation for challenging real-world questions. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Findings of the Association for Computati...

    2024

  43. [51]

    Unioqa: A unified framework for knowledge graph question answering with large language models

    Zhuoyang Li, Liran Deng, Hui Liu, Qiaoqiao Liu, and Junzhao Du. Unioqa: A unified framework for knowledge graph question answering with large language models. arXiv preprint arXiv:2406.02110, 2024

    2024 arXiv

  44. [52]

    Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls

    Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, et al. Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls. Advances in Neural Information Processing Systems, 36, 2024

    2024

  45. [53]

    $ R^3$-NL2GQL: A Model Coordination and Knowledge Graph Alignment Approach for NL2GQL

    Yuhang Zhou, Yu He, Siyu Tian, Yuchen Ni, Zhangyue Yin, Xiang Liu, Chuanjun Ji, Sen Liu, Xipeng Qiu, Guangnan Ye, et al. $ R^3$-NL2GQL: A Model Coordination and Knowledge Graph Alignment Approach for NL2GQL. arXiv preprint arxiv:2311.01862, 2024. 19

    2024 arXiv

  46. [54]

    Aligning large language models to a domain-specific graph database

    Yuanyuan Liang, Keren Tan, Tingyu Xie, Wenbiao Tao, Siyuan Wang, Yunshi Lan, and Weining Qian. Aligning large language models to a domain-specific graph database. arXiv preprint arXiv:2402.16567, 2024

    2024 arXiv

  47. [55]

    Spcql: A semantic pars- ing dataset for converting natural language into cypher

    Aibo Guo, Xinyi Li, Guanchen Xiao, Zhen Tan, and Xiang Zhao. Spcql: A semantic pars- ing dataset for converting natural language into cypher. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, pages 3973–3977, 2022

    2022

  48. [56]

    Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav...

    2019

  49. [57]

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Proce...

    2018

  50. [58]

    Synthet2c: Generating synthetic data for fine-tuning large language models on the text2cypher task

    Ziije Zhong, Linqing Zhong, Zhaoze Sun, Qingyun Jin, Zengchang Qin, and Xiaofan Zhang. Synthet2c: Generating synthetic data for fine-tuning large language models on the text2cypher task. arXiv preprint arXiv:2406.10710, 2024

    2024 arXiv

  51. [59]

    Variational reasoning for question answering with knowledge graph

    Yuyu Zhang, Hanjun Dai, Zornitsa Kozareva, Alexander Smola, and Le Song. Variational reasoning for question answering with knowledge graph. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018

    2018

  52. [60]

    Introducing the neo4j text2cypher (2024) dataset, November 2024

    Makbule Gulcin Ozsoy, Leila Messallem, and Jon Besga. Introducing the neo4j text2cypher (2024) dataset, November 2024. Published on the Neo4j Developer Blog

    2024

  53. [61]

    Reconciliation of rdf* and property graphs

    Olaf Hartig. Reconciliation of rdf* and property graphs. arXiv preprint arXiv:1409.3288, 2014

    2014 arXiv

  54. [62]

    Rdf data in property graph model

    Dominik Tomaszuk. Rdf data in property graph model. In Research Conference on Metadata and Semantics Research, pages 104–115. Springer, 2016

    2016

  55. [63]

    S2x: graph-parallel querying of rdf with graphx

    Alexander Schätzle, Martin Przyjaciel-Zablocki, Thorsten Berberich, and Georg Lausen. S2x: graph-parallel querying of rdf with graphx. In Biomedical Data Management and Graph Online Querying: VLDB 2015 Workshops, Big-O (Q) and DMAH, Waikoloa, HI, USA, August 31–September 4, 20...

    2015

  56. [64]

    Mapping rdf databases to property graph databases

    Renzo Angles, Harsh Thakkar, and Dominik Tomaszuk. Mapping rdf databases to property graph databases. IEEE Access, 8:86091–86110, 2020

    2020

  57. [65]

    Mapping rdf graphs to property graphs

    Shota Matsumoto, Ryota Yamanaka, and Hirokazu Chiba. Mapping rdf graphs to property graphs. arXiv preprint arXiv:1812.01801, 2018

    2018 arXiv

  58. [66]

    Rdf and property graphs interoperabil- ity: Status and issues

    Renzo Angles, Harsh Thakkar, and Dominik Tomaszuk. Rdf and property graphs interoperabil- ity: Status and issues. AMW, 2369:1–11, 2019

    2019

  59. [67]

    Kgtk: a toolkit for large knowl- edge graph manipulation and analysis

    Filip Ilievski, Daniel Garijo, Hans Chalupsky, Naren Teja Divvala, Yixiang Yao, Craig Rogers, Rongpeng Li, Jun Liu, Amandeep Singh, Daniel Schwabe, et al. Kgtk: a toolkit for large knowl- edge graph manipulation and analysis. In The Semantic Web–ISWC 2020: 19th International S...

    2020

  60. [68]

    Rhaenys Targaryen

    Seyed Amir Hosseini Beghaeiraveri, Jose Emilio Labra Gayo, Andra Waagmeester, Ammar Ammar, Carolina Gonzalez, Denise Slenter, Sabah Ul-Hasan, Egon Willighagen, Fiona McNeill, and Alasdair JG Gray. Wikidata subsetting: Approaches, tools, and evaluation. Semantic Web, pages 1–27...

    2023

  61. [2015]

    Association for Computational Linguistics

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.