{"id":"d829b610-26bb-4219-bbea-071b6b5cee5d","arxiv_id":"2412.18702","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"CypherBench provides 11 Wikidata-derived property graphs and 10,000+ text-to-Cypher questions, and state-of-the-art LLMs currently answer only about 60% correctly.","lead":"The authors convert large parts of Wikidata into 11 domain-specific property graphs and test how well LLMs turn natural-language questions into Cypher queries. The outcome is CypherBench, a new benchmark with over 10,000 questions and 7.8 million entities, where the strongest LLM still fails on roughly 40% of queries.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ground-truth Cypher uses entity names as keys without enforcing uniqueness, so name collisions make gold answers ambiguous and execution accuracy a noisy measure of retrieval.","rationale":"The reader's weakest_assumption names the entity-name-as-key issue, and I agree it is the most load-bearing concern. The paper's own evidence supports it: Section 4.5 explicitly chooses to preserve names and defer entity linking; Table 13 defines 'Entity Linking' and 'Incorrect Deduplication' as error categories; Section 6.3 observes models merging distinct entities with identical names; and the evaluation prompt instructs models to output duplicate names for distinct entities. All of these indicate the benchmark is aware of name collisions, yet the paper never measures how often they occur in the ground-truth queries or how they affect EX. If a gold query returns all entities sharing a name while the question intends one specific entity, an accurate model that returns the single intended entity is scored as incorrect, and an inaccurate model that returns every name-match is scored as correct. This corrupts the primary accuracy metric and therefore the empirical support for the central claim that property graph views enable accurate LLM retrieval. The missing SPARQL baseline is a real but secondary issue: it weakens the comparative efficiency claim, not the benchmark's internal validity. Single-run estimates are also secondary; they affect confidence intervals, not correctness. The name-key assumption, by contrast, affects every ground-truth query and the interpretation of every accuracy number. Thus it is the single most load-bearing concern. The appropriate disposition remains CONDITIONAL: the benchmark is a substantial, well-documented resource, but its headline accuracy figures should only be reported after quantifying or mitigating name ambiguity. The reader's verdict already captures this, so no verdict adjustment is needed.","tokens_in":26757,"tokens_out":4184,"duration_ms":39503,"concrete_test":"For each of the 2,488 test questions, extract every entity-name literal used in a property-equality constraint in the ground-truth Cypher. On the corresponding property graph, run a count query: MATCH (n) WHERE n.name = $name RETURN count(n). Record the distribution of counts. For every name with count > 1, manually inspect the question and ground-truth query to determine whether (a) the question's intended referent is uniquely identifiable, and (b) the gold answer set includes multiple distinct entities with that name.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The benchmark's evaluation metric (EX) compares predicted query results against ground-truth query results, and ground-truth Cypher identifies entities by name (e.g., n:Taxon {name:'Leporidae'}). The paper deliberately avoids entity linking (Section 4.5), yet Wikidata does not enforce unique labels. The own error taxonomy (Table 13) includes 'Entity Linking' and 'Incorrect Deduplication', and Section 6.3 attributes a performance drop to models 'merging distinct entities that have the same name'. The prompt (Table 10) even instructs models to repeat names when distinct entities share them. This means a ground-truth query can match multiple nodes with the same name; if the intended entity is one of them, the gold answer may include unintended nodes (or the model may return only the intended one and be scored as wrong). The paper does not quantify how often named entities in the test set collide within their property graph, so the headline EX/PSJS numbers (e.g., 61.58% for claude3.5-sonnet) may not cleanly measure retrieval ability. Because the central claim asserts 'accurate' retrieval over full-scale graphs, and the benchmark is the primary evidence, this ambiguity directly undermines the validity of that claim rather than merely a peripheral comparison.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that modern RDF knowledge graphs such as Wikidata are difficult for LLM-based retrieval because of overly large schemas, use of IRIs, overlapping relation types, and lack of normalization, and proposes converting such graphs into domain-specific property graphs queried via Cypher. It describes an RDF-to-property-graph conversion engine, a template-based pipeline that generates over 10,000 (question, Cypher) pairs across 11 Wikidata-derived graphs, and two evaluation metrics: execution accuracy (EX) and provenance subgraph Jaccard similarity (PSJS). The experimental evaluation covers 12 LLMs in a zero-shot text-to-Cypher setting, with claude3.5-sonnet reaching 61.58% EX and gpt-4o reaching 60.18%, while models under 10B parameters stay below 20%.","tokens_in":26980,"tokens_out":6782,"duration_ms":62866,"significance":"If the benchmark's validity holds, this is a valuable contribution: CypherBench is the first large-scale text-to-Cypher benchmark built on domain-complete Wikidata subsets, and the public dataset, code, Docker deployment, and evaluation scripts make it reproducible. The PSJS metric is a useful addition because it separates graph-matching errors from return-format errors, and the inclusion of global, temporal, union, and optional-match queries fills a gap relative to prior KBQA benchmarks. The construction pipeline is careful in several respects: schema curation, rank filtering, datatype conversion, unit standardization, multi-round LLM verification, and manual inspection of the test set. The main validity risk is the use of entity names as keys in ground-truth queries, which can make gold answers ambiguous when distinct entities share the same name; this needs to be quantified and addressed before the headline accuracies can be interpreted as clean measures of precise retrieval.","major_comments":[{"comment":"The ground-truth Cypher queries identify named entities solely by their name property (e.g., Table 11, Q4–Q6), and §4.5 states that entity names are intentionally preserved to avoid entity linking. Because Wikidata labels are not unique, a MATCH predicate such as {name: '...'} can bind multiple nodes in the same graph, and the paper itself acknowledges this in Table 10 ('if multiple distinct entities share the same name, their names should be repeated as separate entries') and in its error taxonomy (Table 13, Entity Linking; §6.3, Incorrect Deduplication). When a gold query binds several same-named entities, the executed gold result can include unintended rows or aggregate over unintended entities, while a model that correctly isolates the intended entity is scored as wrong. The paper does not quantify how many test instances contain same-name collisions within the relevant graph. Please report the collision rate per graph and per split, and either filter or repair affected instances or add an entity-ID-based evaluation track; without this, the reported EX/PSJS numbers are not a clean measure of retrieval accuracy.","section":"§4.5, Table 13, §6.3, Table 10"},{"comment":"The motivating claim is that RDF graphs are less efficient for LLMs and that property graph views are the solution, but Section 6 only evaluates text-to-Cypher on the converted property graphs. There is no text-to-SPARQL baseline over Wikidata, or over the same domain subsets, that would test the four claimed RDF difficulties (schema size, resource identifiers, overlapping relation types, and lack of normalization). The root-cause argument is therefore a plausible qualitative analysis rather than an empirically supported comparison. I recommend adding a zero-shot SPARQL baseline with comparable schema information and label access, and reporting the same or an equivalent execution-based metric, so the claimed advantage of property graph views is directly tested.","section":"§2.2, §2.4, §6.2"},{"comment":"The evaluation section does not report the sampling temperature or the number of runs for any model, and Table 3 gives a single EX and PSJS value per model. Because LLM generation is stochastic, differences of a few points (e.g., claude3.5-sonnet vs. gpt-4o, or qwen2.5-72b vs. llama3.1-70b) may be within run-to-run noise. Please run each model multiple times, or explicitly set temperature to 0 and state that, and report means, standard deviations, or a significance test so that the model ranking and the headline accuracy figures are reproducible.","section":"§6.1, Table 3"}],"minor_comments":[{"comment":"The phrase 'full-scale modern knowledge graphs' is stronger than what the benchmark actually provides: each graph is a domain-complete subset of Wikidata (7.8M entities out of roughly 114M), not the full Wikidata graph. Please qualify the wording, for example by saying 'domain-complete Wikidata subsets'.","section":"Title and Abstract"},{"comment":"The figures are referenced with pattern symbols and RETURN-template names that are hard to map to the tables in Appendix A; adding explicit legends or labeled axis categories would improve readability.","section":"Figures 4 and 5"},{"comment":"The SpCQL row in Table 6 reports '480k relation types', but the footnote in §7.2 says that OwnThink is stored with a single relation type when loaded into Neo4j. This apparent inconsistency should be clarified.","section":"Table 6 and §7.2"},{"comment":"The paper should specify how the provenance subgraph and the associated `RETURN *` are defined for queries containing UNION, UNWIND, or aggregation, since a single `RETURN *` over such queries is not described.","section":"§5.2, Eq. (2)"},{"comment":"Reference [58] appears to contain a typo: 'Ziije Zhong' should be 'Zijie Zhong'.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"This is a well-constructed and well-documented benchmark paper, and I would be supportive of it once the entity-name ambiguity is quantified and addressed. The missing SPARQL baseline is also important for the framing of the root-cause claim. I have no concerns about citation practice or author conduct."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a solid, useful benchmark and the strongest current resource for text-to-Cypher evaluation at scale. The main caveat is exactly the one in the stress-test note: the gold queries use bare entity names as keys, and the paper never measures how often those names collide in the property graphs. That makes the headline EX/PSJS numbers noisier than they look.\n\nWhat's genuinely new: the scale and coverage. Eleven domain property graphs from a full Wikidata dump, 7.8M entities, 10k+ executable questions spanning 12 graph patterns, including global and temporal queries that previous KBQA benchmarks mostly skip. The PSJS metric is a sensible addition—it separates MATCH-clause accuracy from RETURN-clause formatting, and it behaves well in the error analysis. The paper also ships public data and code, and the construction pipeline is carefully described: schema curation, SPARQL-based transformation with unit standardization and rank filtering, template generation, LLM rewriting with three verification rounds, and author inspection of the test set. The error taxonomy is detailed and useful.\n\nThe soft spots are in proportion: the name-collision issue is real, and it directly affects the validity of the reported execution accuracy. The prompt even tells models to repeat names when distinct entities share one, but the benchmark never quantifies how many names in the test set are non-unique, nor does it provide a fallback (e.g., QIDs or disambiguating context) for gold queries. If a gold query matches multiple nodes with the same name, the model can return the intended entity and still be scored wrong, or return all colliding entities and be scored right. That's not a clean measure of retrieval. It's fixable, but it needs to be addressed before the numbers are taken at face value.\n\nTwo smaller weaknesses: the root-cause claim that RDF is inefficient for LLMs is argued but never tested directly against a SPARQL text-to-query baseline, and the LLM results are single-run point estimates with no variance or temperature reporting. Neither undercuts the benchmark as a resource, but they limit the strength of the causal claims.\n\nWho this is for: anyone working on text-to-Cypher, KBQA, or GraphRAG evaluation. It deserves a serious referee and will likely be a reference dataset in the area, but I'd want the revision to quantify name collisions and ideally provide disambiguated gold entities before relying on its numbers.","headline":"A valuable large-scale text-to-Cypher benchmark whose headline numbers are clouded by unresolved entity-name collisions in the gold queries.","tokens_in":27540,"tokens_out":2814,"would_cite":true,"duration_ms":26029,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Property graph views of Wikidata can be queried by LLMs with Cypher, and a new benchmark shows 61.58% execution accuracy for the best model.","keywords":["text-to-Cypher","knowledge graph question answering","RDF to property graph conversion","LLM graph retrieval","Wikidata","execution accuracy","provenance subgraph Jaccard similarity"],"falsifier":"Take the CypherBench test instances and, for each question that names an entity, count how many distinct nodes in its graph share that exact name. Recompute execution accuracy on the subset where the count is greater than one, and compare it to the singleton-name subset; if the accuracy is markedly lower or the ground-truth queries change under the alternative entities, the clean-precision claim would need qualification.","tokens_in":26549,"feed_emoji":"📊","tokens_out":3223,"duration_ms":32393,"temperature":0.7,"pith_summary":"This paper argues that modern RDF knowledge graphs like Wikidata are inefficient for LLM retrieval because their schemas are far too large, they use unwieldy resource identifiers, their relation types overlap, and they lack normalization. As a fix, it proposes turning an RDF graph into multiple domain-specific property graph views that an LLM can query in Cypher. It instantiates this idea as CypherBench, 11 property graphs containing 7.8 million entities and over 10,000 natural-language to Cypher tasks. The core claim is that this conversion makes precise, full-scale knowledge-graph retrieval practical for LLMs without needing external entity linkers, with the best tested model reaching 61.58% execution accuracy.","feed_headline":"LLMs reach 61.6% accuracy querying Wikidata with Cypher","feed_subtitle":"A property-view conversion makes 7.8M-entity retrieval practical, and the new benchmark scores the gap","key_machinery":"The central machinery is a pipeline that converts RDF triples into schema-enforced property graphs: a curated per-domain schema maps Wikidata QIDs and PIDs to clean node and relation labels, and an engine issues SPARQL queries against a local Wikidata endpoint to fetch conforming triples, then applies datatype conversion, date precision filtering, unit standardization, and rank filtering. On top of this sit the task generation pipeline (templated MATCH and RETURN clause instantiation, LLM rewriting, and verification) and the two evaluation metrics: execution accuracy (EX), comparing returned tables, and provenance subgraph Jaccard similarity (PSJS), isolating how well the MATCH clause finds the right subgraph.","core_discovery":"The paper claims that the obstacle to LLM retrieval from large RDF knowledge graphs is not the graphs' size but their modeling choices, and that a materialized property graph view per domain removes the bulk of the difficulty. The reported evidence is CypherBench, built from a Wikidata April 2024 dump, where the strongest model (claude3.5-sonnet) achieves 61.58% execution accuracy and 80.85% provenance subgraph Jaccard similarity, and gpt-4o achieves 60.18%. The intended consequence is that graph retrieval over full-scale encyclopedic knowledge becomes feasible through text-to-Cypher, a capability the paper says leading LLM frameworks lack for RDF graphs.","pith_inferences":["A likely stress test is entity-name ambiguity: since the pipeline intentionally preserves names and skips entity linking, questions that mention names shared by multiple entities may have ambiguous gold answers, so reported execution accuracy may partly reward guessing the right name rather than retrieving the right entity.","The property graph views are domain-specific, so a question spanning two views (for example, combining movie and politics facts) would fall outside the benchmark; a follow-up could create cross-view or union queries and investigate how well Cypher's composability handles them.","The benchmark could be adapted to other RDF sources (DBpedia, Freebase, domain-specific triplestores) by regenerating schemas and task instances, providing an empirical measure of how transferable the conversion benefit is.","PSJS, as a RETURN-clause-independent score, could be adopted by the broader text-to-SQL community as an auxiliary metric for diagnosing whether errors come from the JOIN structure or the projection/aggregation logic."],"forward_implications":["If the central claim holds, LLMs can retrieve answers from Wikidata-scale graphs by writing Cypher instead of relying on embedding-based top-k retrieval or external entity linkers.","The benchmark separates graph-matching skill from formatting and aggregation skill via PSJS, which could become a standard diagnostic for text-to-query evaluation.","The result that no tested sub-10B-parameter model exceeds 20% execution accuracy suggests CypherBench can serve as a scaling signal for LLM graph reasoning ability.","The RDF-to-property-graph engine, if generalized, gives a reusable bridge from SPARQL-style world knowledge to the property graph interfaces used in enterprise databases.","Because execution accuracy requires running queries, CypherBench provides a testbed for measuring whether LLMs improve at precise retrieval as they grow, rather than just at paraphrase."],"supporting_citations":[{"why":"Supplies the source data: Wikidata, the RDF knowledge graph that the paper transforms into property graph views.","marker":"[2]"},{"why":"Provides the GraphRAG context and motivates the global queries that CypherBench targets as an evaluation gap.","marker":"[1]"},{"why":"Contributes the borrowed execution-accuracy implementation from the Spider text-to-SQL benchmark, which is used to compare returned tables.","marker":"[30]"},{"why":"The Neo4j Text2Cypher dataset that CypherBench contrasts with, drawing the distinction that many of its questions lack executable underlying graphs.","marker":"[60]"},{"why":"KQA Pro, a representative compositional KBQA benchmark with custom logical forms, used as a comparison point for query-pattern coverage and scalability.","marker":"[7]"},{"why":"SPINACH, a recent agentic full-Wikidata SPARQL work, cited as the closest prior effort operating over full-scale Wikidata.","marker":"[50]"}],"fun_headline_variants":["CypherBench: LLMs query 7.8M-node Wikidata at 61.6%","Property views let LLMs hit 61.6% on Wikidata Cypher retrieval","CypherBench: Making full-scale KG retrieval practical for LLMs","61.6% accuracy: LLMs query Wikidata via Cypher after property view","CypherBench: Property views make Wikidata retrieval LLM-friendly"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that an entity's name is enough to identify that entity in the graph, because the pipeline keeps names and purposely avoids entity linking, and Wikidata does not enforce unique labels.","fun_headline_variants_meta":{"raw":{"variants":["CypherBench: LLMs query 7.8M-node Wikidata at 61.6%","Property views let LLMs hit 61.6% on Wikidata Cypher retrieval","CypherBench: Making full-scale KG retrieval practical for LLMs","61.6% accuracy: LLMs query Wikidata via Cypher after property view","CypherBench: Property views make Wikidata retrieval LLM-friendly"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000747,"raw_usage":{"total_tokens":3327,"prompt_tokens":939,"completion_tokens":2388,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":555,"completion_tokens_details":{"reasoning_tokens":2282}},"tokens_in":555,"tokens_out":2388,"duration_ms":14501,"temperature":1.0,"reasoning_tokens":2282,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T04:32:49.240416+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the CypherBench test instances and, for each question that names an entity, count how many distinct nodes in its graph share that exact name. Recompute execution accuracy on the subset where the count is greater than one, and compare it to the singleton-name subset; if the accuracy is markedly lower or the ground-truth queries change under the alternative entities, the clean-precision claim would need qualification.","supporting_citations":[{"cited_title":"Wikidata: a free collaborative knowledgebase","cited_arxiv_id":null,"evidence_quote":"Supplies the source data: Wikidata, the RDF knowledge graph that the paper transforms into property graph views."},{"cited_title":"Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task","cited_arxiv_id":null,"evidence_quote":"Contributes the borrowed execution-accuracy implementation from the Spider text-to-SQL benchmark, which is used to compare returned tables."},{"cited_title":"Introducing the neo4j text2cypher (2024) dataset, November 2024","cited_arxiv_id":null,"evidence_quote":"The Neo4j Text2Cypher dataset that CypherBench contrasts with, drawing the distinction that many of its questions lack executable underlying graphs."},{"cited_title":"KQA pro: A dataset with explicit compositional programs for complex question answering over knowledge base","cited_arxiv_id":null,"evidence_quote":"KQA Pro, a representative compositional KBQA benchmark with custom logical forms, used as a comparison point for query-pattern coverage and scalability."},{"cited_title":"SPINACH: SPARQL-based information navigation for challenging real-world questions","cited_arxiv_id":null,"evidence_quote":"SPINACH, a recent agentic full-Wikidata SPARQL work, cited as the closest prior effort operating over full-scale Wikidata."}],"review_version":1}