Auto-Cypher: Improving LLMs on Cypher generation via LLM-supervised generation-verification framework

Aman Tiwari; Masoud Hashemi; Sathwik Tejaswi Madhusudhan; Shiva Krishna Reddy Malay; Vikas Yadav

REVIEW 4 major objections 6 minor 37 references

Auto-Cypher: Improving LLMs on Cypher generation via LLM-supervised generation-verification framework

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that an LLM-supervised pipeline that reverse-fills synthetic Neo4j databases can generate high-quality Text2Cypher training data, lifting open 7B/8B models by up to 40 percentage points on the SynthCypher test split and…

desk verdict A clever reverse-filling synthetic data pipeline for Text2Cypher with real gains, but the circular validation keeps the headline numbers from being taken at face value. read the letter →

arxiv 2412.12612 v2 pith:256UQHJU submitted 2024-12-17 cs.CL cs.AIcs.IRcs.LG

Aman Tiwari , Shiva Krishna Reddy Malay , Vikas Yadav , Masoud Hashemi , Sathwik Tejaswi Madhusudhan This is my paper · ORCID

classification cs.CLcs.AIcs.IRcs.LG

keywords SyntheticDataText2CypherLargeLanguageModelsGraphDatabasesCypherQueryGenerationKnowledgeGraphsNeo4jNaturalInterfaces

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to close a data gap: translating natural language into Cypher, Neo4j's query language, lags behind Text2SQL because no large validated training set exists. It claims that an entirely LLM-supervised pipeline can manufacture such a set by generating 700 domains, 109 query types, schemas, questions, and dummy answers, then filling a synthetic Neo4j database so each dummy answer really is the query result. Only question–Cypher pairs whose queries execute and return the expected answer are kept, producing the SynthCypher dataset with 29.8k instances. Fine-tuning 7B/8B open models on this data, mixed with instruction data, is reported to raise Text2Cypher accuracy by up to 40 percentage points on the SynthCypher test split and about 30 on an adapted SPIDER-Cypher benchmark. The authors caution that the results are limited to smaller models and that synthetic data may carry or amplify bias.

What carries the argument

The central mechanism is LLM-As-Database-Filler: given a question and a dummy ground-truth answer, synthetic database-population code is generated so that the dummy answer is the intended result of the question while negative data points serve as distractors, making query correctness executable in a real Neo4j engine. Supporting machinery includes 109 query taxonomies used to prompt question diversity, four-step chain-of-thought Cypher generation, and an LLM-as-judge execution match that accepts a result if it contains the information the question asks for. This converts semantic correctness into an execution check, which is the load-bearing idea behind the pipeline.

What would settle it

Sample several hundred SynthCypher pairs at random, have human annotators judge whether the dummy answer contains exactly the information the question requests, and check whether the stored Cypher query returns that answer; if a substantial share are judged unfaithful, execution validation is not actually guaranteeing semantic correctness.

Watch

Extended reading notes

Core claim

The central claim is that high-quality Text2Cypher data can be generated without human annotation by making the database conform to the answer, rather than deriving the answer from the database. In the LLM-As-Database-Filler step, for each generated natural-language question an LLM writes a plausible ground-truth answer; GPT-4-generated Python code populates an empty Neo4j database with positive entries that satisfy the answer and up to five negative entries that do not. A separate LLM then produces a Cypher query through a four-stage chain-of-thought process, and the query is retained only if executing it on that database returns the dummy ground truth, judged by a GPT-4-based execution-match checker. The paper reports that models of 7B/8B scale fine-tuned on the resulting SynthCypher data achieve large gains over their instruction-tuned baselines, and that SynthCypher outperforms the main existing GPT-4o-generated dataset when training size is equalized.

Load-bearing premise

Everything rests on the assumption that the LLM-written dummy ground truth is a faithful answer to the natural-language question, because any Cypher query that returns that answer passes validation even if a human would answer differently.

Editorial extensions

If this is right

SynthCypher gives the Text2Cypher community a public 29.8k-example training set with 109 query types, diverse domains, schemas, populated Neo4j databases, Cypher queries, and execution results.
Open 7B/8B models fine-tuned on SynthCypher reach over 67% on the SynthCypher test split, where their base or instruction-tuned versions sat in the teens to thirties.
The pipeline's automation means the dataset can be extended to more domains and query types without human labeling, as long as the validation step has an executable database to check against.
Adapting SPIDER into SPIDER-Cypher provides a second, cross-domain benchmark for comparing Text2Cypher systems.
Existing generic instruction-tuning data such as UltraChat and MagiCoder is insufficient by itself for Cypher generation, indicating that task-specific data is the binding constraint.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Inference: the dummy-answer-and-fill-the-database strategy should transfer to other executable query languages such as SQL, SPARQL, or GraphQL, and to any code-generation setting where a natural-language spec can be paired with a dummy output and an executable check.
Inference: because the validation signal is an LLM-written answer, the real test of SynthCypher is how much of the 40-point gain survives on human-written questions with human-judged correctness; the paper's own limitation paragraph notes only 7B/8B models were tried and the SPIDER test set may overlap pretraining data.
Inference: the schema and question distribution are seeded by a small set of example domains and by the generator models' prior, so performance on private real-world Neo4j workloads is the natural next stress test.
Editorial note: the paper attributes SPIDER to (Deng et al., 2020) in the introduction and to (Yu et al., 2018) in the experimental setup; the benchmark itself is the Yu et al. dataset, and the inconsistency is worth correcting.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Worth a look. The genuinely new piece is LLM-As-Database-Filler: generate a plausible dummy answer, then populate a Neo4j database so that answer is literally the correct query result, and keep only Cypher queries whose execution matches it. That is a smart trick for getting executable ground truth at scale, and it is not in the prior Neo4j Labs or Synth2C work. The released SynthCypher dataset (29.8k instances, 109 query types) is a real resource for an underserved task, and the fine-tuning gains—up to 40% on their test split and 30% on adapted SPIDER—are consistent across Llama-3.1-8B, Mistral-7B, Qwen2-7B, and code variants. That consistency is the strongest evidence the data is doing something useful.

The soft spot is real: the validation loop only checks that a generated query returns the LLM's dummy answer. If Mixtral misread the question or chose an arbitrary answer, a query that "passes" can be semantically wrong. No human verification of question–answer pairs is reported. The test split comes from the same pipeline and is judged by the same GPT-4 prompt, so the 40% number partly reflects learning to satisfy the pipeline's pseudo-ground truth. The SPIDER adaptation is more external, but the ground-truth Cypher there is also author-generated and judged by the same LLM, so it does not fully break the circularity. There are also internal inconsistencies in dataset counts (29.8k vs 25.8k vs 2k/4k test) and no error bars or training code.

None of this kills the paper. For an under-resourced task, a public dataset and a reproducible pipeline are a contribution even if the absolute numbers are optimistic. The fixes are straightforward: human-annotate a sample of validation pairs, evaluate on an independent benchmark, and run multiple seeds. I would send it to review and let the authors tighten the evaluation.

Referee Report

4 major / 6 minor

Summary. The paper presents Auto-Cypher, an LLM-supervised pipeline for generating synthetic Text2Cypher training data. The pipeline generates graph schemas, natural language questions, dummy ground-truth answers, and then reverse-populates a Neo4j database so that the dummy answer is the query result. A Cypher query is generated and retained only if executing it against the populated database returns the dummy answer, as judged by GPT-4. The resulting dataset, SynthCypher, contains 29.8k training and 2k test instances across 109 query types and 700 domains. The authors fine-tune several 7B/8B open-source LLMs on SynthCypher and report improvements of up to 40% on a SynthCypher test split and up to 30% on an author-adapted version of the SPIDER benchmark. The main empirical claim is that supervised fine-tuning on this pipeline-generated dataset substantially improves Text2Cypher performance.

Significance. If the dataset is genuinely high quality, this is a valuable contribution: it provides a large, diverse, publicly released Text2Cypher resource, demonstrates consistent gains across multiple 7B/8B models, and introduces a novel validation idea (LLM-As-Database-Filler) that could inform future synthetic-data pipelines. The paper also takes a step toward benchmarking Text2Cypher by adapting SPIDER. However, the central quality claim rests on a self-referential correctness signal, and the evaluation uses the same LLM judge that generated the data. The reported gains, while plausible and consistent, are therefore not yet established as genuine Text2Cypher improvements rather than distribution matching. The paper would be substantially strengthened by adding human verification of a sample of question–answer pairs and by evaluating on an external, human-curated benchmark with human-validated ground truth.

major comments (4)

[Section 3, Steps 2–5; Appendices D–E; Figure 7] The correctness signal in the pipeline is self-referential. The dummy ground-truth answer is generated by Mixtral; the Neo4j database is reverse-populated so that this dummy answer is the correct query result; and Step 5 accepts a Cypher query only if its execution returns that dummy answer, with GPT-4 as judge (Figure 14). Because the dummy answer itself defines what 'correct' means, a query can pass validation even if the dummy answer does not faithfully answer the user's question. The paper does not report any human verification of the question–answer pairs. Since the test split is produced by the same pipeline and scored with the same LLM judge, the reported gains on SynthCypher may partly reflect the model learning to match this specific pseudo-ground-truth distribution rather than genuinely improved Text2Cypher ability. The authors should provide a human evaluation on a random sample (e.g., 300–500 examples) establishing that the ground-truth answers correctly answer the natural language questions, and report agreement between human judgments and the GPT-4 judge.
[Section 4, Metric] The evaluation uses an LLM-as-Judge version of exact match with no human validation or inter-annotator agreement study. The test set is generated by the same pipeline and evaluated with the same LLM judge, so the evaluation is in-distribution by construction and may overestimate true performance. The authors should report human evaluation on a sample of the SynthCypher test set and on the SPIDER-adapted test set, and ideally evaluate on an external benchmark with human-verified Cypher ground truth.
[Section 4, Data Setup] The reported data split numbers are inconsistent: the text states 25.8k training samples and a 4k test set, while Table 1 reports 29,838 training and 2,000 test samples, and the abstract says 29.8k instances. This discrepancy makes it unclear which split was actually used in Table 2 and Figure 3. The authors should correct the numbers and specify the exact data split used in each experiment.
[Section 4, SPIDER adaptation] The adapted SPIDER benchmark is defined by the authors, but the manuscript does not describe how the ground-truth Cypher queries were generated or whether they were validated by execution against a populated graph database or by human review. Without this information, the SPIDER results do not fully break the circularity of the pipeline: a query that returns the author-defined result may still be semantically wrong. The authors should provide the construction details and, ideally, human verification of the ground-truth Cypher queries.

minor comments (6)

[Appendix B] In the Schema Generation Process, 'yeilds' should be 'yields'.
[Table 2 caption] The caption says 'Funetuning'; this should be 'Fine-tuning'.
[Table 2 header] The column header 'SPIDERBase' is ambiguous; please clarify whether it refers to the SPIDER-adapted benchmark or a base-model evaluation.
[Figure 3] The downsampling procedure used to match training sizes in the comparison against the Neo4j Labs dataset should be described (random seed, number of runs, variance across runs), since the comparison in Figure 3 could be sensitive to the subsample.
[Section 3, Step 1] The 25% manual review of schemas is not described in detail, and it is unclear how the remaining 75% of schemas were validated beyond GPT-4; please clarify the criteria used for the manual review.
[References] The inline citations '(tom, 2024)' and '(neo, 2024)' should be replaced with conventional author-year references.

Circularity Check

2 steps flagged · score 5.0 of 10

Correctness is defined as matching an LLM-generated dummy answer that the pipeline reverse-engineers into the database; the SynthCypher test split is graded by the same LLM-as-Judge, and the SPIDER adaptation only partially breaks the loop.

self definitional [Section 3, Steps 2-5; Appendix D; Figure 2]
"A dummy ground truth answer for each query is also generated. In the next stage, we fill the database with entries including this dummy answer as the right answer for the question. ... To the best of our knowledge, this strategy of filling the database conditioned on a arbitrarily chosen dummy ground truth has not been explored in literature before. ... The results are compared to the expected ground truth, and only queries that return correct results are retained ... GPT-4 is used as a judge to validate the retrieved data against the ground-truth and ensure correctness."

The correctness target is an LLM-generated dummy answer; the Neo4j database is reverse-filled so that the dummy answer is the query result. A candidate Cypher query is accepted if and only if its execution output matches the dummy answer, as judged by GPT-4. Therefore the validation step does not check whether the query answers the natural-language question; it checks whether the query returns a value that the pipeline itself manufactured as the answer. The claim that Reverse-filling ensures Cypher-code correctness is thus defined as consistency with an arbitrarily chosen input, not as semantic correctness against an external human standard.
other [Section 4, Data Setup and Metric; Table 2]
"For testing, we employed a separate dataset of 4k samples, covering all 109 query types across 165 schemas not included in train. ... As an additional test dataset, we also adapt the popular SPIDER-SQL (Yu et al., 2018) for Text2Cypher by modeling each table as a node and foreign key relationships. ... Metric: We use an LLM-as-Judge version (prompt14) of Exact Match where GPT-4o gives a score of 1 if all requested information in the question is present in the results from the execution, and 0 otherwise."

The SynthCypher test split is produced by the same LLM-supervised pipeline, so its ground-truth answers are dummy answers generated the same way as the training data, and an LLM-as-Judge of the same kind (GPT-4/GPT-4o) used to filter the training data scores the test predictions. The reported 40% improvement is therefore measured against the pipeline's own pseudo-ground-truth distribution rather than an externally verified answer set. The SPIDER adaptation provides some independent grounding, but its Cypher ground truth is also author-generated and scored by the same GPT-4 judge, so this only partially breaks the circularity.

full rationale

The paper's central data-quality claim rests on the LLM-As-Database-Filler step: a dummy ground truth is generated by an LLM, the database is filled so that dummy is the correct query result, and a query is retained only if its execution matches that dummy. This makes the validation loop self-referential by construction: 'correct' means 'matches the answer the pipeline chose and encoded.' The 40% gain on the SynthCypher test split is measured against ground truths produced by the same pipeline and judged by the same LLM-as-Judge used for filtering, so it is partly an in-distribution fit to the pipeline's own pseudo-ground-truth. The SPIDER adaptation is an external source of questions and partially mitigates the circularity, but its Cypher ground truth is author-generated and the same GPT-4 judge scores it, so it does not fully break the loop. There is no load-bearing self-citation chain or uniqueness theorem; the training improvements are empirical and could be real. Score 5 reflects one central step that reduces by construction plus an evaluation loop that is only partially broken.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No fitted constants appear in the central claim; the design choices (109 query types, up to 5 negative data points, 25% manual schema vetting) are pipeline decisions, not free parameters in the statistical sense. The main assumptions concern the trustworthiness of LLM-generated ground truth and LLM-based validation, which are the fragile links in the claimed data quality. No new theoretical entities are introduced.

assumptions (4)

domain assumption A dummy ground truth generated by an LLM from question and schema faithfully captures the correct interpretation of the natural language question.
Invoked in Step 2/D (Figure 7) and used as the target for all later validation; no human verification of answer semantics is performed.
domain assumption Executing a Cypher query on a reverse-filled synthetic Neo4j database and comparing the output with the dummy ground truth using GPT-4 as judge is sufficient to certify query correctness.
Invoked in Step 5 and Figure 14; the comparison checks agreement with the LLM-generated answer, not with an external semantic parse.
domain assumption Modeling SPIDER tables as nodes and foreign keys as relationships yields a valid Text2Cypher benchmark.
Section 4; this adaptation is author-defined and not validated against human-written Cypher.
ad hoc to paper The 25% manual review of schemas ensures that the remaining 75% of schemas are also coherent and real-world useful.
Section B.2; the manual review is partial and not described quantitatively, yet the data-quality claim relies on it.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Auto-Cypher: Improving LLMs on Cypher generation via LLM-supervised generation-verification framework." pith.science (2026). https://pith.science/paper/256UQHJU

@misc{pith2026241212612,
  author       = {Pith},
  title        = {Pith review of: Auto-Cypher: Improving LLMs on Cypher generation via LLM-supervised generation-verification framework},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/256UQHJU}},
  note         = {Machine review of arXiv:2412.12612}
}

read the original abstract

Graph databases like Neo4j are gaining popularity for handling complex, interconnected data, over traditional relational databases in modeling and querying relationships. While translating natural language into SQL queries is well-researched, generating Cypher queries for Neo4j remains relatively underexplored. In this work, we present an automated, LLM-Supervised, pipeline to generate high-quality synthetic data for Text2Cypher. Our Cypher data generation pipeline introduces LLM-As-Database-Filler, a novel strategy for ensuring Cypher query correctness, thus resulting in high quality generations. Using our pipeline, we generate high quality Text2Cypher data - SynthCypher containing 29.8k instances across various domains and queries with varying complexities. Training open-source LLMs like LLaMa-3.1-8B, Mistral-7B, and QWEN-7B on SynthCypher results in performance gains of up to 40% on the Text2Cypher test split and 30% on the SPIDER benchmark, adapted for graph databases.

Figures

Figures reproduced from arXiv: 2412.12612 by the authors.

**Figure 2.** Overview of the SynthCypher data generation pipeline, illustrating domain and schema creation, query and ground truth generation, database population, Cypher query generation, and validation steps. truth has not been explored in literature before. Reverse filling the database in this way enables execution of Cypher queries to check for execution success and Cypher-code correctness. Appendix E(8,9) provides more de… view at source ↗

**Figure 3.** Evaluation on SynthCypher and SPIDER test splits from Llama3.1-8B fine-tuned with equal train size of downsampled SynthCypher (ours) data and Neo4j Text2Cypher data. (1) Need of Text2Cypher datasets - Both off-theshelf instruct LLMs and our finetuned LLMs on base IFT datasets achieve very low performance. Thus, highlighting lack of Text2Cypher alignment of code LLMs and need of more Text2Cypher IFT datasets. (2) E… view at source ↗

Figures from the paper (11 more)

**Figure 4.** Figure 4: Skeleton schema generation step using Mixtral-8*22B [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

**Figure 5.** Figure 5: Complete schema generation step using Mixtral-8*22B [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

**Figure 6.** Figure 6: Question generation step using Mixtral-8*22B [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]

**Figure 7.** Figure 7: Synthetic ground truth generation step using Mixtral-8*22B [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]

**Figure 8.** Figure 8: Code plan generation step using Gpt-4 11 [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]

**Figure 9.** Figure 9: Python code generation step using Gpt-4 12 [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]

**Figure 10.** Figure 10: Cypher generation: Analyse question step using Gpt-4 [PITH_FULL_IMAGE:figures/full_fig_p013_10.png]

**Figure 11.** Figure 11: Cypher generation: Relate user request to the schema step using Gpt-4 [PITH_FULL_IMAGE:figures/full_fig_p014_11.png]

**Figure 12.** Figure 12: Cypher generation: Incorporate Cypher best practices step using Gpt-4 [PITH_FULL_IMAGE:figures/full_fig_p015_12.png]

**Figure 13.** Figure 13: Cypher generation: Final cypher generation step using Gpt-4 [PITH_FULL_IMAGE:figures/full_fig_p016_13.png]

**Figure 14.** Figure 14: Execution Match - LLM-as-Judge 17 [PITH_FULL_IMAGE:figures/full_fig_p017_14.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

37 extracted references · 35 canonical work pages

[1]

Select Nodes and Relationships: Based on the query type, choose nodes and relationships to form the questions

Analyze the Schema: Examine the provided schema and identify relevant nodes and relationships. Select Nodes and Relationships: Based on the query type, choose nodes and relationships to form the questions

work page
[2]

https://neo4j.com/

Neo4j. https://neo4j.com/. Accessed: 2024- 09-12. Xiang Deng, Ahmed Hassan Awadallah, Christopher Meek, Oleksandr Polozov, Huan Sun, and Matthew Richardson. 2020. Structure-grounded pretraining for text-to-sql. arXiv preprint arXiv:2010.12773. Xiang Deng, Ahmed Hassan Awadallah, Christopher Meek, Oleksandr Polozov, Huan Sun, and Matthew Richardson. 2021. ...

arXiv 2024
[3]

Cover Key Aspects: Each question should focus on distinct parts of the schema, such as relationships between nodes, node properties, or node types

work page
[4]

Random Selection: Randomly select nodes or relationships when forming each question, ensuring diversity in the coverage

Vary Complexity: Ensure the questions range from basic to advanced, covering various levels of query complexity. Random Selection: Randomly select nodes or relationships when forming each question, ensuring diversity in the coverage

work page
[5]

Ensure no two questions are similar

Generate Diverse Questions: Create 20 questions, each addressing different aspects of the schema. Ensure no two questions are similar

work page
[6]

1 million

Clarity and Relevance: All questions should be clear, unambiguous, and reflective of what a human would ask. Important: * Ensure each question includes all the information necessary for a meaningful answer. * Generate exactly 20 questions, ensuring they cover different aspects of the schema and that none are repetitive. Type of query for which questions n...

work page
[7]

UUID Usage in Negative Data: Assign UUIDs to variables before using them in the queries for negative data

work page
[8]

2024-01-01

Specific Values: When generating questions involving values like date, time, money, name, or location, use appropriate placeholder values (e.g., "2024-01-01" for a date. "John Smith" for name etc). Be creative!

work page 2024

Show all 37 references

[9]

For example, if the question mentions 1 million, use 1000000; for 1.2 million, use 1200000

Correct Range of Values: When populating fields like money, votes, or similar data, ensure they align with the question. For example, if the question mentions 1 million, use 1000000; for 1.2 million, use 1200000. Key Points to Remember: - Order of Execution: Ensure that nodes ...
[10]

Understand which entities are crucial to construct the ground truth answer

Analyze the User Question and Schema: Identify relevant nodes, relationships, and fields based on the schema and user question. Understand which entities are crucial to construct the ground truth answer
[11]

Include both the ground truth data and additional negative data points

Plan Data Population: Develop a structured plan that describes how the data will be populated. Include both the ground truth data and additional negative data points
[12]

- Creating negative data points that do not match the answer but help ensure the test is comprehensive

Write Cypher Queries: Provide the exact Cypher queries for: - Creating nodes and relationships for the ground truth answer. - Creating negative data points that do not match the answer but help ensure the test is comprehensive
[13]

Include details like names, summaries, and other fields, making sure the negative data does not overlap with the ground truth

Comprehensive Negative Data: For the negative data points, ensure the information is random , and distinctly different from the ground truth. Include details like names, summaries, and other fields, making sure the negative data does not overlap with the ground truth
[14]

This ensures that the negative data is limited and doesn 't overwhelm the test case

Limit Negative Data Points: Do not create more than 5 negative data points. This ensures that the negative data is limited and doesn 't overwhelm the test case
[15]

Specify which fields require unique values, using UUIDs or similar approaches

Unique Fields for Negative Data: Fields like IDs, names, locations, or titles should be unique, especially in negative data points. Specify which fields require unique values, using UUIDs or similar approaches. Ensure this applies only to negative data; the ground truth must n...
[17]

Use the `MATCH` statement before creating relationships to ensure that the nodes exist and the correct connections are established

Relationship Creation: Create relationships between nodes using their IDs. Use the `MATCH` statement before creating relationships to ensure that the nodes exist and the correct connections are established
[19]

Analyze the schema and user question: Identify relevant nodes, fields, and relationships needed to answer the question
[20]

Refer to the code plan: Follow the provided plan for structuring the data generation code
[21]

Create relationships and nodes: Ensure all required relationships and nodes are generated in the database
[22]

Write the Python code in a function `create_data()`: Return a list of Cypher queries that populate the DB to support the query validation
[23]

No execution logic required: The function should return only the list of queries, not execute them
[24]

Use real timestamps: Any fields like timestamps must reflect actual values
[25]

Ground truth must satisfy the query: Ensure that only the ground truth data satisfies all conditions, and negative data does not
[26]

Ensure negative data points are not more than five

Generate up to 5 negative data points: Each negative example should differ entirely from the ground truth (e.g., UUIDs, random names, summaries). Ensure negative data points are not more than five
[27]

Code Writing Suggestions: - Avoid errors with f-strings by using string concatenation or `.format()` when needed

Use `MATCH` to ensure relationship correctness: Ensure relationships are created by matching node IDs before defining relationships. Code Writing Suggestions: - Avoid errors with f-strings by using string concatenation or `.format()` when needed. - Assign UUIDs to variables be...
[28]

**Identify Nodes and Relationships**: - Examine the schema to identify the different types of nodes (entities) and relationships (edges) between them
[29]

**Node Properties**: - Note the properties (attributes) of each node type
[30]

**Relationship Properties**: - Note the properties of each relationship type
[31]

**Indexes and Constraints**: - Check for any indexes or constraints that might be relevant for query optimization
[32]

- Understand what the user needs, keeping in mind the eventual answer

**Break Down User Question**: - Analyze the user 's question step by step, using the provided schema as grounding. - Understand what the user needs, keeping in mind the eventual answer. - For units like 1 million or 1 dozen, convert them to their base forms (e.g., 1 million to...
[33]

- Ensure that any indexes and constraints are considered when formulating your response

**Formulate the Response**: - Use the identified nodes, relationships, and their properties to inform your understanding of the user 's question. - Ensure that any indexes and constraints are considered when formulating your response. - Formulate a clear breakdown of the user ...
[34]

Keep track of these nodes

Identify which nodes (entities) from the given schema are important in answering the user question and forming the correct Cypher query. Keep track of these nodes. Whenever any kind of ID is present in a node, make sure to add it so the final answer includes it along with othe...
[35]

Do not create imaginary relationships; only consider the relationships that are present in the schema

For all identified important nodes, list all relationships related to those nodes and entities individually. Do not create imaginary relationships; only consider the relationships that are present in the schema
[36]

Do not create imaginary properties; only consider the properties that are present in the schema

For all identified important nodes and relationships, list and filter all properties related to those nodes and entities individually. Do not create imaginary properties; only consider the properties that are present in the schema. Whenever any kind of ID is present in a node,...
[37]

Cypher generation plan

Identify and filter out only the nodes, relationships, and properties which are important and relevant to answering the user 's question and creating the correct Cypher query, given the schema. List out all the important nodes, relationships, and properties that are required t...
[2022]

What is the total sales in USD for Apples in the California market and who made the most sales?

Training language models to follow instruc- tions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744. Liang Shi, Zhengju Tang, Nan Zhang, Xiaotong Zhang, and Zhi Yang. 2024. A survey on employing large language models for text-to-sql tasks. ...

2024 arXiv
[2024]

https://huggingface.co/datasets/ tomasonjo/text2cypher-gpt4o-clean?row=0

Huggingface: tomasonjo/text2cypher-gpt4o- clean. https://huggingface.co/datasets/ tomasonjo/text2cypher-gpt4o-clean?row=0. Accessed: 2024-09-12

2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

Select Nodes and Relationships: Based on the query type, choose nodes and relationships to form the questions

Analyze the Schema: Examine the provided schema and identify relevant nodes and relationships. Select Nodes and Relationships: Based on the query type, choose nodes and relationships to form the questions

work page

[2] [2]

https://neo4j.com/

Neo4j. https://neo4j.com/. Accessed: 2024- 09-12. Xiang Deng, Ahmed Hassan Awadallah, Christopher Meek, Oleksandr Polozov, Huan Sun, and Matthew Richardson. 2020. Structure-grounded pretraining for text-to-sql. arXiv preprint arXiv:2010.12773. Xiang Deng, Ahmed Hassan Awadallah, Christopher Meek, Oleksandr Polozov, Huan Sun, and Matthew Richardson. 2021. ...

arXiv 2024

[3] [3]

Cover Key Aspects: Each question should focus on distinct parts of the schema, such as relationships between nodes, node properties, or node types

work page

[4] [4]

Random Selection: Randomly select nodes or relationships when forming each question, ensuring diversity in the coverage

Vary Complexity: Ensure the questions range from basic to advanced, covering various levels of query complexity. Random Selection: Randomly select nodes or relationships when forming each question, ensuring diversity in the coverage

work page

[5] [5]

Ensure no two questions are similar

Generate Diverse Questions: Create 20 questions, each addressing different aspects of the schema. Ensure no two questions are similar

work page

[6] [6]

1 million

Clarity and Relevance: All questions should be clear, unambiguous, and reflective of what a human would ask. Important: * Ensure each question includes all the information necessary for a meaningful answer. * Generate exactly 20 questions, ensuring they cover different aspects of the schema and that none are repetitive. Type of query for which questions n...

work page

[7] [7]

UUID Usage in Negative Data: Assign UUIDs to variables before using them in the queries for negative data

work page

[8] [8]

2024-01-01

Specific Values: When generating questions involving values like date, time, money, name, or location, use appropriate placeholder values (e.g., "2024-01-01" for a date. "John Smith" for name etc). Be creative!

work page 2024