Leveraging Graph Structures and Large Language Models for End-to-End Synthetic Task-Oriented Dialogues

Bruno Yun; Fr\'ed\'eric Armetta; Hugo Imbert; Maya Medjad; Rapha\"el Szymocha

REVIEW 3 major objections 4 minor 20 references

Leveraging Graph Structures and Large Language Models for End-to-End Synthetic Task-Oriented Dialogues

T0 review · 3 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read GraphTOD turns a JSON graph into task-oriented dialogue data.

desk verdict GraphTOD is a sensible, small extension of SynTOD whose JSON interface is genuinely useful, but the evaluation never measures graph adherence—the one thing that makes the framework different from ordinary LLM chit-chat. read the letter →

arxiv 2501.11977 v1 pith:EGACRL3Y submitted 2025-01-21 cs.CL cs.AI

Maya Medjad , Hugo Imbert , Bruno Yun , Raphaël Szymocha , Frédéric Armetta This is my paper · ORCID

classification cs.CLcs.AI

keywords task-orienteddialoguesyntheticdatagenerationlargelanguagemodelsstatetransitiongraphJSONtwo-agentsimulationUniEvalevaluation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

GraphTOD is a pipeline that generates synthetic task-oriented dialogues from a user-supplied action transition graph written in JSON. The paper's central claim is that this graph-only specification lets non-technical users create training data for task-oriented dialogue systems without writing custom prompts or code. Two large-language-model agents, one playing the system and one the user, walk the graph turn by turn: the user agent chooses an allowed action from the current node, and the system agent detects that intent, optionally calls an API, and advances to the next node. The authors report that 150 generated conversations across four domains score between 0.820 and 0.882 on UniEval naturalness, coherence, and understandability, comparable to a human-in-the-loop LLM baseline. If this holds, the cost and complexity bottleneck of dataset creation is substantially reduced.

What carries the argument

The load-bearing object is the action transition graph $G = (V, Ac, E, t, s, f)$ specified in JSON. It encodes the task flow, and it also defines, at each node, the set of allowed actions the user agent may choose and the system agent must recognize. The pipeline pairs this graph with two LLM agents and five prompt templates: four for the system (intent detection, API-driven knowledge collection, unrecognized-intent reply, continuation or closing) and one for the user (persona-grounded utterance generation). The graph is what links the two agents, so dialogue generation becomes a walk—random action selection, intent detection, API calls, node transition—rather than free-form conversation. User-agent preferences are generated automatically from the graph, and the system agent's knowledge database accumulates retrieved API results during the dialogue.

What would settle it

Generate dialogues with GraphTOD, then parse each turn and check whether (i) the user's utterance corresponds to an action allowed at the current node, (ii) the system's detected intent equals that action, and (iii) the next node equals the transition prescribed by the graph; a non-negligible fraction of violating turns, or a demonstration that random graph violations produce equally high quality scores, would settle whether the central claim holds.

Watch

Extended reading notes

Core claim

The discovery the paper argues for is that a finite action transition graph, specified as JSON, is enough structure to drive two LLM agents into producing coherent task-oriented dialogues end to end. Formally the graph is $G = (V, Ac, E, t, s, f)$: nodes, actions, allowed actions per non-final node, transition function, start and final states; a subset of actions are function calls that fetch external knowledge via APIs. The system agent runs two reasoning steps per turn—intent detection using the current node's allowed actions, then response generation through one of several prompt templates—while the user agent generates utterances from a persona and the selected action. The authors report UniEval mean scores of 0.820–0.882 across the Doctor, RentCar, Hotel, and Recipe scenarios, and state these are on par with human-in-the-loop LLM generation approaches, at a fraction of the human annotation burden.

Load-bearing premise

The whole approach depends on the two language-model agents actually following the graph—choosing an allowed next action, recognizing it, and moving to the node the graph points to—and the paper does not measure how often they do.

Editorial extensions

If this is right

A non-technical user can produce a domain-specific dialogue dataset by writing a JSON graph, without custom prompts or code.
The generated dialogues are scored on par with a human-in-the-loop LLM baseline on naturalness, coherence, and understandability, despite requiring far less human effort.
The same five prompt templates cover Recipe, Hotel, RentCar, and Doctor domains, so adding a new domain reduces to supplying a new graph.
The system agent's LLM-based intent detection and API calls make the pipeline tolerant of user phrasing that does not exactly match the action names.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because the interface is just a JSON graph, the pipeline could be rerun with open-weight or local LLMs, and the reported quality scores would either hold or not; that experiment would show whether the graph structure or the underlying model is doing the work.
The automatic metrics used here measure surface quality, not task-flow correctness; a cheap companion check would parse generated dialogues and verify that each turn's action is allowed at the current node and that the next node equals the graph's transition, and correlating that adherence rate with the quality scores would reveal whether high surface quality can coexist with structural violations
If GraphTOD scales, the same JSON-graph abstraction could generate deliberate edge cases—off-graph actions, misrecognized intents, or API failures—giving downstream systems negative training examples that the current pipeline does not produce.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

GraphTOD is worth a look if you work on synthetic task-oriented dialogue. The core idea is simple: take SynTOD's state-transition-graph-plus-LLM recipe, let users specify the graph in JSON, auto-generate personas from the graph, and use a generalized prompting template so the two agents (user and system) walk the graph turn by turn. That is a real usability improvement, and the formalization of the pipeline in Section 2 is clear enough that someone could reimplement it from the paper. The promised GitHub code and demo video also count in its favor.

The soft spots are real, and the stress-test note gets the biggest one right: graph adherence is never measured. The framework's entire claim to structure is that the user agent picks an action in Ac+_v and the system agent detects that intent and follows t(v,a). If the LLMs drift from the graph, the dialogue can still score well on UniEval fluency while failing the actual task flow. The paper reports only UniEval naturalness, coherence, and understandability—none of which detects whether the right node transition happened. Example 1 is a single hand-picked trace. The conclusion's phrase "fully automated and fault-tolerant" is unsupported. This is a load-bearing gap, not a nit.

The evaluation is also thinner than the abstract implies: 150 dialogues, about 37 per domain, means with no variance or significance testing, and the LAPS baseline is matched only for Recipe—the other three GraphTOD domains have no baseline at all. The claim that GraphTOD "reports similar performances" holds only for the one comparable cell. And "significantly lowering the cost and complexity" is asserted without any cost or usability measurement. Those are all fixable issues: add a graph-adherence metric (e.g., fraction of turns where the system's detected intent matches the selected action), run matched baselines, report error bars or human judgments, and actually measure setup time.

Is the central argument sound? Yes, conditionally. The framework is plausible, the JSON interface is a legitimate contribution to an established program, and the evaluation problems are ones that revision can address. This is not a paper whose math or method collapses; it is a paper whose claims outrun its evidence.

For a reader: if you are building task-oriented chatbots and want cheap synthetic data, this is a promising tool to try, and the code will help. For a researcher, it reads like a solid workshop paper with a roadmap to a stronger conference paper. It deserves a serious referee, but the referee should insist on measuring graph adherence before accepting the central claim.

Referee Report

3 major / 4 minor

Summary. The paper introduces GraphTOD, a pipeline that uses two LLM agents (system and user) to generate task-oriented dialogues by navigating a user-provided action transition graph in JSON. The system agent performs LLM-based intent detection and API calls, while the user agent generates utterances from a selected action. The authors report UniEval naturalness, coherence, and understandability scores for 150 generated dialogues across four domains (Recipe, Hotel, RentCar, Doctor) and claim, on the strength of those means, that GraphTOD produces high-quality dialogues comparable to the LAPS human-in-the-loop approach. The paper concludes that the pipeline is fully automated and fault-tolerant.

Significance. If the evaluation were adequate, GraphTOD would be a useful contribution: it lowers the entry barrier for synthetic task-oriented dialogue dataset creation by letting non-technical users specify a graph in JSON, it includes automatic persona/preference generation, and the open-source code and demo support reproducibility. However, the current evidence is not yet sufficient to support the central claim. The strengths are architectural and practical rather than empirical, and the empirical claims need to be either substantially strengthened or appropriately softened.

major comments (3)

[Section 3, Table 1] The claim that GraphTOD "performs consistently well overall" and "reports similar performances" to LAPS is not supported by the reported numbers. The LAPS baseline is scored on Recipe and Movie, while GraphTOD is scored on Recipe, Hotel, RentCar, and Doctor; only the Recipe domain permits a direct comparison, and even there the differences are of the same order as the inter-domain variation among GraphTOD's own scores. With roughly 37 dialogues per domain and only mean UniEval scores reported, there is no way to assess whether the differences are meaningful. Please report score distributions with variance, run significance tests, and either add LAPS scores for the other three domains or restrict the "similar to LAPS" claim to the Recipe domain.
[Sections 2, 3 and Example 1] The framework's core differentiator is that both agents navigate a specified action transition graph, with the system agent detecting the user's intent and performing the transition t(v,a). This property is never measured. UniEval fluency scores cannot detect whether a user utterance corresponds to the randomly selected action a in Ac+_v or whether the system's detected intent and node transition match the graph. Example 1 is a single hand-picked illustration, not a failure-rate estimate. The conclusion's assertion that the pipeline is "fully automated and fault-tolerant" (Section 4) is therefore unsupported. Please add a quantitative graph-adherence evaluation, for instance manual annotation or a second LLM checking action/node alignment over all generated dialogues, and report per-domain adherence or error rates.
[Section 3] The UniEval dimensions used (naturalness, coherence, understandability) are fluency-oriented and do not measure task success, which is essential for task-oriented dialogues. The paper does not report whether the system retrieved the correct API results, reached the final state, or satisfied the user's goal. A task-completion metric (or at least a final-state and final-intent success rate) should be added before claiming "high-quality dialogues" for TOD training data.

minor comments (4)

[Section 2] In the dialogue history definition, "at timej ≥ 0" should read "at time j ≥ 0".
[References] References [15] and [20] are the same paper (Joko et al., "Doing personal laps") and the duplicate entry should be removed.
[Section 3 / footnote 5] The prompt templates and transition graphs are only available through the GitHub repository; the paper would be more self-contained if these artifacts were also included in an appendix or supplementary material, especially since the evaluation depends on them.
[Figure 1] The caption "Representation of one turn of generation of the GraphTOD generation pipeline" is redundant; suggest "One turn of the GraphTOD generation pipeline."

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the pipeline is evaluated with the external UniEval metric and no self-citation chain or fitted-parameter-as-prediction step is present.

full rationale

GraphTOD is a constructive system paper rather than a derivation. The core claim that the framework generates high-quality task-oriented dialogues is checked against external UniEval naturalness, coherence, and understandability scores on 150 generated dialogues, and against an external baseline, LAPS. There is no parameter fitted to the evaluation data and then renamed as a prediction; the generated dialogues are direct outputs of the LLM pipeline, not quantities implied by the evaluation metric. The authors refer readers to their own GitHub repository for prompt templates and graphs, but this is a pointer to implementation details, not load-bearing mathematical support, and no argument in the paper depends on a self-citation. The skeptical concern that graph adherence is never quantitatively measured is an evidence/validation gap, not a circular step: a missing measurement does not make the claim equivalent to its inputs. Likewise, the Section 4 statement that the pipeline is 'fully automated and fault-tolerant' is unsupported by the reported metrics, but unsupported assertiveness is not circularity. Accordingly, the circularity score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The framework introduces no new scientific entities, forces, or conserved quantities. The core machinery (LLM agents, transition graphs, personas) is constructed from existing components. The main hidden costs are design choices (prompt templates, persona parameters) rather than fitted parameters.

assumptions (3)

domain assumption GPT-4 (the LLM used) reliably follows the prompt templates to produce coherent, on-task utterances and correct intent detection.
The whole pipeline depends on LLM instruction-following; no analysis of success rates or failure cases is provided.
domain assumption UniEval's naturalness, coherence, and understandability scores are valid proxies for dialogue quality.
The paper uses UniEval as the sole quality measure, without human evaluation or correlation analysis.
domain assumption The JSON graph format is expressive enough for the intended task flows and usable by non-technical users.
The paper claims accessibility but provides no user study or expressiveness analysis.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Leveraging Graph Structures and Large Language Models for End-to-End Synthetic Task-Oriented Dialogues." pith.science (2026). https://pith.science/paper/EGACRL3Y

@misc{pith2026250111977,
  author       = {Pith},
  title        = {Pith review of: Leveraging Graph Structures and Large Language Models for End-to-End Synthetic Task-Oriented Dialogues},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EGACRL3Y}},
  note         = {Machine review of arXiv:2501.11977}
}

read the original abstract

Training task-oriented dialogue systems is both costly and time-consuming, due to the need for high-quality datasets encompassing diverse intents. Traditional methods depend on extensive human annotation, while recent advancements leverage large language models (LLMs) to generate synthetic data. However, these approaches often require custom prompts or code, limiting accessibility for non-technical users. We introduce GraphTOD, an end-to-end framework that simplifies the generation of task-oriented dialogues. Users can create dialogues by specifying transition graphs in JSON format. Our evaluation demonstrates that GraphTOD generates high-quality dialogues across various domains, significantly lowering the cost and complexity of dataset creation.

Figures

Figures reproduced from arXiv: 2501.11977 by the authors.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 12 canonical work pages

[1]

Hello, it’s GPT-2 - how can I help you? towards the use of pretrained language models for task-oriented dialogue systems

Pawel Budzianowski and Ivan Vulic. Hello, it’s GPT-2 - how can I help you? towards the use of pretrained language models for task-oriented dialogue systems. In Alexandra Birch, Andrew M. Finch, Hiroaki Hayashi, Ioannis Konstas, Thang Luong, Graham Neubig, Yusuke Oda, and Katsuhito Sudoh, editors,Proceedings of the 3rd Workshop on Neural Generation and Tra...

work page 2019
[2]

Neural approaches to conversational AI

Jianfeng Gao, Michel Galley, and Lihong Li. Neural approaches to conversational AI. F ound. Trends Inf. Retr ., 13(2-3):127–298, 2019

work page 2019
[3]

End-to-end task-oriented dialogue: A survey of tasks, methods, and future directions

Libo Qin, Wenbo Pan, Qiguang Chen, Lizi Liao, Zhou Yu, Yue Zhang, Wanxiang Che, and Min Li. End-to-end task-oriented dialogue: A survey of tasks, methods, and future directions. arXiv preprint arXiv:2311.09008, 2023

arXiv 2023
[4]

Task-oriented dialogue system as natural language generation

Weizhi Wang, Zhirui Zhang, Junliang Guo, Yinpei Dai, Boxing Chen, and Weihua Luo. Task-oriented dialogue system as natural language generation. In Proceedings of the 45th international ACM SIGIR conference on research and development in information retrieval, pages 2698–2703, 2022

work page 2022
[5]

Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset, 2020

Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset, 2020

work page 2020
[6]

Mistral 7b

Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023

arXiv
[7]

Language models are few-shot learners

Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020

arXiv 2005
[8]

A survey on recent advances in conversational data generation, 2024

Heydar Soudani, Roxana Petcu, Evangelos Kanoulas, and Faegheh Hasibi. A survey on recent advances in conversational data generation, 2024

work page 2024

Show all 20 references

[9]

Are llms all you need for task-oriented dialogue?, 2023

V ojtˇech Hudeˇcek and Ondˇrej Dušek. Are llms all you need for task-oriented dialogue?, 2023

2023
[10]

Survey of hallucination in natural language generation

Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1–38, 2023

2023
[11]

Toad: Task-oriented automatic dialogs with diverse response styles, 2024

Yinhong Liu, Yimai Fang, David Vandyke, and Nigel Collier. Toad: Task-oriented automatic dialogs with diverse response styles, 2024

2024
[12]

Simulating task-oriented dialogues with state transition graphs and large language models

Chris Samarinas, Pracha Promthaw, Atharva Nijasure, Hansi Zeng, Julian Killingback, and Hamed Zamani. Simulating task-oriented dialogues with state transition graphs and large language models. CoRR, abs/2404.14772, 2024

2024 arXiv
[13]

Recent advances and challenges in task-oriented dialog systems

Zheng Zhang, Ryuichi Takanobu, Qi Zhu, MinLie Huang, and XiaoYan Zhu. Recent advances and challenges in task-oriented dialog systems. Science China Technological Sciences, 63(10):2011–2027, 2020

2011
[14]

Multiwoz – a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling, 2020

Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gaši´c. Multiwoz – a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling, 2020

2020
[15]

de Vries, Jeff Dalton, and Faegheh Hasibi

Hideaki Joko, Shubham Chatterjee, Andrew Ramsay, Arjen P. de Vries, Jeff Dalton, and Faegheh Hasibi. Doing personal laps: Llm-augmented dialogue construction for personalized multi-session conversational search. In Proceedings of the 47th International ACM SIGIR Conference on ...

2024
[16]

Lucid: Llm-generated utterances for complex and interesting dialogues, 2024

Joe Stacey, Jianpeng Cheng, John Torr, Tristan Guigue, Joris Driesen, Alexandru Coca, Mark Gaynor, and Anders Johannsen. Lucid: Llm-generated utterances for complex and interesting dialogues, 2024

2024
[17]

Towards a unified multi-dimensional evaluator for text generation, 2022

Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Pengfei Liu, Chenguang Zhu, Heng Ji, and Jiawei Han. Towards a unified multi-dimensional evaluator for text generation, 2022

2022
[18]

Bleu: a method for automatic evaluation of machine translation

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics , ACL ’02, page 311–318, USA, 2002. Association for Computational...

2002
[19]

Rouge: A package for automatic evaluation of summaries

Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81, 2004

2004
[20]

de Vries, Jeff Dalton, and Faegheh Hasibi

Hideaki Joko, Shubham Chatterjee, Andrew Ramsay, Arjen P. de Vries, Jeff Dalton, and Faegheh Hasibi. Doing personal laps: Llm-augmented dialogue construction for personalized multi-session conversational search. In Proceedings of the 47th International ACM SIGIR Conference on ...

2024

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

Hello, it’s GPT-2 - how can I help you? towards the use of pretrained language models for task-oriented dialogue systems

Pawel Budzianowski and Ivan Vulic. Hello, it’s GPT-2 - how can I help you? towards the use of pretrained language models for task-oriented dialogue systems. In Alexandra Birch, Andrew M. Finch, Hiroaki Hayashi, Ioannis Konstas, Thang Luong, Graham Neubig, Yusuke Oda, and Katsuhito Sudoh, editors,Proceedings of the 3rd Workshop on Neural Generation and Tra...

work page 2019

[2] [2]

Neural approaches to conversational AI

Jianfeng Gao, Michel Galley, and Lihong Li. Neural approaches to conversational AI. F ound. Trends Inf. Retr ., 13(2-3):127–298, 2019

work page 2019

[3] [3]

End-to-end task-oriented dialogue: A survey of tasks, methods, and future directions

Libo Qin, Wenbo Pan, Qiguang Chen, Lizi Liao, Zhou Yu, Yue Zhang, Wanxiang Che, and Min Li. End-to-end task-oriented dialogue: A survey of tasks, methods, and future directions. arXiv preprint arXiv:2311.09008, 2023

arXiv 2023

[4] [4]

Task-oriented dialogue system as natural language generation

Weizhi Wang, Zhirui Zhang, Junliang Guo, Yinpei Dai, Boxing Chen, and Weihua Luo. Task-oriented dialogue system as natural language generation. In Proceedings of the 45th international ACM SIGIR conference on research and development in information retrieval, pages 2698–2703, 2022

work page 2022

[5] [5]

Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset, 2020

Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset, 2020

work page 2020

[6] [6]

Mistral 7b

Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023

arXiv

[7] [7]

Language models are few-shot learners

Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020

arXiv 2005

[8] [8]

A survey on recent advances in conversational data generation, 2024

Heydar Soudani, Roxana Petcu, Evangelos Kanoulas, and Faegheh Hasibi. A survey on recent advances in conversational data generation, 2024

work page 2024