Pith. sign in

REVIEW 4 major objections 5 minor 17 references

FLOW-BENCH: Towards Conversational Generation of Enterprise Workflows

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims plain-English instructions can be translated into BPMN workflows with high accuracy by having an LLM first generate a Python intermediate representation, and it contributes a new dataset, FLOW-BENCH, to test this.

desk verdict A genuinely useful small benchmark for conversational workflow editing, but the evaluation covers only the Python IR leg; the BPMN converters are unvalidated black boxes. read the letter →

arxiv 2505.11646 v1 pith:UE6YHOHR submitted 2025-05-16 cs.AI cs.SE

Evelyn Duesterwald , Siyu Huo , Vatche Isahagian , K.R. Jayaram , Ritesh Kumar , Vinod Muthusamy , Punleuk Oum , Debashish Saha
show 2 more authors
Gegi Thomas Praveen Venkateswaran
This is my paper · ORCID
classification cs.AIcs.SE
keywords businessprocessautomationnaturallanguagetoworkflowLLMPythonintermediaterepresentationBPMNdatasetretrieval-augmentedgeneration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that enterprise workflows can be authored and edited conversationally: a user describes a change in natural language, and a pipeline of retrievers, an LLM, and deterministic converters produces a standard BPMN process definition. The key design move is to have the LLM generate a concise Python intermediate representation of the workflow, rather than the verbose BPMN XML directly, and to convert that IR to BPMN with deterministic software. The paper also releases FLOW-BENCH, a dataset of 101 paired natural-language utterances and workflow representations, along with the claim that the best configuration reaches an exact-match accuracy of 0.83 in-domain and 0.79 cross-domain on this dataset. A sympathetic reader would care because it points to a path where business process automation no longer requires drag-and-drop expertise or hand-written XML.

What carries the argument

The central object is the Python intermediate representation (Python IR): a constrained subset of Python consisting of assignments, if-statements, for and while loops, and function calls, where each function call names a catalogued API (for example, GitHub_Issue__3_0_0__create_Issue()). Because BPMN XML is on average 25 times longer than the equivalent Python, the IR shrinks the LLM's output space and takes advantage of LLMs' strength in code generation. The other load-bearing pieces are the retrieval modules—Activities_Search, a fine-tuned embedding retriever that selects relevant API activities to include in the prompt, and CE_Retriever, a cross-encoder that selects few-shot demonstrations—and the deterministic converters PY2BPMN, BPMN2PY, and DIFF2BPMN that map between the IR and BPMN, and compute incremental updates when a user edits an existing workflow.

What would settle it

Run a round-trip test on all 101 FLOW-BENCH cases: take each BPMN representation, convert it to Python IR with BPMN2PY, convert back with PY2BPMN, and check that the two BPMN files are structurally identical. If any case fails to round-trip—for example a loop or conditional is lost in the conversion—then the deterministic converters are not lossless, and the claim that FLOW-GEN produces correct BPMN from the generated IR collapses.

Watch

Extended reading notes

Core claim

The paper's central claim is that retrieval-augmented prompting of off-the-shelf LLMs can translate natural-language workflow instructions into a constrained Python intermediate representation (Python IR) that matches the ground-truth workflow logic, and that this IR can then be converted into standard BPMN by deterministic modules. On the 101 FLOW-BENCH build-step tests, the best pipeline—Activities_Search activity retrieval with TopK=50 and CE_Retriever demonstration retrieval with TopK=5—together with the Mistral-large model achieves an exact match of 0.83 in-domain and 0.79 cross-domain, and near-zero hallucination rates below 0.01. The paper argues this two-stage NL-to-IR-to-BPMN design outperforms direct BPMN generation because LLMs are far more competent at Python code than at verbose BPMN XML, and because the IR makes errors easy to catch before final conversion.

Load-bearing premise

The deterministic PY2BPMN, BPMN2PY, and DIFF2BPMN modules—the only components that convert the Python IR into BPMN and back—are assumed to be correct, lossless, and complete, but their algorithms are never specified or tested in the paper; if they mishandle loops, conditionals, or user tasks, the final BPMN need not correspond to the generated IR.

Editorial extensions

If this is right

  • Conversational workflow editing becomes practical: users can state incremental changes such as add, delete, or replace and receive updated BPMN without drag-and-drop.
  • The two-stage design means errors can be caught at the IR level before committing to BPMN, simplifying validation and debugging of generated workflows.
  • The FLOW-BENCH dataset gives the community a common benchmark of 101 build-step tests with prior and expected sequences plus BPMN artifacts.
  • A cross-domain exact match of 0.79 suggests the approach generalizes to activities not seen in the demonstrations, reducing the need for retraining on new domains.
  • API grounding through retrieval keeps hallucination rates low (below 0.01 for the best retriever), making generated workflows more trustworthy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The true bottleneck is likely the LLM's Python-generation ability, not the retrieval or conversion stages; improving code LLMs should transfer directly to workflow quality.
  • The same Python-IR architecture could extend to decision logic (DMN) and to other imperative process notations, since the IR is notation-agnostic.
  • A larger and more diverse benchmark would be needed to confirm the 0.8-level accuracy holds beyond the 101 curated cases sourced from two automation platforms.
  • Because the exact-match metric is measured on the IR and the BPMN conversion is internal to the paper, independent verification would require the deterministic converters to be released.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces FLOW-BENCH, a dataset of 101 incremental business-workflow build steps, each consisting of a natural-language utterance, a prior Python intermediate representation (IR), an expected Python IR, and BPMN representations of both prior and expected workflows. It also presents FLOW-GEN, a pipeline that retrieves relevant activities and demonstration examples, prompts an LLM to generate a Python IR for the utterance, and then uses deterministic modules (PY2BPMN, BPMN2PY, DIFF2BPMN) to convert between the IR and BPMN. The evaluation compares three activity retrievers, two demonstration retrievers, and eight LLMs, with Mistral-large achieving exact match scores of 0.83 in-domain and 0.79 cross-domain on the benchmark's test cases. The paper claims that FLOW-GEN translates natural language into business process definitions such as BPMN and DMN, and reports a deployment in IBM's Watsonx Orchestrate.

Significance. If the results hold, FLOW-BENCH would be a useful public resource for a relatively underexplored task, and FLOW-GEN would demonstrate a practical retrieval-augmented prompting recipe for generating workflow logic. The paper's strengths include a public dataset, an evaluation across multiple LLMs and retrievers, and a cross-domain evaluation that attempts to control for demonstration overlap. However, the central evidence gap is substantial: the headline exact-match scores are computed only on the Python IR, while the paper's stated goal is generation of BPMN and DMN artifacts. The deterministic converters that produce BPMN are not specified, implemented, or tested, and the DMN claim is not accompanied by any module or evaluation. The best-performing activity retriever is a fine-tuned model with undisclosed training data, creating a leakage risk. These issues do not necessarily invalidate the IR-generation results, but they mean the paper does not currently support its central claim of producing correct business process definitions.

major comments (4)
  1. [§4.3, Tables 1–3] The evaluation metrics Exact Match and Syntax F1 are computed on the generated Python IR against the ground-truth Python IR, not on the final BPMN output. The abstract and Section 1 claim translation to BPMN and DMN, but no BPMN-level accuracy, no round-trip test of PY2BPMN/BPMN2PY/DIFF2BPMN, and no DMN evaluation appear anywhere in the manuscript. Because the deterministic converters are described only at a high level in Sections 3.2–3.3 and Figure 2, a perfectly matching IR could still yield incorrect BPMN if those converters mishandle loops, conditionals, user tasks, or incremental diffs. The paper needs to either release and test these converters or substantially narrow its claims to IR generation.
  2. [§3.4, Table 1] Activities_Search, which drives the headline results in Table 3, is described as a 'custom model fine-tuned to generate better embeddings for the activity retrieval task,' but the training data, fine-tuning procedure, and any overlap with the 101 FLOW-BENCH test cases are not disclosed. Since FLOW-BENCH was authored by the same team, this creates a concrete data-leakage risk that could inflate the reported recall and exact-match numbers. The authors should disclose the fine-tuning data, describe how overlap with the test set was prevented, and ideally report results with a retriever that has no access to benchmark-derived training data.
  3. [§4.1–4.2, Tables 1–2] The configuration used for the headline results (Activities_Search with TopK=50 and CE_Retriever with TopK=5) is selected by inspecting performance on the same 101 test cases used for the final evaluation. There is no train/validation/test split, so the reported numbers may reflect overfitting to the test set through hyperparameter selection. The authors should either use a held-out validation set for choosing TopK and retriever combinations or clearly report the selection procedure and its effect on the final numbers.
  4. [§1, §3.1, §7] The abstract and introduction repeatedly promise DMN support ('BPMN and DMN', 'BPMN-compliant XML or DMN decision tables'), but the manuscript never describes a DMN module, a DMN representation, or any DMN evaluation. This is a mismatch between the stated scope and the actual content. Either a DMN converter and its evaluation need to be added, or the DMN claim should be removed from the paper's contributions.
minor comments (5)
  1. [§2, Manual Labeling] The text says 'manually crafted Python IR snippets and generated the corresponding BPMN specification,' but it does not state which tool or converter produced the BPMN from the IR. Since the gold BPMN is used in the dataset, clarifying this provenance would help readers assess the dataset's validity.
  2. [§4.3, Table 3 caption] The caption contains a typo: 'Activi-ties_Search (TopK=50) as and CE_Retriever(TopK=5) as Activities and demos retrievers respectively' should read 'as the activity and demo retrievers, respectively.'
  3. [§2, Quality Control] The phrase 'worfklow activities' contains a typo; it should be 'workflow activities.'
  4. [§3.5, Demonstration Retrievers] The sentence 'if the user is updating an existing workflow, only the demos containing a prior sequence are selected' is unclear because all demos in FLOW-BENCH contain a prior sequence. The intended filtering criterion should be stated more precisely.
  5. [§6, Related Work] The related-work section discusses AutoFlow, APA, and WorkflowLLM, but it does not mention any prior benchmarks for NL-to-BPMN or NL-to-workflow generation, which would help position FLOW-BENCH relative to existing evaluation resources.

Circularity Check

2 steps flagged · score 6.0 of 10

Evaluation is partly in-sample: hyperparameters and the best LLM are chosen on the same 101 test cases that are then scored, and the in-domain demo pool can supply the ground-truth answer.

  1. fitted input called prediction [Section 4.1-4.3 (Tables 1-3)]
    "All experiments were conducted over the 101 FLOW-BENCH test cases. ... For the remainder of these experiments we consider retrieving five demonstrations using CE_Retriever. ... We use Activities_Search as the activity retriever with TopK=50 and CE_Retriever as the demonstration retriever with TopK=5."

    The configuration reported in Table 3 was selected because Tables 1 and 2 showed that Activities_Search TopK=50 and CE_Retriever TopK=5 gave the highest Exact Match on the same 101 test cases. The headline 0.83/0.79 numbers are therefore in-sample optima, not held-out predictions. The best-performing LLM in Table 3 is likewise identified by the same test-set scores. Choosing hyperparameters and the model on the test set and then reporting the test-set score as the system's performance makes the reported prediction statistically forced rather than an independent estimate.

  2. self definitional [Section 3.2 and Section 4.3]
    "Concurrently, the utterance is used to select the most relevant demonstrations from the dataset. ... All experiments were conducted over the 101 FLOW-BENCH test cases. ... for in-domain, activities present in ground-truth may be present in selected demos."

    The demonstration catalog is FLOW-BENCH, and the evaluation set is the entire 101-test FLOW-BENCH. Each test case contains an utterance paired with its expected Python sequence, exactly the (utterance, code) form used as a demonstration. The in-domain condition does not exclude the current test case's own ground-truth pair from the retrieved demos (cross-domain only filters by activities). A cross-encoder demo retriever would rank the identical utterance/ground-truth pair highly, so the LLM can copy the expected sequence from the prompt. Under that mechanism, the generated IR equals the expected IR by construction, so part of the in-domain Exact Match measures retrieval of the answer rather than independent generation.

full rationale

The paper's core pipeline is NL -> Python IR -> BPMN/DMN, with Exact Match and Syntax F1 computed only on the Python IR. Two elements of the evaluation are circular or in-sample. First, the final configuration (Activities_Search TopK=50, CE_Retriever TopK=5, and even the best LLM) is selected by maximizing Exact Match on the same 101 FLOW-BENCH test cases that are then reported in Table 3; the headline 0.83/0.79 is thus a test-set optimum rather than a prediction on unseen data. Second, demonstrations are retrieved from the same FLOW-BENCH corpus that constitutes the test set, and the in-domain condition explicitly permits ground-truth activities in the demos; since each test case's utterance/expected-sequence pair is a valid demo, the retriever can supply the answer itself, making part of the in-domain score a copy rather than a generation. The cross-domain score is less exposed to exact-copy leakage because it filters demos by ground-truth activities, but it still inherits the test-set hyperparameter selection. The BPMN/DMN half of the claim is not circular but unsupported: PY2BPMN, BPMN2PY, and DIFF2BPMN are never specified or tested, no DMN module or metric appears, and all reported scores are IR-only. I find no self-citation chain or imported uniqueness theorem. Overall, the circularity is in the evaluation procedure rather than in the derivation itself, and it is partial rather than total.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper contributes a dataset and an empirical system, but the central claims rest on several unproven operational assumptions: the correctness of the unshown deterministic converters, the validity of the manually constructed ground truth, and the integrity of the undisclosed fine-tuned retriever. No mathematical derivation is attempted, so the ledger is dominated by domain assumptions rather than free parameters.

free parameters (3)
  • Activities_Search fine-tuned embedding model = unspecified parameters
    A custom model is fine-tuned for activity retrieval (Section 3.4) but the training data, objective, and validation split are not disclosed; it is a major tunable component in the evaluation.
  • TopK_activities = 50
    Selected based on retrieval and exact-match performance in Table 1 on the same 101 test cases; this tunes the pipeline to the benchmark.
  • TopK_demos = 5
    Selected based on exact match in Table 2 across the same benchmark; beyond 5 demos performance degrades.
assumptions (4)
  • domain assumption Python syntax is a faithful intermediate representation for the business processes in FLOW-BENCH
    Section 3.1 asserts Python IR captures workflow logic more compactly than BPMN; no proof is offered, and BPMN-specific concepts like swimlanes and roles are excluded from the dataset.
  • ad hoc to paper The deterministic PY2BPMN, BPMN2PY, and DIFF2BPMN modules correctly and losslessly convert between Python IR and BPMN
    These modules are central to FLOW-GEN (Sections 3.2-3.3, Figure 2) but are not specified, released, or evaluated; their correctness is assumed.
  • domain assumption Manually crafted BPMN and Python IR ground truths in FLOW-BENCH are correct
    Section 2 states manual labeling and generation of Python IR and BPMN; no inter-annotator agreement or external validation is reported.
  • domain assumption Exact Match, defined as syntactically and semantically matching the ground truth, is a valid measure of workflow correctness
    Section 4 uses Exact Match as the headline metric, but the matching algorithm is not described; semantic equivalence is asserted, not operationalized.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FLOW-BENCH: Towards Conversational Generation of Enterprise Workflows." pith.science (2026). https://pith.science/paper/UE6YHOHR

@misc{pith2026250511646,
  author       = {Pith},
  title        = {Pith review of: FLOW-BENCH: Towards Conversational Generation of Enterprise Workflows},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UE6YHOHR}},
  note         = {Machine review of arXiv:2505.11646}
}
read the original abstract

Business process automation (BPA) that leverages Large Language Models (LLMs) to convert natural language (NL) instructions into structured business process artifacts is becoming a hot research topic. This paper makes two technical contributions -- (i) FLOW-BENCH, a high quality dataset of paired natural language instructions and structured business process definitions to evaluate NL-based BPA tools, and support bourgeoning research in this area, and (ii) FLOW-GEN, our approach to utilize LLMs to translate natural language into an intermediate representation with Python syntax that facilitates final conversion into widely adopted business process definition languages, such as BPMN and DMN. We bootstrap FLOW-BENCH by demonstrating how it can be used to evaluate the components of FLOW-GEN across eight LLMs of varying sizes. We hope that FLOW-GEN and FLOW-BENCH catalyze further research in BPA making it more accessible to novice and expert users.

Figures

Figures reproduced from arXiv: 2505.11646 by the authors.

Figure 1
Figure 1. Example of FLOW-BENCH test case Expected Sequence elements of each test. Generating accurate pythonic function calls in a FLOW-BENCH test by an LLM requires knowledge of existing APIs and descriptions. Thus, we also provide a separate file containing a list of APIs along with their descriptions. An example of the API and the description is shown below. { "id": "Jira_Issue__2_0_0__retrievewithwhere_Issue", "descripti… view at source ↗
Figure 2
Figure 2. FLOW-GEN overview. The top part (in black) depicts the steps to generate a new workflow based on a user utterance. The bottom part (in blue) are the additional steps to update an existing workflow based on an utterance. 3.3 Incremental Flow Updates Let us now consider the case where there is already an existing workflow, and the user issues an ut￾terance to incrementally edit the workflow. The bottom portion of [PI… view at source ↗
Figure 3
Figure 3. Deployment in WxO production environment [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: shows an example of a flow expressed as a BPMN diagram along with the corresponding BPMN code shown in [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: BPMN code corresponding to example in [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: BPMN code corresponding to the prior sequence of Figure [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: BPMN code corresponding to the final output of Figure [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 8 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    decision model and notation

    Thierry Biard, Alexandre Le Mauff, Michel Bigand, and Jean-Pierre Bourey. 2015. Separation of decision modeling from business process modeling using new “decision model and notation”(dmn) for automating operational decision-making. In Risks and Resilience of Collaborative Networks: 16th IFIP WG 5.5 Working Conference on Virtual Enterprises, PRO-VE 2015, A...

    work page 2015

  4. [4]

    Michele Chinosi and Alberto Trombetta. 2012. Bpmn: An introduction to the standard. Computer Standards & Interfaces, 34(1):124--134

    work page 2012

  5. [5]

    Shengda Fan, Xin Cong, Yuepeng Fu, Zhong Zhang, Shuyan Zhang, Yuanwei Liu, Yesai Wu, Yankai Lin, Zhiyuan Liu, and Maosong Sun. 2024. https://api.semanticscholar.org/CorpusID:273950328 Workflowllm: Enhancing workflow orchestration capability of large language models . ArXiv, abs/2411.05451

    arXiv 2024

  6. [6]

    Alexander Grosskopf, Gero Decker, and Mathias Weske. 2009. The process: business process modeling using BPMN. Meghan-Kiffer Press

    work page 2009

  7. [7]

    IBM App Connect . 2025. IBM Automation Explorer . https://explorer.automation.ibm.com/?type=template. Online; accessed 2025

    work page 2025

  8. [8]

    Zelong Li, Shuyuan Xu, Kai Mei, Wenyue Hua, Balaji Rama, Om Raheja, Hao Wang, He Zhu, and Yongfeng Zhang. 2024. https://api.semanticscholar.org/CorpusID:271270428 Autoflow: Automated workflow generation for large language model agents . ArXiv, abs/2407.12821

    arXiv 2024

Show all 17 references
  1. [9]

    Weiwen Liu, Xu Huang, Xingshan Zeng, Xinlong Hao, Shuai Yu, Dexun Li, Shuai Wang, Weinan Gan, Zhengying Liu, Yuanqing Yu, Zezhong Wang, Yuxian Wang, Wu Ning, Yutai Hou, Bin Wang, Chuhan Wu, Xinzhi Wang, Yong Liu, Yasheng Wang, and 8 others. 2024. https://api.semanticscholar.or...

    2024 arXiv

  2. [10]

    Marketwatch . 2022. Business process management growth . https://www.marketwatch.com/press-release/business-process- management-market-size\\-growth-with-top-leading-players-growth-\ - factors-global-trends-industry-share\\-and-forecast-2022-2031-2022-08-18. Online; accessed 2022

    2022

  3. [11]

    Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun

    Yujia Qin, Shi Liang, Yining Ye, Kunlun Zhu, Lan Yan, Ya-Ting Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Runchu Tian, Ruobing Xie, Jie Zhou, Marc H. Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2023. https://api.semanticscholar.org/CorpusID:260334759 T...

    2023 arXiv

  4. [12]

    Weixi Tong and Tianyi Zhang. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.1118 C ode J udge: Evaluating code generation with large language models . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 20032--20051, Miami, Flori...

    2024 doi

  5. [13]

    watsonx Orchestrate . 2025. watsonx Orchestrate: AI for business productivity . https://www.ibm.com/products/watsonx-orchestrate. Online; accessed 2025

    2025

  6. [14]

    Shuyuan Xu, Zelong Li, Kai Mei, and Yongfeng Zhang. 2024. Core: Llm as interpreter for natural language programming, pseudo-code programming, and flow programming of ai agents. arXiv e-prints, pages arXiv--2405

    2024

  7. [15]

    Patil, Ion Stoica, and Joseph E

    Fanjia Yan, Huanzhi Mao, Charlie Cheng-Jie Ji, Tianjun Zhang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. 2024. Berkeley function calling leaderboard

    2024

  8. [16]

    Yining Ye, Xin Cong, Shizuo Tian, Jian Cao, Hao Wang, Yujia Qin, Ya-Ting Lu, Heyang Yu, Huadong Wang, Yankai Lin, Zhiyuan Liu, and Maosong Sun. 2023. https://api.semanticscholar.org/CorpusID:265295561 Proagent: From robotic process automation to agentic process automation . Ar...

    2023 arXiv

  9. [17]

    Zapier Apps . 2025. Find ways for Zapier to handle repetitive tasks in the apps you use every day. https://zapier.com/explore. Online; accessed 2025

    2025

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.