Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

What is the Best Process Model Representation? A Comparative Analysis for Process Modeling with Large Language Models

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Pith's one-line read The paper presents the first head-to-head comparison of process model representations for LLM-based process modeling, concluding that Mermaid is the most suitable overall while BPMN text yields the most faithful generated models.

desk verdict Useful first head-to-head benchmark for process model representations in LLM pipelines, but the 'Mermaid is best' headline rests on author-scored grades; the PMG comparison is the more solid result. read the letter →

arxiv 2507.11356 v1 pith:PKKJUL5J submitted 2025-07-15 cs.CL

classification cs.CL
keywords processmodelrepresentationlargelanguagemodelsgenerationbusinessmodelingPModatasetBPMNMermaidPMEsimilarity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Process modeling with large language models has grown quickly, but each research group has used its own output notation, evaluation set, and metric, so no one knew whether one process model representation is better than another. This paper tries to answer that question directly: it builds the PMo Dataset, 55 gold-standard process descriptions each paired with a model in nine different representations, and compares all nine under one standardized prompting and scoring protocol. On the paper's own terms, the answer is split: Mermaid is the most suitable all-round representation for LLM-based process modeling, while BPMN text gives the most faithful process model generation when measured by element similarity. The authors argue that choosing a representation is itself a design decision that changes both how usable a model format is and how well an LLM can reproduce a process from text.

What carries the argument

The load-bearing mechanism is the PMo Dataset plus its conversion-and-evaluation pipeline. Fifty-five expert-validated BPMN models are automatically converted into nine process model representations (BPMN, BPMN process, Graphviz, Mermaid, PME, Simplified XML, POWL code, BPMN text, and JSON branches), where a PMR is a notation for expressing a process model, such as XML, JSON, Python code, or diagram syntax. Every PMG experiment then runs under one standardized prompt template with LLaMA-3.3-70b, and outputs are scored by element counts and by PME similarity, a two-step measure that semantically matches generated elements to ground-truth elements using sentence embeddings and then computes the Dice–Sørensen coefficient between the sets.

What would settle it

Re-run the PMG experiment on a dataset in which all 55 descriptions include swimlanes or message flows representable in every PMR, and check whether BPMN text still has the highest PME similarity; if its lead disappears on the full set, the reported superiority is an artifact of the evaluable subset.

Watch

Extended reading notes

Core claim

The central discovery is a comparative result rather than a new algorithm: across six suitability criteria (token compactness, expressiveness, human readability, visualizability, usability, and extensibility), Mermaid scores highest on average, driven by its combination of extreme compactness and direct visualization; yet when the same 55 descriptions are given to an open LLM with identical prompts, the branching formats BPMN text and JSON branches yield models with clearly higher PME similarity to the ground truth, with BPMN text best overall. The paper also uncovers a systematic under-generation tendency: generated models have roughly eight fewer nodes than the ground truth on average, exclusive gateways are cut by about half, and parallel gateways by about two thirds. Representations that natively encode branching structure partially counteract this drop, which the authors take as evidence that the representation itself shapes generation fidelity.

Load-bearing premise

The comparison assumes the automatic conversion of the ground-truth BPMN models into all nine representations is faithful, and that the subset of models convertible into the branching PMRs, about one third of the dataset, is representative enough for a fair PMG comparison.

Editorial extensions

If this is right

  • New or existing PMo tools should consider Mermaid as the default working representation for interactive modeling, because it combines the best compactness with direct visualization.
  • PMG systems should prefer branching representations such as BPMN text or JSON branches as generation targets, since these partially offset the LLM's tendency to omit gateways.
  • Process modeling pipelines can be staged by representation: use Mermaid for elicitation and communication, then switch to BPMN text for faithful generation from text.
  • Future PMG evaluations should report the target PMR explicitly, because the representation alone changes element counts and similarity scores by a large margin.
  • The PMo Dataset and the unified protocol provide a common benchmark on which future PMG approaches can be compared against each other.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's two-thirds conversion failure means BPMN text's PMG advantage is measured only on the roughly one-third of models that are representable in branching formats; extending the comparison to swimlanes and message flows could shrink or reverse the gap.
  • A direct testable extension would be to prompt the same LLM to generate both Mermaid and BPMN text for identical descriptions and convert both outputs to BPMN before scoring, isolating the representation effect from the dataset-subset effect.
  • The suitability grades for six criteria are author-assigned; using the released dataset to collect multi-expert ratings would turn the Mermaid result from a claim into an independently reproducible ranking.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces the PMo Dataset of 55 process descriptions paired with ground-truth models in nine process model representations (PMRs), and reports a comparative evaluation of these PMRs for LLM-based process modeling. Suitability is assessed along six author-defined criteria (token compactness, expressivity, human readability, visualization, usability, extensibility), and process model generation (PMG) performance is measured under a standardized prompting setup with LLaMA-3.3-70b using element counts and PME similarity. The paper concludes that Mermaid is the most suitable PMR for PMo with LLMs, while BPMN text yields the best PMG results in terms of process element similarity.

Significance. The paper targets a real gap: PMRs used in LLM-based process modeling have not been systematically compared, and existing PMG evaluations are fragmented. The PMo Dataset and the public code and prompts are useful community assets, and the use of an open-source LLM with a standardized prompting protocol is a methodological strength. The PMG comparison, based on computed PME similarity with semantic matching, is more objective than the suitability grading and produces a concrete, falsifiable ranking. However, the headline suitability conclusion rests on subjective author-assigned grades, and the PMG conclusion is supported by a single run on a restricted subset of the data, so the empirical evidence is currently thinner than the abstract implies.

major comments (3)
  1. [Section 4.1, Table 4; Section 5] The central claim that Mermaid is the most suitable PMR for PMo with LLMs rests on the six grade columns of Table 4, whose values are assigned by the authors. The paper itself concedes in Section 5 that this evaluation "remains primarily subjective." No inter-rater reliability, sensitivity analysis, or alternative weighting is reported. Several individual grades are contestable: for example, BPMN text receives 1/5 for visualization despite being convertible to BPMN, JSON branches receives a schema-support bonus while BPMN text, which is XML-based, does not, and Mermaid and POWL code both receive 5/5 for token compactness. Because the average scores are close (Mermaid 4.00, Graphviz 3.67, BPMN process 3.50), modest changes to one or two grades could change the ranking. I ask the authors to provide a robustness analysis, such as a sensitivity sweep over grades or a structured multi-expert grading protocol with agreement metrics, before the Mermaid conclusion can be considered supported.
  2. [Section 3.3, Table 6] The PMG result that BPMN text is best is computed on a restricted subset: Section 3.3 states that for branching PMRs, two thirds of the BPMN models cannot be converted. The PME similarity scores for BPMN text and JSON branches are therefore based on roughly one third of the dataset, and these models may be systematically simpler than the non-convertible ones. The paper does not report the PME scores of the other PMRs on the same subset, so the comparison may conflate PMR quality with subset difficulty. Additionally, Table 6 reports a single run per prompt with no variance estimates or significance tests; with temperature 0.2 and one run, the differences between BPMN text (0.54), JSON branches (0.53), and Mermaid (0.48) may not be stable. I recommend re-evaluating all PMRs on the intersection of convertible models and reporting results over multiple seeds with confidence intervals or a significance test.
  3. [Section 3.3] The ground-truth PMR models are produced by automatic conversion from BPMN, with validation by converting back to BPMN, but the paper does not report any quantitative or manual verification that the conversions preserve the process semantics relevant for evaluation. The paper also notes that unsupported information is ignored (e.g., conditions in POWL code) and that models are excluded when conversion would lose significant information. Since PME similarity in Section 3.5 is computed against these converted ground truths, errors or omissions introduced during conversion directly affect the reported scores. I ask the authors to present a conversion-quality check, such as a sample of manually inspected conversions or a comparison of element coverage before and after conversion, to support the assumption that the converted ground truths are faithful.
minor comments (5)
  1. [Section 3.4] There is a duplicated word in the sentence "a PMR should should also support direct graphical visualization."
  2. [Table 4] The column header "Vizualisable" is misspelled; it should be "Visualizable" or "Visualization capability" for consistency with the text.
  3. [Section 5] The sentence "BPMN text, in particular, achieves the strongest similarity with ground truth elements for both raw numbers and semantic content" conflates two different measurements: element counts (Table 5) and PME similarity (Table 6). Please phrase this as referring to both element-count differences and PME similarity scores separately.
  4. [Section 3.2, Table 2] The distinction between "BPMN" and "BPMN process" is clear in the text, but Table 2 would benefit from a note that the difference is the presence or absence of the BPMN diagram definition, as stated in Section 3.2.
  5. [Section 3.5] The semantic matching threshold of 0.7 is stated as "experimentally defined" without a reference or local justification. Since this threshold affects all PME similarity scores, a brief explanation or a sensitivity check would help the reader assess its impact.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the comparison is empirical and its scoring limitations are explicitly disclosed.

full rationale

This paper is an empirical comparative study rather than a derivation, so the main circularity patterns do not apply. The headline conclusions—Mermaid best suited for PMo and BPMN text best for PMG—are supported by measured length statistics, element-coverage computations, and PME similarity scores computed against ground-truth models originating from independent datasets (Mangler, PMo Benchmark, PET-7, BPMN for research, CCC19). The suitability ranking in Table 4 relies on grades assigned by the authors, but the paper explicitly acknowledges this subjectivity in Section 5 as a limitation, which makes it a disclosed weakness rather than a hidden circular step. No load-bearing self-citations are present: the reference list contains no works by the present authors (Brissard, Cuppens, Zouaq). The PME semantic-matching threshold of 0.7 is inherited from prior work [19] and is not re-fitted to this dataset, so it is not a fitted input presented as a prediction. The restricted coverage for branching PMRs (two thirds of BPMN models cannot be converted) is a genuine external-validity concern but is openly reported and does not constitute circular reasoning. Consequently, the paper does not reduce to its own inputs by construction.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

No new physical or formal entities are introduced; the PMo Dataset is a new benchmark artifact rather than an invented entity in the sense of a new postulated object. The main load-bearing commitments are the chosen evaluation criteria, the representative set of PMRs, the faithfulness of automatic conversions, and the representativeness of a single LLM.

free parameters (1)
  • Semantic matching threshold = 0.7
    Threshold for considering two process elements semantically equivalent in PME similarity; described as experimentally defined in Section 3.5, no tuning procedure or validation reported.
assumptions (4)
  • domain assumption The six requirements (token compactness, expressivity, human readability, visualization, usability, extensibility) are the relevant criteria for LLM-based PMo.
    Section 3.4 defines these requirements without external validation; they underpin Table 4 and the Mermaid conclusion.
  • domain assumption The nine selected PMRs are representative of PMRs used in current PMG approaches.
    Section 3.2 selects PMRs based on usage in existing approaches and diversity, but representativeness is not proven.
  • ad hoc to paper Automatic BPMN-to-PMR conversions preserve process semantics despite information loss.
    Section 3.3: converters validated by round-trip transformation, but unsupported information is ignored and branching PMRs cannot represent two thirds of models.
  • domain assumption LLaMA-3.3-70b with standard prompting is representative for evaluating PMG performance across PMRs.
    Section 3.5 uses a single open-source LLM and one prompt template; no evidence that rankings generalize across models or prompts.

how reviews work

0 comments
Cite this review

Pith. "Pith review of What is the Best Process Model Representation? A Comparative Analysis for Process Modeling with Large Language Models." pith.science (2026). https://pith.science/paper/PKKJUL5J

@misc{pith2026250711356,
  author       = {Pith},
  title        = {Pith review of: What is the Best Process Model Representation? A Comparative Analysis for Process Modeling with Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PKKJUL5J}},
  note         = {Machine review of arXiv:2507.11356}
}
read the original abstract

Large Language Models (LLMs) are increasingly applied for Process Modeling (PMo) tasks such as Process Model Generation (PMG). To support these tasks, researchers have introduced a variety of Process Model Representations (PMRs) that serve as model abstractions or generation targets. However, these PMRs differ widely in structure, complexity, and usability, and have never been systematically compared. Moreover, recent PMG approaches rely on distinct evaluation strategies and generation techniques, making comparison difficult. This paper presents the first empirical study that evaluates multiple PMRs in the context of PMo with LLMs. We introduce the PMo Dataset, a new dataset containing 55 process descriptions paired with models in nine different PMRs. We evaluate PMRs along two dimensions: suitability for LLM-based PMo and performance on PMG. \textit{Mermaid} achieves the highest overall score across six PMo criteria, whereas \textit{BPMN text} delivers the best PMG results in terms of process element similarity.

Figures

Figures reproduced from arXiv: 2507.11356 by the authors.

Figure 1
Figure 1. Four Process Model Representations (PMRs) of the same process model. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Assessing the Business Process Modeling Competences of Large Language Models

    cs.SE 2026-01 conditional novelty 6.0 of 10

    Open-source LLMs can produce BPMN process models that rival human experts on syntax and readability, but they lag on semantic accuracy and frequently generate invalid BPMN-XML.

Reference graph

Works this paper leans on

19 extracted references · 11 canonical work pages · cited by 1 Pith paper

  1. [1]

    In: Delgado, A., Slaats, T

    Apaydin, K., Zisgen, Y.: Local Large Language Models for Business Process Modeling. In: Delgado, A., Slaats, T. (eds.) Process Mining Workshops. pp. 605–609. Springer Nature Switzerland, Cham (2025). https://doi.org/10.1007/978-3-031-82225-4_44

  2. [2]

    PET: An Annotated Dataset for Process Extraction from Natural Language Text

    Bellan, P., van der Aa, H., Dragoni, M., Ghidini, C., Ponzetto, S.P.: PET: An Annotated Dataset for Process Extraction from Natural Language Text (Jun 2022). https://doi.org/10. 48550/arXiv.2203.04860 12 A. Brissard et al

  3. [3]

    Camunda: BPMN for Research Dataset (Dec 2015)

  4. [4]

    Evaluating the Process Modeling Abilities of Large Language Models -- Preliminary Foundations and Results

    Fettke, P., Houy, C.: Evaluating the Process Modeling Abilities of Large Language Models – Preliminary Foundations and Results (Mar 2025). https://doi.org/10.48550/arXiv.2503.13520

  5. [5]

    In: Modellierung 2024 Satellite Events

    Forell, M., Schüler, S.: Modeling meets Large Language Models. In: Modellierung 2024 Satellite Events. p. 10.18420/modellierung2024. Gesellschaft für Informatik e.V. (2024)

  6. [6]

    https://doi.org/10.48550/arXiv.2307.09923

    Grohs,M.,Abb,L.,Elsayed,N.,Rehse,J.R.:LargeLanguageModelscanaccomplishBusiness Process Management Tasks (Jul 2023). https://doi.org/10.48550/arXiv.2307.09923

  7. [7]

    Software and Systems Modeling21(5), 1877–1906 (Oct 2022)

    Ivanchikj, A., Serbout, S., Pautasso, C.: Live process modeling with the BPMN Sketch Miner. Software and Systems Modeling21(5), 1877–1906 (Oct 2022). https://doi.org/10. 1007/s10270-022-01009-w

  8. [8]

    In: Daniel, F., Zugal, S

    Ivanov, S., Kalenkova, A., Aalst, van der, W.: BPMNDiffViz : A tool for BPMN models comparison. In: Daniel, F., Zugal, S. (eds.) Proceedings of the Demo Session of the 13th International Conference on Business Process Management (BPM 2015, Innsbruck, Austria, August 31-September 3, 2015). pp. 35–39. CEUR Workshop Proceedings (2015)

Show all 19 references
  1. [9]

    https://doi.org/10.48550/arXiv.2304.11065

    Klievtsova,N.,Benzin,J.V.,Kampik,T.,Mangler,J.,Rinderle-Ma,S.:ConversationalProcess Modeling:CanGenerativeAIEmpowerDomainExpertsinCreatingandRedesigningProcess Models? (Jan 2024). https://doi.org/10.48550/arXiv.2304.11065

  2. [10]

    In: BPM2024 (Sep 2024)

    Kopke, J., Safan, A.: Efficient LLM-Based Conversational Process Modeling. In: BPM2024 (Sep 2024)

  3. [11]

    https://doi.org/10.48550/arXiv.2408.08892

    Kourani, H., Berti, A., Hennrich, J., Kratsch, W., Weidlich, R., Li, C.Y., Arslan, A., Schuster, D., van der Aalst, W.M.P.: Leveraging Large Language Models for Enhanced Process Model Comprehension (Sep 2024). https://doi.org/10.48550/arXiv.2408.08892

  4. [12]

    https://doi.org/10.48550/arXiv.2412.00023

    Kourani,H.,Berti,A.,Schuster,D.,vanderAalst,W.M.P.:EvaluatingLargeLanguageModels on Business Process Modeling: Framework, Benchmark, and Self-Improvement Analysis (Nov 2024). https://doi.org/10.48550/arXiv.2412.00023

  5. [13]

    https://doi.org/10.48550/arXiv.2403.07541

    Kourani, H., Berti, A., Schuster, D., van der Aalst, W.M.P.: Process Modeling With Large Language Models (Apr 2024). https://doi.org/10.48550/arXiv.2403.07541

  6. [14]

    https://doi.org/10.48550/arXiv.2403.04327

    Kourani, H., Berti, A., Schuster, D., van der Aalst, W.M.P.: ProMoAI: Process Modeling with Generative AI (Mar 2024). https://doi.org/10.48550/arXiv.2403.04327

  7. [15]

    In: 2023 International Joint Conference on Neural Networks (IJCNN)

    Li,X.,Ni,L.,Li,R.,Liu,J.,Zhang,M.:MaD:ADatasetforInterview-basedBPMinBusiness Process Management. In: 2023 International Joint Conference on Neural Networks (IJCNN). pp. 1–8 (Jun 2023). https://doi.org/10.1109/IJCNN54540.2023.10191898

  8. [16]

    https://doi.org/10.48550/arXiv.2408.01916

    Lin,L.,Jin,Y.,Zhou,Y.,Chen,W.,Qian,C.:MAO:AFrameworkforProcessModelGeneration with Multi-Agent Orchestration (Aug 2024). https://doi.org/10.48550/arXiv.2408.01916

  9. [17]

    https://doi.org/10.5281/zenodo.7783492

    Mangler, J., Klievtsova, N.: Textual Process Descriptions and Corresponding BPMN Models (Mar 2023). https://doi.org/10.5281/zenodo.7783492

  10. [18]

    https://doi.org/10.4121/UUID: C923AF09-CE93-44C3-ACE0-C5508CF103AD

    Munoz-Gama, J., de la Fuente, R.R., Sepúlveda, M.M., Fuentes, R.R.: Confor- mance Checking Challenge 2019 (CCC19) (Feb 2019). https://doi.org/10.4121/UUID: C923AF09-CE93-44C3-ACE0-C5508CF103AD

  11. [19]

    https://doi.org/10

    Voelter, M., Hadian, R., Kampik, T., Breitmayer, M., Reichert, M.: Leveraging Generative AI for Extracting Process Models from Multimodal Documents (Jun 2024). https://doi.org/10. 48550/arXiv.2406.04959

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.