Pith. sign in

REVIEW 3 major objections 6 minor 3 cited by

Evaluating Large Language Models on Business Process Modeling: Framework, Benchmark, and Self-Improvement Analysis

T0 review · 3 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Claude-3.5-Sonnet achieves the highest average quality score, 0.93, on a 20-process LLM business process modeling benchmark.

desk verdict Useful benchmark and careful empirical work, but the loop cap in the simulated logs could flip the headline ranking; fixable, and worth refereeing. read the letter →

arxiv 2412.00023 v1 pith:T2IF3MTT submitted 2024-11-17 cs.DB

Humam Kourani , Alessandro Berti , Daniel Schuster , Wil M.P. van der Aalst This is my paper · ORCID
classification cs.DB
keywords businessprocessmodelinglargelanguagemodelsPOWLconformancecheckingbenchmarkself-improvementminingLLMevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to show that large language models can turn plain-English business process descriptions into formally sound process models, and that this capability can be measured, compared, and improved. The authors extend their earlier LLM-based modeling framework, in which a model writes Python code that builds a process model in the Partially Ordered Workflow Language (POWL), and they benchmark 16 LLMs on 20 hand-written process descriptions with ground-truth models. Quality is scored automatically by conformance checking: each generated model is compared against event logs simulated from the ground truth, and the harmonic mean of fitness and precision becomes the quality score. The paper reports that Claude-3.5-Sonnet scores highest at 0.93, close to the ground-truth average of 0.98, that fewer error-correction iterations correlate with higher quality, and that prompting an LLM to critically revise its own output helps most for models that start out weaker. The self-evaluation experiments, the paper notes, use only two LLMs and are not claimed to generalize.

What carries the argument

The load-bearing object is POWL, the Partially Ordered Workflow Language, an intermediate process representation whose hierarchical structure guarantees soundness by construction, so any model the LLM builds is a valid starting point for BPMN or Petri net export. Around it, the framework wraps prompt engineering (role prompting, knowledge injection, few-shot examples, negative prompting), a restricted set of Python ModelGenerator functions the LLM is allowed to call, and an iterative error-handling loop that distinguishes critical errors from adjustable ones. The evaluation machinery is conformance checking: event logs are simulated from each ground-truth POWL model, the generated model is replayed against those logs, and the harmonic mean of fitness and precision gives a single quality score in [0,1].

What would settle it

Recompute the benchmark's quality scores with event logs generated under unequal decision probabilities (for example, 90/10 at each choice) or loops allowed to run more than twice; if the model ranking and the error-handling/quality correlation change materially, the reported conclusions are artifacts of the simulation assumptions rather than stable properties of the LLMs.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central claim is that within this automated process-modeling framework, LLMs can produce high-quality executable process models from natural-language descriptions, and the differences between models are systematic rather than random. Claude-3.5-Sonnet achieves the highest average quality score (0.93), followed by O1-Preview (0.92) and O1-Mini (0.91), with the ground-truth models at 0.98. The paper also claims there is a notable correlation between error-handling performance and model quality: models that generate valid code in fewer iterations tend to score higher, and it finds stable average quality within model families, so speed-optimized variants do not necessarily sacrifice quality. For self-improvement, the paper claims that output self-optimization, guided by a deliberately restrictive prompt, improves average quality, with the largest gains for lower-performing models (GPT-4o, Gemini-1.5-Flash-002), while self-evaluation and input optimization are unreliable in this setting.

Load-bearing premise

The load-bearing premise is that the 20 hand-written process descriptions and their ground-truth POWL models, together with event logs simulated under equal decision probabilities and two-iteration loop limits, are representative enough of real business process modeling to support general conclusions about which LLMs are best.

Editorial extensions

If this is right

  • If the benchmark is representative, organizations automating process modeling should choose models that generate correct code on the first attempt and repair errors quickly, because that behavior correlates with final model quality.
  • The correlation between iteration count and quality means improving code-generation and self-repair ability is a direct lever for improving process-model quality.
  • Output self-optimization with a restrictive 'only change if genuinely beneficial' prompt is a low-cost way to lift weak models, at the price of occasional small quality declines.
  • Models in the same family cluster in quality, so family-level comparisons are more reliable than single-run comparisons when choosing among LLMs.
  • Conformance checking against simulated logs gives a fully automatable scoring pipeline that can be rerun as new LLMs appear.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the simulated logs assume equal decision probabilities and at most two loop iterations, the scores reward models that reproduce every structural variant equally; a deployment with skewed branch frequencies or longer loops could shift both fitness and precision.
  • The 20-process benchmark is small and hand-written; an obvious testable extension is a larger, real-world corpus with logs taken from actual executions, which would tell whether the ranking holds outside hand-crafted descriptions.
  • The self-evaluation result that a weaker model selects the wrong candidate in 16-17 of 20 cases hints that weak models cannot reliably judge their own work, so self-selection should probably be reserved for stronger models.
  • A natural next experiment is to combine output self-optimization with the adversarial error-handling loop and measure whether the gains persist when the model is also under pressure to fix code.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper extends the authors' earlier LLM-based process modeling framework by adding a benchmark and self-improvement analysis. The benchmark consists of 20 business process descriptions paired with ground-truth POWL models and simulated event logs; the authors evaluate 16 LLMs by measuring how well each LLM-generated POWL model conforms to the simulated logs, using the harmonic mean of fitness and precision as a quality score. The main findings are that Claude-3.5-Sonnet achieves the highest average quality score (0.93), that error-handling performance is positively correlated with model quality, and that output self-optimization improves quality primarily for models with initially lower performance. The paper also introduces the ProMoAI tool and makes the evaluation data and results publicly available.

Significance. If the ranking and self-improvement findings hold, the paper provides a useful benchmark and methodology for evaluating LLMs on a structured modeling task, with a credible external ground truth in the form of simulated logs and objective conformance metrics. The use of a sound intermediate representation (POWL), the breadth of included LLMs, and the public availability of data and code are clear strengths. However, the loop-truncation bias in the simulated logs, the absence of uncertainty/statistical evidence for the headline comparisons, and the limited support for the output-optimization claim currently temper the conclusions. The benchmark size (20 processes, mostly hand-written by the authors) also limits external validity, but this is a limitation rather than a fatal flaw.

major comments (3)
  1. [5.1.3] The simulation caps loop iterations at two, and footnote 1 concedes that a perfect score is unattainable 'due to the presence of loops,' with the ground truth itself scoring only 0.98. This means the evaluation target is the finite 1- and 2-iteration truncation of each process, not the ground-truth process itself. A generated POWL model that represents a loop as a bounded structure (e.g., one do execution followed by an xor between repeating once more and exiting) covers every trace in the simulated log and adds no extra behavior, so it can receive fitness 1.0 and precision 1.0, strictly beating the correct unbounded-loop model. Since the reported top-model differences in Table 2 (e.g., Claude-3.5-Sonnet 0.93 versus O1-Mini 0.91) are at the 0.01-0.02 scale, the headline ranking may reflect fit to the truncated log rather than true process-modeling quality. Please quantify the number of loop-bearing processes, report per-process scores, rerun the evaluation with logs allowing more loop iterations or with a behavioral equivalence/distance measure that is invariant to loop truncation, and show how the Table 2 ordering changes.
  2. [5.2.2] The claim of 'a notable correlation between error-handling performance and model quality' is not supported by any correlation coefficient, significance test, or scatterplot; the text moves directly from Tables 1 and 2, which report only aggregated averages over 20 processes, to a causal-sounding conclusion. Please compute a per-process correlation (e.g., Spearman) between iterations/errors and quality scores, report its value and confidence interval, and limit the claim accordingly. Similarly, the statement that 'consistent quality trends within similar model groups' are observed needs variance information: Table 2 reports average scores without any per-process spread, and the self-evaluation section shows that individual runs vary (e.g., 0.86-0.88 for Gemini-1.5-Pro-002), so the averages alone do not establish within-group consistency.
  3. [6.3] Table 6 is the only quantitative evidence for the paper's conclusion that output optimization is 'promising,' but it reports only averages, maximum improvement, and maximum decline for n=20, with no confidence interval or paired test. GPT-4o's maximum improvement of +0.84 strongly suggests that a single large outlier drives its average gain of +0.05, and the text does not state how many of the 20 cases improved, worsened, or stayed the same. Please report the full per-case distribution of quality differences, include a paired permutation test or bootstrap confidence interval, and clarify whether the 'without error refinement loop' setting makes these results comparable to the Section 5 experiments.
minor comments (6)
  1. [4] There is a typo in the sentence 'Currently ProMoAI in integrated with three LLM providers'; it should read 'is integrated.'
  2. [5.1.3] The sentence claiming that 'each simulated event log includes one instance of every unique trace variant possible' should be reconciled with the fact that the logs are generated probabilistically; if only one instance per variant is retained, the equal-probability assumption is unnecessary and should be stated as a trace-coverage choice rather than a sampling distribution.
  3. [6.1] The 'Subset Match' column in Table 4 needs a precise definition; it is unclear whether a model selected by the LLM counts as a subset match when it is tied within the 0.02 buffer, and how ties are broken.
  4. [5.2.2] Table 2 would be much more informative if it reported per-process quality scores or at least standard deviations/confidence intervals, especially because the paper's own Section 6.1 demonstrates nontrivial run-to-run variability for at least one model.
  5. [5.1.2] The benchmark's 20 processes, 18 of which were created by the authors, are a useful starting point but should be discussed more explicitly as a limitation regarding generalizability to real-world business process modeling tasks.
  6. [5] The GitHub link is given as a plain URL; please provide a versioned archive link or DOI so that the exact benchmark artifacts used in this paper remain accessible.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: quality scores are measured against externally simulated ground-truth logs, and no parameter is fitted to the claimed ranking or self-improvement effects.

full rationale

The benchmark derives each LLM quality score from conformance checking (fitness and precision, Section 5.1.3) of freshly generated POWL code against event logs simulated from ground-truth POWL models; the logs are generated once from the ground truth and do not depend on the LLM outputs, so the evaluation target is not an input to the model. The ground-truth score of 0.98 is reported separately (Table 2), and footnote 1 explains why 1.0 is not attainable with loops. No fitted parameter is renamed as a prediction, and no quantity in the ranking or in the self-improvement comparisons is defined in terms of the LLM outputs being judged. The framework and benchmark reuse the authors' own POWL representation [9] and prior framework [8], but these citations support tooling choices (soundness guarantees and code-generation scaffolding) rather than being invoked as a uniqueness theorem or as the source of the empirical ranking; the central claims are checkable against released logs and PM4Py conformance metrics. The loop cap in Section 5.1.3 is a benchmark-validity concern (it may reward bounded under-approximations), not a circularity, because the quality score is openly defined relative to the simulated log and no hidden equivalence is imported. There is therefore no circular step to quote.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

No numerical fitting occurs in this paper. The hand-chosen settings are experimental thresholds and simulation assumptions, while POWL and the framework are adopted from the authors' prior work. The ground-truth logs and conformance metrics are external to the LLM outputs, so the ledger is dominated by domain assumptions about the benchmark rather than invented entities.

free parameters (5)
  • Error-handling iteration thresholds = 10 for adjustable errors, plus 5 additional iterations
    Chosen by hand in Section 5.1.4. These thresholds determine which models are marked as failures and affect the reported error-handling metrics.
  • Buffer for best-model matching = 0.02
    Used in Section 6.1.2 to decide when two quality scores count as equal. Changing this buffer changes the subset and exact match counts.
  • Maximum loop iterations in simulated logs = 2
    Section 5.1.3 limits loops to two iterations to keep logs finite. This shapes the ground-truth behavior and therefore all fitness and precision scores.
  • Equal decision probabilities in simulation = assumption: p equal for each branch
    Section 5.1.3 assumes all decision points are uniformly distributed. Real processes with skewed branches would produce different logs and different conformance results.
  • Number of candidate runs for self-evaluation = 4
    Section 6.1.1 generates four candidates per process. More or fewer runs would change the measured benefit of self-evaluation.
assumptions (4)
  • domain assumption Conformance checking metrics (fitness and precision) computed with PM4Py accurately reflect the quality of generated process models.
    Invoked in Section 5.1.3 to score every generated model. If these metrics mis-rank models, all comparative claims are affected.
  • domain assumption The 20 hand-authored ground-truth models are correct and representative of real business processes.
    Section 5.1.2 describes the processes as designed by the authors. No external validation, inter-rater agreement, or sourcing from a standard corpus is reported.
  • ad hoc to paper Simulated event logs with equal decision probabilities and a loop cap of two iterations cover the behavior needed for fair conformance checking.
    Section 5.1.3 states these assumptions. They constrain the ground truth and therefore every quality score in the paper.
  • domain assumption Providing the ground-truth activity labels in the prompt does not unduly inflate quality scores.
    Section 5.1.3 standardizes labels to enable conformance checking. Supplying the exact activity labels removes a realistic modeling burden and may make scores overly optimistic.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Evaluating Large Language Models on Business Process Modeling: Framework, Benchmark, and Self-Improvement Analysis." pith.science (2026). https://pith.science/paper/T2IF3MTT

@misc{pith2026241200023,
  author       = {Pith},
  title        = {Pith review of: Evaluating Large Language Models on Business Process Modeling: Framework, Benchmark, and Self-Improvement Analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/T2IF3MTT}},
  note         = {Machine review of arXiv:2412.00023}
}
read the original abstract

Large Language Models (LLMs) are rapidly transforming various fields, and their potential in Business Process Management (BPM) is substantial. This paper assesses the capabilities of LLMs on business process modeling using a framework for automating this task, a comprehensive benchmark, and an analysis of LLM self-improvement strategies. We present a comprehensive evaluation of 16 state-of-the-art LLMs from major AI vendors using a custom-designed benchmark of 20 diverse business processes. Our analysis highlights significant performance variations across LLMs and reveals a positive correlation between efficient error handling and the quality of generated models. It also shows consistent performance trends within similar LLM groups. Furthermore, we investigate LLM self-improvement techniques, encompassing self-evaluation, input optimization, and output optimization. Our findings indicate that output optimization, in particular, offers promising potential for enhancing quality, especially in models with initially lower performance. Our contributions provide insights for leveraging LLMs in BPM, paving the way for more advanced and automated process modeling techniques.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Beyond Generalist LLMs: Specialist Agentic Systems for Structured Code Workflow Execution

    cs.SE 2026-07 conditional novelty 6.0 of 10

    A specialist BPMN-to-agent pipeline beats general-purpose coding agents on tool-use accuracy, latency, and token cost for deterministic business workflows.

  2. Assessing the Business Process Modeling Competences of Large Language Models

    cs.SE 2026-01 conditional novelty 6.0 of 10

    Open-source LLMs can produce BPMN process models that rival human experts on syntax and readability, but they lag on semantic accuracy and frequently generate invalid BPMN-XML.

  3. What is the Best Process Model Representation? A Comparative Analysis for Process Modeling with Large Language Models

    cs.CL 2025-07 conditional novelty 6.0 of 10

    A new dataset and head-to-head comparison of nine process model representations with LLMs finds Mermaid best for general use and BPMN text best for generation.

Reference graph

Works this paper leans on

44 extracted references · 26 canonical work pages · cited by 3 Pith papers

  1. [1]

    In: Rosing, M., Scheel, H., Scheer, A

    Rosing, M., White, S., Cummins, F., Man, H.: Business process model and nota- tion - BPMN. In: Rosing, M., Scheel, H., Scheer, A. (eds.) The Complete Business Process Handbook: Body of Knowledge from Process Modeling to BPM, Vol- ume I, pp. 429–453. Morgan Kaufmann/Elsevier, Massachusets, USA (2015). https://doi.org/10.1016/B978-0-12-799959-3.00021-5

    doi:10.1016/b978-0-12-799959-3.00021-5 2015

  2. [2]

    van Hee, K.M., Sidorova, N., van der Werf, J.M.E.M.: Business process modeling using Petri nets. Trans. Petri Nets Other Model. Concurr. 7, 116–161 (2013) https://doi.org/10.1007/978-3-642-38143-0 4

    doi:10.1007/978-3-642-38143-0 2013

  3. [3]

    CoRR abs/2303.08774 (2023) https://doi

    OpenAI: GPT-4 technical report. CoRR abs/2303.08774 (2023) https://doi. org/10.48550/ARXIV.2303.08774 2303.08774

    arXiv doi:10.48550/arxiv.2303.08774 2023

  4. [4]

    CoRR abs/2312.11805 (2023) https://doi.org/10.48550/ARXIV.2312.11805 2312.11805

    Rohan Anil et al.: Gemini: A family of highly capable multimodal models. CoRR abs/2312.11805 (2023) https://doi.org/10.48550/ARXIV.2312.11805 2312.11805

    arXiv doi:10.48550/arxiv.2312.11805 2023

  5. [5]

    In: Zhou, Z.-H

    Li, J., Tang, T., Zhao, W.X., Wen, J.-R.: Pretrained language model for text gen- eration: A survey. In: Zhou, Z.-H. (ed.) Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pp. 4492–4499. International Joint Conferences on Artificial Intelligence Organization, California, USA (2021). https://doi.org/10.24963...

    work page doi:10.24963/ijcai.2021/612 2021

  6. [6]

    In: IEEE High Performance Extreme Computing Conference, 29 HPEC 2023, Boston, MA, USA, September 25-29, 2023, pp

    Vidan, A., Fiedler, L.H.: A composable just-in-time programming framework with LLMs and FBP. In: IEEE High Performance Extreme Computing Conference, 29 HPEC 2023, Boston, MA, USA, September 25-29, 2023, pp. 1–8. IEEE, New York, USA (2023). https://doi.org/10.1109/HPEC58863.2023.10363587

    arXiv 2023

  7. [7]

    In: The Eleventh Interna- tional Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023

    Zhou, Y., Muresanu, A.I., Han, Z., Paster, K., Pitis, S., Chan, H., Ba, J.: Large language models are human-level prompt engineers. In: The Eleventh Interna- tional Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, Online (2023)

    work page 2023

  8. [8]

    In: van der Aa, H., Bork, D., Schmidt, R., Sturm, A

    Kourani, H., Berti, A., Schuster, D., van der Aalst, W.M.P.: Process model- ing with large language models. In: van der Aa, H., Bork, D., Schmidt, R., Sturm, A. (eds.) Enterprise, Business-Process and Information Systems Mod- eling - 25th International Conference, BPMDS 2024, and 29th International Conference, EMMSAD 2024, Limassol, Cyprus, June 3-4, 2024...

    doi:10.1007/978 2024

Show all 44 references
  1. [9]

    In: Francescomarino, C.D., Burattin, A., Janiesch, C., Sadiq, S

    Kourani, H., van Zelst, S.J.: POWL: partially ordered workflow language. In: Francescomarino, C.D., Burattin, A., Janiesch, C., Sadiq, S. (eds.) Business Process Management - 21st International Conference, BPM 2023, Utrecht, The Netherlands, September 11-15, 2023, Proceedings....

    2023

  2. [10]

    In: Betz, S

    Dunzer, S., Stierle, M., Matzner, M., Baier, S.: Conformance checking: a state-of- the-art literature review. In: Betz, S. (ed.) Proceedings of the 11th International Conference on Subject-Oriented Business Process Management, S-BPM ONE 2019, Seville, Spain, June 26-28, 2019, ...

    2019

  3. [11]

    In: Vizzari, G., Palmonari, M., Orlandini, A

    Bellan, P., Dragoni, M., Ghidini, C.: A qualitative analysis of the state of the art in process extraction from text. In: Vizzari, G., Palmonari, M., Orlandini, A. (eds.) Proceedings of the AIxIA 2020 Discussion Papers Workshop Co-located with the the 19th International Confer...

    2020

  4. [12]

    A. R. Gon¸ calves, J.C., Santoro, F.M., Bai˜ ao, F.A.: Let me tell you a story - on how to build process models. J. Univers. Comput. Sci. 17(2), 276–295 (2011) https://doi.org/10.3217/JUCS-017-02-0276

    2011 doi

  5. [13]

    In: Mouratidis, H., Rolland, C

    Friedrich, F., Mendling, J., Puhlmann, F.: Process model generation from natu- ral language text. In: Mouratidis, H., Rolland, C. (eds.) Advanced Information Systems Engineering - 23rd International Conference, CAiSE 2011, London, UK, June 20-24, 2011. Proceedings. Lecture Not...

    2011 doi

  6. [14]

    Sholiq, S., Sarno, R., Astuti, E.S.: Generating BPMN diagram from textual requirements. J. King Saud Univ. Comput. Inf. Sci. 34(10 Part B), 10079–10093 (2022) https://doi.org/10.1016/J.JKSUCI.2022.10.007

    2022 doi

  7. [15]

    In: Syriani, E., Sahraoui, H.A., Lara, J., Abrah˜ ao, S

    Ivanchikj, A., Serbout, S., Pautasso, C.: From text to visual BPMN process mod- els: design and evaluation. In: Syriani, E., Sahraoui, H.A., Lara, J., Abrah˜ ao, S. (eds.) MoDELS ’20: ACM/IEEE 23rd International Conference on Model Driven Engineering Languages and Systems, Vir...

    2020

  8. [16]

    Springer, Germany (2011)

    van der Aalst, W.M.P.: Process Mining - Discovery, Conformance and Enhance- ment of Business Processes. Springer, Germany (2011). https://doi.org/10.1007/ 978-3-642-19345-3

    2011

  9. [17]

    In: van der Aa, H., Bork, D., Proper, H.A., Schmidt, R

    Busch, K., Rochlitzer, A., Sola, D., Leopold, H.: Just tell me: Prompt engi- neering in business process management. In: van der Aa, H., Bork, D., Proper, H.A., Schmidt, R. (eds.) Enterprise, Business-Process and Information Systems Modeling - 24th International Conference, BP...

    2023

  10. [19]

    CoRR abs/2408.17316 (2024) https://doi.org/10.48550/ARXIV.2408.17316 2408.17316

    Norouzifar, A., Kourani, H., Dees, M., van der Aalst, W.M.P.: Bridg- ing domain knowledge and process discovery using large language models. CoRR abs/2408.17316 (2024) https://doi.org/10.48550/ARXIV.2408.17316 2408.17316

    2024 doi arXiv

  11. [20]

    CoRR abs/2408.08892 (2024) https: //doi.org/10.48550/ARXIV.2408.08892 2408.08892

    Kourani, H., Berti, A., Hennrich, J., Kratsch, W., Weidlich, R., Li, C., Arslan, A., Schuster, D., van der Aalst, W.M.P.: Leveraging large language models for enhanced process model comprehension. CoRR abs/2408.08892 (2024) https: //doi.org/10.48550/ARXIV.2408.08892 2408.08892

    2024 doi arXiv

  12. [21]

    Chen, S., Liao, H.: Bert-log: Anomaly detection for system logs based on pre- trained language model. Appl. Artif. Intell.36(1) (2022) https://doi.org/10.1080/ 08839514.2022.2145642

    2022

  13. [22]

    In: Burstein, J., Doran, C., Solorio, T

    Devlin, J., Chang, M., Lee, K., Toutanova, K.: BERT: pre-training of deep bidi- rectional transformers for language understanding. In: Burstein, J., Doran, C., Solorio, T. (eds.) Proceedings of the 2019 Conference of the North American 31 Chapter of the Association for Computa...

    2019

  14. [23]

    Gesellschaft f¨ ur Informatik e.V

    Muff, F., Fill, H.-G.: Limitations of ChatGPT in Conceptual Modeling: Insights from Experiments in Metamodeling. Gesellschaft f¨ ur Informatik e.V. (2024). https://doi.org/https://dl.gi.de/handle/20.500.12116/43782

    2024

  15. [24]

    In: Francescomarino, C.D., Burattin, A., Janiesch, C., Sadiq, S.W

    Klievtsova, N., Benzin, J., Kampik, T., Mangler, J., Rinderle-Ma, S.: Conver- sational process modelling: State of the art, applications, and implications in practice. In: Francescomarino, C.D., Burattin, A., Janiesch, C., Sadiq, S.W. (eds.) Business Process Management Forum -...

    2023 doi

  16. [25]

    CoRR abs/2407.17478 (2024) https://doi.org/10.48550/ARXIV.2407.17478 2407.17478

    Ziche, C., Apruzzese, G.: LLM4PM: A case study on using large language models for process modeling in enterprise organizations. CoRR abs/2407.17478 (2024) https://doi.org/10.48550/ARXIV.2407.17478 2407.17478

    2024 doi arXiv

  17. [26]

    In: Fahland, D., Jim´ enez-Ram ´ ırez, A., Kumar, A., Mendling, J., Pentland, B.T., Rinderle-Ma, S., Slaats, T., Versendaal, J., Weber, B., Weske, M., Winter, K

    Fontenla-Seco, Y., Winkler, S., Gianola, A., Montali, M., Pen ´ ın, M.L., Diz, A.J.B.: The droid you’re looking for: C-4pm, a conversational agent for declarative pro- cess mining. In: Fahland, D., Jim´ enez-Ram ´ ırez, A., Kumar, A., Mendling, J., Pentland, B.T., Rinderle-Ma,...

    2023

  18. [27]

    In: Weerdt, J.D., Pufahl, L

    Grohs, M., Abb, L., Elsayed, N., Rehse, J.: Large language models can accomplish business process management tasks. In: Weerdt, J.D., Pufahl, L. (eds.) Business Process Management Workshops - BPM 2023 International Workshops, Utrecht, The Netherlands, September 11-15, 2023, Re...

    2023

  19. [28]

    Fill, H., Fettke, P., K¨ opke, J.: Conceptual modeling and large language mod- els: Impressions from first experiments with ChatGPT. Enterp. Model. Inf. Syst. Archit. Int. J. Concept. Model. 18, 3 (2023) https://doi.org/10.18417/EMISA. 18.3

    2023 doi

  20. [29]

    CoRR abs/2407.13244 (2024) https://doi.org/10.48550/ARXIV.2407.13244 2407.13244 32

    Berti, A., Kourani, H., van der Aalst, W.M.P.: PM-LLM-Benchmark: Evaluating large language models on process mining tasks. CoRR abs/2407.13244 (2024) https://doi.org/10.48550/ARXIV.2407.13244 2407.13244 32

    2024 doi arXiv

  21. [30]

    Fournier, F., Limonad, L., Skarbovsky, I.: Towards a Benchmark for Causal Business Process Reasoning with LLMs (2024)

    2024

  22. [31]

    Fahland, D., Fournier, F., Limonad, L., Skarbovsky, I., Swevels, A.J.E.: How well can large language models explain business processes? (2024)

    2024

  23. [32]

    In: 6th International Conference on Process Mining, ICPM 2024, Kgs

    Rebmann, A., Schmidt, F.D., Glavas, G., van der Aa, H.: Evaluating the abil- ity of llms to solve semantics-aware process mining tasks. In: 6th International Conference on Process Mining, ICPM 2024, Kgs. Lyngby, Denmark, October 14-18, 2024, pp. 9–16. IEEE, New York, USA (2024...

    2024

  24. [33]

    In: 5th International Confer- ence on Process Mining, ICPM 2023, Rome, Italy, October 23-27, 2023, pp

    Kourani, H., Schuster, D., van der Aalst, W.M.P.: Scalable discovery of partially ordered workflow models with formal guarantees. In: 5th International Confer- ence on Process Mining, ICPM 2023, Rome, Italy, October 23-27, 2023, pp. 89–96. IEEE, New York, USA (2023). https://d...

    2023 arXiv

  25. [34]

    Lecture Notes in Business Informa- tion Processing, vol

    Leemans, S.J.J.: Robust Process Mining with Guarantees - Process Discovery, Conformance Checking and Enhancement. Lecture Notes in Business Informa- tion Processing, vol. 440. Springer, Germany (2022). https://doi.org/10.1007/ 978-3-030-96655-3

    2022

  26. [35]

    CoRR abs/2305.14688 (2023) https://doi.org/10.48550/ARXIV.2305.14688 2305.14688

    Xu, B., Yang, A., Lin, J., Wang, Q., Zhou, C., Zhang, Y., Mao, Z.: Expert- prompting: Instructing large language models to be distinguished experts. CoRR abs/2305.14688 (2023) https://doi.org/10.48550/ARXIV.2305.14688 2305.14688

    2023 doi arXiv

  27. [36]

    In: Pesquita, C., Skaf-Molli, H., Efthymiou, V., Kirrane, S., Ngonga, A., Collarana, D., Cerqueira, R., Alam, M., Trojahn, C., Hertling, S

    Martino, A., Iannelli, M., Truong, C.: Knowledge injection to counter large lan- guage model (LLM) hallucination. In: Pesquita, C., Skaf-Molli, H., Efthymiou, V., Kirrane, S., Ngonga, A., Collarana, D., Cerqueira, R., Alam, M., Trojahn, C., Hertling, S. (eds.) The Semantic Web...

    2023 doi

  28. [37]

    Brown et al.: Language models are few-shot learners

    Tom B. Brown et al.: Language models are few-shot learners. In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., Lin, H. (eds.) Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-...

    2020

  29. [38]

    In: Cabanillas, C., Garmann-Johnsen, N.F., Koschmider, A

    Bellan, P., van der Aa, H., Dragoni, M., Ghidini, C., Ponzetto, S.P.: PET: an annotated dataset for process extraction from natural language text tasks. In: Cabanillas, C., Garmann-Johnsen, N.F., Koschmider, A. (eds.) Business Pro- cess Management Workshops - BPM 2022 Internat...

    2022 doi

  30. [39]

    CoRR abs/2305.16807 (2023) https://doi.org/10.48550/ARXIV.2305.16807 2305.16807

    Miyake, D., Iohara, A., Saito, Y., Tanaka, T.: Negative-prompt inver- sion: Fast image inversion for editing with text-guided diffusion models. CoRR abs/2305.16807 (2023) https://doi.org/10.48550/ARXIV.2305.16807 2305.16807

    2023 doi arXiv

  31. [40]

    In: Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI 2024, Jeju, South Korea, August 3-9, 2024, pp

    Kourani, H., Berti, A., Schuster, D., van der Aalst, W.M.P.: ProMoAI: Process modeling with generative AI. In: Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI 2024, Jeju, South Korea, August 3-9, 2024, pp. 8708–8712. ijcai.org, ...

    2024

  32. [41]

    In: van der Aalst, W.M.P., Bergenthum, R., Car- mona, J

    Berti, A., van der Aalst, W.M.P.: Reviving token-based replay: Increasing speed while improving diagnostics. In: van der Aalst, W.M.P., Bergenthum, R., Car- mona, J. (eds.) Proceedings of the International Workshop on Algorithms & Theories for the Analysis of Event Data, Satel...

    2019

  33. [42]

    In: Hull, R., Mendling, J., Tai, S

    Munoz-Gama, J., Carmona, J.: A fresh look at precision in process confor- mance. In: Hull, R., Mendling, J., Tai, S. (eds.) BPM 2010. Proceedings. LNCS, vol. 6336, pp. 211–226. Springer, Germany (2010). https://doi.org/10.1007/ 978-3-642-15618-2 16

    2010

  34. [43]

    Berti, A., van Zelst, S.J., Schuster, D.: PM4Py: A process mining library for Python. Softw. Impacts 17, 100556 (2023) https://doi.org/10.1016/J.SIMPA. 2023.100556

    2023

  35. [44]

    In: Ku, L., Martins, A., Srikumar, V

    Zhang, X., Peng, B., Tian, Y., Zhou, J., Jin, L., Song, L., Mi, H., Meng, H.: Self- alignment for factuality: Mitigating hallucinations in llms via self-evaluation. In: Ku, L., Martins, A., Srikumar, V. (eds.) Proceedings of the 62nd Annual Meeting of the Association for Compu...

    2024

  36. [45]

    CoRR abs/2407.10457 (2024) https://doi.org/10.48550/ARXIV.2407.10457 2407.10457 34

    Song, Y., Wang, G., Li, S., Lin, B.Y.: The good, the bad, and the greedy: Evalu- ation of llms should not ignore non-determinism. CoRR abs/2407.10457 (2024) https://doi.org/10.48550/ARXIV.2407.10457 2407.10457 34

    2024 doi arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.