Pith. sign in

REVIEW 3 major objections 5 minor 45 references

On the Potential of Large Language Models to Solve Semantics-Aware Process Mining Tasks

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Fine-tuned large language models can solve semantics-aware process mining tasks, while off-the-shelf or few-shot LLMs largely cannot.

desk verdict A genuinely useful benchmark and fine-tuning study for semantics-aware process mining, with relative results that hold up; the absolute 'semantic understanding' claim outruns the gold-standard validity. read the letter →

arxiv 2504.21074 v1 pith:L4Q6H37A submitted 2025-04-29 cs.DB cs.AI

Adrian Rebmann , Fabian David Schmidt , Goran Glavaš , Han van der Aa This is my paper · ORCID
classification cs.DBcs.AI
keywords processmininglargelanguagemodelssemantics-awaresemanticanomalydetectionnextactivitypredictiondiscoveryfine-tuningin-contextlearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that large language models can carry out process mining tasks that require understanding what activities mean, not just counting how often one activity follows another. The authors define five such tasks: judging whether a whole trace is anomalous, judging whether one activity may eventually follow another, predicting the next activity from a prefix, discovering a directly-follows graph from an activity set, and discovering a process tree from an activity set. They build large benchmark datasets for all five by unfolding a corpus of process models into all allowed execution sequences and deriving gold labels. Their experiments show that the same open LLMs that score near chance with few-shot prompting jump to macro F1 scores of 0.79, 0.88, and 0.69 on the three classification tasks and to fitness scores around 0.80 to 0.84 on the two discovery tasks once fine-tuned, outperforming a fine-tuned encoder-only language model on every classification task.

What carries the argument

The argument is carried by a benchmark-construction pipeline plus a fine-tuning protocol. The corpus starts from a large public collection of English business-process diagrams; each diagram is converted into a sound block-structured workflow net, and each net is unfolded into all its allowed activity sequences, with loops executed at most once. Those sequences are the source of gold labels for the five tasks: trace-level and relation-level semantic anomaly detection, next-activity prediction, directly-follows graph discovery, and process-tree discovery; the same nets are translated into process trees as gold standards for the process-tree task. On the modeling side, the load-bearing mechanism is low-rank adaptation (LoRA) of decoder LLMs: classification is cast as constrained generation over the allowed class tokens, and discovery as free generation of edge lists or tree notation, so both kinds of tasks can be fine-tuned without updating all billions of parameters. Output quality is measured by footprint-based fitness, which compares the behavioral footprints of predicted and gold directly-follows graphs or process trees.

What would settle it

Recompute the evaluation on a quality-assured subset: take a random sample of test models, have process experts judge whether each gold-standard trace, relation, and process tree is semantically correct, and rerun the fine-tuned models only on instances with expert-verified gold standards. If the fine-tuned LLMs no longer beat the encoder baseline or the random baseline on that subset, the claim that fine-tuning unlocks semantic process understanding would be refuted.

Watch

Extended reading notes

Core claim

The central claim is that the process knowledge needed for semantics-aware process mining is already present in large language models but is not accessible through prompting alone; supervised fine-tuning unlocks it. With in-context learning, the models hover at or near a random baseline on trace-level and activity-pair anomaly detection (F1 0.44 to 0.53) and manage only weak next-activity prediction (best F1 0.32), while their discovered graphs and trees score fitness 0.52 to 0.61. After fine-tuning, Llama-3 and Mistral reach macro F1 of 0.79 on trace-level anomaly detection, 0.88 on activity-pair anomaly detection, and 0.69 on semantic next-activity prediction, and fitness 0.80 to 0.81 on directly-follows graph discovery and 0.83 to 0.84 on process-tree discovery. The fine-tuned LLMs outperform fine-tuned RoBERTa on every classification task, and on the discovery tasks they learn the few global ordering constraints of a process more effectively than the many local pairwise relations.

Load-bearing premise

The entire evaluation assumes the gold-standard process models created from the source diagrams correctly encode the semantics of the underlying processes, so that a trace, relation, or generated model matching the gold standard is semantically right; the paper itself notes in Section 7.3.1 that some gold standards are not semantically correct.

Editorial extensions

If this is right

  • A fine-tuned LLM can perform semantic checks without access to historical event logs, since inputs are only activity labels plus a trace, relation, or activity set; this makes the approach usable for processes with scarce or missing logs.
  • The near-random few-shot results imply that task-specific fine-tuning, not prompt engineering, is the practical condition for deploying LLMs on such process mining tasks.
  • Because fine-tuned LLMs surpass a fine-tuned encoder on all classification tasks, decoder LLMs should be the default choice for semantics-aware process mining rather than smaller bidirectional encoders.
  • On process-tree discovery the LLMs learn global process structure well enough to reach fitness 0.83 to 0.84, which suggests that LLM-generated process trees could serve as semantic models for conformance checking against observed traces.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: fine-tuning on all five tasks jointly, as the authors propose as future work, may produce a single multi-task model whose semantic competence transfers across tasks, potentially reducing the data needed for new process mining tasks.
  • Beyond the paper: the paper's own observation that some gold-standard models are not semantically correct, such as a travel-reconciliation model that allows payment verification after payment approval, means the absolute fitness and F1 numbers likely underestimate LLM capability on a cleaner benchmark; the fine-tuning-versus-prompting gap is the more robust finding.
  • Beyond the paper: a natural testable extension is to plug a fine-tuned semantic scorer into an existing frequency-based next-activity predictor and check whether rejecting semantically invalid predictions improves accuracy on real logs.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper defines five semantics-aware process mining tasks (T-SAD, A-SAD, S-NAP, S-DFD, S-PTD), constructs benchmarking datasets from a large corpus of sap-sam BPMN models, and evaluates open-source LLMs (Llama-3-8B, Mistral-7B) under both in-context learning and supervised fine-tuning, with RoBERTa and rule-based/random baselines. The central empirical finding is that ICL performs poorly on these tasks, whereas fine-tuned LLMs achieve macro F1 scores of 0.79 (T-SAD), 0.88 (A-SAD), and 0.69 (S-NAP), and fitness scores of 0.80–0.84 on the two generation tasks, consistently outperforming fine-tuned RoBERTa on classification and exceeding the random baseline on generation.

Significance. If the benchmark gold standards are valid, this is a valuable contribution: it is the first systematic comparison of fine-tuned open LLMs against encoder and rule-based baselines on five well-defined semantics-aware process mining tasks. The paper ships public datasets, training scripts, and evaluation code, uses process-level split to avoid leakage, and reports three-seed variance. The relative ranking (fine-tuning > ICL > random; LLMs > RoBERTa) is robust across tasks and seeds. The main weakness is that the gold standards themselves are asserted, not demonstrated, to be semantically correct; the paper's own admission of problematic models in the corpus undermines the absolute performance claims as evidence of semantic understanding. The results are still useful as a benchmark of agreement with a plausible process-model collection, but the headline claim of 'strong performance' on semantics-aware tasks is not yet secured.

major comments (3)
  1. [Sections 4.1.1, 7.3.1, and 9] The load-bearing assumption of the evaluation is that the sap-sam-derived gold standards correctly encode the semantics of the underlying processes. The corpus creation filters only for sound block-structured workflow nets, which is a syntactic quality filter; it does not establish semantic correctness. The paper itself states in Section 7.3.1 that 'the process models in the employed collection do not always represent semantically correct models', giving a travel-reconciliation example where the gold tree allows verification, payment, authorization, archiving, and acceptance in parallel. Section 9's plan to redo evaluation on a 'quality-assured model collection' confirms that the current gold standards are not trustworthy for absolute measurement. Since every label and gold graph/tree in T-SAD, A-SAD, S-NAP, S-DFD, and S-PTD is derived from these models, the reported F1 and fitness scores do not by themselves establish semantic understanding. Please either re-score the test set on a human-validated subset of models or provide an analysis showing that the semantic errors in the gold standards do not systematically inflate or deflate the reported scores.
  2. [Section 7.3.1 (and Table 5/6)] The in-depth analysis further illustrates the benchmark-validity problem. The paper shows a prescription-fulfillment example where the generated tree differs from the gold tree in the ordering of 'Check insurance coverage' and 'Fill prescription', and a travel process where the LLM's sequential tree is described as 'not ideal either'. These cases demonstrate that a low fitness score is not necessarily a semantic error, and a high fitness score is not necessarily semantic correctness. The discussion of per-process-type and per-industry fitness should therefore be interpreted as agreement with a specific gold collection rather than as evidence of domain-level semantic competence. I recommend reporting results on the subset of models that pass a semantic-quality review and separately quantifying the fraction of gold models with identified semantic problems.
  3. [Sections 7.1.2 and 7.2.2] The abstract and conclusion state that fine-tuned LLMs 'achieve strong performance' on semantics-aware tasks. Given the gold-standard validity issue, the safe claim is that they achieve strong agreement with the gold-standard models derived from sap-sam. The distinction matters because the paper's own examples show that multiple reasonable models can exist for the same activity set, and the gold standard is not always one of them. I ask that the absolute performance claims be tempered or, ideally, accompanied by a human-evaluation study on a sample of outputs, so that 'strong performance' refers to semantic correctness rather than benchmark agreement.
minor comments (5)
  1. [Section 6, Large Language Models] The text refers to 'Mistral-2 in its 7 billion parameter version', but the cited model is Mistral-7B-Instruct-v0.2; please correct the model name for reproducibility and consistency.
  2. [Table 6] The 'Retail' row appears twice with identical values (0.88, 9.03); one of these rows should be removed.
  3. [Sections 4.2.2 and 7.2.2] The task abbreviation 'S-DPT' is used instead of 'S-PTD' in the S-PTD dataset description and in the discussion of fine-tuning results; please standardize the abbreviation.
  4. [Section 7.2.1] The text mentions 'S-SPT variations'; this appears to be a typo for 'S-PTD'.
  5. [Throughout] There are inconsistent spellings of 'RoBERTa' (e.g., 'RoBER T a', 'RobERTa' in Tables 3 and 7 and running text); these should be unified.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the evaluation is an external benchmark against sap-sam gold standards; the paper's own caveat about imperfect gold models is a validity concern, not a circularity.

full rationale

The paper's central claim is empirical: fine-tuned LLMs outperform ICL, random guessing, a rule-based baseline, and RoBERTa on five semantics-aware process mining tasks. The gold labels for all five tasks are derived from the sap-sam process model collection (Sec. 4.1.1 and 4.2), which is external to the LLMs under test. No label is a function of the models' own outputs, no fitted parameter is renamed as a prediction, and the train/validation/test split is performed at the process-model level to prevent behavioral leakage (Sec. 6). The T-SAD, A-SAD, S-NAP, S-DFD, and S-PTD gold standards are generated from workflow nets and process trees, i.e., from independently existing process diagrams, not from the LLM knowledge or from the task definitions themselves. The authors do cite their own prior work ([1], [2], [8]), but only as baseline algorithms or as pointers to earlier task formulations; these citations are not used to justify the paper's central claim, and the comparisons are externally falsifiable. The paper itself concedes in Sec. 7.3.1 that some sap-sam models 'do not always represent semantically correct models' and plans a future quality-assured evaluation in Sec. 9; this is a threat to the validity of the absolute scores as evidence of semantic understanding, but it is not a circularity, because the measured quantities still come from an external gold standard. The derivation chain is therefore self-contained with respect to circularity.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces five task definitions and datasets, not new physical or formal entities. Its central claims rest on domain assumptions about the semantic validity of the process models used to generate gold standards and about the evaluation metrics.

assumptions (4)
  • domain assumption The filtered sap-sam diagrams form a sound, representative process behavior corpus.
    Section 4.1.1 keeps only English BPMN diagrams convertible to sound block-structured workflow nets with unique activity sets. The paper later notes in Section 7.3.1 that some models in this collection are not semantically correct.
  • domain assumption Gold-standard labels (valid/anomalous, next activity, DFG, process tree) derived from the process model M are the correct semantic ground truth.
    Section 4.2 computes labels from the model M, equating 'semantically valid' with 'allowed by the model'. Section 7.3.1 calls this equation into question by giving counterexamples where the model itself is semantically wrong.
  • domain assumption Footprint-based fitness is a valid proxy for semantic correctness of discovered DFGs and process trees.
    Section 6 measures generation quality by comparing behavioral footprints between gold and generated models. Two structurally different models can have identical footprints, and semantically wrong models can still score high.
  • standard math Block-structured workflow nets can be translated to process trees without loss (Inductive Miner correspondence).
    Section 4.2.2 states the translation is straightforward due to the block-structured restriction, citing Inductive Miner [18].

how reviews work

0 comments
Cite this review

Pith. "Pith review of On the Potential of Large Language Models to Solve Semantics-Aware Process Mining Tasks." pith.science (2026). https://pith.science/paper/L4Q6H37A

@misc{pith2026250421074,
  author       = {Pith},
  title        = {Pith review of: On the Potential of Large Language Models to Solve Semantics-Aware Process Mining Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/L4Q6H37A}},
  note         = {Machine review of arXiv:2504.21074}
}
read the original abstract

Large language models (LLMs) have shown to be valuable tools for tackling process mining tasks. Existing studies report on their capability to support various data-driven process analyses and even, to some extent, that they are able to reason about how processes work. This reasoning ability suggests that there is potential for LLMs to tackle semantics-aware process mining tasks, which are tasks that rely on an understanding of the meaning of activities and their relationships. Examples of these include process discovery, where the meaning of activities can indicate their dependency, whereas in anomaly detection the meaning can be used to recognize process behavior that is abnormal. In this paper, we systematically explore the capabilities of LLMs for such tasks. Unlike prior work, which largely evaluates LLMs in their default state, we investigate their utility through both in-context learning and supervised fine-tuning. Concretely, we define five process mining tasks requiring semantic understanding and provide extensive benchmarking datasets for evaluation. Our experiments reveal that while LLMs struggle with challenging process mining tasks when used out of the box or with minimal in-context examples, they achieve strong performance when fine-tuned for these tasks across a broad range of process types and industries.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

45 extracted references · 31 canonical work pages

  1. [1]

    Information Systems 102, 101824 (2021)

    van der Aa, H., Rebmann, A., Leopold, H.: Natural language-based detection of semantic execution anomalies in event logs. Information Systems 102, 101824 (2021)

    work page 2021

  2. [2]

    In: BPM, pp

    Caspary, J., Rebmann, A., van der Aa, H.: Does this make sense? machine learning-based detection of semantic anomalies in business processes. In: BPM, pp. 163–179 (2023). Springer

    work page 2023

  3. [3]

    arXiv preprint arXiv:2408.17316 (2024)

    Norouzifar, A., Kourani, H., Dees, M., Aalst, W.M.: Bridging domain knowl- edge and process discovery using large language models. arXiv preprint arXiv:2408.17316 (2024)

    arXiv 2024

  4. [4]

    In: International Conference on Business Process Management, pp

    Busch, K., Kampik, T., Leopold, H.: xsemad: Explainable semantic anomaly detection in event logs using sequence-to-sequence models. In: International Conference on Business Process Management, pp. 309–327 (2024). Springer

    work page 2024

  5. [5]

    In: BPM, pp

    Berti, A., Schuster, D., van der Aalst, W.M.: Abstractions, scenarios, and prompt definitions for process mining with llms: A case study. In: BPM, pp. 427–439 (2023). Springer

    work page 2023

  6. [6]

    preprint arXiv:2307.09909 (2023)

    Jessen, U., Sroka, M., Fahland, D.: Chit-chat or deep talk: Prompt engineering for process mining. preprint arXiv:2307.09909 (2023)

    arXiv 2023

  7. [7]

    In: BPMDS, pp

    Estrada-Torres, B., del-R´ ıo-Ortega, A., Resinas, M.: Mapping the landscape: Exploring large language model applications in business process management. In: BPMDS, pp. 22–31 (2024). Springer

    work page 2024

  8. [8]

    In: 2024 6th International Conference on Process Mining (ICPM), pp

    Rebmann, A., Schmidt, F.D., Glavaˇ s, G., van der Aa, H.: Evaluating the ability of llms to solve semantics-aware process mining tasks. In: 2024 6th International Conference on Process Mining (ICPM), pp. 9–16 (2024). IEEE

    work page 2024

Show all 45 references
  1. [9]

    In: Process Mining Handbook, pp

    van der Aalst, W.M.: Foundations of process discovery. In: Process Mining Handbook, pp. 37–75 (2022). Springer 28

    2022

  2. [10]

    Electronic Notes in Theoretical Computer Science 121, 3–21 (2005)

    van der Aalst, W.M.P., Medeiros, A.K.A.: Process mining and security: Detect- ing anomalous process executions and checking process conformance. Electronic Notes in Theoretical Computer Science 121, 3–21 (2005)

    2005

  3. [11]

    Information Systems 38(1), 33–44 (2013)

    Bezerra, F., Wainer, J.: Algorithms for anomaly detection of traces in logs of process aware information systems. Information Systems 38(1), 33–44 (2013)

    2013

  4. [12]

    In: Process Mining Handbook, pp

    van der Aalst, W.M.: Process mining: a 360 degree overview. In: Process Mining Handbook, pp. 3–34 (2022). Springer

    2022

  5. [13]

    Artificial Intelligence Review 55(2), 801–827 (2022)

    Neu, D.A., Lahann, J., Fettke, P.: A systematic literature review on state-of-the- art deep learning methods for process prediction. Artificial Intelligence Review 55(2), 801–827 (2022)

    2022

  6. [14]

    Evermann, J., Rehse, J.-R., Fettke, P.: Predicting process behaviour using deep learning. Dec. Support Systems 100, 129–140 (2017)

    2017

  7. [15]

    In: BPM, pp

    Pfeiffer, P., Lahann, J., Fettke, P.: Multivariate business process representation learning utilizing gramian angular fields and convolutional neural networks. In: BPM, pp. 327–344 (2021). Springer

    2021

  8. [16]

    Zenodo (2024)

    Rebmann, A., Schmidt, F.D., Glavaˇ s, G., van der Aa, H.: Process Behavior Cor- pus and Benchmarking Datasets. Zenodo (2024). https://zenodo.org/records/ 14273161

    2024

  9. [17]

    In: ICPM Workshops, pp

    Sola, D., Warmuth, C., Sch¨ afer, B., Badakhshan, P., Rehse, J.-R., Kampik, T.: SAP Signavio academic models: A large process model dataset. In: ICPM Workshops, pp. 453–465 (2023). Springer

    2023

  10. [18]

    In: Application and The- ory of Petri Nets and Concurrency: 34th International Conference, PETRI NETS 2013, Milan, Italy, June 24-28, 2013

    Leemans, S.J., Fahland, D., Van Der Aalst, W.M.: Discovering block-structured process models from event logs-a constructive approach. In: Application and The- ory of Petri Nets and Concurrency: 34th International Conference, PETRI NETS 2013, Milan, Italy, June 24-28, 2013. Pro...

    2013

  11. [19]

    Advances in neural information processing systems 30 (2017)

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need. Advances in neural information processing systems 30 (2017)

    2017

  12. [20]

    In: NAACL, pp

    Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In: NAACL, pp. 4171– 4186 (2019). ACL

    2019

  13. [21]

    arXiv:1907.11692 (2019)

    Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., Stoyanov, V.: Roberta: A robustly optimized bert pretraining approach. arXiv:1907.11692 (2019)

    2019 arXiv

  14. [22]

    : Language models are few-shot learners

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., 29 Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. : Language models are few-shot learners. Advances in neural information processing systems 33, 1877–1901 (2020)

    2020

  15. [23]

    arXiv:2302.13971 (2023)

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozi` ere, B., Goyal, N., Hambro, E., Azhar, F., et al.: Llama: Open and efficient foundation language models. arXiv:2302.13971 (2023)

    2023 arXiv

  16. [24]

    : Super- naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks

    Wang, Y., Mishra, S., Alipoormolabashi, P., Kordi, Y., Mirzaei, A., Arunk- umar, A., Ashok, A., Dhanasekaran, A.S., Naik, A., Stap, D., et al. : Super- naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks. In: EMNLP (2022)

    2022

  17. [25]

    arXiv:2308.10792 (2023)

    Zhang, S., Dong, L., Li, X., Zhang, S., Sun, X., Wang, S., Li, J., Hu, R., Zhang, T., Wu, F., et al.: Instruction tuning for large language models: A survey. arXiv:2308.10792 (2023)

    2023

  18. [26]

    arXiv:2301.00234 (2022)

    Dong, Q., Li, L., Dai, D., Zheng, C., Wu, Z., Chang, B., Sun, X., Xu, J., Sui, Z.: A survey on in-context learning. arXiv:2301.00234 (2022)

    2022 arXiv

  19. [27]

    In: International Conference on Learning Representations (2021)

    Hu, E.J., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al.: Lora: Low-rank adaptation of large language models. In: International Conference on Learning Representations (2021)

    2021

  20. [28]

    56, (2018)

    Carmona, J., Dongen, B., Solti, A., Weidlich, M.: Conformance Checking vol. 56, (2018). Springer

    2018

  21. [29]

    In: Interna- tional Conference on Learning Representations (2018)

    Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. In: Interna- tional Conference on Learning Representations (2018)

    2018

  22. [30]

    In: International Conference on Enterprise Design, Operations, and Computing, pp

    Bellan, P., Dragoni, M., Ghidini, C.: Extracting business process entities and relations from text using pre-trained language models and in-context learning. In: International Conference on Enterprise Design, Operations, and Computing, pp. 182–199 (2022). Springer

    2022

  23. [31]

    arXiv preprint arXiv:2110.03754 (2021)

    Bellan, P., Dragoni, M., Ghidini, C., Aa, H., Ponzetto, S.P.: Process extraction from text: benchmarking the state of the art and paving the way for future challenges. arXiv preprint arXiv:2110.03754 (2021)

    2021 arXiv

  24. [32]

    Rebmann, A., van der Aa, H.: Enabling semantics-aware process mining through the automatic annotation of event logs. Inf. Syst. 110, 102111 (2022)

    2022

  25. [33]

    In: ICPM, pp

    Kecht, C., Egger, A., Kratsch, W., R¨ oglinger, M.: Event log construction from customer service conversations using natural language inference. In: ICPM, pp. 144–151 (2021). IEEE

    2021

  26. [34]

    In: BPM, pp

    Grohs, M., Abb, L., Elsayed, N., Rehse, J.-R.: Large language models can 30 accomplish business process management tasks. In: BPM, pp. 453–465 (2023). Springer

    2023

  27. [35]

    arXiv:2403.07541 (2024)

    Kourani, H., Berti, A., Schuster, D., van der Aalst, W.M.: Process modeling with large language models. arXiv:2403.07541 (2024)

    2024 arXiv

  28. [36]

    In: International Conference on Business Process Management, pp

    Klievtsova, N., Benzin, J.-V., Kampik, T., Mangler, J., Rinderle-Ma, S.: Con- versational process modelling: state of the art, applications, and implications in practice. In: International Conference on Business Process Management, pp. 319–336 (2023). Springer

    2023

  29. [37]

    In: International Conference on Business Process Management, pp

    Ziche, C., Apruzzese, G.: Llm4pm: A case study on using large language models for process modeling in enterprise organizations. In: International Conference on Business Process Management, pp. 472–483 (2024). Springer

    2024

  30. [38]

    In: International Conference on Cooperative Information Systems, pp

    Nour Eldin, A., Assy, N., Anesini, O., Dalmas, B., Gaaloul, W.: A decom- posed hybrid approach to business process modeling with llms. In: International Conference on Cooperative Information Systems, pp. 243–260 (2024). Springer

    2024

  31. [39]

    In: International Conference on Process Mining, pp

    Fani Sani, M., Sroka, M., Burattin, A.: Llms and process mining: Challenges in rpa: Task grouping, labelling and connector recommendation. In: International Conference on Process Mining, pp. 379–391 (2023). Springer

    2023

  32. [40]

    In: BPMDS, pp

    Berti, A., Kourani, H., H¨ afke, H., Li, C.-Y., Schuster, D.: Evaluating large language models in process mining: Capabilities, benchmarks, and evaluation strategies. In: BPMDS, pp. 13–21 (2024). Springer

    2024

  33. [41]

    arXiv:2407.13244 (2024)

    Berti, A., Kourani, H., van der Aalst, W.M.: PM-LLM-Benchmark: Evaluating large language models on process mining tasks. arXiv:2407.13244 (2024)

    2024 arXiv

  34. [42]

    preprint arXiv:2404.13076 (2024)

    Panickssery, A., Bowman, S.R., Feng, S.: LLM evaluators recognize and favor their own generations. preprint arXiv:2404.13076 (2024)

    2024 arXiv

  35. [43]

    arXiv preprint arXiv:2410.03255 (2024)

    Busch, K., Leopold, H.: Towards a benchmark for large language models for business process management tasks. arXiv preprint arXiv:2410.03255 (2024)

    2024 arXiv

  36. [44]

    arXiv preprint arXiv:2412.00023 (2024)

    Kourani, H., Berti, A., Schuster, D., Aalst, W.M.: Evaluating large language mod- els on business process modeling: Framework, benchmark, and self-improvement analysis. arXiv preprint arXiv:2412.00023 (2024)

    2024 arXiv

  37. [45]

    KI-K¨ unstliche Intelligenz, 1–15 (2024) 31

    Kampik, T., Warmuth, C., Rebmann, A., Agam, R., Egger, L.N., Gerber, A., Hoffart, J., Kolk, J., Herzig, P., Decker, G., et al.: Large process models: A vision for business process management in the age of generative AI. KI-K¨ unstliche Intelligenz, 1–15 (2024) 31

    2024

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.