REVIEW 4 major objections 5 minor 26 references
Leveraging Machine Learning and Enhanced Parallelism Detection for BPMN Model Generation from Text
T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read A 15-document extension to the PET corpus raises AND-gateway F1 from 0% to 23% and gives 89.2% element F1 in the full pipeline.
desk verdict A useful small dataset and pipeline for BPMN extraction, but the headline AND-gateway improvement is not supported by a controlled comparison. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is LESCHNEIDER, a manually authored extension of the PET dataset: 15 process descriptions, 91 sentences, and 32 AND-gateway annotations built by drafting BPMN diagrams from real-world sources and then writing text that follows the Seven Process Modeling Guidelines. It carries the argument by rebalancing the AND-gateway label distribution and by providing textual markers such as 'and simultaneously' that the models can learn for parallel structure detection. Around it, the pipeline combines token-level NER (CRF, BERT, or RoBERTa), a CatBoost relation classifier over mention-pair features, coreference-based entity resolution, and a directed-graph renderer that outputs Business Process Model and Notation (BPMN) diagrams.
What would settle it
Take a held-out set of process descriptions from unrelated domains that express parallelism with connectors other than 'and simultaneously' (for example, 'while', 'in parallel', or 'at the same time') and compare B-AND Gateway recall for the combined-corpus CRF. If recall drops to near zero, the measured improvement is an artifact of the dataset's phrasing rather than a general parallel-structure capability.
Extended reading notes
Core claim
The paper's central claim is that the LESCHNEIDER dataset, 15 manually drafted and annotated process descriptions containing 32 AND gateways, fixes a critical data imbalance in the PET dataset (8 AND vs 117 XOR gateways) and thereby enables BPMN extraction models to recognize parallel splits in text. In the authors' experiments, a CRF trained on the combined corpus improves B-AND Gateway F1 from 0% to 23% while most other labels keep their performance, and BERT-base-cased is the best NER model because larger models overfit the small corpus. The assembled pipeline, using CatBoost with random over-sampling for relation extraction and coreference resolution for entity merging, produces BPMN diagrams with aggregate element F1 of 89.2% and relation F1 of 73.7% on six test documents. The authors also report that the flow relation (F1 62%) and implicit gateway closures remain the main bottlenecks.
Load-bearing premise
The load-bearing premise is that the 15 hand-written LESCHNEIDER documents, annotated by the authors with no reported check that other annotators would label them the same way, are a valid sample of how real process texts express parallelism.
Editorial extensions
If this is right
- Models trained on the combined PET plus LESCHNEIDER corpus should detect AND gateways in unseen texts, where PET-only models currently score 0% F1.
- The LESCHNEIDER additions make parallelism a measurable, trainable target instead of a rule-based afterthought.
- BERT-base-cased is the recommended NER backbone for this data size; larger BERT and RoBERTa variants overfit and should be avoided unless more data is added.
- The full pipeline can serve as a reproducible baseline for end-to-end BPMN extraction, with element F1 around 89% and relation F1 around 74%.
- Relation extraction, especially the flow relation at 62% F1, is the current bottleneck for accurate diagram generation.
Reading between the lines
- Because the 15 LESCHNEIDER documents were written by the authors, the reported gain may be tied to the specific phrase 'and simultaneously'; parallel connectors like 'while', 'in tandem', or 'at the same time' should be tested to see whether the improvement generalizes.
- The paper reports no inter-annotator agreement for the new annotations, so an independent re-annotation of the same documents would clarify how much of the F1 gain is signal rather than a single annotator's style.
- A natural next experiment is to keep the pipeline fixed and vary only the training corpus, replacing LESCHNEIDER with an equal-size corpus of parallel-heavy documents from a different domain; the B-AND Gateway F1 delta would measure dataset value versus corpus size.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes an automated pipeline for generating BPMN models from text, combining preprocessing, NER (CRF and fine-tuned BERT/RoBERTa), relation extraction with CatBoost, entity resolution, and diagram generation. Its main contribution is the LESCHNEIDER dataset, 15 manually authored and annotated documents adding 32 AND gateways to the PET dataset, intended to improve detection of parallel structures. Three experiments address whether PET is sufficiently diverse, whether the augmented corpus improves AND-gateway detection, and whether BERT or RoBERTa is more effective. Reported results include a weighted-average CRF F1 of 0.72, a B-AND F1 improvement from 0.00 to 0.23 after adding LESCHNEIDER, BERT-base-cased as the best NER model, and a full-pipeline element F1 of 89.2% with relation F1 of 73.7%.
Significance. If the claims hold, the pipeline and dataset are a modest but useful step toward automated BPMN generation, addressing a real gap: AND gateways are severely underrepresented in the PET dataset. The paper reports exact F1 values and explicitly acknowledges limitations such as the CatBoost flow-relation F1 of 62% and difficulties with implicit gateway closures, which is commendable. The manuscript also states that code and datasets will be made available through GitHub, which would support reproducibility if the link is provided. However, the significance is currently tempered by an evaluation design that does not yet establish the central parallelism claim: the main comparison is uncontrolled, the gold standard is author-created with no inter-annotator agreement, and the pipeline test documents are not described in terms of provenance or holding-out.
major comments (4)
- [§5.2, Table 3] The RQ2 claim that adding LESCHNEIDER 'significantly improved' AND-gateway detection rests on an uncontrolled comparison. Baseline (a) is a 5-fold cross-validation on PET alone, while Model 3 (c) is a 5-fold cross-validation on the combined PET+LESCHNEIDER corpus; these columns are evaluated on different test folds, so the 0.00-to-0.23 B-AND F1 improvement conflates added data volume with added parallel-structure content. The symmetric ablation needed to answer RQ2 is missing: hold out the same LESCHNEIDER folds and compare a CRF trained on PET only with one trained on PET+LESCHNEIDER. The one available cross-corpus condition, CRF Model 2 trained on all PET and tested on all LESCHNEIDER, yields 0.00 B-AND F1, indicating no transfer of AND-gateway knowledge from PET to LESCHNEIDER. Without per-fold scores or intervals, the reported improvement cannot be distinguished from split noise, so the central parallelism claim is not yet established.
- [§3.2] The LESCHNEIDER dataset is both the training enrichment and the evaluation gold standard for RQ2, and it was authored and annotated by the same team with no reported inter-annotator agreement. Because the texts were written to include parallel structures via explicit markers such as 'and simultaneously' (§3.2), a model trained on part of LESCHNEIDER can match the annotation style of the remaining part without learning a generalizable notion of concurrency. The paper should report annotation guidelines, inter-annotator agreement, and ideally evaluate on an externally produced test set; at minimum it should justify that the manually drafted markers reflect naturally occurring process language. This is load-bearing because the dataset itself is the paper's main contribution.
- [§5.4, Table 4] The pipeline evaluation does not state the provenance or holding-out of the six test documents. If these documents are LESCHNEIDER texts that were used to train the NER model, the reported element F1 of 89.2% and relation F1 of 73.7% could be in-sample estimates. Please specify whether the six documents come from PET, LESCHNEIDER, or a separate set, and confirm that they were excluded from all training and hyperparameter selection. Without this information the headline pipeline numbers cannot be interpreted.
- [§5.2 and §5.4] All F1 values are point estimates from five-fold cross-validation with no per-fold breakdown, confidence intervals, or significance tests. Given the tiny positive class for AND gateways (8 in PET, 40 in LESCHNEIDER), a B-AND F1 of 0.23 could easily arise from sampling noise. Reporting per-fold scores and an interval or significance test would materially strengthen the comparison and is necessary to support the RQ2 conclusion.
minor comments (5)
- [Abstract and §3.2] The paper reports '32 new AND Gateways' while Table 2 lists 40 AND Gateway mentions; please clarify whether these are gateway elements or mentions and reconcile the counts.
- [§4.2] The text 'For the CRP' appears to be a typo for 'For the CRF'.
- [§5.1] The metric formulas are formatted as 'PrecisionPrecision=...' and would be clearer with standard equation formatting.
- [Title] The title contains an extra space in 'T ext'; please proofread the final text.
- [Table 1 and §4.3] Table 1 notes that the PET dataset was updated to version 1.1 by the authors, but the paper does not specify where this updated version is archived; please provide version information and a repository link to support reproducibility.
Circularity Check
The central AND-gateway improvement is reported across different test sets built from the authors' own annotated corpus, so the 0% to 23% headline partly reduces to training and testing on the same self-authored dataset rather than an independent prediction.
-
fitted input called prediction
[Section 5.5 Discussion; cf. Table 3 in Section 5.2]
"Second, in response to RQ2, we mitigated some limitations of the PET dataset by integrating the LESCHNEIDER dataset, which significantly improved the model's ability to detect parallel structures. This integration resulted in a notable rise in the F1 score for B-AND Gateway from 0% to 23% and an increase in precision to 50%."
The 0% is Table 3(a), the PET-only 5-fold cross-validation, whose test folds are PET documents containing only 8 AND mentions. The 23% is Table 3(c), the 5-fold cross-validation on the combined PET+LESCHNEIDER corpus, whose test folds include LESCHNEIDER documents containing 40 AND mentions. LESCHNEIDER was written and manually annotated by the same authors, with AND gateways introduced via explicit phrases like 'and simultaneously' (Section 3.2). Thus the model is trained and evaluated on the same authorial style and marker distribution. The paper's own CRF Model 2 (trained on PET, tested on LESCHNEIDER) still scores 0.00 on B-AND, confirming that the reported 'rise' appears only when the self-authored corpus is present on both sides of the train/test split.
full rationale
Most of the paper is a straightforward empirical pipeline: baseline CRF, BERT/RoBERTa fine-tuning with 5-fold cross-validation, CatBoost relation extraction with sampling, and a pipeline evaluation on six documents. These parts are internally consistent and are not built on a self-citation chain; for example, the superiority of BERT-base-cased is an empirical result, not a definitional one. The circularity risk is concentrated in the central RQ2 claim about parallelism detection. The headline '0% to 23%' B-AND F1 improvement compares two different experimental conditions: the PET-only 5-fold CV, tested on PET documents that contain only 8 AND mentions, and the combined PET+LESCHNEIDER 5-fold CV, tested on a corpus where 40 of 48 AND mentions come from LESCHNEIDER. Since LESCHNEIDER was authored and annotated by the same team, and the AND gateways were introduced through explicit lexical markers such as 'and simultaneously', a model trained on LESCHNEIDER folds can match the marker style of the held-out LESCHNEIDER folds without learning a generalizable notion of concurrency. The paper even reports that a model trained on all of PET and tested on LESCHNEIDER achieves B-AND F1 = 0.00, which shows the gain is tied to having the same self-authored corpus on both sides of the split. This is partial circularity of the evaluation rather than a fully closed derivation: the held-out folds are genuinely separate documents, and the pipeline evaluation on elements and relations is not itself caught in the same reduction. A score of 5 reflects that the central claim's headline evidence reduces substantially to the dataset construction and test-set shift, while leaving some independent empirical content in the rest of the paper.
Assumptions & free parameters
free parameters (4)
- Learning rate =
Grid of 2e-5, 3e-5, 4e-5, 5e-5, selected by F1
- Number of epochs =
Average optimal epochs across folds plus 2
- Batch size =
8
- Data sampling configuration =
Negative sampling rates, SMOTE, and ROS not fully reported
assumptions (4)
- domain assumption Token-level IOB tagging can capture BPMN entities and relations.
- domain assumption The authors' 15 manually annotated documents are correct gold standard annotations.
- ad hoc to paper The sources used to draft diagrams are representative of real-world business process text.
- standard math Cross-validation on the combined dataset estimates performance on unseen process descriptions.
Cite this review
Pith. "Pith review of Leveraging Machine Learning and Enhanced Parallelism Detection for BPMN Model Generation from Text." pith.science (2026). https://pith.science/paper/RBSHOODF
@misc{pith2026250708362,
author = {Pith},
title = {Pith review of: Leveraging Machine Learning and Enhanced Parallelism Detection for BPMN Model Generation from Text},
year = {2026},
howpublished = {\url{https://pith.science/paper/RBSHOODF}},
note = {Machine review of arXiv:2507.08362}
}
read the original abstract
Efficient planning, resource management, and consistent operations often rely on converting textual process documents into formal Business Process Model and Notation (BPMN) models. However, this conversion process remains time-intensive and costly. Existing approaches, whether rule-based or machine-learning-based, still struggle with writing styles and often fail to identify parallel structures in process descriptions. This paper introduces an automated pipeline for extracting BPMN models from text, leveraging the use of machine learning and large language models. A key contribution of this work is the introduction of a newly annotated dataset, which significantly enhances the training process. Specifically, we augment the PET dataset with 15 newly annotated documents containing 32 parallel gateways for model training, a critical feature often overlooked in existing datasets. This addition enables models to better capture parallel structures, a common but complex aspect of process descriptions. The proposed approach demonstrates adequate performance in terms of reconstruction accuracy, offering a promising foundation for organizations to accelerate BPMN model creation.
Figures
Reference graph
Works this paper leans on
-
[1]
Automated generation of business process models from natural language input,
F. Friedrich, “Automated generation of business process models from natural language input,” M. Sc., School of Business and Economics. Humboldt-Universität zu Berli, 2010
work page 2010
-
[2]
(2014) About the business process model and notation specification version 2.0.2
OMG. (2014) About the business process model and notation specification version 2.0.2. Accessed 2/01/2024. [Online]. Available: https://www.omg.org/spec/BPMN
work page 2014
-
[3]
J. Neuberger, L. Ackermann, and S. Jablonski, “Beyond rule-based named entity recognition and relation extraction for process model generation from natural language text,” inCooPIS 2023, ser. LNCS, vol. 14353. Springer, 2023, pp. 179–197
work page 2023
-
[4]
Process model generation from natural language text,
F. Friedrich, J. Mendling, and F. Puhlmann, “Process model generation from natural language text,” inAdvanced Information Systems Engineering 2011, ser. Lecture Notes in Computer Science, vol. 6741. Springer, 2011, pp. 482–496
work page 2011
-
[5]
P. Bellan, M. Dragoni, and C. Ghidini, “Extracting business process entities and relations from text using pre-trained language models and in-context learning,” vol. 13585 LNCS. Springer, 2022, pp. 182–199
work page 2022
-
[6]
P. Bellan, M. Dragoni, C. Ghidini, H. van der Aa, and S. P. Ponzetto, “Process extraction from text: Benchmarking the state of the art and paving the way for future challenges,” arXiv preprint arXiv:2110.03754, 2021. 12 Phuong et al
arXiv 2021
-
[7]
Bert: Pre-training of deep bidirectional transformers for language understanding,
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” 2019. [Online]. Available: https://arxiv.org/abs/1810.04805
arXiv 2019
-
[8]
Roberta: A robustly optimized BERT pretraining approach,
Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “Roberta: A robustly optimized BERT pretraining approach,”CoRR, vol. abs/1907.11692, 2019. [Online]. Available: http://arxiv.org/abs/1907.11692
arXiv 1907
Show all 26 references
-
[9]
A universal prompting strategy for extracting process model information from natural language text using large language models,
J. Neuberger, L. Ackermann, H. van der Aa, and S. Jablonski, “A universal prompting strategy for extracting process model information from natural language text using large language models,” inICCM. Springer, 2024, pp. 38–55
2024
-
[10]
Large language models can accomplish business process management tasks,
M. Grohs, L. Abb, N. Elsayed, and J.-R. Rehse, “Large language models can accomplish business process management tasks,” inBPM. Springer, 2023, pp. 453–465
2023
-
[11]
Process modeling with large language models,
H. Kourani, A. Berti, D. Schuster, and W. M. van der Aalst, “Process modeling with large language models,” inICBPMDS. Springer, 2024, pp. 229–244
2024
-
[12]
PET: an annotated dataset for process extraction from natural language text tasks,
P. Bellan, H. van der Aa, M. Dragoni, C. Ghidini, and S. P. Ponzetto, “PET: an annotated dataset for process extraction from natural language text tasks,” inBPM W orkshops 2022, vol. 460. Springer, 2022, pp. 315–321
2022
-
[13]
A comprehensive investigation of bpmn models generation from textual requirements—techniques, tools and trends,
B. Maqbool, F. Azam, M. W. Anwar, W. H. Butt, J. Zeb, I. Zafar, A. K. Nazir, and Z. Umair, “A comprehensive investigation of bpmn models generation from textual requirements—techniques, tools and trends,” vol. 514. Springer Verlag, 2018, pp. 543–557
2018
-
[14]
Information extraction,
R. Grishman, “Information extraction,”IEEE Intelligent Systems, vol. 30, no. 5, pp. 8–15, 2015
2015
-
[15]
Catboost: unbiased boosting with categorical features,
L. Prokhorenkova, G. Gusev, A. Vorobev, A. V. Dorogush, and A. Gulin, “Catboost: unbiased boosting with categorical features,” vol. 31. Curran Associates, Inc., 2018
2018
-
[16]
A survey on deep learning for named entity recognition,
J. Li, A. Sun, J. Han, and C. Li, “A survey on deep learning for named entity recognition,” IEEE T ransactions on Knowledge and Data Engineering, vol. 34, no. 1, pp. 50–70, 2022
2022
-
[17]
Conditional random fields: Probabilistic models for segmenting and labeling sequence data,
J. Lafferty, A. Mccallum, and F. Pereira, “Conditional random fields: Probabilistic models for segmenting and labeling sequence data,” 6 2001. [Online]. Available: https://api.semanticscholar.org/CorpusID:219683473
2001
-
[18]
How to fine-tune bert for text classification?
C. Sun, X. Qiu, Y. Xu, and X. Huang, “How to fine-tune bert for text classification?”
-
[19]
Conditional random fields: An introduction,
H. M. Wallach, “Conditional random fields: An introduction,” 2 2004
2004
-
[20]
Text chunking using transformation-based learning,
L. A. Ramshaw and M. P. Marcus, “Text chunking using transformation-based learning,” 1995
1995
-
[21]
Seven process modeling guidelines (7pmg),
J. Mendling, H. A. Reijers, and W. M. van der Aalst, “Seven process modeling guidelines (7pmg),” Information and Software T echnology, vol. 52, pp. 127–136, 2 2010
2010
-
[22]
Aconceptforgeneratingbusinessprocessmodels from natural language description,
K.Honkisz,K.Kluza,andP.Wiśniewski,“Aconceptforgeneratingbusinessprocessmodels from natural language description,” vol. 11061 LNAI. Springer Verlag, 2018, pp. 91–103
2018
-
[23]
Glove: Global vectors for word representation,
J. Pennington, R. Socher, and C. D. Manning, “Glove: Global vectors for word representation,” inEMNLP 2014, 2014, pp. 1532–1543
2014
-
[24]
Smote: Synthetic minority over-sampling technique,
N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer, “Smote: Synthetic minority over-sampling technique,”Journal of Artificial Intelligence Research, vol. 16, pp. 321–357, 2002
2002
-
[25]
How much language is enough? theoretical and practical use of the business process modeling notation,
M. z. Muehlen and J. Recker, “How much language is enough? theoretical and practical use of the business process modeling notation,” inAdvanced Information Systems Engineering. Springer, 2008, pp. 465–479
2008
-
[2020]
Available: https://arxiv.org/abs/1905.05583
[Online]. Available: https://arxiv.org/abs/1905.05583
1905 arXiv
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.