{"id":"30e66080-e7b5-4e9f-8c4b-060fe58c8ffd","arxiv_id":"2412.00023","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A benchmark of 20 business processes and 16 large language models finds Claude-3.5-Sonnet produces the highest-quality process models, and suggests that output optimization improves weaker models.","lead":"This paper tests 16 large language models on turning text descriptions of business processes into formal process models, using a 20-process benchmark with known correct answers. It offers a reusable evaluation setup for process modeling and evidence that letting an LLM critique and revise its own output can improve weaker models.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Loop cap in simulated logs (Sec 5.1.3) makes quality scores reward bounded-loop under-approximations; top-model gaps (0.93 vs 0.91) are at the scale this bias could flip.","rationale":"The paper's central claim has three observable parts: Claude-3.5-Sonnet is best, error-handling correlates with quality, and output-optimization helps weaker models. All three rest on the quality score defined in Section 5.1.3. The most load-bearing, least secure component is the evaluation metric itself. The reader flagged the simulated-log assumptions as a representativeness issue; the concern here is sharper: it threatens internal validity. Because the log contains one instance of every trace variant with at most two loop iterations, the reference standard is a truncated language. Correctly modeling an unbounded loop is penalized (precision drops), and a bounded re-implementation is rewarded (perfect score). This is not merely a matter of whether the benchmark generalizes to real processes; it is a question of whether Table 2 measures what it claims to measure even on the 20 benchmark processes. The fact that the ground-truth models score 0.98 while a bounded approximation could score 1.0 demonstrates the inconsistency. The paper has genuine strengths: the framework is described in detail, code and data are released, the use of POWL guarantees soundness, and the self-improvement experiments are honestly discussed with limitations. None of these, however, insulate the ranking from this metric bias. The proposed check is inexpensive: regenerate logs with a third loop iteration (or use alignments against the ground-truth model) and recompute the scores. If the ranking and margins persist, the concern is resolved; if not, the headline claim should be downgraded. This is addressable, so CONDITIONAL rather than REJECT is appropriate.","tokens_in":23587,"tokens_out":12262,"duration_ms":177806,"concrete_test":"Regenerate the event logs from the ground-truth POWL models with loop iterations capped at 3 (or compute conformance by alignment against the full ground-truth POWL model instead of a finite log), then recompute Table 2. Also report per-process standard deviations and a paired significance test between Claude-3.5-Sonnet and O1-Preview/O1-Mini. If the top position or the 0.01-0.02 margins do not survive, the ranking in Table 2 is an artifact of the 2-iteration log cap.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 5.1.3 simulates event logs from the ground-truth POWL models under two assumptions: equal decision probabilities and loops limited to at most two iterations. POWL loops are unbounded, so the log is a strict finite subset of the ground-truth language. The paper's own footnote 1 concedes that a perfect score is unattainable 'due to the presence of loops,' and the ground truth scores 0.98, not 1.0. The consequence is that the evaluation target is not the ground-truth process but its 1- and 2-iteration truncation. A generated model that realizes a loop as a bounded structure (e.g., do followed by a choice to repeat once more) covers every log trace and has no extra behavior, yielding fitness 1.0 and precision 1.0, a score of 1.0 that is strictly better than the ground truth. Such a model is semantically wrong (it cannot do a third iteration), yet the metric rewards it. Different LLMs generate structurally different models (Figures 3-6), and some may systematically under-approximate loops. The reported top-model differences in Table 2 (Claude-3.5-Sonnet 0.93 vs. O1-Mini 0.91) are exactly the 0.01-0.02 scale at which this bias could change the ordering. Thus the headline ranking may reflect fit to the truncated log rather than process-modeling quality.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper extends the authors' earlier LLM-based process modeling framework by adding a benchmark and self-improvement analysis. The benchmark consists of 20 business process descriptions paired with ground-truth POWL models and simulated event logs; the authors evaluate 16 LLMs by measuring how well each LLM-generated POWL model conforms to the simulated logs, using the harmonic mean of fitness and precision as a quality score. The main findings are that Claude-3.5-Sonnet achieves the highest average quality score (0.93), that error-handling performance is positively correlated with model quality, and that output self-optimization improves quality primarily for models with initially lower performance. The paper also introduces the ProMoAI tool and makes the evaluation data and results publicly available.","tokens_in":23896,"tokens_out":4861,"duration_ms":57805,"significance":"If the ranking and self-improvement findings hold, the paper provides a useful benchmark and methodology for evaluating LLMs on a structured modeling task, with a credible external ground truth in the form of simulated logs and objective conformance metrics. The use of a sound intermediate representation (POWL), the breadth of included LLMs, and the public availability of data and code are clear strengths. However, the loop-truncation bias in the simulated logs, the absence of uncertainty/statistical evidence for the headline comparisons, and the limited support for the output-optimization claim currently temper the conclusions. The benchmark size (20 processes, mostly hand-written by the authors) also limits external validity, but this is a limitation rather than a fatal flaw.","major_comments":[{"comment":"The simulation caps loop iterations at two, and footnote 1 concedes that a perfect score is unattainable 'due to the presence of loops,' with the ground truth itself scoring only 0.98. This means the evaluation target is the finite 1- and 2-iteration truncation of each process, not the ground-truth process itself. A generated POWL model that represents a loop as a bounded structure (e.g., one do execution followed by an xor between repeating once more and exiting) covers every trace in the simulated log and adds no extra behavior, so it can receive fitness 1.0 and precision 1.0, strictly beating the correct unbounded-loop model. Since the reported top-model differences in Table 2 (e.g., Claude-3.5-Sonnet 0.93 versus O1-Mini 0.91) are at the 0.01-0.02 scale, the headline ranking may reflect fit to the truncated log rather than true process-modeling quality. Please quantify the number of loop-bearing processes, report per-process scores, rerun the evaluation with logs allowing more loop iterations or with a behavioral equivalence/distance measure that is invariant to loop truncation, and show how the Table 2 ordering changes.","section":"5.1.3"},{"comment":"The claim of 'a notable correlation between error-handling performance and model quality' is not supported by any correlation coefficient, significance test, or scatterplot; the text moves directly from Tables 1 and 2, which report only aggregated averages over 20 processes, to a causal-sounding conclusion. Please compute a per-process correlation (e.g., Spearman) between iterations/errors and quality scores, report its value and confidence interval, and limit the claim accordingly. Similarly, the statement that 'consistent quality trends within similar model groups' are observed needs variance information: Table 2 reports average scores without any per-process spread, and the self-evaluation section shows that individual runs vary (e.g., 0.86-0.88 for Gemini-1.5-Pro-002), so the averages alone do not establish within-group consistency.","section":"5.2.2"},{"comment":"Table 6 is the only quantitative evidence for the paper's conclusion that output optimization is 'promising,' but it reports only averages, maximum improvement, and maximum decline for n=20, with no confidence interval or paired test. GPT-4o's maximum improvement of +0.84 strongly suggests that a single large outlier drives its average gain of +0.05, and the text does not state how many of the 20 cases improved, worsened, or stayed the same. Please report the full per-case distribution of quality differences, include a paired permutation test or bootstrap confidence interval, and clarify whether the 'without error refinement loop' setting makes these results comparable to the Section 5 experiments.","section":"6.3"}],"minor_comments":[{"comment":"There is a typo in the sentence 'Currently ProMoAI in integrated with three LLM providers'; it should read 'is integrated.'","section":"4"},{"comment":"The sentence claiming that 'each simulated event log includes one instance of every unique trace variant possible' should be reconciled with the fact that the logs are generated probabilistically; if only one instance per variant is retained, the equal-probability assumption is unnecessary and should be stated as a trace-coverage choice rather than a sampling distribution.","section":"5.1.3"},{"comment":"The 'Subset Match' column in Table 4 needs a precise definition; it is unclear whether a model selected by the LLM counts as a subset match when it is tied within the 0.02 buffer, and how ties are broken.","section":"6.1"},{"comment":"Table 2 would be much more informative if it reported per-process quality scores or at least standard deviations/confidence intervals, especially because the paper's own Section 6.1 demonstrates nontrivial run-to-run variability for at least one model.","section":"5.2.2"},{"comment":"The benchmark's 20 processes, 18 of which were created by the authors, are a useful starting point but should be discussed more explicitly as a limitation regarding generalizability to real-world business process modeling tasks.","section":"5.1.2"},{"comment":"The GitHub link is given as a plain URL; please provide a versioned archive link or DOI so that the exact benchmark artifacts used in this paper remain accessible.","section":"5"}],"recommendation":"major_revision","confidential_remarks":"The loop-truncation issue identified in Section 5.1.3 is the most serious technical concern: it could plausibly change the top-model ordering, since the reported gaps are the same order of magnitude as the bias. The authors' extensive use of their own POWL and ProMoAI is a reasonable tooling choice and is transparently cited, not a circularity problem. The paper fits the scope of cs.DB and can be made publishable with a focused revision that re-analyzes the benchmark under a more loop-robust evaluation and adds appropriate statistical support for the headline claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. The paper delivers a genuinely useful benchmark for LLM-based process modeling — 20 processes with ground-truth POWL models, simulated logs, 16 LLMs scored via conformance checking — and the empirical setup is mostly careful. But the evaluation metric has a loop bias that could change the top of the leaderboard, and the headline ranking should be treated as provisional until it is addressed.\n\nWhat's new: the 20-process benchmark itself, the evaluation of 16 LLMs from five vendors, and the comparison of three self-improvement strategies (self-evaluation, input optimization, output optimization). The benchmark and code are on GitHub, which is a real plus. The conformance-checking methodology is sensible: fitness and precision against simulated logs give an external yardstick, not the LLM's own judgment. The authors also show good instincts in their limitations section — footnote 1 concedes that a perfect score is unattainable because of loops, and Section 6.1 explicitly notes that the self-evaluation findings rest on two LLMs. That honesty is worth crediting.\n\nThe soft spot is load-bearing. Section 5.1.3 simulates logs with two assumptions: equal decision probabilities and loops capped at two iterations. POWL loops are unbounded, so the log is a strict finite subset of the ground-truth language. The consequence is that the evaluation target is not the process; it is the truncated log. A generated model that realizes a loop as a bounded structure covers every trace in the log and has no extra behavior, so it scores fitness 1.0 and precision 1.0 — a perfect 1.0, strictly better than the ground truth's own 0.98. That model is semantically wrong, because it cannot do a third iteration, yet the metric rewards it. Different LLMs produce structurally different models, so this bias can plausibly flip the reported ordering: the top gap (Claude-3.5-Sonnet 0.93 vs O1-Mini 0.91) is exactly the 0.01–0.02 scale at which this effect could matter.\n\nTwo smaller issues: the 'notable correlation' between error-handling performance and quality in Section 5.2.2 is asserted without a coefficient or significance test, and Table 2 reports averages without confidence intervals. Both are addressable and should be fixed in revision.\n\nWho this is for: the BPM and process mining communities, and anyone selecting an LLM for process modeling work. The benchmark is worth having, but the ranking should be read as provisional. This paper deserves a serious referee; the loop-bias flaw is real but fixable — run a sensitivity analysis with a larger loop cap or a different simulation strategy, report variance, and soften the generalizability claims.\n\nI'd accept it for peer review and push for a revision that addresses the loop bias.","headline":"Useful benchmark and careful empirical work, but the loop cap in the simulated logs could flip the headline ranking; fixable, and worth refereeing.","tokens_in":24422,"tokens_out":2802,"would_cite":true,"duration_ms":25964,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Claude-3.5-Sonnet achieves the highest average quality score, 0.93, on a 20-process LLM business process modeling benchmark.","keywords":["business process modeling","large language models","POWL","conformance checking","benchmark","self-improvement","process mining","LLM evaluation"],"falsifier":"Recompute the benchmark's quality scores with event logs generated under unequal decision probabilities (for example, 90/10 at each choice) or loops allowed to run more than twice; if the model ranking and the error-handling/quality correlation change materially, the reported conclusions are artifacts of the simulation assumptions rather than stable properties of the LLMs.","tokens_in":23371,"feed_emoji":"🧩","tokens_out":5793,"duration_ms":56567,"temperature":0.7,"pith_summary":"This paper sets out to show that large language models can turn plain-English business process descriptions into formally sound process models, and that this capability can be measured, compared, and improved. The authors extend their earlier LLM-based modeling framework, in which a model writes Python code that builds a process model in the Partially Ordered Workflow Language (POWL), and they benchmark 16 LLMs on 20 hand-written process descriptions with ground-truth models. Quality is scored automatically by conformance checking: each generated model is compared against event logs simulated from the ground truth, and the harmonic mean of fitness and precision becomes the quality score. The paper reports that Claude-3.5-Sonnet scores highest at 0.93, close to the ground-truth average of 0.98, that fewer error-correction iterations correlate with higher quality, and that prompting an LLM to critically revise its own output helps most for models that start out weaker. The self-evaluation experiments, the paper notes, use only two LLMs and are not claimed to generalize.","feed_headline":"Claude-3.5-Sonnet tops 16-LLM process-modeling benchmark","feed_subtitle":"LLMs that fix errors faster produce better process models, and output self-optimization helps weaker models most.","key_machinery":"The load-bearing object is POWL, the Partially Ordered Workflow Language, an intermediate process representation whose hierarchical structure guarantees soundness by construction, so any model the LLM builds is a valid starting point for BPMN or Petri net export. Around it, the framework wraps prompt engineering (role prompting, knowledge injection, few-shot examples, negative prompting), a restricted set of Python ModelGenerator functions the LLM is allowed to call, and an iterative error-handling loop that distinguishes critical errors from adjustable ones. The evaluation machinery is conformance checking: event logs are simulated from each ground-truth POWL model, the generated model is replayed against those logs, and the harmonic mean of fitness and precision gives a single quality score in [0,1].","core_discovery":"On the paper's own terms, the central claim is that within this automated process-modeling framework, LLMs can produce high-quality executable process models from natural-language descriptions, and the differences between models are systematic rather than random. Claude-3.5-Sonnet achieves the highest average quality score (0.93), followed by O1-Preview (0.92) and O1-Mini (0.91), with the ground-truth models at 0.98. The paper also claims there is a notable correlation between error-handling performance and model quality: models that generate valid code in fewer iterations tend to score higher, and it finds stable average quality within model families, so speed-optimized variants do not necessarily sacrifice quality. For self-improvement, the paper claims that output self-optimization, guided by a deliberately restrictive prompt, improves average quality, with the largest gains for lower-performing models (GPT-4o, Gemini-1.5-Flash-002), while self-evaluation and input optimization are unreliable in this setting.","pith_inferences":["Because the simulated logs assume equal decision probabilities and at most two loop iterations, the scores reward models that reproduce every structural variant equally; a deployment with skewed branch frequencies or longer loops could shift both fitness and precision.","The 20-process benchmark is small and hand-written; an obvious testable extension is a larger, real-world corpus with logs taken from actual executions, which would tell whether the ranking holds outside hand-crafted descriptions.","The self-evaluation result that a weaker model selects the wrong candidate in 16-17 of 20 cases hints that weak models cannot reliably judge their own work, so self-selection should probably be reserved for stronger models.","A natural next experiment is to combine output self-optimization with the adversarial error-handling loop and measure whether the gains persist when the model is also under pressure to fix code."],"forward_implications":["If the benchmark is representative, organizations automating process modeling should choose models that generate correct code on the first attempt and repair errors quickly, because that behavior correlates with final model quality.","The correlation between iteration count and quality means improving code-generation and self-repair ability is a direct lever for improving process-model quality.","Output self-optimization with a restrictive 'only change if genuinely beneficial' prompt is a low-cost way to lift weak models, at the price of occasional small quality declines.","Models in the same family cluster in quality, so family-level comparisons are more reliable than single-run comparisons when choosing among LLMs.","Conformance checking against simulated logs gives a fully automatable scoring pipeline that can be rerun as new LLMs appear."],"supporting_citations":[{"why":"Supplies the earlier LLM-based process modeling framework that this paper extends with a benchmark and self-improvement analysis.","marker":"[8]"},{"why":"Defines POWL, the intermediate representation whose soundness guarantees make the framework's quality claims possible.","marker":"[9]"},{"why":"Provides the conformance checking methodology used to compare generated models against ground-truth event logs.","marker":"[10]"},{"why":"Defines the fitness metric used as one component of the harmonic-mean quality score.","marker":"[41]"},{"why":"Defines the precision metric used as the other component of the harmonic-mean quality score.","marker":"[42]"},{"why":"Provides the process mining library used to compute the conformance metrics.","marker":"[43]"},{"why":"Supplies source process descriptions used in few-shot learning and one of the benchmark processes.","marker":"[38]"}],"fun_headline_variants":["LLMs can model business processes, Claude-3.5-Sonnet best","Claude-3.5-Sonnet leads 16-LLM process modeling test","Error-fixing speed predicts LLM process model quality","Output self-optimization boosts weak LLMs in BPM","16 LLMs benchmarked for business process modeling"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the 20 hand-written process descriptions and their ground-truth POWL models, together with event logs simulated under equal decision probabilities and two-iteration loop limits, are representative enough of real business process modeling to support general conclusions about which LLMs are best.","fun_headline_variants_meta":{"raw":{"variants":["LLMs can model business processes, Claude-3.5-Sonnet best","Claude-3.5-Sonnet leads 16-LLM process modeling test","Error-fixing speed predicts LLM process model quality","Output self-optimization boosts weak LLMs in BPM","16 LLMs benchmarked for business process modeling"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000577,"raw_usage":{"total_tokens":2720,"prompt_tokens":939,"completion_tokens":1781,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":555,"completion_tokens_details":{"reasoning_tokens":1690}},"tokens_in":555,"tokens_out":1781,"duration_ms":12794,"temperature":1.0,"reasoning_tokens":1690,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T19:01:14.936952+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the benchmark's quality scores with event logs generated under unequal decision probabilities (for example, 90/10 at each choice) or loops allowed to run more than twice; if the model ranking and the error-handling/quality correlation change materially, the reported conclusions are artifacts of the simulation assumptions rather than stable properties of the LLMs.","supporting_citations":[{"cited_title":"In: Francescomarino, C.D., Burattin, A., Janiesch, C., Sadiq, S","cited_arxiv_id":null,"evidence_quote":"Defines POWL, the intermediate representation whose soundness guarantees make the framework's quality claims possible."},{"cited_title":"In: Betz, S","cited_arxiv_id":null,"evidence_quote":"Provides the conformance checking methodology used to compare generated models against ground-truth event logs."},{"cited_title":"In: van der Aalst, W.M.P., Bergenthum, R., Car- mona, J","cited_arxiv_id":null,"evidence_quote":"Defines the fitness metric used as one component of the harmonic-mean quality score."},{"cited_title":"In: Hull, R., Mendling, J., Tai, S","cited_arxiv_id":null,"evidence_quote":"Defines the precision metric used as the other component of the harmonic-mean quality score."}],"review_version":1}