The reviewed record of science sign in
Pith

arxiv: 2607.06175 · v1 · pith:DUP4VEOD · submitted 2026-07-07 · cs.CL · cs.AI· cs.LG

Improving LLM-Generated Process Model Quality Through Reinforcement Learning: The Role of Reward Function Design

Alexander Rombach , Chantale Lauer , Nijat Mehdiyev This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel 2026-07-08 14:15 UTCglm-5.2pith:DUP4VEODrecord.jsonopen to challenge →

classification cs.CL cs.AIcs.LG
keywords qualitymodelrewardprocessdesignfunctionarchitectureautomated
0
0 comments X

The pith

Equal reward weights beat targeted weights in multi-dimensional RL

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper investigates how to design reward functions when quality is multi-dimensional, using the generation of BPMN business process models from natural language as a testbed. The authors train two LLM families (Llama 3.1 8B and Qwen 2.5 14B) under 48 configurations using Group Sequence Policy Optimization (GSPO), with rewards derived from 38 automated metrics spanning syntactic correctness, pragmatic comprehensibility, and semantic fidelity. The central finding is that equal weighting of quality dimensions consistently outperforms targeted weighting: emphasizing a specific dimension in the reward function not only fails to improve that dimension but can cause the model to collapse into a narrow, low-quality output mode. This effect is as large as the decision to apply reinforcement learning at all. The paper also finds that design choices interact strongly with model architecture: an invalidity penalty is essential for one model but irrelevant for the other, and supervised fine-tuning initialization is indispensable for one architecture but counterproductive for another. The authors argue that reward composition is a primary determinant of optimization outcomes, not a secondary tuning detail, and that these findings generalize to any structured generation task where quality is assessed along multiple automated dimensions.

Core claim

The paper's core discovery is that in multi-dimensional reward optimization for structured generation, equal weighting of quality dimensions is not merely a safe default but actively superior to targeted weighting. Targeted weighting destabilizes the group-relative advantage signal that GSPO relies on: when one dimension dominates the reward, candidate rankings are driven by variation along a single axis, which is often insufficient to produce informative gradients, causing the policy to drift or collapse. The authors identify a recurring failure mode they call policy collapse, where the model converges to a narrow, low-diversity output template regardless of input. This collapse appears in三

What carries the argument

The central object carrying the argument is the composite reward function R(w, p; y) = w_syn·r_syn + w_pra·r_pra + w_sem·r_sem for valid outputs, with penalty p for invalid outputs. The six reward configurations (R_avg through R_5) systematically vary the weight vector w and the penalty p, allowing the authors to isolate the effects of dimensional weighting, penalty structure, and implementation pathway. GSPO's group-relative normalization is the mechanism through which reward composition exerts its effects: by ranking candidates within each group rather than against an absolute baseline, the structure of the reward function directly shapes which quality distinctions are detectable and thus,

If this is right

  • Practitioners applying RL to any structured generation task with multiple quality metrics should default to equal reward weighting unless they have specific evidence that targeted weighting helps their model and task.
  • The invalidity penalty should be treated not only as a validity enforcement mechanism but as an implicit diversity regularizer whose effect depends on the base model's prior validity behavior.
  • The standard SFT-then-RL pipeline should not be assumed universally optimal; a pilot evaluation comparing SFT-initialized and untrained bases may reveal that skipping SFT yields better results for some architectures.
  • Reward function design choices should be co-tuned with model architecture and initialization strategy rather than transferred across model families.
  • The finding that reward composition effects can be as large as the decision to apply RL suggests that future work on RL for structured generation should report and ablate reward configurations as systematically as the authors do here.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The failure of targeted weighting may reflect a general property of group-relative RL methods: when the reward signal is dominated by a single dimension, the within-group ranking becomes uninformative if candidates show little variation on that dimension. This could extend to other group-relative methods beyond GSPO.
  • The observation that pragmatic quality is the most easily optimized dimension (because it rewards simpler outputs) suggests that reward accessibility is partly determined by the direction of the gradient in policy space: dimensions that can be improved by moving toward simpler, more compact outputs are more accessible than those requiring specific structural changes.
  • The template collapse observed under the no-penalty condition for Qwen resembles reward hacking, but with the distinction that the template achieves high reward under the defined objective. This suggests that multi-dimensional reward functions may need explicit diversity-preserving mechanisms beyond penalty terms, particularly when the model's validity behavior is variable.
  • The architecture-dependent effects (penalty essential for one model but not the other, SFT indispensable for one but counterproductive for the other) suggest that the interaction between reward design and model architecture may be governed by the model's prior disposition toward the target task, which could potentially be assessed before training through zero-shot evaluation.

Load-bearing premise

The evaluation uses a single generation per sample across 105 held-out instances with no repeated runs, meaning within-configuration variance cannot be estimated. This is load-bearing because the statistical significance claims rest on paired permutation tests that assume each single observed score is a reliable point estimate, yet the paper reports dramatic distributional changes (such as a syntactic standard deviation of 0.002 for one Qwen configuration) that could be arte

What would settle it

If repeated generation runs revealed that the within-configuration variance is large relative to the between-configuration differences reported, particularly for the targeted-weighting failure and the penalty effect on Qwen, the central claims about equal weighting superiority and penalty importance would be weakened.

Figures

Figures reproduced from arXiv: 2607.06175 by Alexander Rombach, Chantale Lauer, Nijat Mehdiyev.

Figure 1
Figure 1. Figure 1: RQ1: Distributional comparison of quality scores between SFT-only baseline and GSPO configurations (𝑅avg, 𝑅1 , 𝑅2 ). Top row: Llama 3.1 8B. Bottom row: Qwen 2.5 14B. Violins show the kernel density estimate; embedded box plots indicate median and interquartile range; diamond markers denote the mean. All GSPO configurations use the SFT base without grammar enforcement. toward pragmatic quality (e.g., 𝑅2 wit… view at source ↗
Figure 2
Figure 2. Figure 2: RQ2: Distributional comparison of quality scores across reward weighting schemes. Top row: Llama 3.1 8B. Bottom row: Qwen 2.5 14B. 𝑅1 (equal) maintains broad, high distributions, while weighted variants (𝑅3–𝑅5 ) produce narrower distributions shifted toward lower values, most dramatically for Llama’s 𝑅3 , which collapses to a tight cluster at 0.662 despite emphasizing syntactic quality. Syntactic Pragmatic… view at source ↗
Figure 3
Figure 3. Figure 3: Radar charts comparing quality profiles of reward weighting schemes (SFT base, no grammar). For Llama (left), 𝑅1 (equal) produces the largest balanced triangle. For Qwen (right), weighted variants extend the pragmatic axis while compressing the syntactic axis, regardless of which dimension receives the highest weight. : Preprint submitted to Elsevier Page 15 of 21 [PITH_FULL_IMAGE:figures/full_fig_p015_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: RQ2: Penalty effect on Qwen 2.5 14B quality distributions (𝑅1 : 𝑝=−1 vs. 𝑅2 : 𝑝=0, SFT base, no grammar). Without the penalty (𝑅2 , red), the syntactic distribution collapses to a near-uniform band at 0.750 (std=0.002), while the pragmatic distribution forms a razor-thin spike at 0.964. The penalty (𝑅1 , dark) preserves distributional diversity across all dimensions. For Llama (not shown), the two conditio… view at source ↗
Figure 5
Figure 5. Figure 5: RQ3: Effect of base model initialization on RL outcomes under 𝑅1 for Llama 3.1 8B (no grammar). The SFT-initialized base (dark) produces high syntactic quality (0.926), while the BPM-unadapted base (red), i.e., the original instruction-tuned model without BPM-specific SFT, collapses to 0.664, a gap of 0.262, one of the two largest effects in this study. Pragmatic quality shows a smaller but significant gap… view at source ↗
read the original abstract

Large language models (LLMs) can generate BPMN process models from natural-language descriptions, yet supervised fine-tuning (SFT) limits their output quality to the patterns present in the training data. Reinforcement learning (RL) can optimize beyond this ceiling using external quality measures, but how the reward function should be designed when quality is multi-dimensional remains unexplored. We present a systematic investigation of reward function design for RL-based process model generation, training two LLM families (Llama~3.1 8B, Qwen~2.5 14B) under 48 configurations using Group Sequence Policy Optimization with rewards derived from an automated evaluation framework comprising 38 metrics across syntactic, pragmatic, and semantic quality. Three findings emerge. First, RL significantly improves pragmatic and syntactic quality while preserving semantic fidelity, reducing output variability by more than sixfold. Second, equal reward weighting consistently outperforms targeted weighting: emphasizing a specific dimension fails to improve it and can collapse the model into a low-quality mode. Third, design choices interact with model architecture in non-trivial ways: the invalidity penalty is essential for one model but irrelevant for the other, and SFT initialization is indispensable for one architecture but counterproductive for another. These results demonstrate that reward composition is a primary determinant of optimization outcomes, with effects as large as the decision to apply RL itself. The findings generalize to any structured generation task where quality is assessed along multiple automated dimensions. We release our implementation and experimental code at https://github.com/chlauer99/RL_for_process_modeling.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

4 major / 8 minor

Summary. This paper investigates reward function design for reinforcement learning (RL) applied to LLM-based BPMN process model generation. Using Group Sequence Policy Optimization (GSPO) with rewards derived from the BEF4LLM quality framework (38 metrics across syntactic, pragmatic, and semantic dimensions), the authors train Llama 3.1 8B and Qwen 2.5 14B under 48 configurations varying reward weighting, invalidity penalty, base model initialization, and grammar-constrained decoding. Three main findings emerge: (1) GSPO significantly improves pragmatic and syntactic quality while preserving semantic fidelity, with dramatic variability reduction; (2) equal reward weighting consistently outperforms targeted (3:1:1) weighting, which can cause policy collapse; (3) design choices interact with model architecture in non-trivial ways. The experimental design is systematic, with paired permutation tests and Bonferroni correction. The paper releases code and experimental artifacts.

Significance. The paper addresses a genuine gap: systematic investigation of multi-dimensional reward function design for structured generation, where most prior RLVR work uses single-dimensional rewards. The finding that equal weighting outperforms targeted weighting is counter-intuitive and practically useful. The inclusion of mathematically equivalent configurations (R_avg vs R1) as a natural experiment on training variance is a methodological strength, as is the release of reproducible code. The cross-architecture analysis (two model families, two initialization strategies) provides useful generality evidence. The work is relevant beyond BPMN to any structured generation task with multi-dimensional quality assessment.

major comments (4)
  1. §4.4 and Table 5: The R_avg vs R1 natural experiment reveals substantial uncontrolled training variance. These configurations are mathematically equivalent (both compute the unweighted mean of three quality scores with p=-1), yet for Llama they produce significantly different syntactic outcomes (Δ=0.059, p_adj<2×10^-5, favoring R1: 0.926 vs 0.869). The authors acknowledge this in §6.3 but interpret it cautiously. However, several moderate effects that support the central 'equal weighting consistently outperforms' claim are within 0.3-0.5× of this variance estimate: R1 vs R5 for Llama semantic (Δ=0.027, Table 6), R1 vs R4 for Qwen pragmatic (Δ=0.023), R1 vs R5 for Qwen semantic (Δ=0.019). The claim of 'consistently outperforms' and the abstract's statement that 'reward composition effects are as large as the decision to apply RL itself' rest on single-run pairwise comparisons where some效应
  2. §4.4: The single generation per evaluation sample (no repeated runs) is a load-bearing limitation. The paper states this 'limits the ability to estimate within-configuration variance' but the statistical significance claims via paired permutation tests assume the single observed score per sample is a reliable point estimate. Given that the R_avg-R1 gap demonstrates training-run variance of ~0.06, the authors should either (a) explicitly bound which effect sizes are robust given this variance floor, or (b) temper the 'consistently outperforms' language for effects near the noise floor. The largest effects (e.g., R1 vs R3 for Llama syntactic: Δ=0.264) are clearly robust, but the blanket claim needs qualification.
  3. §3.3 and §3.4: The reward functions are linear combinations of BEF4LLM metrics, and evaluation uses the same BEF4LLM framework. The authors note this is not a logical circularity (the reward is a weighted composite; evaluation reports per-dimension scores). This is correct, but the concern is deeper: the reward signal and evaluation metric share the same underlying 38 metrics, meaning RL optimization directly improves the measured quantities. The paper does not discuss whether improvements on BEF4LLM metrics correspond to genuine process model quality improvements as perceived by human modelers, or whether RL could exploit metric artifacts. The Qwen R2 template collapse (§5.2.2, syntactic std=0.002) is actually evidence of such metric exploitation. The authors should discuss this risk more explicitly and note that human evaluation would strengthen the claims.
  4. Table 5: Several configurations have substantially different validity counts (e.g., Llama SFT-only n=36 vs R1 n=102; Llama R5 n=38 vs R1 n=102). The paired permutation tests are conditioned on joint validity (§4.5), meaning comparisons between configurations with very different validity rates are conducted on restricted, potentially biased subsets. The paper notes this but does not report how many samples are in the joint validity set for each comparison. For the R1 vs R5 comparison (Llama), R5 has only 38 valid outputs — the joint set could be as small as ~35, substantially reducing statistical power. The authors should report joint validity counts for each pairwise comparison in Table 6.
minor comments (8)
  1. §3.4, Table 2: The distinction between R_avg ('aggregated scalar') and R1 ('equal, decomposed') is unclear. The text states they are 'mathematically equivalent' but 'correspond to separate implementation pipelines,' but the implementation difference is not described. What specifically differs in the reward computation pipeline?
  2. §5.2.1: The term 'policy collapse' is introduced informally. While the phenomenon is well-described, a brief formal characterization (e.g., convergence to a low-entropy output distribution) would strengthen the analysis.
  3. Figure 1: The violin plots for SFT-only Llama show n=36 for syntactic and pragmatic but n=35 for semantic. This discrepancy is unexplained. Is this due to additional translation failures for semantic evaluation?
  4. §6.4: The grammar-constrained decoding analysis is described as 'supplementary' and summarized briefly. Given that it is one of the four experimental factors in the 48-configuration matrix, the results should be reported more fully, at least in an appendix or supplementary material.
  5. §4.1: The training set (1,552 samples) is described as a mix of German and English, assembled from 'multiple publicly available BPMN datasets, supplemented with synthetically generated samples.' The proportion of synthetic data and its potential impact on generalization are not discussed.
  6. Table 6: The R_avg vs R1 comparison is not included in the consolidated pairwise tests table, despite being discussed in §6.3. It should be added for completeness.
  7. §3.5: The choice of 3:1:1 weighting ratio is justified as balancing 'experimental discriminability with training stability,' but no sensitivity analysis is provided. The authors acknowledge this in §6.5 but it weakens the claim that targeted weighting 'consistently fails' — gentler ratios might succeed.
  8. Abstract: 'reducing output variability by more than sixfold' — this refers specifically to pragmatic quality standard deviation for Llama (0.085 to 0.013). The abstract should clarify this is dimension-specific rather than a general variability reduction.

Simulated Author's Rebuttal

4 responses · 2 unresolved

We thank the referee for a careful and substantive review. The comments identify genuine limitations in our statistical claims and evaluation methodology. We agree with the core of each comment and will revise accordingly. Below we address each point in turn.

read point-by-point responses

  1. Referee: §4.4 and Table 5: The R_avg vs R1 natural experiment reveals substantial uncontrolled training variance. Several moderate effects supporting 'equal weighting consistently outperforms' are within 0.3-0.5× of this variance estimate. The claim of 'consistently outperforms' and the abstract's statement rest on single-run pairwise comparisons.

    Authors: The referee is correct that the R_avg–R1 gap (Δ=0.059 for Llama syntactic) provides a lower bound on training-run variance, and that several effects supporting the 'consistently outperforms' claim are of comparable or smaller magnitude. We have re-examined each pairwise comparison in Table 6 against this variance floor. The largest effects are clearly robust: R1 vs R3 for Llama syntactic (Δ=0.264), R1 vs R4 for Llama pragmatic (Δ=0.120), R1 vs R5 for Llama syntactic (Δ=0.128), and the R1-vs-SFT-only comparisons (Δ≥0.092 for syntactic, Δ≥0.116 for pragmatic). These exceed the variance estimate by a factor of 2–4× and are unlikely to be artifacts. However, the referee correctly identifies effects near the noise floor: R1 vs R5 for Llama semantic (Δ=0.027), R1 vs R4 for Qwen pragmatic (Δ=0.023), and R1 vs R5 for Qwen semantic (Δ=0.019) are all within 0.3–0.5× of the R_avg–R1 gap. We will revise the manuscript to explicitly classify effects as robust (Δ ≥ 2× the R_avg–R1 variance estimate) versus tentative (Δ < 2×), and to qualify the 'consistently outperforms' language accordingly. The abstract's claim that 'reward composition effects are as large as the decision to apply RL itself' is supported by the robust effects (e.g., R3 collapse at Δ=0.264 vs R1-vs-SFT Δ=0.092), but we will add a qualifier noting that this holds for the largest effects, not all pairwise comparisons. revision: yes

  2. Referee: §4.4: Single generation per evaluation sample is a load-bearing limitation. The authors should either (a) explicitly bound which effect sizes are robust given this variance floor, or (b) temper the 'consistently outperforms' language for effects near the noise floor.

    Authors: We agree. The single-generation protocol means we cannot estimate within-configuration variance from repeated sampling, and the R_avg–R1 comparison is our only direct evidence of training-run variance. We will take approach (a): we will add a subsection to Section 6.3 that explicitly establishes the R_avg–R1 gap as a variance floor, classifies each Table 6 comparison as robust or tentative relative to this floor, and states which specific claims are robust versus which require confirmation through multi-seed replication. We will also temper the blanket 'consistently outperforms' language in the abstract, Section 5.2.1 summary, and conclusion to 'consistently outperforms for the largest effects; smaller effects require multi-seed confirmation.' We acknowledge that multi-seed training (at minimum 3 seeds per configuration) would be the proper remedy, and we will note this as the most important future work item. Given the 48-configuration experimental matrix, this was not computationally feasible for the current study, but we recognize it as essential for confirming the smaller effects. revision: yes

  3. Referee: §3.3 and §3.4: The reward signal and evaluation metric share the same underlying 38 metrics, meaning RL optimization directly improves the measured quantities. The paper does not discuss whether improvements correspond to genuine process model quality improvements as perceived by human modelers, or whether RL could exploit metric artifacts. The Qwen R2 template collapse is evidence of such exploitation.

    Authors: This is a fair and important concern. We agree that the shared metric basis between reward and evaluation constitutes a form of evaluation–reward coupling that is stronger than typical RLVR settings (where, e.g., training rewards and test metrics are the same pass/fail signal, but the concern is less salient because the metric is unambiguous). The Qwen R2 template collapse (syntactic std=0.002, pragmatic spike at 0.964) is indeed a concrete example of metric exploitation: the model discovered a degenerate structure that scores well on the metrics while being useless as a process model. We will add a dedicated discussion in Section 6.2 that (1) explicitly acknowledges the reward–evaluation coupling as a limitation, (2) uses the R2 collapse as a case study of metric exploitation, (3) notes that the equal-weight R1 configuration's preserved output diversity (visible in the violin plots) provides indirect evidence against exploitation for the main results, but is not a substitute for independent evaluation, and (4) states that human evaluation would substantially strengthen the claims and should be conducted in future work. We will also note that the BEF4LLM metrics are rule-based and inspectable rather than learned proxies, which limits (but does not eliminate) the risk of exploitation compared to learned reward models. We cannot fully resolve this concern without human evaluation, which is beyond the scope of the current revision. revision: partial

  4. Referee: Table 5: Several configurations have substantially different validity counts. The paired permutation tests are conditioned on joint validity, meaning comparisons between configurations with very different validity rates are conducted on restricted, potentially biased subsets. The authors should report joint validity counts for each pairwise comparison in Table 6.

    Authors: The referee is correct that joint validity counts are essential for interpreting the statistical tests, particularly for comparisons involving configurations with very different validity rates (e.g., R1 n=102 vs R5 n=38 for Llama SFT). We will add a column to Table 6 reporting the joint validity count (n_joint) for each pairwise comparison. We will also add a brief discussion of the potential selection bias introduced by conditioning on joint validity: when one configuration has much lower validity, the joint set is restricted to the 'easier' samples where both models succeed, which may attenuate or amplify observed differences. For the R1 vs R5 (Llama) comparison specifically, we expect n_joint to be approximately 35–38 (the intersection of 102 and 38 valid outputs), which substantially reduces statistical power and means the Δ=0.027 semantic difference should be interpreted very cautiously—consistent with our response to the first comment. We will flag comparisons where n_joint is substantially smaller than either configuration's individual validity count as having reduced reliability. revision: yes

standing simulated objections not resolved
  • The reward–evaluation coupling concern (Comment 3) cannot be fully resolved without human evaluation of generated process models, which is beyond the scope of this revision. We can acknowledge the limitation and provide indirect evidence (output diversity under R1, inspectability of rule-based metrics), but we cannot provide direct evidence that BEF4LLM improvements correspond to human-perceived quality improvements.
  • Multi-seed training to establish within-configuration variance (Comments 1 and 2) is the proper remedy for the training-variance concern, but is computationally infeasible within the revision timeframe given the 48-configuration matrix. We will bound effect sizes using the R_avg–R1 variance floor as an interim measure.

Circularity Check

0 steps flagged

No significant circularity: reward and evaluation share the BEF4LLM framework, but the paper tests how different reward compositions affect outcomes rather than claiming high reward implies high quality by definition.

full rationale

The paper's reward functions (Eq. 12, Table 2) are linear combinations of three BEF4LLM quality scores (r_syn, r_pra, r_sem) with varying weights, and evaluation uses the same BEF4LLM framework. This creates a surface-level concern that the reward signal and evaluation metric are the same framework. However, this is not circular in the logical sense the analyzer targets. The paper's central claim is that equal reward weighting outperforms targeted weighting — i.e., it is testing how different compositions of the reward affect downstream model quality, not claiming that high reward scores predict high evaluation scores (which would be trivially true by construction). The key empirical findings are genuinely non-trivial: emphasizing a dimension in the reward can *decrease* that dimension's evaluation score (e.g., R3 syntax-weighted produces the *lowest* syntactic quality for Llama, 0.662 vs R1's 0.926). If the result were circular — if the reward simply optimized the evaluation metric by construction — targeted weighting would improve the targeted dimension, not collapse it. The paper also includes R_avg vs R1 as a natural experiment on implementation variance, and tests architecture-dependent effects (penalty, SFT initialization) that cannot be reduced to the reward-evaluation overlap. The BEF4LLM framework [27] is cited for metric definitions but is not invoked as a uniqueness theorem or load-bearing mathematical fact that forbids alternatives. The self-citation to [27] (Lauer et al., with overlapping author Rombach) provides the evaluation framework, but the paper's contributions are about how different reward compositions interact with RL training dynamics, which is an empirical question not determined by the framework definition. One minor concern: the paper does not evaluate against any external benchmark independent of BEF4LLM, so all quality claims are relative to that framework. But this is a scope limitation, not circularity — the findings about reward composition effects (weighting failure, penalty as diversity regularizer, architecture-dependent initialization) are substantive empirical results that do not reduce to the framework's definitions.

Axiom & Free-Parameter Ledger

10 free parameters · 5 axioms · 0 invented entities

The paper introduces no new entities, particles, forces, or postulated objects. All components (GSPO, BEF4LLM, LoRA, Llama, Qwen) are from prior work. The reward functions are linear combinations of existing metrics. The 'policy collapse' and 'template convergence' phenomena are descriptive labels for observed behaviors, not new theoretical constructs. Free parameters are experimental design choices rather than fitted values. The main ad hoc assumptions are the single-epoch training duration and the 3:1:1 weighting ratio, both of which could affect the generality of the findings.

free parameters (10)
  • Reward weights (w_syn, w_pra, w_sem) = R1: (1/3,1/3,1/3); R3-R5: (3/5,1/5,1/5) etc.
    Chosen by design to test equal vs. targeted weighting; not fitted to data but selected as experimental conditions.
  • Invalidity penalty p = -1 or 0
    Two values chosen to test presence/absence of negative penalty; not fitted.
  • LoRA rank (SFT) = 8
    Standard choice, not tuned.
  • LoRA rank (GSPO) = 16
    Increased from SFT for additional capacity; chosen by heuristic.
  • Learning rate (SFT) = 1e-4
    Standard value for LoRA fine-tuning.
  • Learning rate (GSPO) = 5e-5
    Reduced from SFT for conservative RL updates.
  • Temperature = 0.7
    Sampling temperature for candidate generation.
  • Top-p = 0.95
    Nucleus sampling parameter.
  • K (generations per prompt) = 4
    Number of candidates per group in GSPO.
  • BEF4LLM pragmatic thresholds = Four per metric from literature [39,38,4,11]
    Empirically validated thresholds from prior work, not fitted in this paper.
axioms (5)
  • domain assumption BEF4LLM framework [27] provides valid automated quality metrics for BPMN models across syntactic, pragmatic, and semantic dimensions
    Invoked in §3.3 as the foundation for reward signals and evaluation. The framework is from prior work by overlapping authors but is independently published with defined metrics.
  • domain assumption GSPO [44] is a suitable RL algorithm for sequence-level optimization of structured outputs
    Adopted in §3.2 without comparison to alternatives (PPO, GRPO, DPO). Justified by alignment with holistic quality evaluation.
  • domain assumption The three quality dimensions (syntactic, pragmatic, semantic) capture the relevant quality space for BPMN models
    Invoked throughout via the SIQ framework [36]. Standard in BPM literature.
  • ad hoc to paper Single-epoch GSPO training is sufficient to observe reward design effects without reward overfitting
    §4.3: 'single-epoch GSPO training is intended to reduce the risk of reward overfitting.' No validation that additional epochs would not change the relative ordering of configurations.
  • ad hoc to paper The 3:1:1 weighting ratio is sufficient to test whether targeted weighting can improve a target dimension
    §3.5 acknowledges 'more moderate ratios (e.g., 2:1:1) may produce insufficient differentiation.' The failure of targeted weighting may be specific to this aggressive ratio.

pith-pipeline@v1.1.0-glm · 35022 in / 3258 out tokens · 598612 ms · 2026-07-08T14:15:11.193242+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

48 extracted references · 48 canonical work pages · 8 internal anchors

  1. [1]

    Concrete Problems in AI Safety

    Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., Mané, D., 2016. Concrete problems in AI safety. arXiv preprint arXiv:1606.06565arXiv:1606.06565

    work page internal anchor Pith review Pith/arXiv arXiv 2016

  2. [2]

    A comparative survey of business process similarity measures

    Becker, M., Laue, R., 2012. A comparative survey of business process similarity measures. Computers in Industry 63, 148–167. doi:10.1016/j.compind.2011.11.003

    work page doi:10.1016/j.compind.2011.11.003 2012

  3. [3]

    Specializing large language models for process modeling via rein- forcement learning with verifiable and universal rewards

    Berti, A., Wang, X., Kourani, H., van der Aalst, W.M.P., 2025. Specializing large language models for process modeling via rein- forcement learning with verifiable and universal rewards. doi:10. 21203/rs.3.rs-7646566/v1

    work page 2025

  4. [4]

    An Evaluation of Thresholds for Business Process Model Metrics

    Boomsma, R.D., 2009. An Evaluation of Thresholds for Business Process Model Metrics. Master’s thesis. University of Twente

    work page 2009

  5. [5]

    What is the Best Process Model Representation? A Comparative Analysis for Pro- cess Modeling with Large Language Models, in: van de Weerd, I., Estrada Torres, B., van der Aa, H

    Brissard, A., Cuppens, F., Zouaq, A., 2026. What is the Best Process Model Representation? A Comparative Analysis for Pro- cess Modeling with Large Language Models, in: van de Weerd, I., Estrada Torres, B., van der Aa, H. (Eds.), Business Process Man- agement Workshops, Springer Nature Switzerland, Cham. pp. 55–

    work page 2026

  6. [6]

    URL:https://link.springer.com/10.1007/978-3-032-13426-4_5, doi:10.1007/978-3-032-13426-4_5

    work page doi:10.1007/978-3-032-13426-4_5

  7. [7]

    Çelikmasat, G., Özgövde, A., Aydemir, F.B., 2026. Generating Business Process Models with Open Source Large Language Models Using Instruction Tuning, in: Product-Focused Software Process Im- provement: 26th International Conference, PROFES 2025, Salerno, Italy, December 1–3, 2025, Proceedings, Springer-Verlag, Berlin, Heidelberg. pp. 269–284. doi:10.1007/...

    work page doi:10.1007/978-3-032-12089-2_17 2026

  8. [8]

    A fast and elitistmultiobjectivegeneticalgorithm:NSGA-II

    Deb, K., Pratap, A., Agarwal, S., Meyarivan, T., 2002. A fast and elitistmultiobjectivegeneticalgorithm:NSGA-II. IEEETransactions on Evolutionary Computation 6, 182–197. doi:10.1109/4235.996017

    work page doi:10.1109/4235.996017 2002

  9. [9]

    Dijkman, R., Dumas, M., van Dongen, B., Käärik, R., Mendling, J.,

    work page

  10. [10]

    Information Systems 36, 498–516

    Similarityofbusinessprocessmodels:Metricsandevaluation. Information Systems 36, 498–516. doi:10.1016/j.is.2010.09.006

    work page doi:10.1016/j.is.2010.09.006 2010

  11. [11]

    Formal semantics and automated analysis of BPMN process models

    Dijkman, R.M., Dumas, M., Ouyang, C., 2007. Formal semantics and automated analysis of BPMN process models. URL:https: //api.semanticscholar.org/CorpusID:7918800

    work page 2007

  12. [12]

    Measuring simi- larity between business process models, in: Seminal Contributions to Information Systems Engineering: 25 Years of CAiSE

    van Dongen, B., Dijkman, R., Mendling, J., 2013. Measuring simi- larity between business process models, in: Seminal Contributions to Information Systems Engineering: 25 Years of CAiSE. Springer, pp. 405–419. doi:10.1007/978-3-642-36926-1_33

    work page doi:10.1007/978-3-642-36926-1_33 2013

  13. [13]

    Quality of Business Process Models Expressed in BPMN

    Ekstedt, J., 2015. Quality of Business Process Models Expressed in BPMN. Master’s thesis. KTH, School of Information and Commu- nication Technology. URL:https://www.diva-portal.org/smash/get/ diva2:830545/FULLTEXT01.pdf

    work page 2015

  14. [14]

    Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., Kiela, D.,

    work page

  15. [15]

    KTO: Model Alignment as Prospect Theoretic Optimization

    KTO: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306

    work page internal anchor Pith review Pith/arXiv arXiv

  16. [16]

    Permutation, Parametric, and Bootstrap Tests of Hypotheses

    Good, P.I., 2005. Permutation, Parametric, and Bootstrap Tests of Hypotheses. 3rd ed., Springer. doi:10.1007/b138696. :Preprint submitted to Elsevier Page 20 of 21 Improving LLM-Generated Process Model Quality Through Reinforcement Learning

    work page doi:10.1007/b138696 2005

  17. [17]

    DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

    Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al., 2025. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv preprint arXiv:2501.12948

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  18. [18]

    A practical guide to multi-objective reinforcementlearningandplanning

    Hayes, C.F., Rădulescu, R., Bargiacchi, E., Källström, J., Macfar- lane, M., Reymond, M., Verstraeten, T., Zintgraf, L.M., Dazeley, R., Heintz, F., et al., 2022. A practical guide to multi-objective reinforcementlearningandplanning. AutonomousAgentsandMulti- Agent Systems 36, 26. doi:10.1007/s10458-022-09552-y

    work page doi:10.1007/s10458-022-09552-y 2022

  19. [19]

    Automatically Gen- erating BPMN 2.0 Process Models from Natural Language Process Descriptions: Challenges, Framework, Quality Assessment

    Hörner, L.F., Möller, M., Reichert, M., 2026. Automatically Gen- erating BPMN 2.0 Process Models from Natural Language Process Descriptions: Challenges, Framework, Quality Assessment. Busi- ness & Information Systems Engineering 68, 59–83. doi:10.1007/ s12599-025-00983-x

    work page 2026

  20. [20]

    LoRA: Low-rank adaptation of large language models, in: International Conference on Learning Representations

    Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., 2022. LoRA: Low-rank adaptation of large language models, in: International Conference on Learning Representations. URL:https://openreview.net/forum?id=nZeVKeeFYf9

    work page 2022

  21. [21]

    RL-Struct:Alightweightreinforcementlearning framework for reliable structured output in LLMs

    Hu,R.,etal.,2024. RL-Struct:Alightweightreinforcementlearning framework for reliable structured output in LLMs. arXiv preprint arXiv:2512.00319

    work page arXiv 2024

  22. [22]

    Large process models: A vision for business process management in the age of generative AI

    Kampik, T., Warmuth, C., Rebmann, A., Agam, R., Egger, L.N.P., Gerber, A., Hoffart, J., Kolk, J., Herzig, P., Decker, G., et al., 2025. Large process models: A vision for business process management in the age of generative AI. KI – Künstliche Intelligenz 39, 81–95

    work page 2025

  23. [23]

    Conversational process modelling: State of the art, applica- tions,andimplicationsinpractice,in:BusinessProcessManagement Forum, Springer

    Klievtsova, N., Benzin, J.V., Kampik, T., Mangler, J., Rinderle-Ma, S., 2023. Conversational process modelling: State of the art, applica- tions,andimplicationsinpractice,in:BusinessProcessManagement Forum, Springer. pp. 319–336. doi:10.1007/978-3-031-41623-1_19

    work page doi:10.1007/978-3-031-41623-1_19 2023

  24. [24]

    Efficient LLM-based conversational process modeling, in: Business Process Management Workshops, Springer

    Köpke, J., Safan, A., 2025. Efficient LLM-based conversational process modeling, in: Business Process Management Workshops, Springer. pp. 259–270

    work page 2025

  25. [25]

    Pro- MoAI: Process modeling with generative AI, in: Proceedings of the Thirty-ThirdInternationalJointConferenceonArtificialIntelligence

    Kourani,H.,Berti,A.,Schuster,D.,vanderAalst,W.M.P.,2024. Pro- MoAI: Process modeling with generative AI, in: Proceedings of the Thirty-ThirdInternationalJointConferenceonArtificialIntelligence. doi:10.24963/ijcai.2024/1014

    work page doi:10.24963/ijcai.2024/1014 2024

  26. [26]

    Process models repre- sentingknowledgeforaction:Arevisedqualityframework

    Krogstie, J., Sindre, G., Jørgensen, H., 2006. Process models repre- sentingknowledgeforaction:Arevisedqualityframework. European Journal of Information Systems 15, 91–102. doi:10.1057/palgrave. ejis.3000598

    work page doi:10.1057/palgrave 2006

  27. [27]

    Tulu 3: Pushing Frontiers in Open Language Model Post-Training

    Lambert, N., Morrison, J., Pyatkin, V., Huang, S., Ivison, H., Brah- man,F.,Miranda,L.J.V.,Liu,A.,Dziri,N.,Lyu,S.,etal.,2024. Tülu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  28. [28]

    Human-CenteredEvalua- tion of an LLM-Based Process Modeling Copilot: A Mixed-Methods Study with Domain Experts

    Lauer,C.,Pfeiffer,P.,Mehdiyev,N.,2026a. Human-CenteredEvalua- tion of an LLM-Based Process Modeling Copilot: A Mixed-Methods Study with Domain Experts. URL:https://arxiv.org/abs/2603. 12895,arXiv:2603.12895

    work page arXiv

  29. [29]

    Conver- sational Business Process Modeling using LLMs: Initial Results and Challenges,in:EMISA2025.GesellschaftfürInformatike.V.,Bonn, pp

    Lauer, C., Pfeiffer, P., Rombach, A., Mehdiyev, N., 2025. Conver- sational Business Process Modeling using LLMs: Initial Results and Challenges,in:EMISA2025.GesellschaftfürInformatike.V.,Bonn, pp. P3+2—-O4+5. doi:10.18420/EMISA2025_03

    work page doi:10.18420/emisa2025_03 2025

  30. [30]

    Assessing the Business Process Modeling Competences of Large Language Models

    Lauer,C.,Pfeiffer,P.,Rombach,A.,Mehdiyev,N.,2026b. Assessing thebusinessprocessmodelingcompetencesoflargelanguagemodels. arXiv preprint arXiv:2601.21787arXiv:2601.21787

    work page internal anchor Pith review Pith/arXiv arXiv

  31. [31]

    CodeRL: Mastering code generation through pretrained models and deep reinforcement learning, in: Advances in Neural Information Processing Systems, pp

    Le, H., Wang, Y., Gotmare, A.D., Savarese, S., Hoi, S.C.H., 2022. CodeRL: Mastering code generation through pretrained models and deep reinforcement learning, in: Advances in Neural Information Processing Systems, pp. 21314–21328

    work page 2022

  32. [32]

    Understandingquality in conceptual modeling

    Lindland,O.I.,Sindre,G.,Sølvberg,A.,1994. Understandingquality in conceptual modeling. IEEE Software 11, 42–49. doi:10.1109/52. 268955

    work page doi:10.1109/52 1994

  33. [33]

    Textual process descriptions and corresponding BPMN models

    Mangler, J., Klievtsova, N., 2023. Textual process descriptions and corresponding BPMN models. doi:10.5281/zenodo.7783492

    work page doi:10.5281/zenodo.7783492 2023

  34. [34]

    Metrics for business process models, in: Metrics forProcessModels:EmpiricalFoundationsofVerification,ErrorPre- diction,andGuidelinesforCorrectness.Springer,Berlin,Heidelberg, pp

    Mendling, J., 2008. Metrics for business process models, in: Metrics forProcessModels:EmpiricalFoundationsofVerification,ErrorPre- diction,andGuidelinesforCorrectness.Springer,Berlin,Heidelberg, pp. 103–133. doi:10.1007/978-3-540-89224-3_4

    work page doi:10.1007/978-3-540-89224-3_4 2008

  35. [35]

    Policy invariance under reward transformations: Theory and application to reward shaping, in:ProceedingsoftheSixteenthInternationalConferenceonMachine Learning, pp

    Ng, A.Y., Harada, D., Russell, S., 1999. Policy invariance under reward transformations: Theory and application to reward shaping, in:ProceedingsoftheSixteenthInternationalConferenceonMachine Learning, pp. 278–287

    work page 1999

  36. [36]

    Business process model and notation (BPMN), version 2.0.2

    Object Management Group, 2013. Business process model and notation (BPMN), version 2.0.2. OMG Standard. URL:https: //www.omg.org/spec/BPMN

    work page 2013

  37. [37]

    Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C.L., Mishkin,P.,Zhang,C.,Agarwal,S.,Slama,K.,Ray,A.,Schulman,J., Hilton,J.,Kelton,F.,Miller,L.,Simens,M.,Askell,A.,Welinder,P., Christiano,P.,Leike,J.,Lowe,R.,2022. Traininglanguagemodelsto follow instructions with human feedback, in: Proceedings of the 36th InternationalConference onNeuralInformati...

    work page 2022

  38. [38]

    Direct preference optimization: Your language modelissecretlyarewardmodel,in:AdvancesinNeuralInformation Processing Systems

    Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C.D., Finn, C., 2023. Direct preference optimization: Your language modelissecretlyarewardmodel,in:AdvancesinNeuralInformation Processing Systems

    work page 2023

  39. [39]

    Businessprocessquality management, in: Handbook on Business Process Management 1: Introduction, Methods, and Information Systems

    Reijers,H.A.,Mendling,J.,Recker,J.,2010. Businessprocessquality management, in: Handbook on Business Process Management 1: Introduction, Methods, and Information Systems. Springer, Berlin, Heidelberg, pp. 167–185. doi:10.1007/978-3-642-00416-2_8

    work page doi:10.1007/978-3-642-00416-2_8 2010

  40. [40]

    Evaluation of BPMN models quality—a family of exper- iments, in: Third International Conference on Evaluation of Novel Approaches to Software Engineering, pp

    Rolón, E., Garcia, F., Ruiz, F., Piattini, M., Visaggio, C.A., Canfora, G., 2008. Evaluation of BPMN models quality—a family of exper- iments, in: Third International Conference on Evaluation of Novel Approaches to Software Engineering, pp. 56–63

    work page 2008

  41. [41]

    Acase study about the improvement of business process models driven by indicators

    Sánchez-González,L.,García,F.,Ruiz,F.,Piattini,M.,2017. Acase study about the improvement of business process models driven by indicators. Software and Systems Modeling 16, 759–788. doi:10. 1007/s10270-015-0482-0

    work page 2017

  42. [42]

    Busi- ness process model improvement based on measurement activities, in: International Conference on Evaluation of Novel Software Ap- proaches to Software Engineering, SCITEPRESS

    Sánchez-González, L., Ruiz, F., García, F., Piattini, M., 2011. Busi- ness process model improvement based on measurement activities, in: International Conference on Evaluation of Novel Software Ap- proaches to Software Engineering, SCITEPRESS. pp. 104–113

    work page 2011

  43. [43]

    Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O.,

    work page

  44. [44]

    Proximal Policy Optimization Algorithms

    Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347

    work page internal anchor Pith review Pith/arXiv arXiv

  45. [45]

    DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

    Shao,Z.,Wang,P.,Zhu,Q.,Xu,R.,Song,J.,Bi,X.,Zhang,H.,Zhang, M., Li, Y.K., Wu, Y., Guo, D., 2024. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  46. [46]

    On a quest for good process models: The cross- connectivitymetric, in:AdvancedInformationSystems Engineering, Springer

    Vanderfeesten, I., Reijers, H.A., Mendling, J., van der Aalst, W.M.P., Cardoso, J., 2008. On a quest for good process models: The cross- connectivitymetric, in:AdvancedInformationSystems Engineering, Springer. pp. 480–494. doi:10.1007/978-3-540-69534-9_36

    work page doi:10.1007/978-3-540-69534-9_36 2008

  47. [47]

    AprocesssemanticsforBPMN,in: Liu,S.,Maibaum,T.,Araki,K.(Eds.),FormalMethodsandSoftware Engineering, Springer, Berlin, Heidelberg

    Wong,P.Y.H.,Gibbons,J.,2008. AprocesssemanticsforBPMN,in: Liu,S.,Maibaum,T.,Araki,K.(Eds.),FormalMethodsandSoftware Engineering, Springer, Berlin, Heidelberg. pp. 355–374

    work page 2008

  48. [48]

    Group Sequence Policy Optimization

    Zheng, C., Liu, S., Li, M., Chen, X.H., Yu, B., Gao, C., Dang, K., Liu, Y., Men, R., Yang, A., Zhou, J., Lin, J., 2025. Group Sequence Policy Optimization. URL:https://arxiv.org/abs/2507. 18071,arXiv:2507.18071. :Preprint submitted to Elsevier Page 21 of 21

    work page internal anchor Pith review Pith/arXiv arXiv 2025