{"id":"2b67620d-52e4-4589-8679-ac912bde4bb5","arxiv_id":"2509.00834","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A differentiable LTLf-based loss, using Gumbel-Softmax sampling and DeepDFA, improves rule compliance and accuracy of autoregressive suffix predictors for business process traces.","lead":"This paper adds a mathematical rule book, written in temporal logic, into the training of a neural network that predicts the next steps of a business process. The result is a model that produces more rule-following and more accurate forecasts, tested on three real-world process logs.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Main evidence for the central claim is compromised: LTLf constraints are mined from the test set and test traces violating them are removed, so the 'prior knowledge' is not prior and satisfaction is measured on a filtered set.","rationale":"The reader's weakest assumption is exactly the load-bearing concern: the LTLf knowledge is extracted from the test set and the test set is filtered to contain only traces satisfying it, so the evaluation protocol cannot support the claim that training-time injection of prior knowledge improves satisfaction and accuracy. I agree with the reader's assessment. The method itself is plausible and the mathematical formulation is coherent, but the reported empirical evidence is confounded by test-set leakage and filtering bias. This does not require changing the reader's CONDITIONAL verdict: the paper should be accepted only if the experimental protocol is revised to use training-only or expert-provided constraints and an unfiltered test set. The lack of error bars and hyperparameter reporting are secondary issues, not the primary reason for the conditional verdict.","tokens_in":15948,"tokens_out":3900,"duration_ms":48950,"concrete_test":"Re-run the full experiment with constraints mined from the training split only (or using fixed, expert-authored Declare formulas), and evaluate all models on the complete, unfiltered test set. Report SAT and DL for RNN, RNN+LLL, and RNN+GLL, plus the number of traces removed under the original protocol. If the logic-loss models still improve both metrics on the unfiltered test set, the leakage/filtering explanation is ruled out; if the gap shrinks or reverses, the original conclusion is an artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim—that injecting LTLf knowledge at training time improves suffix similarity and constraint satisfaction—rests on the protocol in §4.1. There, Declare constraints are extracted from the test set with 85% support, and 'traces that violate the extracted LTLf model are removed from both the training and test set.' This makes the LTLf model a posterior summary of the test labels rather than independent prior knowledge, and it guarantees by construction that every test trace satisfies the formula. The satisfaction-rate metric is therefore measured only on traces that conform to the formula the model was explicitly trained to satisfy, while the base RNN is denied access to this test-derived formula. The reported gains in SAT and DL could reflect test-set leakage or the distribution shift caused by filtering, not a general benefit of training-time logic injection. The paper also does not report how many traces were removed, so the size and bias of the filtered evaluation set are unknown. Without an evaluation using knowledge that is genuinely independent of the test set and an unfiltered test set, the central claim is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a neuro-symbolic approach to suffix prediction in business process monitoring. It integrates LTLf constraints into the training of an autoregressive next-activity predictor by defining a differentiable logic loss, with a local variant (Lloc) that penalizes transitions into permanently failing DFA states and a global variant (Lglob) that uses Gumbel-Softmax sampling and DeepDFA to maximize the Monte Carlo estimate of suffix satisfaction. The combined loss is L = αLD + (1 − α)Lφ. The authors evaluate RNN, RNN+LLL, and RNN+GLL on three event logs under increasing noise, reporting Damerau-Levenshtein similarity and LTLf satisfaction rate. The central claim is that incorporating LTLf knowledge at training time improves both suffix similarity and constraint satisfaction. However, Section 4.1 describes a protocol in which Declare constraints are mined from the test set and traces violating them are removed from both training and test sets, which compromises the empirical evaluation.","tokens_in":16170,"tokens_out":4031,"duration_ms":53138,"significance":"If the empirical claim were soundly supported, the paper would make a useful contribution: a model-agnostic way to inject temporal-logic knowledge into training of symbolic sequence generators, with two loss variants that are clearly motivated and backed by public code. The conceptual machinery—DeepDFA encoding, Gumbel-Softmax differentiable sampling, and the local/global loss decomposition—is coherent and potentially reusable beyond BPM. The paper also provides a reproducibility link and transparent dataset statistics. However, the current experimental protocol does not establish the claimed benefit. Because the LTLf model is derived from the test set and the test set is filtered to satisfy it, the reported satisfaction gains are partly guaranteed by construction, and the comparison against the base RNN is biased. The evaluation needs to be redone with genuinely prior knowledge and an unfiltered test set before the central claim can be accepted.","major_comments":[{"comment":"The protocol violates the 'prior knowledge' assumption that the paper's method relies on. The text states: 'Knowledge is extracted from the test set in the form of Declare constraints using the Declare Miner... traces that violate the extracted LTLf model are removed from both the training and test set.' This makes the LTLf formula a posterior summary of the test labels, not background knowledge available before data collection. It also removes non-conforming test traces, so the satisfaction-rate metric is computed only on traces that satisfy the formula by construction. The base RNN is denied access to this test-derived formula while being evaluated on the filtered set, whereas the proposed models are explicitly trained to maximize satisfaction of that same formula. Consequently, the reported SAT improvements and even the DL comparisons are confounded by test-set leakage and distributio","section":"4.1 Experimental Setup"},{"comment":"The main success metric is circular with the training objective. The global loss is Lglob = −log(ˆPθ⊨ϕ), where ˆPθ⊨ϕ is the empirical satisfaction probability over sampled suffixes. The paper then reports the satisfaction rate of predicted traces on a test set that has been filtered to satisfy exactly the same LTLf model. Thus, SAT values near 100% are expected for the GLL method and are weak evidence that training-time knowledge injection works. Satisfaction-rate differences between methods are largely a consequence of the experimental setup. The authors should report compliance on the original unfiltered test set, and ideally with respect to independently held-out constraints, to provide meaningful evidence.","section":"4.2 Empirical Results; Eq. (14)-(15)"},{"comment":"The paper does not report the number of traces removed from the training/test sets, nor any dispersion measures for the results of the 15 runs. Table 2 gives average epochs without standard deviations or significance tests, so the claim that knowledge 'accelerates model convergence' is not quantified. Figure 2 and Figure 3 do not show confidence intervals or error bars. Given the small number of datasets and the filtering step, these omissions make it difficult to assess whether observed differences are robust. The authors should report the fraction of filtered traces and standard errors/confidence intervals, and ideally perform statistical tests for the main comparisons.","section":"4.1-4.2 and Table 2"}],"minor_comments":[{"comment":"The set of free hyperparameters (α, τ, N, Declare minimum support) is listed, but the exact values used for each dataset are not reported. This makes reproduction harder even with the linked code. Please include a table or appendix with the chosen values.","section":"4.1 Experimental Setup"},{"comment":"The claim of being 'the first to integrate temporal knowledge in the generation of multi-step symbolic sequences at training time' is too strong given STLnet [6], which is acknowledged but dismissed on domain grounds. The statement should be qualified to discrete domains or LTLf specifically.","section":"5 Related Work"},{"comment":"The paper correctly notes that Lloc only penalizes permanent violations (case (i)) and not other non-satisfying traces (case (ii)). This is a real limitation; the authors should discuss how often case (ii) occurs in the experiments, especially since the DFA failure states are few (Table 1).","section":"3.2 Local Guidance"},{"comment":"Figures 2 and 3 are referenced before they appear in the text; please reorder or fix the cross-reference. Also, the figures would benefit from error bars or shaded confidence regions.","section":"4.2 Empirical Results"},{"comment":"Minor typos and notation inconsistencies exist, e.g., 'subsymbolic' (Section 2.1) and the use of both '˜σ' and 'σ' in Eq. (8). A careful proofreading pass is recommended.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The core methodological idea is interesting and the formal development is sound, but the experimental evaluation as designed cannot support the central claim. The main fix—using training-set-derived or externally supplied knowledge and evaluating on the unfiltered test set—is within the scope of a major revision. I would not reject, but the paper cannot be accepted without substantial re-analysis of the experiments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nTwo things to know before you read this paper. First, the training-time logic losses are coherent and the local loss is a genuinely useful trick. Second, the main experimental claim is compromised by the way they built the test set: the LTLf constraints are mined from the test set, and then test traces that violate them are deleted. So the 'prior knowledge' is not prior, and the satisfaction-rate gains are partly guaranteed by construction.\n\nWhat's actually new and good: the local activity-level loss, which uses failing states of the DFA to push the model away from irreversible violations, is a clean idea and easy to reuse. The global loss via DeepDFA and Gumbel-Softmax is a sensible extension of their earlier workshop work. The paper is written clearly, the math is solid, and they ship code and data.\n\nNow the soft spots, in order of severity. The protocol in Section 4.1 says constraints are extracted from the test set with 85% support and traces that violate them are removed from both training and test. That means the knowledge is a summary of the test labels, and the test set is filtered to be compliant with it. The base RNN is trained on the same filtered data but without the formula, so the comparison shows the value of having the test formula during training, not the value of prior knowledge. They don't report how many traces were removed, so the size and bias of the filtered set are unknown. This is a load-bearing flaw, not a minor caveat.\n\nAlso missing: values for alpha, tau, and N; error bars or significance tests for the 15 runs; and any comparison against inference-time constrained decoding, which is the main alternative method they cite. Those are smaller issues.\n\nWho this is for: people working on neuro-symbolic sequence generation or predictive process monitoring will find the local loss worth thinking about, and the failure is educational. But don't take the current numbers at face value.\n\nRecommendation: this deserves a serious referee, not a desk reject. The core idea is worth discussing, and the evaluation flaw is fixable. A revision with expert-authored or training-only constraints, an unfiltered test set, and full hyperparameter reporting could become a solid paper.","headline":"The local loss is a clean idea, but the main empirical claim is built into the protocol: LTLf constraints are mined from the test set and the test set is filtered to satisfy them.","tokens_in":16694,"tokens_out":4369,"would_cite":false,"duration_ms":48158,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that adding a differentiable logic loss to autoregressive suffix predictors improves both edit-distance accuracy and satisfaction of temporal-logic constraints.","keywords":["Predictive process monitoring","Suffix prediction","Neuro-symbolic AI","LTLf","Differentiable automata","Logic loss","Gumbel-Softmax","Declare constraints"],"falsifier":"Run the same comparison with the temporal rules mined only from the training split (or supplied by an expert) and keep all test traces in the evaluation, including those that violate the rules. If the logic-trained models no longer beat the data-only baseline on satisfaction or on Damerau-Levenshtein similarity, the reported improvement is an artifact of test-derived filtering.","tokens_in":15806,"feed_emoji":"🧠","tokens_out":10294,"duration_ms":116057,"temperature":0.7,"pith_summary":"The paper tries to show that a neural model predicting the future portion of a business-process trace learns better when logical rules about the process are folded into training rather than checked only after generation. Its recipe adds a differentiable logic-loss term to the standard next-event prediction loss: a temporal-logic rule is translated into a finite-state automaton, embedded as a differentiable layer, and complete predicted suffixes are sampled with Gumbel-Softmax so gradients flow through both accuracy and rule compliance. On three real-world event logs, this hybrid training yields predicted suffixes closer to the true continuation by edit distance and far more likely to satisfy the rules, with compliance near 100 percent even when 40 percent of training labels are corrupted. The same recipe is offered as a general way to inject temporal knowledge into any autoregressive symbolic sequence generator.","feed_headline":"Training with temporal-logic rules lifts suffix compliance to ~100%","feed_subtitle":"Adding logic constraints during training keeps predictions accurate and compliant even at 40% label noise.","key_machinery":"Load-bearing is the combined loss L = αLD + (1 − α)Lφ. LD is per-token cross-entropy. Lφ uses an LTLf formula translated into a deterministic finite automaton (extended with an end-of-trace marker and, for the local variant, failing states), DeepDFA — a tensor representation that makes acceptance differentiable over probabilistically grounded traces — and Gumbel-Softmax sampling, which yields near-one-hot suffixes whose gradients back-propagate. Lloc penalizes entering a failing state at each step; Lglob maximizes a Monte Carlo estimate of the fraction of sampled traces the automaton accepts.","core_discovery":"The paper claims that adding a differentiable logic-loss term to the usual cross-entropy loss makes an autoregressive suffix predictor produce continuations that are closer to the observed ones and far more likely to satisfy stated temporal rules. The loss translates the LTLf rule into a deterministic finite automaton, embeds it as a differentiable tensor layer, and samples complete suffixes with Gumbel-Softmax so gradients flow through sampling. A local variant penalizes moves into automaton states from which satisfaction is impossible; a global variant maximizes a Monte Carlo estimate of satisfaction. On three real-world event logs with up to 40 percent label noise, logic-trained models st","pith_inferences":["The test-derived mining protocol makes the reported compliance rates an upper bound; re-running with constraints written before seeing test traces is the direct way to test how much of the gain is genuinely prior knowledge.","The convergence speedup hints that the logic loss acts as a regularizer; if so, the accuracy gap between logic-trained and data-only models should widen as label noise increases and narrow on clean logs.","The self-loop alphabet extension becomes impractical for very large vocabularies, so transferring the recipe to large language models would require a different way to handle unconstrained symbols.","A combined local-plus-global loss is a natural next step for mixed safety/liveness rule sets; the authors list it as future work."],"forward_implications":["Models trained with either logic loss satisfy the mined temporal rules at close to 100 percent even when 40 percent of training events are relabeled as noise.","Adding the logic loss does not reduce suffix similarity to the ground truth; reported Damerau-Levenshtein similarity stays at least as high as the data-only RNN.","The global logic loss consistently reaches convergence in fewer training epochs than the data-only baseline, up to roughly a third of the epochs in some configurations.","Because the method only needs an autoregressive next-symbol distribution, it transfers to any sequence predictor, not just the LSTM used in the experiments.","The local loss gives step-level feedback but requires automata with failing states; the global loss handles arbitrary formulas at higher computational cost."],"supporting_citations":[{"why":"Supplies the Gumbel-Softmax reparameterization that keeps sampled suffixes differentiable.","marker":"[15]"},{"why":"Supplies the differentiable automaton layer used to evaluate trace satisfaction.","marker":"[26]"},{"why":"Translates LTLf formulas into the deterministic finite automata the method tensorizes.","marker":"[28]"},{"why":"Defines LTLf semantics and the equivalence to deterministic finite automata that the method relies on.","marker":"[19]"},{"why":"Mines the Declare constraints used as background knowledge in the experiments.","marker":"[38]"},{"why":"Shows Declare formulas produce polynomial-size automata, making the preprocessing practical.","marker":"[37]"},{"why":"Defines Declare, the constraint language whose rules are mined and converted to LTLf.","marker":"[35]"},{"why":"Defines the edit-distance metric used to compare predicted and ground-truth suffixes.","marker":"[16]"}],"fun_headline_variants":["Temporal-logic loss lifts suffix compliance and accuracy","Differentiable LTLf training enforces rules in suffix predictions","Neuro-symbolic PPM: logic-aware loss improves sequence endings","Gumbel-Softmax sampling plus temporal logic yields compliant suffixes","Training with LTLf constraints boosts suffix prediction correctness"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The reported compliance gain rests on the rules being mined from the test set at 85 percent support and on traces that violate them being removed from both training and test data, so part of the satisfaction rate is built into the evaluation rather than coming from independent prior knowledge.","fun_headline_variants_meta":{"raw":{"variants":["Temporal-logic loss lifts suffix compliance and accuracy","Differentiable LTLf training enforces rules in suffix predictions","Neuro-symbolic PPM: logic-aware loss improves sequence endings","Gumbel-Softmax sampling plus temporal logic yields compliant suffixes","Training with LTLf constraints boosts suffix prediction correctness"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000751,"raw_usage":{"total_tokens":3162,"prompt_tokens":711,"completion_tokens":2451,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":455,"completion_tokens_details":{"reasoning_tokens":2381}},"tokens_in":455,"tokens_out":2451,"duration_ms":21648,"temperature":1.0,"reasoning_tokens":2381,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T13:10:36.607822+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same comparison with the temporal rules mined only from the training split (or supplied by an expert) and keep all test traces in the evaluation, including those that violate the rules. If the logic-trained models no longer beat the data-only baseline on satisfaction or on Damerau-Levenshtein similarity, the reported improvement is an artifact of test-derived filtering.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Gumbel-Softmax reparameterization that keeps sampled suffixes differentiable."},{"cited_title":"De Giacomo, M","cited_arxiv_id":null,"evidence_quote":"Defines LTLf semantics and the equivalence to deterministic finite automata that the method relies on."},{"cited_title":"Alman, C","cited_arxiv_id":null,"evidence_quote":"Mines the Declare constraints used as background knowledge in the experiments."},{"cited_title":"Westergaard, Better algorithms for analyzing and enacting declar- ative workflow languages using ltl, in: S","cited_arxiv_id":null,"evidence_quote":"Shows Declare formulas produce polynomial-size automata, making the preprocessing practical."},{"cited_title":"Pesic, W","cited_arxiv_id":null,"evidence_quote":"Defines Declare, the constraint language whose rules are mined and converted to LTLf."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the edit-distance metric used to compare predicted and ground-truth suffixes."}],"review_version":1}