{"id":"5b60d6e7-26f9-451d-b7d3-42dda9a9ed8e","arxiv_id":"2608.04271","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Large language models can flag contradictions and errors in business documentation and extract process structures, but the reported evidence is thin, partly synthetic, and lacks ground-truth metrics for the hardest task.","lead":"The paper describes MIRABELLE, a system that uses large language models to find contradictions, errors, and structural flaws in business process documentation such as user guides and work instructions. The authors report initial experiments on three tasks, from flagging bad instructions to converting an Oracle purchasing guide into Petri nets.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Synthetic evaluation data—especially work instructions generated by the same GPT-OSS:250B model that is then scored—does not establish transfer to real business documentation; a real-world rerun is needed.","rationale":"The reader's weakest_assumption and my concern align. The strongest claim depends on the 0.96 numbers; those numbers depend on GPT-OSS:250B-generated test data and SCC proxies. Even if the pipeline is sound, the evidence as reported cannot distinguish 'LLMs catch real documentation errors' from 'LLMs catch errors that match the LLM's own text-generation habits.' The proposed real-data rerun would settle that. I also note a secondary internal inconsistency: Section 3.2 reports precision 1.0 and error-set accuracy 0.98 while Table 1 lists both as 0.96; re-running with a confusion matrix would resolve this as well. Given the concrete architecture and honest caveats, CONDITIONAL remains the appropriate verdict, and my read does not change it.","tokens_in":10250,"tokens_out":4701,"duration_ms":44908,"concrete_test":"Re-run Section 3.2's exact prompt and temperature (0.1) with 100 work instructions from real manufacturing SOPs/user guides, half containing errors inserted and independently verified by domain experts (same error classes: unit conversions, qualifier conflicts, numerical mistakes). Report a full confusion matrix and compare precision/accuracy to Table 1. Separately, re-run Section 3.1 with contradiction pairs mined from genuine ERP/SOP documentation embedded in a named host document of comparable length, reporting accuracy with confidence intervals. If both remain near 0.96, the transfer claim survives; if either drops materially, the synthetic/proxy datasets are the explanation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is a transfer claim: off-the-shelf LLMs find real documentation errors. Section 3.2's headline 0.96/0.96 rests entirely on 100 work instructions generated by GPT-OSS:250B and then evaluated by GPT-OSS:250B at temperature 0.1. When the same model writes the errors and judges them, the scores reflect consistency with the model's own generation priors, not ability to catch human-authored errors. Section 3.1 has the same external-validity gap in a different form: SCC contradiction pairs are not business text, the host document is unspecified, and the authors explicitly note this 'might give the LLM an unfair advantage.' Because Section 4 extrapolates to large-scale automated ERP documentation processing, the evaluation data is load-bearing. This is a correctable gap, not an internal inconsistency, but until a real-data rerun appears the reported accuracy cannot support the generalization.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes MIRABELLE/DPT, a pipeline that uses off-the-shelf LLMs to process business-process documentation. It reports three experiments: (i) contradiction detection by injecting Stanford Contradiction Corpora pairs into a host document and asking LLMs to identify inconsistent chunk pairs; (ii) classification of 100 synthetic work instructions, half with injected subtle errors, generated by GPT-OSS:250B and later scored by the same model; and (iii) an end-to-end pipeline that converts Chapter 15 of the Oracle Purchasing User Guide into CPN-Py Petri Net JSON. The paper reports high accuracy and precision for (i) and (ii), and qualitative success with quantified runtimes for (iii), and concludes that LLMs significantly accelerate formalization and vulnerability discovery.","tokens_in":10398,"tokens_out":5258,"duration_ms":45884,"significance":"If the empirical claims transferred to real enterprise documentation, the system would be practically valuable: it combines document ingestion, LLM-based extraction, graph/Petri-net formalization, and downstream logic analysis in one pipeline, and it compares quantized and non-quantized backends on realistic runtimes. The paper is transparent about several limitations (SCC is not business text; work instructions are synthetic), and the pipeline engineering is described in enough detail to replicate. However, the current evaluation does not yet support the transfer claim: the two error-detection experiments use synthetic distributions that are either generated by the same model under test or assembled from non-business text, and the Petri-net experiment has no quantitative accuracy assessment. The contribution is therefore an initial system demonstration rather than a validated empirical result.","major_comments":[{"comment":"The headline precision/accuracy of 0.96 is computed on a corpus of 100 work instructions that were generated by GPT-OSS:250B, the same model that is then evaluated at temperature 0.1. This does not establish transfer to real, human-authored erroneous work instructions: the model's ability to find errors injected according to its own generation priors may reflect consistency with those priors rather than general error-detection ability. Since the claims in §4 about 'high accuracy in detecting nuanced conversion and range errors within work instructions' rest entirely on this dataset, the evaluation is load-bearing. I request a rerun on human-authored or independently collected erroneous work instructions, with per-category (conversion, range, unit) results and error bars, or a clear reframing of the result as a self-consistency check.","section":"§3.2, Table 1"},{"comment":"The contradiction-detection experiment injects SCC pairs into an unspecified 'host document' and reports only a figure without exact accuracy values in the text. The authors themselves state that SCC text is 'not pure business text' and 'might give the LLM an unfair advantage,' so the experiment does not support the strong conclusion that 'LLMs can be effective for uncovering documentation contradictions buried deeply in business text.' Moreover, no baseline, precision/recall breakdown, or statistical characterization is provided. Please specify the host document and injection protocol, report exact per-condition accuracies (and ideally precision/recall/F1), and re-evaluate on real SOP/user-guide text or on business-style contradiction pairs before claiming validity.","section":"§3.1, Figure 3"},{"comment":"The Petri-net extraction experiment reports processing times but no quantitative evaluation of correctness for the generated CPN-Py JSON. Statements such as 'high fidelity', 'most accurate and coherent', and 'usable but occasionally incomplete' are not backed by any metric, ground-truth comparison, or human evaluation protocol. As a result, the conclusion that the pipeline 'significantly accelerates the transition from unstructured enterprise documentation to rigorous, machine-readable formats' is not yet supported. I request a scoring scheme for generated workflows (e.g., structural alignment with a reference model, place/transition precision and recall, or expert ratings on a defined scale) and a per-model accuracy table to accompany the runtime table.","section":"§3.3, Table 2"},{"comment":"The numbers in §3.2 and Table 1 are inconsistent. The text says the LLM 'correctly validating all the clean instructions' and that precision was '1.0 (or 0.96 if we let the zero-day error ride)', then reports accuracy '0.98' for the error set, while Table 1 lists both Precision and Accuracy as 0.96. The definitions of precision and accuracy are also not given (e.g., accuracy over all 100 instructions, or over the 50 faulty ones only). Please reconcile the reported scores and define the metrics precisely; this affects the credibility of the central 0.96 claim.","section":"§3.2, Table 1 vs text"}],"minor_comments":[{"comment":"Section 3.1 refers to GPT-OSS:120B in the model list and Figure 3, while §3.2 uses GPT-OSS:250B and the scaling discussion mentions GPT-OSS (250B parameters); clarify whether these are two distinct checkpoints or a typo.","section":"§3.1, §3.2"},{"comment":"The x-axis labels '1..10' and the plotted series are not defined in the caption; please state what each run/series represents and add replicate counts or error bars.","section":"Figure 3"},{"comment":"The phrase 'zero-day error' is informal and undefined; replace it with standard terminology such as 'undetected annotation error' or 'false positive against the original annotation.'","section":"§3.2"},{"comment":"The abstract claims 'attributed graph representations' and 'formal logic approaches' that are processed downstream, but the reported experiments stop at Petri-net JSON generation; clarify the scope so readers do not expect end-to-end vulnerability reports in this paper.","section":"Abstract, §3.3"},{"comment":"The caption spells 'MIRABELE' while the text uses 'MIRABELLE'; fix the typo.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is an early system/experience report rather than a fully validated empirical study. The central gap—same-model synthetic evaluation and the absence of quantitative Petri-net accuracy—is fixable in a revision, but the authors should either add a real-data evaluation or substantially weaken the generalization claims in §4. I would also ask the editor to verify that the venue's expectations for empirical validation are consistent with this level of evidence, since the current version is likely to be judged as preliminary by many reviewers."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper has a sensible idea and honest reporting, but the evidence doesn't yet support the claim about large-scale automated ERP documentation processing. The strongest experiments lean on self-generated test data, which makes the reported accuracy numbers hard to trust as measures of real-world performance.\n\nWhat's new: the specific integration of off-the-shelf LLMs into a pipeline for finding contradictions, flagging work-instruction errors, and generating Petri nets from ERP docs. The contradiction experiment shows a clear chunk-size trend and an honest caveat that SCC isn't business prose. The work-instruction experiment, despite its circularity, includes a nice detail: the LLM caught a genuine error in a supposedly clean instruction that the authors had missed. The Petri net case study demonstrates a real end-to-end pipeline on an Oracle user guide, with a believable qualitative comparison across quantized and non-quantized models.\n\nThe soft spots are proportionate to how soft they are. The circularity in Section 3.2 is the biggest issue: GPT-OSS:250B generates the 50 erroneous work instructions, and then the same model flags them. That measures self-consistency with the model's own generation priors, not the ability to catch human-authored errors. The authors acknowledge the SCC caveat in Section 3.1 but not the deeper problem that the host document is unspecified, which makes it hard to judge whether the injection setup resembles real documentation. The Petri net extraction has no quantitative evaluation at all, yet Section 4 calls it \"successful generation.\" Also, exact accuracy numbers with variance are missing from Section 3.1; we get a figure and a single .96 in Table 1, with no sense of stability across runs.\n\nWho is this for? Practitioners building LLM-based document-processing pipelines will get useful engineering hints: chunk size matters, smaller chunks helped, quantization trade-offs are real, and a well-designed prompt with logprob-based filtering can scale. Researchers studying LLM evaluation will find a cautionary example of self-generated test data.\n\nI'd send this to peer review at an applied venue or workshop, with the expectation that the authors address the self-generation issue and add some external or human-annotated data. It's an honest work-in-progress, not a fundamentally invalid approach, but the current support is too thin for the broad claims in Section 4.","headline":"Plausible system, weak evaluation: the same LLM that writes the test data shouldn't be the one scoring it.","tokens_in":10944,"tokens_out":1431,"would_cite":false,"duration_ms":15450,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Off-the-shelf LLMs, without fine-tuning, can find contradictions and subtle numeric errors in business documentation and automatically turn ERP user guides into executable Petri nets.","keywords":["business process documentation","large language models","contradiction detection","work instruction errors","Petri net generation","ERP user guides","vulnerability discovery","LLM-as-classifier"],"falsifier":"Run the work-instruction flagging prompt on a set of genuinely human-authored erroneous instructions from an equipment manual or field service report (errors like wrong pressure conversions or contradictory torque limits). If accuracy on those instructions falls clearly below the reported 0.96, the claim that the approach generalizes from synthetic instructions to real business documentation is falsified.","tokens_in":9988,"feed_emoji":"🔍","tokens_out":6294,"duration_ms":54671,"temperature":0.7,"pith_summary":"The paper tries to establish that off-the-shelf LLMs, used without fine-tuning, can serve as the extraction and error-detection front end of a business-logic vulnerability pipeline. Across three experiments, the authors show that LLMs can find injected contradictions buried in long documents, flag subtle unit-conversion and range-limit errors in work instructions, and turn a chapter of a real ERP user guide into a Petri net model in machine-readable JSON. The payoff: business processes, like software, have defects, and much of that defect evidence sits in verbose natural-language documentation that is slow for humans and brittle for classical NLP to analyze. If the results transfer, a largely automated pipeline can convert such documentation into formal models that downstream logic analyzers can check for flaws.","feed_headline":"LLMs catch hidden errors in business process docs","feed_subtitle":"Off-the-shelf models flag contradictory instructions and turn ERP guides into Petri nets.","key_machinery":"The carrying mechanism is the Documentation Processing Toolkit (DPT), a set of LLM-driven stages built around a shared idea: break documents into chunks, prompt a general-purpose LLM to classify or extract, then feed structured output into a formal-model downstream. For contradiction detection, the paper pairs chunks, asks the model whether two passages are inconsistent, and uses logprob outputs on single yes/no tokens (LLM-as-classifier) to cull unlikely pairs and avoid checking all quadratic many pairs. For work instructions, a single prompt asks for a brief explanation of any inconsistency or the response 'none'. For ERP guides, the pipeline normalizes PDFs to markdown, performs structure-aware hierarchical chunking, extracts references and user-action summaries, and synthesizes JSON for a Petri net simulation toolchain. The repeated finding is that chunk size and model scale, not task-specific training, drive accuracy.","core_discovery":"The central claim is that the hypothesis that LLMs can be effective for uncovering documentation contradictions buried deeply in business text is valid, and that the same models also flag subtle numeric errors in short work instructions and reconstruct executable process structure from real user guides. In the contradiction task, GPT-OSS:120B with a 512-token chunk size performed best, accuracy rose as chunk size shrank and as model size grew, and small models failed. In the work-instruction task, the LLM achieved precision 0.96 and accuracy 0.96 over 100 synthetic instructions, even catching one error the authors had missed in a 'clean' instruction. In the ERP pipeline, all tested backends extracted references and action summaries with high fidelity, while the largest non-quantized model produced the most coherent Petri-net-compatible JSON; quantized models were faster but sometimes structurally inconsistent. Together these results support the claim that LLMs significantly accelerate the transition from unstructured enterprise documentation to rigorous, machine-readable formats.","pith_inferences":["A natural next experiment is to test the contradiction-detection step with contradiction pairs rewritten in full business prose (torque limits, tolerance ranges, sign-off rules) rather than encyclopedia-style sentences, to see whether the accuracy advantage holds without the lexical-context cue.","The logprob-based LLM-as-classifier culling could be made cheaper still by pre-filtering chunk pairs with embedding similarity, reserving the LLM for pairs that are semantically close but lexically different.","A two-tier deployment is implied by the results: a fast quantized model for broad extraction and a large non-quantized model to verify structured workflow output, which could bring the accuracy of the pipeline to controlled-information environments at acceptable throughput."],"forward_implications":["Contradiction detection in business documentation can be made practical by choosing smaller chunk sizes (512 tokens in this study) and larger models, at the cost of more chunk pairs to check.","Quantized models are a viable throughput option for extraction tasks, but structured workflow synthesis (Petri net generation) still favors a larger, non-quantized model.","The same general-purpose LLM, without fine-tuning, can handle three different document-processing jobs (contradiction classification, work-instruction flagging, and process extraction) with the same style of prompting.","The extracted Petri-net JSON can serve as input to simulation, verification, and auditing tools, closing the gap between natural-language documentation and formal logic-based fault detection."],"supporting_citations":[{"why":"supplies the contradiction corpus whose pairs are injected into host documents to simulate documentation errors.","marker":"[1]"},{"why":"prior LLM-based iterative workflow generation baseline that motivates the pipeline's structured output approach.","marker":"[6]"},{"why":"recent vision-language extraction of BPMN from images, a related but different baseline for process recovery.","marker":"[7]"},{"why":"classic NLP approach for process model generation from natural language, the limited baseline this work extends.","marker":"[14]"},{"why":"NLP-based text-to-BPMN approach claiming 81% accuracy, cited as the state of the art the LLM pipeline compares against.","marker":"[16]"},{"why":"underpins the LLM-as-classifier logprob technique used to cull contradiction chunk pairs at scale.","marker":"[23]"}],"fun_headline_variants":["LLMs expose hidden flaws in business process docs","AI flags contradictions buried in enterprise guides","Business docs' hidden errors surfaced by LLMs","LLMs turn messy documentation into clean processes","Large language models sniff out process vulnerabilities"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation data is assumed to resemble real business documentation errors: the contradiction corpus is not business text (which the authors note may give the LLM an unfair advantage), and the erroneous work instructions were generated by the same family of large language model that was later scored, so if real human-authored documentation errors look different, the reported accuracies may not transfer.","fun_headline_variants_meta":{"raw":{"variants":["LLMs expose hidden flaws in business process docs","AI flags contradictions buried in enterprise guides","Business docs' hidden errors surfaced by LLMs","LLMs turn messy documentation into clean processes","Large language models sniff out process vulnerabilities"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000201,"raw_usage":{"total_tokens":1394,"prompt_tokens":976,"completion_tokens":418,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":592,"completion_tokens_details":{"reasoning_tokens":352}},"tokens_in":592,"tokens_out":418,"duration_ms":4625,"temperature":1.0,"reasoning_tokens":352,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T00:05:56.054751+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the work-instruction flagging prompt on a set of genuinely human-authored erroneous instructions from an equipment manual or field service report (errors like wrong pressure conversions or contradictory torque limits). If accuracy on those instructions falls clearly below the reported 0.96, the claim that the approach generalizes from synthetic instructions to real business documentation is falsified.","supporting_citations":[{"cited_title":"Finding contradictions in text","cited_arxiv_id":null,"evidence_quote":"supplies the contradiction corpus whose pairs are injected into host documents to simulate documentation errors."},{"cited_title":"POWL: partially ordered workflow language","cited_arxiv_id":null,"evidence_quote":"prior LLM-based iterative workflow generation baseline that motivates the pipeline's structured output approach."},{"cited_title":"Process Model Generation from Natural Lan- guage Text","cited_arxiv_id":null,"evidence_quote":"classic NLP approach for process model generation from natural language, the limited baseline this work extends."},{"cited_title":"A Machine Translation Like Approach to Generate Business Process Model from Textual Description","cited_arxiv_id":null,"evidence_quote":"NLP-based text-to-BPMN approach claiming 81% accuracy, cited as the state of the art the LLM pipeline compares against."},{"cited_title":"Language models are few -shot learners","cited_arxiv_id":null,"evidence_quote":"underpins the LLM-as-classifier logprob technique used to cull contradiction chunk pairs at scale."}],"review_version":1}