{"id":"bfc12720-0712-47b7-855f-996a908670ed","arxiv_id":"2508.20021","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"FairLoop distills predictive neural models into editable decision trees, lets users cut out unfair rules, and fine-tunes the model on corrected labels for business process monitoring.","lead":"FairLoop is a new software tool that lets domain experts inspect a neural network's decision logic as a decision tree, edit out unfair rules, and retrain the model on the edited labels. It targets fairness in predictive business process monitoring, where removing a sensitive attribute blindly can break useful predictions.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Core mechanism relies on unmeasured distillation fidelity and no fairness evaluation; 'eliminate the bias' is not demonstrated.","rationale":"The reader's weakest_assumption correctly identifies the unmeasured link between the decision-tree edit and the MLP's actual behavior. My stress-test concurs: the paper's central claim of 'eliminating bias' depends on two unverified steps—distillation fidelity and the efficacy of relabeling/fine-tuning. The paper itself flags the lack of fairness metrics and user studies, which supports a CONDITIONAL verdict. The software artifact and architecture are plausible, but no quantitative evidence demonstrates that the tool actually removes bias; the demonstration is anecdotal. I see no internal inconsistency or obvious fatal flaw, only a missing empirical core. Therefore, no change to the reader's CONDITIONAL verdict is warranted; it should remain conditional pending the concrete measurements proposed. I agree with the reader rather than only partial because the same load-bearing assumption is named and the same missing evidence is identified.","tokens_in":4697,"tokens_out":2727,"duration_ms":34409,"concrete_test":"Reproduce the simulated cancer-screening demo from the public repository. (1) Compute fidelity of the distilled tree to the original MLP (agreement rate on all prefixes, and class-wise F1). (2) Measure a standard fairness metric, e.g., demographic parity of 'refuse screening' predictions for female vs. male prefixes, before the edit, after the tree edit and fine-tuning, and after a control retraining with gender removed entirely. (3) Check whether the fine-tuned model's refusal-rate disparity substantially decreases and whether the distilled tree from the fine-tuned model no longer contains the biased branch. If fidelity is low (<95% agreement) or the disparity fails to shrink, the central claim collapses.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing assumption is that the distilled decision tree is a faithful surrogate of the MLP and that the fine-tuning step actually transfers the edited tree's logic back into the MLP. Section 2 states the tree is built from 'all possible prefixes in the event log and the corresponding prediction made by the MLP,' but no fidelity metric (e.g., tree-MLP agreement on held-out prefixes) is reported. Without this, removing a rule such as 'gender = female -> refuse screening' may edit a branch that does not correspond to the MLP's real decision pathway, so the relabeled training data may not reflect true MLP changes. Furthermore, the relabeled data are derived from the tree, which is derived from the MLP; any approximation error in the surrogate is propagated into the new labels. After fine-tuning, the MLP could still rely on gender (if gender remains a feature), or it could fail to unlearn the original bias entirely. The paper explicitly admits the 'absence of built-in metrics for quantifying the fairness of predictive models,' so even the authors cannot verify that the intervention achieves its stated goal. The simulated demo shows a screenshot, not before/after measurements of prediction distributions or fairness. Therefore, the central claim that FairLoop 'eliminate[s] the bias' is unsupported by the evidence presented.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents FairLoop, a web-based tool for human-in-the-loop bias mitigation in predictive business process monitoring (PBPM). The proposed workflow has four steps: train an MLP on an event log for next-activity prediction; distill a decision tree from the MLP using all prefixes in the event log and their MLP predictions; let a domain expert inspect the tree and remove or modify rules judged to encode negative bias (while preserving positive, context-dependent uses of sensitive attributes); and re-label the training data with the edited tree, then fine-tune the original MLP on the relabeled data. The authors argue that this selective, context-aware intervention improves on fairness methods that remove sensitive attributes uniformly. The paper includes an architecture description, a simulated cancer-screening demonstration event log, a screenshot of the interactive tree editor, and links to public code, a video, and a live demo. The authors explicitly state that no user study has been conducted and that the tool currently includes no built-in fairness metrics; the demonstration is assessed by manual inspection of the distilled tree.","tokens_in":4979,"tokens_out":3714,"duration_ms":47011,"significance":"If the central mechanism works, FairLoop would be a practically useful contribution: it offers a graphical, human-guided workflow that lets PBPM users inspect black-box MLP decisions through distilled trees and apply context-sensitive fairness interventions rather than blanket removal of sensitive attributes. The modular architecture and public code are strengths, and the paper is honest about its current limitations. However, the load-bearing claim—that the distilled tree faithfully represents the MLP and that editing it and fine-tuning on relabeled data actually removes bias—is not quantitatively demonstrated anywhere in the manuscript. The paper reports no distillation fidelity, no fairness metric, no before/after prediction comparison, and no accuracy/F1 figures, despite stating that such metrics are shown in the interface. The significance of the contribution therefore remains conditional: the tool is plausible and demonstrable, but the evidence supplied does not substantiate the claim that FairLoop enables fairer predictions.","major_comments":[{"comment":"The decision tree is built from \"all possible prefixes in the event log and the corresponding prediction made by the MLP,\" but no measure of distillation fidelity is reported. The entire intervention depends on the tree being a faithful surrogate of the MLP: if the tree misrepresents the MLP's decision logic, then removing a rule such as \"gender = female -> refuse screening\" may alter a branch the MLP does not actually use, and the relabeled training data will not reflect the intended change. Please report tree–MLP agreement on held-out prefixes (and, if available, tree size/depth or fidelity by output class), and show the before/after distilled trees for the simulated log to demonstrate that the edit changes the surrogate in the intended place.","section":"Section 2, Distillation step"},{"comment":"The paper's abstract and Section 1 claim that the re-labeling and fine-tuning procedure is used \"to eliminate the bias,\" but Section 4 explicitly notes the \"absence of built-in metrics for quantifying the fairness of predictive models\" and that interventions are assessed \"solely through manual inspection of the distilled decision tree.\" The only demonstration is a screenshot (Fig. 3); no quantitative before/after comparison is provided. Please add a concrete fairness evaluation for the simulated cancer-screening log—e.g., demographic parity or equalized odds of the next-activity prediction with respect to gender, before and after the intervention—together with the accuracy/F1/precision/recall numbers that Section 2 says are displayed. Without such measurements, the central claim that FairLoop removes bias is unsupported.","section":"Section 4, Limitations and demonstration"},{"comment":"Even if the tree is an accurate surrogate, the pipeline has a potential circularity: the tree is derived from the MLP, the edited tree generates new labels, and the MLP is fine-tuned on those labels. The MLP may converge to the edited behavior largely by construction on the training prefixes, but it is not shown whether the fine-tuned model generalizes the edit to unseen prefixes or merely memorizes the relabeled training set. It is also not shown whether the model simply redirects its decision to correlated features (e.g., other attributes correlated with gender). Please report fine-tuning details (epochs, learning rate, data split) and a holdout evaluation showing that predictions on affected prefixes actually change, plus an ablation comparing fine-tuning on edited-tree labels versus simply retraining without the sensitive attribute.","section":"Section 2, Re-labeling and fine-tuning"}],"minor_comments":[{"comment":"Typo: \"from even log data\" should be \"from event log data.\"","section":"Section 1, paragraph 4"},{"comment":"The numbered steps 1–8 in the architecture diagram are not explained in the text. A short caption or corresponding paragraph enumerating the steps would make the workflow much easier to follow.","section":"Figure 1"},{"comment":"The phrase \"all possible prefixes\" is ambiguous for a process log: does this include prefixes truncated at every event position, with all case attributes and encoding variants? Clarify the exact construction and how the MLP output is mapped to a discrete target for tree training. Also, the \"custom implementation\" of the decision tree is not described; please specify what operations (removing a node, retraining a subtree) are supported and how they are implemented.","section":"Section 2, decision tree construction"}],"recommendation":"major_revision","confidential_remarks":"The paper is written as a tool/demonstration contribution, and the technical feasibility rests heavily on prior work [11]. For a demo-track venue, the current level of evidence might be acceptable; for a journal-style publication, the missing distillation-fidelity and fairness evaluations are load-bearing. I recommend requiring the quantitative checks described in the major comments before acceptance, rather than rejecting, because the public code and live demo make the system verifiable and the authors have openly acknowledged the main limitations."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"FairLoop is a software demonstration, not a new method. The distill-modify-retrain workflow comes from the authors' earlier paper [11]; the contribution here is a browser-based tool with an editable decision tree, plus a public code repo and demo. The paper is well-written and unusually candid: Section 4 explicitly says there is no user study, no built-in fairness metrics, and only a simulated event log. That honesty earns credit.\n\nWhat the tool does well: the architecture is clean (React frontend, Flask backend, separate prediction and distillation modules), and the interaction model—editing a tree branch and re-labeling—is intuitive for a practitioner. The comparison with Nirdizati and the ProM discrimination-aware tree plugin is fair, and the point about context-aware versus uniform bias removal is legitimate.\n\nThe soft spots are exactly where the reader and stress-test point. The paper claims FairLoop can 'eliminate the bias,' but nowhere measures whether the intervention works. There is no distillation fidelity metric, so we do not know if the tree the user edits actually reflects the MLP's decisions. The re-labeling step propagates any surrogate error, and fine-tuning could just re-learn the old pattern if gender remains a feature. The demo shows a screenshot, not before/after outcome distributions or fairness values. These are not minor quibbles; they are load-bearing for the central claim. The paper's own limitation statement admits the fairness metrics gap, which makes the 'eliminate' wording in the introduction an overstatement.\n\nThat said, this is a workshop demo/resource paper, and within that genre the standard is different. The tool appears real, the code is linked, and the workflow is coherent. The citations are appropriate; relying on their own prior work is fine when that work is the source of the method. No citation issues.\n\nBottom line: the paper is worth a serious referee, but the referee should require either tempering the efficacy claims to 'support guidance toward' fairer predictions, or adding at least one quantitative sanity check—tree-model agreement on held-out prefixes, or a simple demographic parity comparison on the simulated log before/after the edit. As written, it is a solid demo with an unsupported headline claim.","headline":"FairLoop is a useful tool demo with an honest limitations section, but its central fairness claim is unmeasured—worth a referee who will push for either softened wording or one quantitative check.","tokens_in":5454,"tokens_out":2547,"would_cite":false,"duration_ms":27482,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper presents FairLoop, a tool that distills a neural prediction model into an editable decision tree and lets users delete unfairly biased rules before re-labeling data and fine-tuning the model for fairer predictions.","keywords":["predictive business process monitoring","fairness","decision tree distillation","human-in-the-loop","bias mitigation","neural networks","explainable AI","process mining"],"falsifier":"Measure the agreement rate between the distilled decision tree and the MLP on all event-log prefixes; then delete one biased rule, fine-tune, and check whether the MLP's predictions on prefixes matching that rule actually shift relative to a randomly chosen control set. If the agreement is low or the targeted predictions do not move, the loop has failed to transfer the edit.","tokens_in":4637,"feed_emoji":"⚖️","tokens_out":7461,"duration_ms":75717,"temperature":0.7,"pith_summary":"FairLoop is a graphical tool that gives users a way to fix unfair predictions in a process-monitoring neural network without retraining from scratch or banning a sensitive attribute outright. It turns the network's decision logic into an editable decision tree, lets a user delete rules that encode negative bias, then uses the corrected tree to re-label the training data and fine-tune the network. The work's aim is to make context-aware fairness practical: the same attribute (e.g., gender) can stay in the model where it is clinically or operationally necessary while being removed where it is discriminatory. The paper demonstrates the workflow on a simulated cancer-screening event log and argues the approach fills a gap left by uniform bias-removal methods. It does not claim quantitative fairness results; it claims the tool makes selective, human-guided fairness intervention possible.","feed_headline":"FairLoop distills neural nets into editable trees to cut unfair rules","feed_subtitle":"A browser-based loop lets domain experts fix biased predictions without dropping sensitive attributes entirely.","key_machinery":"The editable decision tree: an interpretable tree trained on the set of all event-log prefixes and the MLP's predictions for them, converted into a mutable structure that users can alter by deleting nodes or retraining subtrees. It is the translation layer between the black-box MLP and the human; after the user edits it, its predictions serve as the new target labels for fine-tuning, which is what propagates the edit back into the neural model. The repeated distill-alter-tune cycle is the loop that drives FairLoop.","core_discovery":"The paper's central claim is that a human-in-the-loop cycle—distilling a decision tree from a trained neural network, letting a user delete rules that encode negative bias, re-labeling every event prefix with the edited tree's predictions, and fine-tuning the neural network on that re-labeled data—can remove unfair behavior selectively while preserving the model's predictive expressiveness and its legitimate use of the same sensitive attribute in other contexts. The paper demonstrates this in a browser-based tool on a simulated cancer-screening event log, showing where gender drives a fair screening decision (positive bias) and where it drives an unfair refusal (negative bias). It argues tha","pith_inferences":["The same distill-edit-retrain recipe should transfer to other black-box models and other prediction targets (e.g., outcome or remaining-time prediction) whenever the distilled tree is a faithful enough mirror of the model; the paper leaves that fidelity unmeasured.","Without a built-in fairness metric, users currently judge success by inspecting the tree; adding an automatic before/after comparison of prediction rates per sensitive group would turn the loop from a manual check into a measurable intervention.","If real event logs with sensitive attributes produce larger, messier trees than this simulated log, the human-editing step may become the bottleneck; guiding user attention to the most biased subtrees would then be necessary."],"forward_implications":["A user can remove one discriminatory rule, such as 'gender = female → refuse screening,' while keeping other rules that use gender for clinically necessary decisions.","Because the final predictor is still the MLP, the interpretability of the tree does not come at the cost of the model's predictive capacity.","The iterative loop lets users re-distill after fine-tuning to see whether the edited behavior took effect and to make further adjustments.","The tool lowers the technical barrier for fairness intervention: a domain expert with no programming experience can do the editing in a browser."],"supporting_citations":[{"why":"Defines predictive business process monitoring, the task FairLoop applies to.","marker":"[1]"},{"why":"Supplies the underlying human-in-the-loop distill-edit-retrain method that FairLoop implements and demonstrates.","marker":"[11]"},{"why":"Existing graphical tool for predictive monitoring whose interpretability approach FairLoop extends.","marker":"[12]"},{"why":"Prior fairness-aware process-mining plugin that FairLoop contrasts because it removes sensitive attributes uniformly.","marker":"[13]"},{"why":"Prior fairness approach FairLoop contrasts as not user-involving and uniformly de-biasing.","marker":"[14]"},{"why":"Adversarial fairness approach sharing the uniform-removal limitation FairLoop addresses.","marker":"[15]"}],"fun_headline_variants":["Human-in-the-loop cuts unfair rules selectively in business AI","Distill, edit, retrain: human loop removes biased rules in monitoring","FairLoop: edit decision trees to fine-tune fairer predictions","Business AI fairness: humans refine rules, not drop sensitive data"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The distilled decision tree must faithfully reproduce the MLP's decisions on the relevant prefixes; if it does not, deleting a rule from the tree will not change the MLP's behavior in the intended way.","fun_headline_variants_meta":{"raw":{"variants":["Human-in-the-loop cuts unfair rules selectively in business AI","Distill, edit, retrain: human loop removes biased rules in monitoring","FairLoop: edit decision trees to fine-tune fairer predictions","Business AI fairness: humans refine rules, not drop sensitive data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000498,"raw_usage":{"total_tokens":2205,"prompt_tokens":603,"completion_tokens":1602,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":347,"completion_tokens_details":{"reasoning_tokens":1528}},"tokens_in":347,"tokens_out":1602,"duration_ms":12667,"temperature":1.0,"reasoning_tokens":1528,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T15:12:56.299262+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the agreement rate between the distilled decision tree and the MLP on all event-log prefixes; then delete one biased rule, fine-tune, and check whether the MLP's predictions on prefixes matching that rule actually shift relative to a randomly chosen control set. If the agreement is low or the targeted predictions do not move, the loop has failed to transfer the edit.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines predictive business process monitoring, the task FairLoop applies to."},{"cited_title":"Käppel, J","cited_arxiv_id":null,"evidence_quote":"Supplies the underlying human-in-the-loop distill-edit-retrain method that FairLoop implements and demonstrates."},{"cited_title":"Jorbina, A","cited_arxiv_id":null,"evidence_quote":"Existing graphical tool for predictive monitoring whose interpretability approach FairLoop extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior fairness-aware process-mining plugin that FairLoop contrasts because it removes sensitive attributes uniformly."},{"cited_title":"Achieving Group Fairness through Independence in Predictive Process Monitoring","cited_arxiv_id":"2412.04914","evidence_quote":"Prior fairness approach FairLoop contrasts as not user-involving and uniformly de-biasing."},{"cited_title":"Achieving Fairness in Predictive Process Analytics via Adversarial Learning (Extended Version)","cited_arxiv_id":"2410.02618","evidence_quote":"Adversarial fairness approach sharing the uniform-removal limitation FairLoop addresses."}],"review_version":1}