{"id":"90c0b649-5df0-4f4b-9d57-71a6560ba51c","arxiv_id":"2507.00066","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"An interface knowledge graph combined with logged operator behavior can identify human failure events from error-prone and time-deviated paths, and simple layout metrics can be mapped to interface risk levels, but validation is limited to six students and eight labeled samples.","lead":"This paper presents InSight-R, a framework that maps operator actions in a simulated nuclear control room onto an interface knowledge graph to automatically flag error-prone and unusually slow task paths as human failure events. It also derives simple interface metrics to score interface risk, aiming to reduce reliance on expert judgment in human reliability analysis.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Time-deviated HFE identification rests on an unvalidated duration-to-failure assumption; Section 4.2.2's own evidence does not test it.","rationale":"The reader's weakest assumption matches the load-bearing point. The framework has a coherent pipeline (IE-KG construction, action tracking, error-path mapping), and the error-based HFE identification is at least empirically grounded, but the time-deviated half is not. Section 4.2.2 states the critical monotonicity assumption directly, and the final HFE list shows it is doing work: several time-deviated paths have no observed errors. The descriptive 7-of-12 statistic is not sufficient support, especially with N=6 participants. The PIF estimation problem (8-sample MLP, expert labels, no released code or data) compounds the issue but is secondary to the HFE identification claim. The paper also contains technical inconsistencies (VD equation vs. definition, incomplete IE-KG description, duplicated HFE entries) that increase correctness risk. Overall the conditional verdict is appropriate: the idea merits a larger validation study, but the current evidence does not establish the central claims.","tokens_in":13490,"tokens_out":4050,"duration_ms":38393,"concrete_test":"Re-analyze the Section 4.1 session logs: for every path, pair each participant's task duration with whether an error occurred, and fit a mixed-effects logistic regression with error as outcome and log duration as fixed effect, with random intercepts for participant and path. The time-based HFE claim survives only if the duration coefficient is positive and significant, or if a permutation test shows the time-deviated set is enriched for errors beyond chance. As a stronger check, run the same five scenarios with at least 30 participants and test whether the time-deviated paths identified in Section 4.2.2 predict hold-out errors better than the base error rate. If not, re-label time-deviated paths as workload or efficiency risks rather than HFEs.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.2.2 adopts, rather than derives, the assumption that 'longer execution time implies a higher probability of performance degradation or failure.' This is load-bearing: the final HFE list includes time-deviated paths with no recorded errors (P123, P126, P322, P411), so the HFE set is not merely a summary of observed errors. The claimed automated HFE identification treats slow operation as a failure indicator on the strength of an imported workload correlation (Ref [16]) and the paper's own descriptive observation that 7 of 12 errors occurred on relatively long paths. That observation is not a statistical test, and the overlap between error-deviated and time-deviated path sets is only partial (P122, P211, P212, P321, P414). Without a validated monotonic relationship in this interface context, designating slow paths as HFEs conflates inefficiency with failure and inflates the HFE list. A related but secondary weakness is that the MLP in Section 5.2 is trained on expert-assigned HSI labels from 8 samples, so the PIF part inherits the subjectivity the paper criticizes; however, the time assumption is the more fundamental gap because it affects the primary HFE identification claim for an entire class of paths.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes InSight-R, a framework that combines an interface-embedded knowledge graph (IE-KG) built by AutoGraph with empirical operator behavior data to automate human failure event (HFE) identification and to estimate interface-related performance influencing factors (PIFs). HFE identification is based on two criteria: error-prone operational paths and time-deviated paths detected by lognormal modeling of task durations. PIF estimation uses three interface metrics (visual density, semantic interference density, and interaction span) as inputs to a multi-layer perceptron whose outputs are mapped to IDHEAS-ECA HSI levels. The framework is demonstrated in a human-in-the-loop study with six graduate students in the HTRPMSim simulator and is applied to predict PIF levels for three untested procedures. The paper also presents a designer-user conflict analysis intended to explain when interface problems translate into errors.","tokens_in":13648,"tokens_out":5810,"duration_ms":64887,"significance":"If the framework's claims are supported, it would be a useful step toward reducing reliance on subjective expert judgment in HRA by providing a reproducible, data-driven pipeline from interface structure to risk indicators. The paper deserves credit for a concrete behavioral-data collection pipeline, a clear graph-based representation of interface elements, and an explicit attempt to quantify semantic interference and interaction span. However, the evidence base is currently small: six participants, no statistical validation of the time-to-failure assumption, and an MLP trained and evaluated on only eight expert-assigned samples. The significance is therefore conditional on substantial additional validation of the load-bearing assumptions.","major_comments":[{"comment":"The central claim that time-deviated paths are human failure events rests on the unvalidated assumption that 'longer execution time implies a higher probability of performance degradation or failure.' This assumption is imported from Ref. [16] and IDHEAS-ECA guidance [17], not tested in this experiment. The only supporting evidence is the descriptive observation that 7 of 12 errors occurred on longer-duration paths, and the lognormal model fixes sigma=0.28 without any goodness-of-fit test or sensitivity analysis. Because the final HFE list includes paths that appear only in the time-deviated set (e.g., P126, P322, P411) and are not in the observed error set, the procedure conflates slow performance with failure. Please provide a statistical test of the duration-error relationship, report sensitivity to the threshold and to sigma, or explicitly reframe these as candidate risk indicators rather than confirmed HFEs.","section":"Section 4.2.2"},{"comment":"The MLP classifier is trained and evaluated on a dataset of only eight expert-assigned samples, as shown in the confusion matrix in Figure 11(a) (2 HSI0, 4 HSI1, 2 HSI5). Five-fold cross-validation on eight samples does not provide a reliable estimate of generalization, the reported mean accuracy of 0.875 is obtained without a held-out test set, and the class imbalance makes the result difficult to interpret. Moreover, because the labels are expert judgments, the PIF quantification inherits the very subjectivity the paper criticizes. The relationship between Table 1, which lists dozens of paths with PIF levels, and the eight samples in Figure 11(a) is also unclear. Please report the exact number of training samples, fold assignments, per-class metrics, and an external validation set; alternatively, present the MLP as an illustrative mapping rather than a validated model.","section":"Section 5.2 and Figure 11"},{"comment":"The definition of Visual Density is internally contradictory. The text defines VD as 'the ratio of the total area occupied by visible interface elements to the overall area of the interface,' but Equation (1) gives VD = N_target/N_elements, with N_target always equal to one. The latter is simply the reciprocal of the number of visible elements and is not an area-based density; no area computation appears anywhere in the case study. This metric is load-bearing because VD is one of the three MLP inputs used to predict PIF levels. Please correct the equation, compute an actual area-based density, or rename the metric so that it does not claim to measure visual density.","section":"Section 3.4, Eq. (1)"},{"comment":"The final list of identified HFEs is presented as three separate bullet groups with duplicate entries (P216, P323, P211, P212, P321, P414 appear more than once), and no set operation such as union or intersection is specified. As a result, the reader cannot determine which paths are HFEs from the error-proneness criterion, from the time-deviation criterion, or from both. Please provide a single unambiguous set, with per-path notation of which criterion or criteria triggered identification.","section":"Section 4.2, final HFE list"}],"minor_comments":[{"comment":"The framework name is spelled inconsistently: the title uses 'InSight-R' while Section 3.1 and several other places use 'Insight-R'; please unify the spelling.","section":"Throughout"},{"comment":"The sentence 'For full details on IE-KG construction.' is an incomplete fragment and does not point to an appendix, reference, or supplementary file; please complete the reference or remove the sentence.","section":"Section 4.1"},{"comment":"The claim that '66% occurrence rate' indicates a strong correlation is based on only two outcome errors from three total cases; this should be reported with an explicit caveat about the tiny sample size.","section":"Section 5.3"},{"comment":"The paper states that semantic similarity is computed with text-embedding-ada-002, but the cited reference [18] is the Nomic Embed paper; please correct the citation or use the appropriate model reference.","section":"Reference [18]"},{"comment":"The paragraph about 'unmanned' plants and the hidden role of human factors appears nearly verbatim in both the Introduction and Section 2.3; please keep only one copy.","section":"Section 1 and Section 2.3"}],"recommendation":"major_revision","confidential_remarks":"This is a promising framework paper, but the abstract and conclusions overstate objectivity and validation relative to the evidence. The time-deviation assumption and the eight-sample MLP evaluation are the two load-bearing gaps; both can be addressed by reframing claims and adding sensitivity analyses, but the authors should be asked to do so before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The specific pipeline here is new and worth knowing: InSight-R maps behavioral logs onto an interface-embedded knowledge graph, then flags HFEs from both error-prone and time-deviated paths, and scores interface risk with three simple metrics mapped to IDHEAS-ECA PIF levels. I don't know of another paper that does exactly that. The framework is coherent, and the designer-user conflict discussion is honest—it explicitly says conflict doesn't deterministically cause error and shows examples both ways. Credit where due: this is a sensible step toward reducing expert-judgment dependence in HRA, and the authors acknowledge the student-participant limitation up front.\n\nThe problems are in the evidence, not the idea. The biggest one, as the stress-test note says, is the tail-end time assumption. Section 4.2.2 adopts \"longer execution time implies higher failure probability\" from a co-author's prior modeling study, plus the descriptive observation that 7 of 12 errors landed on longer paths. That is not a statistical test, and the final HFE list includes four paths (P123, P126, P322, P411) with no recorded errors at all. Calling slow operation a human failure event conflates inefficiency with failure, and the paper gives no validation that the relationship holds in this interface context. Second is the MLP: eight expert-assigned samples, 5-fold CV, fold accuracies swinging between 0.625 and 1.0. The VD/SID/IS features are objective, but the labels are expert judgment, so the model inherits the very subjectivity the paper criticizes. The HFE list is also presented as three bullet groups with duplicates and no explanation of how they combine; the VD equation doesn't match its prose definition; one sentence in 4.1 is cut off; and no data or code is released—a real gap for a framework whose selling point is reproducibility.\n\nNone of this kills the idea. It's a solid proof-of-concept that overclaims. The citation pattern is acceptable: the self-citations are the group's own prior HRA work, though note ref [16] is the source of the untested time assumption, so it isn't independent support.\n\nSend it to review. A serious referee can push on the time-to-error relationship and the PIF validation. With a larger dataset, released artifacts, and a real statistical check, the paper could earn its claims.","headline":"A genuinely novel HFE-identification pipeline that maps behavioral logs onto an interface knowledge graph, held back by an untested duration-to-failure assumption and an 8-sample expert-labeled MLP.","tokens_in":14270,"tokens_out":5402,"would_cite":true,"duration_ms":88222,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"InSight-R maps operator clicks onto an interface knowledge graph to automate human-error risk identification","keywords":["Human Reliability Analysis","Human Failure Event Identification","Interface-Embedded Knowledge Graph","Performance Influencing Factors","Interface Complexity","Nuclear Power Control Room","Data-Driven Risk Assessment","AutoGraph"],"falsifier":"A controlled study with licensed operators performing the same procedures, with outcome errors independently determined, would settle the claim: if the time-deviated paths flagged by the lognormal model do not show elevated error rates relative to unflagged paths, the tail-end-risk component of automatic HFE identification collapses.","tokens_in":13200,"feed_emoji":"🧠","tokens_out":5150,"duration_ms":53253,"temperature":0.7,"pith_summary":"This paper claims that human failure events (HFEs) in a digital control room can be identified automatically rather than by expert judgment, by fusing recorded operator behavior with a knowledge graph of the interface. If this works, human reliability analysis becomes reproducible, data-driven, and updatable in real time, and interface-related performance influencing factors (PIFs) can be quantified instead of scored subjectively. The framework, InSight-R, builds an interface-embedded knowledge graph (IE-KG) from the screen layout, maps clicks, cursor paths, errors, and task durations onto its nodes, and flags both error-prone and time-deviated operational paths as candidate HFEs. It also defines three computable metrics, visual density (VD), semantic interference density (SID), and interaction span (IS), and uses them to predict interface-related PIF levels, including for interfaces that have not yet been tested.","feed_headline":"Operator clicks plus an interface graph flag error-prone paths","feed_subtitle":"Data-driven HFE identification replaces expert judgment in control-room reliability analysis.","key_machinery":"The load-bearing object is the interface-embedded knowledge graph (IE-KG), a graph whose nodes are clickable interface elements and whose edges encode hierarchical and logical relations, built automatically by the AutoGraph framework. The argument runs through three tools built on it: error-path identification, which maps recorded errors onto a Bayesian network of procedure paths to find high-error routes; time-path identification, which models each path's task duration as lognormal with a fixed shape parameter of $\\sigma=0.28$ per IDHEAS-ECA guidance and flags the upper tail as tail-end risk; and three procedure-driven metrics, visual density ($VD$), semantic interference density ($SID$) from cosine similarity of element names, and interaction span ($IS$) from normalized cursor travel distance, whose values feed an MLP classifier that predicts interface-related PIF levels. The claim that these pieces interoperate is what makes automated HFE identification and quantitative PIF assessment possible.","core_discovery":"InSight-R's central claim is that an interface-embedded knowledge graph can carry a full human reliability analysis: it turns the interface into a structured graph, attaches empirical behavioral data to graph nodes, and derives HFEs from two signals, paths where operators actually made errors and paths where task duration is abnormally long. The paper argues that this procedure replaces expert elicitation with a mechanism-based, updatable analysis, and that the same graph yields quantitative interface metrics which can be mapped, through a trained classifier, onto the PIF scale used by the IDHEAS-ECA framework. In the reported case study, data-driven HFE identification produced a different list than expert judgment, catching low-complexity but high-variability actions that experts overlook, while expert judgment pointed to high-complexity paths. The paper also reports that designer-user conflict raises error probability statistically but does not determine it: two of three outcome errors occurred on HSI5-classified conflict-prone interfaces, yet some conflicting interfaces were completed without error.","pith_inferences":["Extension the paper leaves implicit: the same IE-KG pipeline is transferable to any screen-based safety-critical task, such as aircraft cockpit displays, anesthesiology workstations, or autonomous-vehicle supervision, because the graph is built from the interface alone and the behavioral data from logs.","The framework's real-time updating property implies that simulators can serve as continuous human-reliability data engines, with every training run refreshing error probabilities and PIF estimates automatically rather than through periodic expert re-scoring.","The paper's conflict analysis suggests a testable design rule: interfaces scoring HSI5 should be treated as necessary, not sufficient, causes of error, and interventions should target conflict-prone elements while preserving affordances that allow adaptive operators to compensate.","A larger operator sample with professional control-room crews would settle whether the lognormal time assumption and the 0.28 shape parameter hold outside students, and whether time-deviated paths actually predict future errors at the claimed rate."],"forward_implications":["Automated HFE lists can be regenerated each time new operator data arrives, allowing human reliability analysis to track evolving error patterns rather than freezing expert judgment.","Interface-related PIF levels become computable from screen layout and task semantics, so untested procedures can be screened for design-induced risk before deployment.","Tail-end timing deviations give a measurable early-warning signal for cognitive overload, so slow paths can be treated as risk precursors rather than only counting mistakes.","The framework's HFE output can be reconciled with expert-based IDHEAS-ECA analysis, giving HRA analysts a quantitative cross-check instead of a single subjective channel.","Design guidance becomes concrete: reduce element count, differentiate semantically similar labels, and shorten cursor travel distances on interfaces predicted as HSI5."],"supporting_citations":[{"why":"Supplies the IDHEAS-ECA macro-cognitive framework and the qualitative HSI PIF definitions onto which the paper's quantitative metrics are mapped.","marker":"[8]"},{"why":"Provides the prior workload-studies evidence that longer task duration correlates with higher cognitive load, underpinning the time-path assumption.","marker":"[16]"},{"why":"Supplies the IDHEAS-ECA technical guidance for modeling human task durations as lognormal with shape parameter 0.28 and for estimating the median from the 95th percentile time.","marker":"[17]"},{"why":"Supplies the text-embedding model used to compute semantic similarity between element names for the semantic interference density metric.","marker":"[18]"},{"why":"Provides the NUREG-1792 expert-based HFE identification process used as the comparison baseline in the discussion.","marker":"[19]"}],"fun_headline_variants":["Interface graph flags operator error paths automatically","Graph-based HFE identification replaces expert judgment","AutoGraph links interface design to operator failure rates","Operator behavior graph reveals hidden error-prone paths"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The time-based half of the automated HFE list stands on the assumption that longer task execution time means a higher chance of degraded performance or failure, an assumption borrowed from workload research and not validated by this experiment.","fun_headline_variants_meta":{"raw":{"variants":["Interface graph flags operator error paths automatically","Graph-based HFE identification replaces expert judgment","AutoGraph links interface design to operator failure rates","Operator behavior graph reveals hidden error-prone paths"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000303,"raw_usage":{"total_tokens":1764,"prompt_tokens":984,"completion_tokens":780,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":600,"completion_tokens_details":{"reasoning_tokens":725}},"tokens_in":600,"tokens_out":780,"duration_ms":8100,"temperature":1.0,"reasoning_tokens":725,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:59:56.979219+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A controlled study with licensed operators performing the same procedures, with outcome errors independently determined, would settle the claim: if the time-deviated paths flagged by the lognormal model do not show elevated error rates relative to unflagged paths, the tail-end-risk component of automatic HFE identification collapses.","supporting_citations":[{"cited_title":"US Nuclear Regulatory Commission, Washington, DC (2020)","cited_arxiv_id":null,"evidence_quote":"Supplies the IDHEAS-ECA macro-cognitive framework and the qualitative HSI PIF definitions onto which the paper's quantitative metrics are mapped."},{"cited_title":"In: International Conference on Nuclear Engineering, vol","cited_arxiv_id":null,"evidence_quote":"Provides the prior workload-studies evidence that longer task duration correlates with higher cognitive load, underpinning the time-path assumption."},{"cited_title":"In: Proceedings of the 2021 International Topical Meeting on Probabilistic Safety Assessment and Analysis (PSA 2021), pp","cited_arxiv_id":null,"evidence_quote":"Supplies the IDHEAS-ECA technical guidance for modeling human task durations as lognormal with shape parameter 0.28 and for estimating the median from the 95th percentile time."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the NUREG-1792 expert-based HFE identification process used as the comparison baseline in the discussion."}],"review_version":1}