REVIEW 4 major objections 4 minor 2 cited by
AI-Driven IRM: Transforming insider risk management with adaptive scoring and LLM-based threat detection
T0 review · 4 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read An autoencoder risk score seeded by a static PRISM baseline and refined through analyst feedback cuts false positives from 42% to 17% and raises true-positive detection from 65% to 85%.
desk verdict Vendor system description with circular evaluation; the reported FP/TP gains are fitting artifacts of retraining on the same analyst feedback used to measure them. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing piece is the autoencoder risk scorer trained on PRISM-labeled activity, assisted by a weighted linear baseline. PRISM sums seven weighted factors — privilege, activity type, application context, IP reputation, business hours, device compliance, and cumulative activity — into a normalized 0–1 score. The AI model replaces the weights with a learned reconstruction baseline: each new event gets a risk score from its reconstruction error, and analysts can adjust any score through $S_{\mathrm{final}} = S_{\mathrm{AI}} + \alpha(S_{\mathrm{user}} - S_{\mathrm{AI}})$. Once enough adjusted instances accumulate, the autoencoder is retrained, prioritizing analyst feedback over its initial PRISM-derived weights. That feedback loop is what converts a static heuristic into an organization-specific, continuously updating risk measure.
What would settle it
Rerun the evaluation on a strict holdout that excludes every event whose analyst adjustment fed the retraining loops, including the 12% of scores analysts changed. If the false positive rate rises back toward 42% or the true positive gain disappears, the reported improvement is feedback-loop fitting rather than detection skill. A companion test: in a live deployment, compare the AI score's ranking against a second analyst's independent judgment on events the model has never been retrained on.
Extended reading notes
Core claim
The central claim is that a hybrid scoring pipeline — a static PRISM baseline that seeds an autoencoder, which then assigns risk by reconstruction error and is incrementally retrained on analyst feedback — detects insider threats better than PRISM alone. On the paper's reported numbers, the AI score cuts the false positive rate from 42% to 17%, raises true positive detection from 65% to 85%, and lowers the false negative rate from 18% to 12% over a 12-week feedback process. The same deployment is said to process up to 10 million log events per day with sub-300ms query latency and to reduce incident resolution time from 45 to 24 minutes. The authors present these results as evidence that continuous human-in-the-loop adaptation, rather than more elaborate static rules, is the scalable path to insider-risk detection.
Load-bearing premise
The reported gains depend on expert-annotated risk labels being correct ground truth and on the final evaluation using events the model had not already seen through its own retraining feedback.
Editorial extensions
If this is right
- If the metric gains hold, security teams would review fewer than half as many false alerts while catching more genuine insider threats, reducing alert fatigue.
- An organization adopting the feedback loop would get a risk score that drifts toward its own analysts' judgment without rewriting policy rules.
- The reported throughput — 10 million log events daily with sub-300ms queries — implies the scoring pipeline can run in real time at enterprise scale.
- Automated enforcement triggered by policy violations (privilege revocation, file-access restriction, device disconnection) would shrink the window between detection and response.
- The context-aware LLM recommendations, if they work as described, give analysts a plain-language reason for each high score, supporting faster triage.
Reading between the lines
- If the central mechanism is correct, the same pattern — a simple explainable baseline, an autoencoder seeded on it, and analyst-correction retraining — should transfer to other alert-triaging domains such as fraud or cloud-cost anomaly detection.
- The paper's reported 12% analyst-adjustment rate is itself testable: if the feedback loop genuinely works, corrections should converge over time, and inter-analyst agreement on high-risk events could be measured to test whether the label noise is low.
- The latency figures suggest the practical bottleneck is log ingestion and indexing, not the autoencoder; this predicts that larger embedding or graph-based features could be added without breaking the sub-300ms budget, which would be a cheap follow-up experiment.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript describes an AI-driven insider risk management system that combines a PRISM rule-based scorer, an autoencoder-based AI risk scorer with human feedback, policy-based violation detection, and LLM-generated recommendations. The authors report large improvements over the PRISM baseline (false positive rate 42% to 17%, true positive rate 65% to 85%, false negative rate 18% to 12%) and operational gains including a 47% reduction in incident response time. The evaluation is based on the CERT dataset augmented with production data and expert annotations.
Significance. If the reported gains were supported by a sound evaluation, the system would be a valuable practical contribution to insider risk management. The paper's strengths include a transparent PRISM scoring equation, a clear description of the human-in-the-loop retraining loop, and preliminary scalability measurements. However, the central comparative claim rests on an evaluation that is circular and lacks any held-out test set, so the empirical contribution cannot be accepted as stated.
major comments (4)
- [Section 3.2.3, Section 4.1, Table 10] The autoencoder is described in Section 3.2.3 as 'initially trained using data from the PRISM framework,' which means PRISM scores serve as training labels. Section 4.1 then claims the AI model outperforms PRISM (Table 10). This is not an independent comparison: the AI model inherits PRISM's scoring behavior and the comparison measures how well the autoencoder reproduced its labels. The authors must evaluate both methods on an independent test set that was neither used for initial training nor for feedback retraining, and report the PRISM and AIRS results separately on that set.
- [Section 4.3, Table 12] Table 12 reports false positives falling from 42% to 30% to 17% across successive feedback loops, while Section 4.1 says the model was 'continuously evaluated on a dataset annotated by field experts and the administrator.' The paper never specifies a train/test split, a temporal gap, or a disjoint user cohort for this evaluation. If the same analyst-feedback instances used for retraining are also used for evaluation, the reported improvements are fitting artifacts rather than detection performance. An explicit description of evaluation data partitioning and a confirmation that test labels were not used in any retraining step are required.
- [Section 3.1] The ground-truth annotation is described with bracketed notes '[internal methodology – may not require citation]', '[custom process – not publicly citable]', and '[practical engineering – internal claim]'. These statements make the annotation protocol and the resulting labels unavailable for independent audit. There is no definition of what constitutes a true insider threat, no inter-annotator agreement measure, and no description of how production streams were labeled. Without a verifiable ground truth, the FP, TP, and FN metrics in Table 10 are not reproducible.
- [Section 4.5, Section 4.6, Tables 14 and 15] The scalability and incident-response improvements (sub-300ms latency at 100,000 logs/sec and a 47% reduction in response time) are reported without any experimental setup details: no hardware configuration, no workload description, no sample size, and no statistical uncertainty. These claims should be labeled as anecdotal or supported by a proper benchmark with reproducible methodology.
minor comments (4)
- [Introduction, Section 1] There is a typo in the introduction: 'A WS IAM' should be 'AWS IAM'.
- [Section 4.5] The text says 'as shown in Table 1' but the performance results are presented in Table 14; the reference should be corrected.
- [References] Reference [13] has inconsistent formatting with a stray 'Glasser, Joshua and Lindauer, Brian' string; the entry should follow the standard author-list format used elsewhere.
- [Section 4.2] Table 11 reports 'Actual Positive Rate' over time, but the term is not defined; it is presumably the true positive rate and should be stated as such for consistency with Table 10.
Circularity Check
The headline FP/TP gains are reported on the analyst-feedback data used for retraining, and the AI-vs-PRISM comparison is not independent because PRISM supplies the training labels.
-
self definitional
[Section 3.2.3, 'AIRS - AI Risk Scoring Algorithm'; Section 4.1, Table 10]
"The AI model is initially trained using data from the PRISM framework. This data serves as labeled input, enabling the model to understand predefined risk patterns and behaviors."
The AI model's training target is the PRISM score, and PRISM is also the baseline it is claimed to outperform in Section 4.1 and Table 10. The comparison is therefore not independent: the model is initialized to reproduce PRISM labels (Table 12 shows the 'Initial Model' FP/FN of 42%/18%, identical to PRISM's 42%/18%), so any later 'outperformance' must be attributed to the analyst-feedback retraining rather than to the AI model versus PRISM. The claim that the AI model beats its own label source is a self-referential comparison, not an independent prediction.
-
fitted input called prediction
[Section 4.3, Table 12 and Section 3.2.3, 'Incremental Model Relearning']
"After the first feedback loop, where analyst corrections were incorporated into retraining, false positives dropped to 30% and false negatives to 15%. Following a second feedback loop, the model further improved, reducing false positives to 17% and false negatives to 12%."
The reported improvements at each loop are measured on the same corrections that were 'incorporated into retraining' (Section 3.2.3: 'User feedback is prioritized over initial training weights'). No held-out split, temporal gap, or disjoint user cohort is described; Section 4.3 says the 60-day testing period is the same period in which analysts adjusted 12% of AI-generated scores. Hence the 59% false-positive reduction and 30% true-positive gain are in-sample fits to the feedback labels, not independent detection performance. The reported metric is the training signal under a different name.
full rationale
The central quantitative claims about detection improvement are circular in two connected ways. First, the AI autoencoder is trained 'using data from the PRISM framework', so the later 'AI vs. PRISM' comparison (Table 10) compares a model against the source of its own initial labels; the initial FP/FN in Table 12 exactly reproduce PRISM's 42%/18%, confirming the dependency. Second, the large gains are obtained by retraining on analyst corrections and then reported after each feedback loop with no held-out evaluation, so the 59% FP reduction and 30% TP gain reduce to fitting the feedback data. The paper's own bracketed annotations in Section 3.1 ('internal methodology – may not require citation', 'custom process – not publicly citable', 'practical engineering – internal claim') indicate the annotation protocol is not externally auditable, which strengthens rather than resolves the concern. The scalability and incident-response claims (Tables 14-15) and policy-violation counts are not entangled in this circularity, though they are not independently verified; the self-citation [17] for CHPDA is not load-bearing for the main scoring claims. Overall, the headline detection improvements are largely circular, so the score is 6 rather than 0.
Assumptions & free parameters
free parameters (6)
- PRISM weights W_p, W_A, W_C, W_IP, W_B, W_D, W_CA =
illustrated as 1; actual values not reported
- Risk thresholds (low/moderate/high) =
0.3, 0.6
- Min-max normalization bounds R_min, R_max =
R_min=0, R_max=100
- Feedback blending factor alpha =
not reported
- Autoencoder architecture and reconstruction threshold =
not reported
- Retraining trigger count n =
not reported
assumptions (5)
- domain assumption Expert-annotated risk scores are correct ground truth.
- domain assumption The CERT synthetic dataset and the unspecified production streams represent real insider-threat behavior for the target deployment.
- domain assumption Autoencoder reconstruction error is a valid monotonic measure of insider risk.
- ad hoc to paper PRISM output is a valid training signal for the AI model and does not bias the later comparison against PRISM.
- domain assumption Analyst feedback improves generalizability rather than encoding the test set.
Cite this review
Pith. "Pith review of AI-Driven IRM: Transforming insider risk management with adaptive scoring and LLM-based threat detection." pith.science (2026). https://pith.science/paper/E2HWSJWF
@misc{pith2026250503796,
author = {Pith},
title = {Pith review of: AI-Driven IRM: Transforming insider risk management with adaptive scoring and LLM-based threat detection},
year = {2026},
howpublished = {\url{https://pith.science/paper/E2HWSJWF}},
note = {Machine review of arXiv:2505.03796}
}
read the original abstract
Insider threats pose a significant challenge to organizational security, often evading traditional rule-based detection systems due to their subtlety and contextual nature. This paper presents an AI-powered Insider Risk Management (IRM) system that integrates behavioral analytics, dynamic risk scoring, and real-time policy enforcement to detect and mitigate insider threats with high accuracy and adaptability. We introduce a hybrid scoring mechanism - transitioning from the static PRISM model to an adaptive AI-based model utilizing an autoencoder neural network trained on expert-annotated user activity data. Through iterative feedback loops and continuous learning, the system reduces false positives by 59% and improves true positive detection rates by 30%, demonstrating substantial gains in detection precision. Additionally, the platform scales efficiently, processing up to 10 million log events daily with sub-300ms query latency, and supports automated enforcement actions for policy violations, reducing manual intervention. The IRM system's deployment resulted in a 47% reduction in incident response times, highlighting its operational impact. Future enhancements include integrating explainable AI, federated learning, graph-based anomaly detection, and alignment with Zero Trust principles to further elevate its adaptability, transparency, and compliance-readiness. This work establishes a scalable and proactive framework for mitigating emerging insider risks in both on-premises and hybrid environments.
Figures
Figures from the paper (4 more)
Forward citations
Cited by 2 Pith papers
-
LLM Harms: A Taxonomy and Discussion
This paper proposes a taxonomy of LLM harms in five categories and suggests mitigation strategies plus a dynamic auditing system for responsible development.
-
LLM-Based Threat Detection and Prevention Framework for IoT Ecosystems
Fine-tuned BERT-small reaches 99.75% test accuracy on combined IoT-23 and TON IoT attack classification, but the claimed real-time prevention advantage over traditional methods is not demonstrated.
Reference graph
Works this paper leans on
-
[20]
Insider Detection using Deep Autoencoder and Variational Autoencoder Neural Networks
Pantelidis, Efthimios, Bendiab, Gueltoum, Shiaeles, Stavros, and Kolokotronis, Nicholas. Insider Detection using Deep Autoencoder and Variational Autoencoder Neural Networks.arXiv preprint arXiv:2109.02568, 2021
work page Pith review arXiv 2021
-
[21]
Humans in AI: The necessity for human-in-the-loop (HILT).Se- curity Magazine, 2024
Bentley, Rick and Sarkar, Abhik. Humans in AI: The necessity for human-in-the-loop (HILT).Se- curity Magazine, 2024. https://www.securitymagazine.com/blogs/14-security-blog/post/ 100798-humans-in-ai-the-necessity-for-human-in-the-loop-hilt
work page 2024
-
[1]
Technical report, Verizon Enter- prise Solutions, 2024
2024 Data Breach Investigations Report (DBIR). Technical report, Verizon Enter- prise Solutions, 2024. https://www.verizon.com/business/resources/reports/ 2024-dbir-data-breach-investigations-report.pdf
work page 2024
-
[2]
2022 Cost of Insider Threats Global Report
Ponemon Institute. "2022 Cost of Insider Threats Global Report." Ponemon Institute, 2022.https://protectera. com.au/wp-content/uploads/2022/03/The-Cost-of-Insider-Threats-2022-Global-Report.pdf
work page 2022
-
[3]
Using AI for Behavioral Analytics in Cybersecurity: Detecting Anomalies and Insider Threats
Malik, Shoaib. "Using AI for Behavioral Analytics in Cybersecurity: Detecting Anomalies and Insider Threats." (2024)
work page 2024
-
[4]
Privacy Framework: A Tool for Improving Privacy through Enterprise Risk Management
NIST. "Privacy Framework: A Tool for Improving Privacy through Enterprise Risk Management." 2020
work page 2020
-
[5]
Azure Active Directory Documentation
Microsoft. Azure Active Directory Documentation. https://learn.microsoft.com/en-us/azure/ active-directory/, 2024
work page 2024
-
[6]
Identity and Access Management (IAM)
Amazon Web Services. Identity and Access Management (IAM). https://docs.aws.amazon.com/IAM/ latest/UserGuide/, 2024. 17 Transforming Insider Risk Management with Adaptive Scoring and LLM-Based Threat Detection
work page 2024
Show all 26 references
-
[7]
Google Workspace Admin Help.https://support.google.com/a/, 2024
Google. Google Workspace Admin Help.https://support.google.com/a/, 2024
2024
-
[8]
GPT-4 Technical Report.https://openai.com/research/gpt-4, 2023
OpenAI. GPT-4 Technical Report.https://openai.com/research/gpt-4, 2023
2023
-
[9]
Market Guide for Insider Risk Management Solutions
Gartner. Market Guide for Insider Risk Management Solutions. https://www.gartner.com/en/documents/4931631 Gartner Research, 2023
2023
-
[10]
Security and privacy controls for federal information systems and organizations
Force, Joint Task, and Transformation Initiative. "Security and privacy controls for federal information systems and organizations."NIST Special Publication800, no. 53 (2013): 8-13. U.S. Department of Commerce, 2020
2013
-
[11]
2023 Data Breach Investigations Report
Verizon. 2023 Data Breach Investigations Report. Technical report, Verizon Enterprise Solutions, 2023. https: //inquest.net/wp-content/uploads/2023-data-breach-investigations-report-dbir.pdf
2023
-
[12]
Gartner Identifies the Top Cybersecurity Trends for 2023
Gartner. Gartner Identifies the Top Cybersecurity Trends for 2023. https://www.gartner.com/en/newsroom/press- releases/04-12-2023-gartner-identifies-the-top-cybersecurity-trends-for-2023 Gartner Research, 2023
2023
-
[13]
In 2013 IEEE Security and Privacy Workshops, pp
, Glasser, Joshua and Lindauer, Brian Bridging the gap: A pragmatic approach to generating insider threat data. In 2013 IEEE Security and Privacy Workshops, pp. 98-104. IEEE, 2013
2013
-
[14]
CERT Insider Threat Tools and Datasets.Technical Report, 2016
Carnegie Mellon University - Software Engineering Institute (CMU-SEI). CERT Insider Threat Tools and Datasets.Technical Report, 2016. Available at: https://resources.sei.cmu.edu/library/asset-view. cfm?assetid=508099
2016
-
[15]
A survey of insider attack detection research
Salem, Mohammad B., Hershkop, Shlomo and Stolfo, Salvatore J. A survey of insider attack detection research. Insider Attack and Cyber Security, pp. 69–90, Springer, 2008
2008
-
[16]
and Frincke, Deborah A
Greitzer, Frank L. and Frincke, Deborah A. Combining traditional cyber security audit data with psychosocial data: Towards predictive modeling for insider threat mitigation.Insider Threats in Cyber Security, pp. 85–113, Springer, 2010
2010
-
[17]
Koli Lokesh and Kalra Shubham and Singh Karanpreet Context-Aware Hybrid Pattern Detection Algorithm (CHPDA) for Real-Time Sensitivity Analysis.V ectorEdge Technical Whitepaper, 2025
2025
-
[18]
Risk assessment model based on user behavior for insider threats.Computers & Security, 77, 428–440, 2018
Liu, S., Wang, W., and Liu, D. Risk assessment model based on user behavior for insider threats.Computers & Security, 77, 428–440, 2018
2018
-
[19]
L., & Hohimer, R
Greitzer, F. L., & Hohimer, R. E. Modeling human behavior to anticipate insider attacksJournal of Strategic Security 4, no. 2 (2011): 25-48
2011
-
[22]
Microsoft Learn.Data Protection Policies in Microsoft Defender for Cloud Apps. 2023. Accessed: 2025-04-28. https://learn.microsoft.com/en-us/defender-cloud-apps/data-protection-policies
2023
-
[23]
Uptycs.Mastering Cloud Security: Understanding the Attack Path. 2023. Accessed: 2025-04-28. https: //www.uptycs.com/blog/mastering-cloud-security/attack-path
2023
-
[24]
Elevating Threat Intelligence: Integrating Context-Aware AI Models for Real-Time Cyber Defense.Wrixte, May 15, 2024
Team Wrixte. Elevating Threat Intelligence: Integrating Context-Aware AI Models for Real-Time Cyber Defense.Wrixte, May 15, 2024. https://wrixte.co/2024/05/15/ elevating-threat-intelligence-integrating-context-aware-ai-models-for-real-time-cyber-defense/
2024
-
[25]
Building Context-Aware Cybersecurity Alerts.TCS Insights, Jan- uary 10, 2025
TATA Consultancy Services. Building Context-Aware Cybersecurity Alerts.TCS Insights, Jan- uary 10, 2025. https://www.tcs.com/insights/topics/cybersecurity-topic/article/ context-aware-cybersecurity
2025
-
[26]
AI Risk Management: Introducing the On-Premises Version of DeepSeek to Keep Your Data In-House.Medium, March 2025
Hiroshi Kinoshita. AI Risk Management: Introducing the On-Premises Version of DeepSeek to Keep Your Data In-House.Medium, March 2025. https://medium.com/@ai2ai/ ai-risk-management-introducing-the-on-premises-version-of-deepseek-to-keep-your-data-in-house-a-a7175910e3c2 18
2025
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.