Predicting Malicious Insider Threat Scenarios Using Organizational Data and a Heterogeneous Stack-Classifier

Adam James Hall; Naghmeh Moradpoor; Nikolaos Pitropakis; William J Buchanan

arxiv: 1907.10272 · v1 · pith:MCBJGA6Gnew · submitted 2019-07-24 · 💻 cs.CR

Predicting Malicious Insider Threat Scenarios Using Organizational Data and a Heterogeneous Stack-Classifier

Adam James Hall , Nikolaos Pitropakis , William J Buchanan , Naghmeh Moradpoor This is my paper

Pith reviewed 2026-05-24 17:09 UTC · model grok-4.3

classification 💻 cs.CR

keywords insider threatmachine learningmeta-classifierCERT r4.2threat predictionlog data preprocessingensemble classifier

0 comments

The pith

A meta-classifier aggregates multiple models to predict a specific insider threat scenario with 96.2 percent accuracy.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper seeks to close the gap in insider threat detection by predicting the scenario where an employee uploads sensitive information to WikiLeaks before departing the organization. It preprocesses the CERT r4.2 organizational log data into daily user summaries and applies a series of machine learning classifiers, including boosting, which are then combined into a meta-classifier. This ensemble achieves superior performance with 96.2% accuracy and an AUC of 0.988, evaluated through confusion matrices and ROC curves, offering a practical methodology for threat prediction from log data.

Core claim

By converting CERT r4.2 logs into daily user summaries and training multiple classifiers that are aggregated into a heterogeneous stack-classifier with boosting, the approach detects the malicious insider threat of leaking data to WikiLeaks prior to leaving with 96.2% accuracy and 0.988 AUC.

What carries the argument

The heterogeneous stack-classifier, formed by aggregating the best performing individual classifiers trained on daily user summaries derived from organizational logs.

If this is right

Daily user summaries from log data provide sufficient features for distinguishing the target threat from normal behavior.
Boosting optimizes the accuracy of the base classifiers before aggregation.
The meta-classifier demonstrates stronger predictive performance than any of its constituent models.
Analysis of confusion matrices and ROC curves validates the high accuracy and AUC values.
This methodology can be used to train classifiers for the specified threat scenario.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The method could be tested on real organizational data to check if performance holds outside the synthetic CERT dataset.
Similar preprocessing and stacking could apply to other insider threat scenarios like unauthorized access or sabotage.
Real-time implementation of daily summary generation might allow for proactive alerts in security systems.
Combining this with additional data sources beyond logs could further improve detection.

Load-bearing premise

That the daily user summaries from the synthetic CERT r4.2 logs contain signals that distinguish the threat scenario in a way that generalizes to real organizations.

What would settle it

Applying the trained meta-classifier to a different dataset of real insider threat incidents and checking if accuracy remains near 96%.

Figures

Figures reproduced from arXiv: 1907.10272 by Adam James Hall, Naghmeh Moradpoor, Nikolaos Pitropakis, William J Buchanan.

**Figure 2.** Figure 2: Histogram showing distribution of threat cases (shown in dark [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Figure 3: Histogram showing distribution of threat cases (shown in dark [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

**Figure 4.** Figure 4: Histogram showing distribution of threat cases (shown in dark [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗

**Figure 5.** Figure 5: Graph demonstrating performance approximation of models A [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗

**Figure 6.** Figure 6: ROC curve showing predictive performance of the meta-learner. [PITH_FULL_IMAGE:figures/full_fig_p005_6.png] view at source ↗

read the original abstract

Insider threats continue to present a major challenge for the information security community. Despite constant research taking place in this area; a substantial gap still exists between the requirements of this community and the solutions that are currently available. This paper uses the CERT dataset r4.2 along with a series of machine learning classifiers to predict the occurrence of a particular malicious insider threat scenario - the uploading sensitive information to wiki leaks before leaving the organization. These algorithms are aggregated into a meta-classifier which has a stronger predictive performance than its constituent models. It also defines a methodology for performing pre-processing on organizational log data into daily user summaries for classification, and is used to train multiple classifiers. Boosting is also applied to optimise classifier accuracy. Overall the models are evaluated through analysis of their associated confusion matrix and Receiver Operating Characteristic (ROC) curve, and the best performing classifiers are aggregated into an ensemble classifier. This meta-classifier has an accuracy of \textbf{96.2\%} with an area under the ROC curve of \textbf{0.988}.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

Standard stack classifier on CERT r4.2 daily summaries reaches 96.2% accuracy for one synthetic insider scenario, but thin validation leaves open whether it captures real signals or dataset artifacts.

read the letter

The main takeaway is that the authors turn CERT r4.2 logs into daily user summaries, train several base classifiers, apply boosting, and combine the best ones into a meta-classifier that hits 96.2% accuracy and 0.988 AUC on the specific wikileaks-upload-before-departure scenario. They also lay out the pre-processing steps explicitly, which is the most concrete part of the work. That feature construction from organizational logs is practical and could be reused by others working on similar log data. The evaluation with confusion matrices and ROC curves is standard and shows the ensemble beating the individuals, as expected. Beyond that, the paper does not introduce new algorithms or a first-principles framework; it applies known ensemble techniques to a well-studied public dataset for a previously examined threat type. Comparable results already exist in the literature. The soft spots sit mainly in the validation. The abstract and methods give no details on train-test splitting, cross-validation folds, or how severe class imbalance was handled. Because the malicious traces in CERT r4.2 are rule-based injections, any feature that tracks volume, timing, or file types tied to those rules can produce high numbers without reflecting generalizable behavior. No external hold-out set or cross-dataset test is described, so the reported performance stays a fitted quantity on this corpus. The daily-summary construction itself risks leakage if the malicious event influences the same day's features. This paper is useful mainly to applied researchers who already work with the CERT dataset and want another baseline or example of daily aggregation. Readers seeking novel methods or robust evidence of generalization will find little. It does not look strong enough for a serious referee process; the validation gaps would need substantial additional work before the central claim can be trusted.

Referee Report

3 major / 1 minor

Summary. The manuscript claims to predict a specific malicious insider threat scenario (uploading sensitive information to Wikileaks before departure) on the synthetic CERT r4.2 dataset. It describes a preprocessing pipeline that converts organizational logs into daily user summaries, trains multiple base classifiers with boosting, and aggregates the strongest models into a meta-classifier that achieves 96.2% accuracy and 0.988 AUC, with evaluation via confusion matrices and ROC curves.

Significance. If the performance numbers survive rigorous validation, the preprocessing methodology for daily summaries and the ensemble construction could provide a reusable template for log-based insider-threat detection. The work also demonstrates the practical aggregation of heterogeneous classifiers, which is a modest but concrete engineering contribution.

major comments (3)

[Evaluation] Evaluation section (and abstract): the reported 96.2% accuracy and 0.988 AUC are presented without any description of the train/test partitioning, cross-validation procedure, or stratification for the extreme class imbalance present in CERT r4.2. Because the central claim rests on these performance figures, the absence of these details makes it impossible to determine whether the numbers reflect genuine out-of-sample prediction or in-sample fitting.
[Methodology] Data-preprocessing section: the daily user summaries are constructed from the full CERT r4.2 logs with no stated safeguards against temporal leakage (e.g., feature statistics computed on the entire timeline rather than strictly within training windows). Given that the malicious traces are rule-based synthetic injections, any feature that encodes log volume, timing, or file-type patterns tied to those rules can produce high accuracy without capturing generalizable behavioral signals.
[Experiments] Experimental design: all results are obtained on a single synthetic corpus (CERT r4.2) with no external hold-out dataset, cross-organization test, or real-world logs. The meta-classifier therefore cannot be shown to generalize beyond the specific injection rules used to generate the target scenario.

minor comments (1)

[Abstract] Abstract: 'wiki leaks' should be written as the single word 'Wikileaks'.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for their constructive and detailed comments. We address each major point below and indicate the changes planned for the revised manuscript.

read point-by-point responses

Referee: [Evaluation] Evaluation section (and abstract): the reported 96.2% accuracy and 0.988 AUC are presented without any description of the train/test partitioning, cross-validation procedure, or stratification for the extreme class imbalance present in CERT r4.2. Because the central claim rests on these performance figures, the absence of these details makes it impossible to determine whether the numbers reflect genuine out-of-sample prediction or in-sample fitting.

Authors: We agree that the Evaluation section does not provide sufficient detail on the experimental protocol. We will revise both the Evaluation section and the abstract to include a complete description of the train/test partitioning, cross-validation procedure, and stratification approach used to handle class imbalance, allowing readers to verify that the reported metrics reflect out-of-sample performance. revision: yes
Referee: [Methodology] Data-preprocessing section: the daily user summaries are constructed from the full CERT r4.2 logs with no stated safeguards against temporal leakage (e.g., feature statistics computed on the entire timeline rather than strictly within training windows). Given that the malicious traces are rule-based synthetic injections, any feature that encodes log volume, timing, or file-type patterns tied to those rules can produce high accuracy without capturing generalizable behavioral signals.

Authors: We acknowledge that the Data-preprocessing section does not explicitly document safeguards against temporal leakage. We will revise this section to describe how daily summaries were constructed while respecting temporal order and ensuring that all feature computations were performed strictly within training windows, thereby clarifying that no future information influenced the models. revision: yes
Referee: [Experiments] Experimental design: all results are obtained on a single synthetic corpus (CERT r4.2) with no external hold-out dataset, cross-organization test, or real-world logs. The meta-classifier therefore cannot be shown to generalize beyond the specific injection rules used to generate the target scenario.

Authors: We recognize that limiting evaluation to a single synthetic dataset restricts claims of broad generalizability. Our contribution centers on the daily-summary preprocessing pipeline and the heterogeneous stacking methodology rather than on cross-dataset validation. We will add an explicit discussion of this limitation in the revised manuscript and outline directions for future work on additional datasets. revision: partial

Circularity Check

0 steps flagged

No significant circularity; empirical ML evaluation on provided dataset.

full rationale

The paper describes an empirical machine-learning pipeline that derives daily user summary features from the CERT r4.2 logs, trains a heterogeneous stack of classifiers, applies boosting, and aggregates them into a meta-classifier whose performance is measured by accuracy and AUC on that same corpus. No mathematical derivation chain, first-principles equations, or uniqueness theorem is presented whose output is definitionally identical to its inputs. The reported 96.2 % accuracy and 0.988 AUC are standard supervised-learning evaluation metrics rather than a 'prediction' that has been forced by construction. No self-citation load-bearing step, ansatz smuggling, or renaming of a known result occurs. The work is therefore self-contained as an applied ML study; any concerns about synthetic-data artifacts or lack of external hold-out belong to correctness or generalization risk, not circularity.

Axiom & Free-Parameter Ledger

2 free parameters · 1 axioms · 0 invented entities

The central performance claim rests on the synthetic CERT labels being representative, on the daily-summary features being informative without post-hoc selection bias, and on a large number of implicit hyperparameters in the classifier stack.

free parameters (2)

daily summary feature definitions and thresholds
Choices of which log events to count and how to aggregate them into daily vectors are required to produce the input matrix.
classifier hyperparameters and boosting rounds
Multiple base learners plus the meta-learner require tuning to reach the stated accuracy.

axioms (1)

domain assumption CERT r4.2 synthetic labels accurately reflect real-world malicious insider behavior for the WikiLeaks scenario.
All training and evaluation depend on these labels being valid proxies.

pith-pipeline@v0.9.0 · 5721 in / 1381 out tokens · 24181 ms · 2026-05-24T17:09:42.487347+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

14 extracted references · 14 canonical work pages · 1 internal anchor

[1]

The Greater The Power , The More Dangerous The Abuse : Facing Malicious Insiders in The Cloud,

N. Pitropakis, C. Lyvas, and C. Lambrinoudakis, “The Greater The Power , The More Dangerous The Abuse : Facing Malicious Insiders in The Cloud,” no. c, pp. 156–161, 2017

work page 2017
[2]

Interested in learning SANS Institute InfoSec Reading Room Defending Against the Wrong Enemy : 2017 SANS Defending Against the Wrong Enemy : 2017 SANS Insider Threat Survey,

Sans Intitute, “Interested in learning SANS Institute InfoSec Reading Room Defending Against the Wrong Enemy : 2017 SANS Defending Against the Wrong Enemy : 2017 SANS Insider Threat Survey,” 2017

work page 2017
[3]

2017 Cost of Data Breach Study, Global Overview,

I. Ponemon, “2017 Cost of Data Breach Study, Global Overview,” IBM Security , no. March, pp. 1–34, 2017. [Online]. Available: https://www-01.ibm.com/common/ssi/ cgi-bin/ssialias?htmlﬁd=SEL03130WWEN{&}

work page 2017
[4]

Insider Threat 2018 Report,

C. Associates, “Insider Threat 2018 Report,” p. 41, 2018. [Online]. Available: https://www.cybersecurity-insiders.com/ wp-content/uploads/2016/09/Insider-Threat-Report-2018.pdf

work page 2018
[5]

Detection and prediction of insider threats to cyber security: a systematic literature review and meta-analysis,

I. A. Gheyas and A. E. Abdallah, “Detection and prediction of insider threats to cyber security: a systematic literature review and meta-analysis,” Big Data Analytics , vol. 1, no. 1, p. 6, 2016. [Online]. Available: http://bdataanalytics. biomedcentral.com/articles/10.1186/s41044-016-0006-0

work page doi:10.1186/s41044-016-0006-0 2016
[6]

Insider Threat Detection Using Graph-Based Approaches,

W. Eberle and L. Holder, “Insider Threat Detection Using Graph-Based Approaches,” 2009 Cybersecurity Applications & Technology Conference for Homeland Security , pp. 237–241, 2009. [Online]. Available: http://ieeexplore.ieee. org/document/4804450/

work page arXiv 2009
[7]

Anomaly instruction detec- tion of masqueraders and threat evaluation using fuzzy logic,

Y . Yingbing and J. H. Graham, “Anomaly instruction detec- tion of masqueraders and threat evaluation using fuzzy logic,” Conference Proceedings - IEEE International Conference on Systems, Man and Cybernetics , vol. 3, pp. 2309–2314, 2007

work page 2007
[8]

Deep learning based attribute classiﬁcation insider threat detection for data secu- rity,

F. Meng, F. Lou, Y . Fu, and Z. Tian, “Deep learning based attribute classiﬁcation insider threat detection for data secu- rity,” in 2018 IEEE Third International Conference on Data Science in Cyberspace (DSC) , June 2018, pp. 576–581

work page 2018
[9]

Deep Learning for Unsupervised Insider Threat Detection in Structured Cybersecurity Data Streams

A. Tuor, S. Kaplan, B. Hutchinson, N. Nichols, and S. Robinson, “Deep Learning for Unsupervised Insider Threat Detection in Structured Cybersecurity Data Streams,” no. 2012, 2017. [Online]. Available: http://arxiv.org/abs/ 1710.00811

work page internal anchor Pith review Pith/arXiv arXiv 2012
[10]

Inside the Mind of the Insider: Towards Insider Threat Detection Using Psychophysiological Signals *,

Y . Hashem, H. Takabi, M. Ghasemigol, and R. Dantu, “Inside the Mind of the Insider: Towards Insider Threat Detection Using Psychophysiological Signals *,” Journal of Internet Services and Information Security , vol. 6, no. 1, pp. 20 – 36, 2016. [Online]. Available: http: //isyou.info/jisis/vol6/no1/jisis-2016-vol6-no1-02.pdf

work page 2016
[11]

Supervised and unsupervised methods to detect insider threat from enterprise social and online activity data,

G. Gavai, K. Sricharan, D. Gunning, J. Hanley, M. Singhal, and R. Rolleston, “Supervised and unsupervised methods to detect insider threat from enterprise social and online activity data,” Journal of Wireless Mobile Networks, Ubiquitous Computing, and Dependable Applications (JoWUA) , vol. 6, no. 4, pp. 47–63, 2015. [Online]. Available: http://isyou.info/...

work page 2015
[12]

Weka 3 - data mining with open source machine learning software in java,

“Weka 3 - data mining with open source machine learning software in java,” https://www.cs.waikato.ac.nz/ml/weka/, ac- cessed: 2018-10-05

work page 2018
[13]

Agent-Based Modeling for Complex Financial Systems have concluded that the dynamics of networked market,

J. P. A. C. M. Wooldridge, “Agent-Based Modeling for Complex Financial Systems have concluded that the dynamics of networked market,” IEEE Intelligent Systems , vol. 33, no. 2, pp. 74–82, 2018

work page 2018
[14]

I. H. Witten, E. Frank, and M. a. Hall, Data Mining: Practical Machine Learning Tools and Techniques (Google eBook), 2013. [Online]. Available: http://books.google.com/ books?id=bDtLM8CODsQC{&}pgis=1

work page 2013

[1] [1]

The Greater The Power , The More Dangerous The Abuse : Facing Malicious Insiders in The Cloud,

N. Pitropakis, C. Lyvas, and C. Lambrinoudakis, “The Greater The Power , The More Dangerous The Abuse : Facing Malicious Insiders in The Cloud,” no. c, pp. 156–161, 2017

work page 2017

[2] [2]

Interested in learning SANS Institute InfoSec Reading Room Defending Against the Wrong Enemy : 2017 SANS Defending Against the Wrong Enemy : 2017 SANS Insider Threat Survey,

Sans Intitute, “Interested in learning SANS Institute InfoSec Reading Room Defending Against the Wrong Enemy : 2017 SANS Defending Against the Wrong Enemy : 2017 SANS Insider Threat Survey,” 2017

work page 2017

[3] [3]

2017 Cost of Data Breach Study, Global Overview,

I. Ponemon, “2017 Cost of Data Breach Study, Global Overview,” IBM Security , no. March, pp. 1–34, 2017. [Online]. Available: https://www-01.ibm.com/common/ssi/ cgi-bin/ssialias?htmlﬁd=SEL03130WWEN{&}

work page 2017

[4] [4]

Insider Threat 2018 Report,

C. Associates, “Insider Threat 2018 Report,” p. 41, 2018. [Online]. Available: https://www.cybersecurity-insiders.com/ wp-content/uploads/2016/09/Insider-Threat-Report-2018.pdf

work page 2018

[5] [5]

Detection and prediction of insider threats to cyber security: a systematic literature review and meta-analysis,

I. A. Gheyas and A. E. Abdallah, “Detection and prediction of insider threats to cyber security: a systematic literature review and meta-analysis,” Big Data Analytics , vol. 1, no. 1, p. 6, 2016. [Online]. Available: http://bdataanalytics. biomedcentral.com/articles/10.1186/s41044-016-0006-0

work page doi:10.1186/s41044-016-0006-0 2016

[6] [6]

Insider Threat Detection Using Graph-Based Approaches,

W. Eberle and L. Holder, “Insider Threat Detection Using Graph-Based Approaches,” 2009 Cybersecurity Applications & Technology Conference for Homeland Security , pp. 237–241, 2009. [Online]. Available: http://ieeexplore.ieee. org/document/4804450/

work page arXiv 2009

[7] [7]

Anomaly instruction detec- tion of masqueraders and threat evaluation using fuzzy logic,

Y . Yingbing and J. H. Graham, “Anomaly instruction detec- tion of masqueraders and threat evaluation using fuzzy logic,” Conference Proceedings - IEEE International Conference on Systems, Man and Cybernetics , vol. 3, pp. 2309–2314, 2007

work page 2007

[8] [8]

Deep learning based attribute classiﬁcation insider threat detection for data secu- rity,

F. Meng, F. Lou, Y . Fu, and Z. Tian, “Deep learning based attribute classiﬁcation insider threat detection for data secu- rity,” in 2018 IEEE Third International Conference on Data Science in Cyberspace (DSC) , June 2018, pp. 576–581

work page 2018

[9] [9]

Deep Learning for Unsupervised Insider Threat Detection in Structured Cybersecurity Data Streams

A. Tuor, S. Kaplan, B. Hutchinson, N. Nichols, and S. Robinson, “Deep Learning for Unsupervised Insider Threat Detection in Structured Cybersecurity Data Streams,” no. 2012, 2017. [Online]. Available: http://arxiv.org/abs/ 1710.00811

work page internal anchor Pith review Pith/arXiv arXiv 2012

[10] [10]

Inside the Mind of the Insider: Towards Insider Threat Detection Using Psychophysiological Signals *,

Y . Hashem, H. Takabi, M. Ghasemigol, and R. Dantu, “Inside the Mind of the Insider: Towards Insider Threat Detection Using Psychophysiological Signals *,” Journal of Internet Services and Information Security , vol. 6, no. 1, pp. 20 – 36, 2016. [Online]. Available: http: //isyou.info/jisis/vol6/no1/jisis-2016-vol6-no1-02.pdf

work page 2016

[11] [11]

Supervised and unsupervised methods to detect insider threat from enterprise social and online activity data,

G. Gavai, K. Sricharan, D. Gunning, J. Hanley, M. Singhal, and R. Rolleston, “Supervised and unsupervised methods to detect insider threat from enterprise social and online activity data,” Journal of Wireless Mobile Networks, Ubiquitous Computing, and Dependable Applications (JoWUA) , vol. 6, no. 4, pp. 47–63, 2015. [Online]. Available: http://isyou.info/...

work page 2015

[12] [12]

Weka 3 - data mining with open source machine learning software in java,

“Weka 3 - data mining with open source machine learning software in java,” https://www.cs.waikato.ac.nz/ml/weka/, ac- cessed: 2018-10-05

work page 2018

[13] [13]

Agent-Based Modeling for Complex Financial Systems have concluded that the dynamics of networked market,

J. P. A. C. M. Wooldridge, “Agent-Based Modeling for Complex Financial Systems have concluded that the dynamics of networked market,” IEEE Intelligent Systems , vol. 33, no. 2, pp. 74–82, 2018

work page 2018

[14] [14]

I. H. Witten, E. Frank, and M. a. Hall, Data Mining: Practical Machine Learning Tools and Techniques (Google eBook), 2013. [Online]. Available: http://books.google.com/ books?id=bDtLM8CODsQC{&}pgis=1

work page 2013