{"id":"f282d834-f0c7-4e6c-8797-4c27d26cf9f2","arxiv_id":"2506.23446","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A transformer encoder trained on benign user activity, with reconstruction errors scored by OCSVM, LOF, or Isolation Forest, is reported to achieve state-of-the-art insider threat detection on CERT datasets.","lead":"Researchers turned CERT insider threat logs into ordered user activity sequences and trained a transformer encoder on normal behavior, flagging deviations as insider attacks. On synthetic benchmark test sets the method reports high recall and low false negative rates, though evaluation gaps make the headline numbers hard to verify.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Test sets include benign r4.2 users that were used for training; no holdout is described, so the reported Test-1/Test-4 metrics may reflect memorization rather than detection.","rationale":"The reader's verdict is REJECT, and our analysis supports that rejection, so the verdict is unchanged. However, the reader's weakest_assumption identifies cross-release transfer and the unspecified fitting procedure as the core problem, while our review finds a more concrete and more damaging flaw: the apparent overlap between benign r4.2 users used for training and those included in Test-1 and Test-4. The paper describes training on all benign r4.2 data and then constructs Test-1 with 30 benign r4.2 users, without any statement of a holdout. The only separation guarantee refers to uniqueness among test sets, not between training and testing. Because the model is a reconstruction-based autoencoder trained with batch size one, it can memorize individual user sequences, making seen benign users trivially 'normal' and inflating separation from malicious users. This directly affects the headline Test-4 metrics. The unspecified fitting of the outlier detectors is a related but distinct issue; if the detectors are fit on test errors, the reported numbers are not a valid estimate of deployment performance. Both concerns weaken the central claim, so no change to the reject verdict is needed.","tokens_in":10987,"tokens_out":6144,"duration_ms":59744,"concrete_test":"Request the user IDs for the benign r4.2 users used in training and in Test-1/Test-4. If any Test-1 benign user appears in the training set, re-run the pipeline with a held-out benign set (e.g., 30 r4.2 users never seen during training) and recompute Table 3. If the reported metrics drop materially, the SOTA claim is invalidated. Additionally, require the authors to state whether OCSVM, LOF, and iForest were fit on training reconstruction errors or on test errors; if test errors, the results are transductive and cannot support a deployable detector.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on the Test-4 numbers in Table 3, yet that test set includes benign users from r4.2 (Table 1), the same release used for training (Sections 3.3 and 3.4). The paper trains the Transformer on 'only benign data' from r4.2 and never states that the 30 benign r4.2 users in Test-1 (and thus in Test-4, since Test-4 appears to be the union of Tests 1-3) were excluded from training. The only separation statement (Section 5) says that each test set is unique and that no overlap exists among the sets; it does not establish train/test disjointness. If these users were in training, the encoder has memorized their sequences, yielding near-zero reconstruction errors and making separation from malicious users trivial. This would inflate the reported 96.61% accuracy, 99.43% recall, and 95.00% AUROC, and it would explain the perfect recall and zero false negative rate on Test-1. A second, compounding issue is that the paper never specifies how OCSVM, LOF, and iForest are fit; if they are fitted to reconstruction errors from the test set itself, the threshold is chosen with knowledge of test contamination, further biasing the headline metrics. Either issue alone makes the state-of-the-art claim unsupported as written.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a User-Based Sequencing (UBS) methodology that converts raw CERT insider-threat logs into per-user tensors of shape [days, sessions, features], then trains a Transformer encoder on benign data to reconstruct these tensors. Reconstruction errors are scored by OCSVM, LOF, and Isolation Forest. The authors report four test sets, including a mixed-release Test-4, and claim state-of-the-art detection performance (96.61% accuracy, 99.43% recall, 96.38% F1, 95% AUROC) and very low false negative and false positive rates.","tokens_in":11257,"tokens_out":8218,"duration_ms":72434,"significance":"If the reported numbers were reproducible and the evaluation were clean, the paper would make a credible contribution to insider-threat detection by demonstrating that sequential modeling with Transformers can outperform tabular and simple autoencoder baselines, and the UBS representation could be reused by other sequence models. The paper is explicit about the architecture, hyperparameters, and the unsupervised nature of the anomaly scorers. However, the lack of a proper train/test split, the unspecified training of the outlier detectors, the absence of error bars, and the uncontrolled comparison with prior work currently preclude accepting the state-of-the-art claim.","major_comments":[{"comment":"The model is trained on benign users from CERT r4.2, but Test-1 includes 30 benign r4.2 users and Test-4 combines Tests 1–3, so those same users are part of the evaluation. The paper never states that these 30 users were excluded from training. The only separation claim (Section 5) is that the test sets do not overlap with each other; it does not establish train/test disjointness. If the benign r4.2 users in Test-1/Test-4 were in the training set, the Transformer can memorize their sequences and produce near-zero reconstruction errors, making anomaly detection trivially easier. This directly affects the headline numbers in Table 3 and the abstract. The authors must either exclude all training users from the test sets or demonstrate performance on a genuinely held-out set of r4.2 benign users.","section":"3.6.3 / Table 1 vs. 3.3–3.4"},{"comment":"The paper does not specify how OCSVM, LOF, and iForest are trained and thresholded. It is essential to know whether these detectors are fit on reconstruction errors of benign training users (a legitimate novelty-detection setup) or on reconstruction errors of the test set (which would leak label information and make the reported metrics optimistic). The hyperparameters for these detectors (e.g., nu/gamma for OCSVM, n_neighbors for LOF, contamination for iForest) are not reported, so the protocol is not reproducible. This is load-bearing because the same reconstruction errors can yield dramatically different F-scores depending on the detector configuration.","section":"3.6.1"},{"comment":"The hyperparameter search is described as a Cartesian product, but no validation split is mentioned. Selecting hyperparameters by optimizing performance on the test sets would inflate the reported results. The authors should describe a validation procedure using only benign data (e.g., a held-out subset of r4.2 benign users) and report the selected hyperparameters.","section":"3.6"},{"comment":"The comparison with prior work is not on equal footing. The baseline models are evaluated on single CERT releases (mostly r4.2), while the proposed model is evaluated on the combined Test-4 set. Differences in test composition, contamination ratio, and preprocessing make the 'state-of-the-art' claim unsupported. The authors should compare on the same evaluation protocol or explicitly quantify the effect of the different test sets.","section":"5.1 / Table 7"},{"comment":"Results are reported for a single run; no standard deviations or multiple seeds are provided. Given the small number of malicious users in some test sets (e.g., 5 in Test-3), the reported metrics are subject to considerable variance. The authors should report mean and variance over multiple runs, or at least specify the random seeds.","section":"Tables 2 and 3"}],"minor_comments":[{"comment":"The analysis of false positive rate is incorrect. For example, Table 4 says 'FPR + 76.01% Reduction in the rate at which true positives are mistakenly overlooked,' but FPR is the rate of false positives among negatives, not a measure of missed true positives (which is FNR). Table 5 similarly labels a 4.28-percentage-point increase in FPR as 'the rate at which true positives are mistakenly overlooked.' These statements need to be corrected.","section":"5 (Tables 4–6)"},{"comment":"The citation [24] is to the Cartesian product algorithm for type inference, not to hyperparameter search; a different reference is needed.","section":"3.6"},{"comment":"The notation 'user_data: U -> T' is ambiguous; it should clearly define that U is the set of users and T is the space of tensors of shape [501,9,35], with a formal mapping from each user to their tensor.","section":"3.5"},{"comment":"The notation '+1', '+2', '+3', '+4' in the baseline names is undefined; these should be replaced with meaningful names or a footnote explaining the notation.","section":"Table 7"},{"comment":"Calling a 5.71% false positive rate 'exceptionally low' is a matter of interpretation, but it is not a technical error.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The paper's central claim is currently unsupported due to the evaluation issues detailed above. The authors should be given the opportunity to revise by re-running experiments with a proper train/test split, specifying the outlier-detector training, and reporting variance. If they cannot provide these, the paper should not be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis paper has a useful idea and an evaluation that does not support it. The UBS data restructuring — packing per-user session counts into a [501,9,35] tensor — is a reasonable way to feed CERT logs into a transformer, and the paper shows that this representation helps an autoencoder as much as the transformer. That is a genuine empirical observation, and if the reported numbers held up, the pipeline would be a decent engineering contribution for a niche but real security problem.\n\nThey do not hold up as reported. The stress-test note is right: the model is trained on benign r4.2 users, and Test-1 (therefore Test-4, which aggregates Tests 1–3) includes 30 benign r4.2 users with no stated exclusion from training. Section 5 only says the test sets are disjoint from each other, not from training. With the encoder trained on those users' sequences, near-zero reconstruction error for them is trivial, making the 99%+ recall and near-zero FNR on Test-4 explainable by memorization rather than detection. This is load-bearing: the headline numbers all rest on Test-4.\n\nCompounding issues: hyperparameters were selected by a Cartesian-product search with no validation split described, so the reported numbers are selection results, not test results. The three outlier detectors (OCSVM, LOF, iForest) are never specified as fit on training reconstruction errors; if they are fit on test errors, that is threshold leakage. There are also no seeds or error bars. The test sets themselves are unrealistic — 70% malicious in Test-1, 45% in Test-4 — so even a clean protocol would not support the deployable-detector framing. Minor but telling: the discussion praises OCSVM for perfect recall while the abstract uses iForest's numbers, and the \"FPR improvement\" in Table 4 appears to confuse false positives with missed true positives.\n\nWhat is worth keeping: sequence-structured inputs help both the transformer and a plain autoencoder over unstructured tabular inputs on this benchmark. That is modest but real. The state-of-the-art claim, however, should not be taken at face value until code and a clean train/test split are released.\n\nThis paper needs a major revision — released code, a proper holdout, a realistic test distribution, and a clear thresholding protocol — before it deserves a serious referee. My recommendation: do not send it out as-is; ask for the redo first.","headline":"The UBS idea is sensible, but the evaluation leaks training users into the test set, so the headline numbers do not support the state-of-the-art claim.","tokens_in":11812,"tokens_out":3378,"would_cite":false,"duration_ms":33997,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A transformer trained only on normal user activity, fed per-user session sequences, flags 99.43% of simulated insider attacks on a combined CERT test set.","keywords":["insider threat detection","transformer encoder","user-based sequencing","anomaly detection","reconstruction error","CERT dataset","one-class SVM","isolation forest"],"falsifier":"Fit OCSVM, LOF, and iForest on reconstruction errors from the benign r4.2 training users only, freeze those detectors, then score the r5.2 and r6.2 users in Test-4 and report per-release recall and FPR; if recall drops materially below 99% or FPR exceeds the reported 0.0571, the headline figures depend on test-set information rather than on the learned notion of normal behavior.","tokens_in":10750,"feed_emoji":"🛡️","tokens_out":7875,"duration_ms":78117,"temperature":0.7,"pith_summary":"The paper argues that insider-threat detection should treat user activity as a sequence, not as isolated events, and that this reframing is what lets deep sequence models succeed. It proposes User-Based Sequencing (UBS), which reshapes CERT audit logs into a per-user tensor of days, sessions, and features, trains a transformer encoder only on benign users, and treats the encoder's reconstruction error as the anomaly signal. Three unsupervised detectors—One-Class SVM, Local Outlier Factor, and Isolation Forest—then separate normal from anomalous users. The central result reported is that this pipeline catches 99.43% of malicious users on a mixed r4.2/r5.2/r6.2 test set with a 0.0057 false-negative rate, while beating tabular and autoencoder baselines on accuracy, recall, F1, and AUROC. If the result holds, organizations with log data in CERT-like form have a concrete recipe for turning raw audit logs into a high-recall insider-threat detector.","feed_headline":"Insider threat detector catches 99.4% of attacks","feed_subtitle":"Per-user session sequences plus transformer reconstruction errors beat tabular baselines on CERT logs.","key_machinery":"The load-bearing object is User-Based Sequencing (UBS): a rearrangement of raw CERT log features into a per-user tensor of shape [501 days × 9 sessions per day × 35 features], so that one user's whole history is one multi-index sequence. A transformer encoder with an initial linear embedding, positional encoding, six encoder blocks, and eight attention heads is trained with batch size one (one user at a time) to reconstruct benign input; mean squared error between input and output defines the reconstruction error. That scalar error per user/session is then handed to One-Class SVM, Local Outlier Factor, or Isolation Forest, each of which decides whether the error pattern is normal or anomalous. The machinery works by turning 'is this user behaving normally?' into 'can the model rebuild this user's behavior from patterns learned on benign users?'.","core_discovery":"The paper's central claim is that transforming structured log data into per-user temporal sequences, then learning to reconstruct those sequences with a transformer encoder trained exclusively on benign activity, yields an insider-threat detector that outperforms both tabular machine learning and conventional autoencoders. On Test-4, a combined test set of 210 benign and 174 malicious users drawn from CERT r4.2, r5.2, and r6.2, the paper reports 96.61% accuracy, 99.43% recall, 96.38% F1-score, and 95.00% AUROC for the transformer pipeline, with false-negative rate 0.0057 and false-positive rate 0.0571. The paper further reports that the same UBS reorganization lifts the autoencoder baseline substantially, which it takes as evidence that the sequencing structure, not the transformer alone, carries much of the benefit.","pith_inferences":["The reported Test-4 metrics assume a test population that is roughly 45% malicious; real deployments have far lower base rates, so thresholds and scores would need recalibration before the accuracy and F1 numbers transfer.","The paper does not isolate the contribution of temporal order from the contribution of per-user distributions; a shuffled-session control would show how much of the gain actually comes from sequence structure.","The autoencoder improvement under UBS suggests a broader recipe: reorganize any structured log into per-user sequences, train any reconstruction model on benign data, and score reconstruction errors with off-the-shelf outlier detectors; this is testable on other behavioral log domains, such as network flows or physical access records.","Reporting per-release results for Test-4 would show which CERT release drives the combined scores; Test-3, with only five malicious users, is the least informative of the four test sets."],"forward_implications":["Presenting CERT logs as per-user temporal sequences improves both the transformer and an autoencoder baseline, so UBS is a reusable data transformation, not a transformer-only trick.","The pipeline is geared to the insider-threat setting where missed attacks are the costliest failure: the reported recall is near-perfect and the false-negative rate is the lowest among the baselines the paper tabulates.","A model trained on benign r4.2 users is reported to detect malicious users drawn from r5.2 and r6.2, indicating cross-release generalization on synthetic CERT data.","Because the three outlier detectors give similar results on the transformer's reconstruction errors, the errors themselves appear to be a strong anomaly signal regardless of the final classifier.","Session-level granularity is meant to catch behavioral shifts that day- or week-level aggregation would smooth away."],"supporting_citations":[{"why":"Supplies the CERT insider threat dataset used for training and all four test sets.","marker":"[19]"},{"why":"Supplies the transformer encoder architecture and self-attention mechanism the method adapts.","marker":"[8]"},{"why":"Supplies the reconstruction-error anomaly-detection principle used to score the encoder.","marker":"[22]"},{"why":"Motivates one-class ensemble style anomaly detection for insider threats.","marker":"[26]"},{"why":"Supports the choice of unsupervised outlier detectors for anomaly detection problems.","marker":"[27]"},{"why":"Supports applying machine-learning anomaly detection to insider threat data.","marker":"[28]"},{"why":"Provides the transformer baseline that the paper's recall numbers are compared against, along with CERT r4.2 statistics.","marker":"[20]"},{"why":"Provides the TRANLOG transformer log-anomaly baseline compared in the discussion.","marker":"[29]"},{"why":"Provides the LSTM anomaly-detection baseline with recall and FPR compared in Table 7.","marker":"[13]"},{"why":"Provides the convolutional autoencoder (CAE) baseline compared in Table 7.","marker":"[30]"}],"fun_headline_variants":["Transformer sequences catch 99.4% of insider threats","Sequential user modeling reaches 99.43% recall on insider threat","Per-user sequence transformer outperforms tabular baselines","Insider threat detection achieves 96.61% accuracy with transformer sequences","User sessions in sequence boost insider threat detection to 95% AUROC"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The model learns normality only from benign users in CERT r4.2, yet its main test set includes users from CERT r5.2 and r6.2, so the result depends on the assumption that benign behavior, and the scale of the resulting reconstruction errors, is comparable across CERT releases.","fun_headline_variants_meta":{"raw":{"variants":["Transformer sequences catch 99.4% of insider threats","Sequential user modeling reaches 99.43% recall on insider threat","Per-user sequence transformer outperforms tabular baselines","Insider threat detection achieves 96.61% accuracy with transformer sequences","User sessions in sequence boost insider threat detection to 95% AUROC"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000942,"raw_usage":{"total_tokens":4025,"prompt_tokens":943,"completion_tokens":3082,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":559,"completion_tokens_details":{"reasoning_tokens":2993}},"tokens_in":559,"tokens_out":3082,"duration_ms":22287,"temperature":1.0,"reasoning_tokens":2993,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:42:17.817760+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fit OCSVM, LOF, and iForest on reconstruction errors from the benign r4.2 training users only, freeze those detectors, then score the r5.2 and r6.2 users in Test-4 and report per-release recall and FPR; if recall drops materially below 99% or FPR exceeds the reported 0.0571, the headline figures depend on test-set information rather than on the learned notion of normal behavior.","supporting_citations":[{"cited_title":"Insider threat test dataset, 2020","cited_arxiv_id":null,"evidence_quote":"Supplies the CERT insider threat dataset used for training and all four test sets."},{"cited_title":"Anovit: Unsupervised anomaly detection and localization with vision transformer-based encoder-decoder","cited_arxiv_id":null,"evidence_quote":"Supplies the reconstruction-error anomaly-detection principle used to score the encoder."},{"cited_title":"Adaptive one-class ensemble-based anomaly detection: An application to insider threats","cited_arxiv_id":null,"evidence_quote":"Motivates one-class ensemble style anomaly detection for insider threats."},{"cited_title":"Machine learning and anomaly detection for insider threat detection","cited_arxiv_id":null,"evidence_quote":"Supports applying machine-learning anomaly detection to insider threat data."},{"cited_title":"Dtitd: An intelligent insider threat detection framework based on digital twin and self-attention based deep learning models","cited_arxiv_id":null,"evidence_quote":"Provides the transformer baseline that the paper's recall numbers are compared against, along with CERT r4.2 statistics."},{"cited_title":"Translog: A unified transformer-based framework for log anomaly detection","cited_arxiv_id":null,"evidence_quote":"Provides the TRANLOG transformer log-anomaly baseline compared in the discussion."},{"cited_title":"Hunting for insider threats using lstm-based anomaly detection","cited_arxiv_id":null,"evidence_quote":"Provides the LSTM anomaly-detection baseline with recall and FPR compared in Table 7."},{"cited_title":"Insider threat prediction based on unsupervised anomaly detection scheme for proactive forensic investigation","cited_arxiv_id":null,"evidence_quote":"Provides the convolutional autoencoder (CAE) baseline compared in Table 7."}],"review_version":1}