Pith. sign in

REVIEW 3 major objections 5 minor 12 references

APT-LLM: Embedding-Based Anomaly Detection of Cyber Advanced Persistent Threats Using Large Language Models

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that a pipeline of pretrained LLM sentence embeddings plus autoencoder reconstruction error detects advanced persistent threats in DARPA TC provenance data better than OC-SVM, DBSCAN, and Isolation Forest, even when…

desk verdict Potentially useful pipeline, but the reported AUC is a best-of-15 selection on test data, so the claimed advantage over baselines is unproven. read the letter →

arxiv 2502.09385 v1 pith:GH64ZI63 submitted 2025-02-13 cs.CR

Sidahmed Benabderrahmane , Petko Valtchev , James Cheney , Talal Rahwan This is my paper · ORCID
classification cs.CR
keywords anomalydetectionadvancedpersistentthreatslargelanguagemodelsautoencodersprovenancetracesDARPATransparentComputingclassimbalanceembedding-based
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a generic pretrained large language model can replace manual feature engineering in detecting advanced persistent threats (APTs) from system provenance traces. The proposed APT-LLM pipeline converts each process's recorded events into a short templated sentence, embeds the sentence with off-the-shelf LLMs, and trains an autoencoder on embeddings of normal behavior; at test time, high reconstruction error flags a process as potentially malicious. On forty highly imbalanced DARPA Transparent Computing datasets spanning Android, Linux, BSD, and Windows, the best LLM-autoencoder combination is reported to outperform OC-SVM, DBSCAN, and Isolation Forest in AUC for most datasets. The significance is that effective APT detection may not require domain-specific feature engineering or fine-tuning.

What carries the argument

The load-bearing mechanism is the mapping of provenance records to short sentences such as 'Process 123 has event open a file and event write to the file and event send network data', followed by mean-pooled or [CLS] embeddings from a frozen transformer LLM. These vectors replace hand-engineered features. The second half is the autoencoder family: a baseline AE, a VAE with a KL-regularized Gaussian latent space, and a DAE trained on noise-corrupted inputs, all augmented with self-attention in the encoder. Reconstruction error on the LLM embedding is the anomaly score, and the paper shows the specific pairing matters, with ALBERT plus VAE reaching AUC 0.95 on one dataset while BERT and RoBERTa underperform.

What would settle it

Run the same autoencoder training on bag-of-words, TF-IDF, or random vector embeddings of the exact same templated sentences. If the AUC of these cheap baselines matches or exceeds APT-LLM, then the LLM's contextual embeddings are not the source of the detection advantage, contradicting the paper's central claim. Alternatively, shuffle the order of events inside each sentence; if detection performance does not drop, the model is not using the sequential semantics that the LLM is supposed to provide.

Watch

Extended reading notes

Core claim

APT-LLM is an unsupervised anomaly detection pipeline whose central claim is that semantic embeddings produced by pretrained language models, when fed to an autoencoder trained only on normal processes, yield higher AUC than classical anomaly detectors under extreme class imbalance. The authors evaluate five LLMs (BERT, ALBERT, DistilBERT, RoBERTa, MiniLM) and three autoencoder variants (AE, VAE, DAE) across forty datasets, with APT activity as low as 0.004% of processes. For each dataset the best LLM-autoencoder pairing is selected and compared against OC-SVM, DBSCAN, and Isolation Forest; APT-LLM consistently achieves the highest AUC scores, for example 0.97 in the Windows Pandex PX dataset versus 0.85, 0.23, and 0.51 for the three baselines.

Load-bearing premise

The framework assumes that a short templated sentence like 'Process 123 has event open a file and event write to the file and event send network data' preserves enough behavioral signal for a pretrained natural-language model to separate stealthy APT activity from normal activity, without any fine-tuning or domain adaptation.

Editorial extensions

If this is right

  • Security analysts could build an APT detector for a new environment from normal-process traces alone, with no manual feature engineering and no fine-tuning of the language model.
  • The pipeline transfers across operating systems and attack scenarios as long as the autoencoder is retrained on local normal data, since the embedding layer is frozen.
  • Smaller LLMs (ALBERT, MiniLM) can match or beat larger ones, so the approach is compatible with resource-constrained deployment.
  • Because the anomaly score is reconstruction error on a textual embedding, flagged processes can be traced back to their templated sentences, giving analysts a natural starting point for inspection.
  • Classical anomaly detectors are still competitive on some BSD datasets, so APT-LLM is best seen as an addition to a detection toolbox rather than a universal replacement.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural experiment would replace the LLM embeddings with bag-of-words or random projections of the same templated sentences; if the AUC gap over classical baselines persists, the semantic richness of the LLM is not what carries detection.
  • Because the templated sentences drop event ordering, timestamps, and argument values, serializing the provenance graph into ordered walks before embedding could improve detection if temporal or dependency structure matters; the paper does not test this.
  • The per-dataset selection of the best LLM-autoencoder pair likely inflates reported AUC; fixing the pair a priori on a validation split would give a more realistic estimate of out-of-the-box performance.
  • Plugging a domain-adapted cybersecurity language model into the same autoencoder pipeline would quantify how much of the advantage comes from general-purpose pretraining versus domain-specific text.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes APT-LLM, a framework that turns provenance traces into short English sentences, embeds those sentences with pretrained LLMs (BERT, ALBERT, DistilBERT, RoBERTa, MiniLM), and runs autoencoder variants (AE, VAE, DAE) on the embeddings for anomaly detection. The authors evaluate 15 LLM-autoencoder combinations on 40 DARPA Transparent Computing datasets spanning four operating systems and two attack scenarios, then select the best combination per dataset for comparison against OC-SVM, DBSCAN, and Isolation Forest. The reported result is that APT-LLM consistently achieves higher AUC than the classical baselines across datasets.

Significance. If the empirical claims were valid, the paper would demonstrate a useful and relatively low-effort recipe: pretrained LLM embeddings plus reconstruction-based anomaly detection for APT discovery in provenance data, evaluated across a broad multi-OS benchmark. The dataset coverage is a genuine strength, and the extreme class-imbalance settings (as low as 0.004% attacks) are realistic and worth studying. However, the central comparison is compromised by a post-hoc selection step in which the best of 15 configurations is chosen after observing test AUC, and no uncertainty quantification is provided. The claimed consistent advantage over baselines is therefore not established, which substantially reduces the significance of the contribution. The paper does not provide code or machine-checked artifacts that would mitigate these concerns.

major comments (3)
  1. [Section IV.E, Fig. 6] The main empirical claim is unsupported by the experimental design. The text states that heatmaps were generated for all 5x3 combinations across datasets and 'for each dataset, we identified the combination yielding the maximum AUC' to serve as APT-LLM. Thus APT-LLM's reported AUC is the maximum over 15 configurations evaluated on the same test data, while each baseline is a single method evaluated once. This post-hoc test-set selection systematically inflates the reported APT-LLM scores, especially because several datasets have only 8 to 46 positive attack records (Table I), so AUC estimates are high-variance. The claim that 'APT-LLM consistently achieves higher AUC scores' (Section IV.E) is therefore not supported as stated. The authors should either fix the pipeline before evaluation (e.g., select the LLM-autoencoder pair on a validation split with no test-set peeking), report all 15 configurations alongside the baselines with appropriate multiple-comparison awareness, or provide a nested cross-validation protocol.
  2. [Section IV.E, Table I] No error bars, repeated runs, or statistical tests are reported for any AUC value. With attack counts as low as 8 per dataset, the AUC estimates have very wide confidence intervals, and the observed differences between APT-LLM and the baselines may not be significant. For example, the reported PX Windows Pandex comparison (APT-LLM 0.97 vs OC-SVM 0.85) involves an extremely small positive class, and no confidence intervals are given. The authors should provide bootstrap confidence intervals, repeated-seed runs, or a statistical significance test (e.g., DeLong's test for paired AUCs) before claiming consistent superiority.
  3. [Section III-B, Section IV-B] The entire framework relies on the assumption that templated sentences of the form 'Process 123 has event open a file and event write to the file and event send network data' preserve enough behavioral signal for pretrained general-purpose LLM embeddings to separate stealthy APT activity from normal activity. The sentence template discards event order, event arguments, and dependency relations, and the paper provides no analysis or ablation showing that the resulting embeddings carry the information needed for detection. Because no domain adaptation or fine-tuning is performed, this is a load-bearing assumption. The authors should either justify it with concrete evidence (e.g., comparisons against embeddings that preserve order or include arguments, or nearest-neighbor checks showing that semantically similar behaviors are embedded close together) or temper the claims accordingly.
minor comments (5)
  1. [Section IV.E] The phrase 'the baseline APT-LLM' appears to be a typo; the selected best configuration is not a baseline but the proposed method's best variant.
  2. [Section III-C and elsewhere] The notation 'V AE' has an inconsistent space and should be written as 'VAE' throughout.
  3. [Section IV.A, Table I] Table I is difficult to parse: the header mixes dataset size, row/column counts, attack counts, and imbalance ratio with different separators, and some rows (e.g., Android 1) appear to have zero columns for PP. A clearer table layout with separate columns for rows, columns, and attack counts would improve readability.
  4. [Section III.D.1] The anomaly threshold tau is said to be 'based on a percentile of ri among known normal samples or tuned via validation metrics,' but the reported results are threshold-free AUC values, while Figure 3 shows a hard threshold. The relationship between the threshold-based visualization and the AUC-based evaluation should be clarified.
  5. [Section IV.E] The paper claims 'significantly improves' in the abstract and conclusion, but no significance test is performed; the wording should be aligned with the absence of statistical inference.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation found: LLM embeddings, autoencoder reconstruction, and anomaly thresholds are all computed externally; the per-dataset best-of-15 test-set selection is a soundness/overfitting concern, not a circular reduction.

full rationale

The claimed derivation chain is not circular. LLM sentence embeddings are produced by pretrained external models f_LM(s_i); no equation defines these embeddings in terms of the detection labels or the final AUC. Autoencoders (AE/VAE/DAE) are trained on normal-process embeddings with standard reconstruction objectives; the anomaly score is reconstruction error compared with an independently set threshold. Thus the framework's outputs are computed from external pretrained weights, normal-data training, and generic reconstruction loss. The main empirical concern is the evaluation protocol: Section IV.E selects, per dataset, the LLM×AE combination with maximum test AUC and reports that as APT-LLM, so the displayed 'APT-LLM' curve is a max-of-15 selection statistic and the comparison against single OC-SVM/DBSCAN/IF baselines is optimistically biased. This is a test-set selection/overfitting problem that threatens the soundness of the claim, but it is not a circular reduction: the selected maximum is not by construction guaranteed to exceed the classical baselines, and no fitted parameter is renamed as an independent prediction. Self-citations [8]-[10] are used only to identify public DARPA TC datasets and scenario names, not as load-bearing evidence for the embedding-anomaly derivation. Accordingly, no circularity step is exhibited.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The framework does not derive any constants from first principles. All performance depends on unstated hyperparameters (tau, beta, k, noise level) and on a per-dataset model-selection rule that fits the test set. No new physical or conceptual entities are introduced.

free parameters (5)
  • Anomaly threshold tau = not specified
    Section III-D1 says the threshold can be a percentile of normal reconstruction errors or tuned via validation; the value is never reported, and flags depend on it.
  • VAE KL weight beta = not specified
    Section III-C2 defines L_VAE = L_recon + beta * L_KL with no value, yet beta changes the latent geometry and reconstruction behavior.
  • Autoencoder latent dimension k = not specified
    Section III-C1 only requires k << d; no layer sizes, depths, learning rates, or epochs are given, so the architecture is underspecified.
  • DAE noise level sigma = not specified
    Section III-C3 adds Gaussian noise eta ~ N(0, sigma^2) or masks features, but the noise level and masking rate are not reported.
  • Per-dataset model selector = the maximum-AUC LLM x AE combination for each of 40 datasets
    Section IV.E selects the best of 5 LLMs x 3 autoencoders after seeing test AUC; this is a hand-chosen, data-dependent selection that inflates the reported APT-LLM score.
assumptions (5)
  • domain assumption Templated English sentences of process events preserve the behavioral signal needed to distinguish APT activity from normal activity.
    Introduced in Section III-B; the entire embedding stage depends on this mapping.
  • domain assumption Pretrained LLM embeddings are meaningful for short synthetic system-event sentences without fine-tuning or domain adaptation.
    Used in Section IV-B, where BERT-family models are applied directly to generated sentences.
  • domain assumption Reconstruction error of autoencoders trained predominantly on normal data is a valid anomaly score at 0.004% class imbalance.
    Assumed in Section III-D; no calibration, error analysis, or class-imbalance handling is presented.
  • domain assumption DARPA TC provenance data processed by the ADAPT ingester has correct normal/anomalous labels and representative records.
    Section IV-A relies on these labels for all AUC calculations; no label audit is reported.
  • domain assumption Standard deep learning training assumptions apply: i.i.d. records, stable embedding distribution, and convergence of the autoencoders.
    Assumed throughout Section III-C and IV-E; no seeds, splits, or convergence diagnostics beyond one loss curve are shown.

how reviews work

0 comments
Cite this review

Pith. "Pith review of APT-LLM: Embedding-Based Anomaly Detection of Cyber Advanced Persistent Threats Using Large Language Models." pith.science (2026). https://pith.science/paper/GH64ZI63

@misc{pith2026250209385,
  author       = {Pith},
  title        = {Pith review of: APT-LLM: Embedding-Based Anomaly Detection of Cyber Advanced Persistent Threats Using Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GH64ZI63}},
  note         = {Machine review of arXiv:2502.09385}
}
read the original abstract

Advanced Persistent Threats (APTs) pose a major cybersecurity challenge due to their stealth and ability to mimic normal system behavior, making detection particularly difficult in highly imbalanced datasets. Traditional anomaly detection methods struggle to effectively differentiate APT-related activities from benign processes, limiting their applicability in real-world scenarios. This paper introduces APT-LLM, a novel embedding-based anomaly detection framework that integrates large language models (LLMs) -- BERT, ALBERT, DistilBERT, and RoBERTa -- with autoencoder architectures to detect APTs. Unlike prior approaches, which rely on manually engineered features or conventional anomaly detection models, APT-LLM leverages LLMs to encode process-action provenance traces into semantically rich embeddings, capturing nuanced behavioral patterns. These embeddings are analyzed using three autoencoder architectures -- Baseline Autoencoder (AE), Variational Autoencoder (VAE), and Denoising Autoencoder (DAE) -- to model normal process behavior and identify anomalies. The best-performing model is selected for comparison against traditional methods. The framework is evaluated on real-world, highly imbalanced provenance trace datasets from the DARPA Transparent Computing program, where APT-like attacks constitute as little as 0.004\% of the data across multiple operating systems (Android, Linux, BSD, and Windows) and attack scenarios. Results demonstrate that APT-LLM significantly improves detection performance under extreme imbalance conditions, outperforming existing anomaly detection methods and highlighting the effectiveness of LLM-based feature extraction in cybersecurity.

Figures

Figures reproduced from arXiv: 2502.09385 by the authors.

Figure 1
Figure 1. T-SNE Visualizations of Embeddings Using Different LLMs. Blue [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 4
Figure 4. Heatmap of AUC Scores for LLM and Autoencoder Combinations: [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 3
Figure 3. Scatter Plot of the AutoEncoder Reconstruction Errors by Sample [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: ROC Curve Comparison for the Best performing models (PE dataset [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: AUC heatmap comparing the performance of APT-LLM, OC-SVM, [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 10 canonical work pages

  1. [1]

    Strategically-motivated advanced persistent threat: Definition, process, tactics and a disinformation model of coun- terattack,

    A. Ahmad, J. Webb et al., “Strategically-motivated advanced persistent threat: Definition, process, tactics and a disinformation model of coun- terattack,” Computers & Security , vol. 86, pp. 402–418, 2019

    work page 2019

  2. [2]

    A systematic literature review on advanced persistent threat behaviors and its detection strategy,

    N. I. Che Mat, N. Jamil, Y . Yusoff, and M. L. Mat Kiah, “A systematic literature review on advanced persistent threat behaviors and its detection strategy,” Journal of Cybersecurity , vol. 10, no. 1, p. tyad023, 2024

    work page 2024

  3. [3]

    A survey on evaluation of large language models,

    Y . Chang, X. Wang, J. Wang, Y . Wu, L. Yang, K. Zhu, H. Chen, X. Yi, C. Wang, Y . Wang et al. , “A survey on evaluation of large language models,” ACM Transactions on Intelligent Systems and Technology , vol. 15, no. 3, pp. 1–45, 2024

    2024

  4. [4]

    Large language models in cybersecurity: State-of-the-art,

    F. N. Motlagh, M. Hajizadeh, M. Majd, P. Najafi, F. Cheng, and C. Meinel, “Large language models in cybersecurity: State-of-the-art,” arXiv preprint arXiv:2402.00891 , 2024

    arXiv 2024

  5. [5]

    A scalable and efficient outlier detection strategy for categorical data,

    A. Koufakou et al. , “A scalable and efficient outlier detection strategy for categorical data,” in 19th IEEE Int Conf on Tools with Artificial Intelligence(ICTAI 2007). IEEE, Oct. 2007, pp. 210–217

    work page 2007

  6. [6]

    The odd one out: Identifying and char- acterising anomalies,

    K. Smets and J. Vreeken, “The odd one out: Identifying and char- acterising anomalies,” in Proceedings of the 2011 SIAM international conference on data mining . SIAM, 2011, pp. 804–815

    work page 2011

  7. [7]

    Outlier detection for transaction databases using association rules,

    K. Narita and H. Kitagawa, “Outlier detection for transaction databases using association rules,” in 2008 The 9th Int Conf on Web-Age Informa- tion Management, 2008, pp. 373–380

    work page 2008

  8. [8]

    A baseline for unsupervised advanced persistent threat detection in system-level provenance,

    G. Berrada et al., “A baseline for unsupervised advanced persistent threat detection in system-level provenance,” Future Generation Computer Systems, vol. 108, pp. 401–413, 2020

    work page 2020

Show all 12 references
  1. [9]

    A rule mining-based advanced persistent threats detection system,

    S. Benabderrahmane et al. , “A rule mining-based advanced persistent threats detection system,” in Proc of IJCAI21, Montreal, Canada , Z. Zhou, Ed. ijcai.org, 2021, pp. 3589–3596

    2021

  2. [10]

    Hack me if you can: Aggregating autoencoders for countering persistent access threats within highly imbalanced data,

    S. Benabderrahmane, N. Hoang et al., “Hack me if you can: Aggregating autoencoders for countering persistent access threats within highly imbalanced data,” Future Gener. Comput. Syst. , vol. 160, pp. 926–941, 2024

    2024

  3. [11]

    A survey of large language models for cyber threat detection,

    Y . Chen et al. , “A survey of large language models for cyber threat detection,” Computers & Security , p. 104016, 2024

    2024

  4. [12]

    Cysecbert: A domain-adapted language model for the cybersecurity domain,

    M. Bayer et al. , “Cysecbert: A domain-adapted language model for the cybersecurity domain,” ACM Transactions on Privacy and Security , vol. 27, no. 2, pp. 1–20, 2024

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.