{"id":"ec01849d-e245-43f5-8c7b-2d81f0729129","arxiv_id":"2505.11547","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"GPT-4's TTP lists match human MITRE labels only 39% by Jaccard similarity, yet an embedding-based attribution model ranks the correct threat actor 7.55 on average out of 29, beating the random baseline of 15.","lead":"This paper tests whether AI language models can extract the attack methods described in hacker group reports and then use those methods to guess which group carried out an attack. It finds the AI-generated TTP lists are noisy but still let a simple pipeline rank the correct group above random chance, which could give cyber defenders a faster triage tool.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim that LLM-generated TTPs enable above-baseline attribution is not actually tested: the attribution pipeline in §IV-B trains and evaluates only on VE-search TTPs, not GPT-4 TTPs.","rationale":"The paper's abstract and conclusion explicitly attribute the above-baseline result to 'these TTPs' referring to LLM-generated TTPs, but the experimental design in Section IV.B.1 and Table I uses vector-embedding search as the sole TTP source for the attribution model. This mismatch is the most load-bearing issue because it directly falsifies the paper's central claim as written: even if every other aspect of the experiment were flawless, the reported rank of 7.55 would not demonstrate anything about the utility of GPT-4-generated TTPs for training an attribution model. The reader's identified weakest assumption (near-duplicate reports across the split) is a valid methodological threat to the VE-based result, but it is secondary: it questions the magnitude of the VE pipeline's advantage, whereas the missing GPT-4 experiment questions whether the main contribution (LLM-based attribution) exists at all. The paper does have useful independent contributions—the end-to-end VE pipeline, the honest characterization of LLM noise, and the frequency correlation finding—which is why the verdict should remain conditional rather than rejected. The recommended concrete test is a single, well-scoped experiment that would settle the concern by either supporting the abstract's claim or requiring its revision.","tokens_in":13805,"tokens_out":5703,"duration_ms":52925,"concrete_test":"Re-run the attribution protocol in §IV-B and Table I using GPT-4-generated TTP counts (from the prompt in Fig. 1) as the only features for both the training weight matrix and the test document scoring, with the same 10 train/validation/test splits, expert priors, and ranking metric. Report the average rank with standard deviation or confidence intervals; if the resulting average rank is not significantly better than 15 (e.g., 95% CI includes 15), the abstract's claim that LLM-generated TTPs are useful for attribution is unsupported and should be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section IV.B.1 states that 'The vector embedding search approach is used for all documents in the training dataset to generate Y_ttp', and Table I reports attribution only for VE-search features, including HyDE augmentation; no row or experiment uses GPT-4 TTPs as input to the weight matrix or to the test documents. The abstract and conclusion, however, claim that LLM-generated TTPs 'still prove useful for training a model that performs above baseline on attribution.' This is an internal inconsistency: the quantitative anchor (average rank 7.55 vs random baseline 15) is evidence for the VE pipeline, not for the headline LLM claim. Because Algorithm 2 is agnostic to how the TTPs are generated, the missing experiment is a straightforward gap, not a theoretical impossibility. Unless GPT-4-generated TTPs are run through the same attribution protocol, the central claim is unsupported.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates two methods for extracting MITRE ATT&CK TTPs from unstructured cyber threat intelligence reports: prompting GPT-4 and a vector-embedding (VE) nearest-neighbor search over TTP definitions. The extracted TTPs are compared with MITRE's human-generated per-actor TTP sets using Jaccard similarity and frequency correlation. An attribution pipeline then constructs normalized per-actor TTP weight matrices from VE-extracted TTPs and ranks threat actors for held-out documents by a dot product (Algorithm 2). The best reported average rank is 7.55 out of 29 actors with HyDE augmentation and an expert prior, compared with a random-guess baseline mean of 15. The abstract and conclusion claim that LLM-generated TTPs, despite their divergence from human-generated sets, are useful for training a model that performs above baseline on attribution.","tokens_in":13970,"tokens_out":8329,"duration_ms":74813,"significance":"The paper's main strength is a concrete, reproducible end-to-end pipeline with publicly linked code, and the VE-based attribution result could constitute a useful decision-support proof-of-concept if it survives stronger baselines and a leakage-free evaluation. The frequency-correlation result (R^2=0.81) is interesting but is not evidence for the attribution claim, and the manuscript itself concedes that GPT-4 may have memorized the MITRE data. The central advertised contribution — that LLM-generated TTPs enable above-baseline attribution — is not supported by the reported experiments, which use only VE-search TTPs in the attribution pipeline.","major_comments":[{"comment":"The attribution experiments in Section IV.B and Table I use only TTPs produced by the vector-embedding search: Section IV.B.1 states that 'The vector embedding search approach is used for all documents in the training dataset to generate Y_ttp,' and Table I reports no condition in which GPT-4-generated TTPs are used either in the weight matrix or in the test documents. The abstract's and conclusion's claim that LLM-generated TTPs 'still prove useful for training a model that performs above baseline on attribution' is therefore not supported by any experiment in the manuscript. Since Algorithm 2 is agnostic to how the TTPs are generated, the missing condition is readily testable; the authors should run the attribution protocol on GPT-4 TTPs or revise the central claim to refer specifically to VE-search TTPs.","section":"IV.B.1; Table I"},{"comment":"The only baseline in Table I is random guessing over the 29 actors (uniform distribution with mean 15). This baseline is too weak to establish 'above baseline' attribution: a prior-only classifier that always predicts the training-set prior, a majority-class baseline, or a model trained on the MITRE human-generated TTPs would be far more informative, particularly because the gain from the uniform-prior condition (10.68) to the expert-prior condition (7.55) suggests that the prior itself may account for much of the improvement. Additional baselines are needed to support the claim.","section":"V.A"},{"comment":"The document-level 70/20/10 split is not validated for report independence. If the same or near-identical incident reports appear in both training and test splits, the per-actor TTP profiles are effectively memorized and the reported average rank of 7.55 would be inflated. The manuscript does not describe deduplication or any overlap analysis, and the external reports added for 12 actors ([33], [34]) are not checked for overlap with the MITRE references. The authors should quantify duplicate and near-duplicate overlap across splits and re-evaluate on a deduplicated split if needed.","section":"IV.B"},{"comment":"The description of the weight matrix is internally inconsistent: the normalized counts are first described as P(y_ttp|t), then the text states that after Bayes' theorem each matrix entry represents P(t|y_ttp), but Algorithm 2 computes pattr = y_bar^T Y without showing how the prior P(t) enters the computation. Since Table I reports results for 'Uniform Expert Prior' and 'Expert Expert Prior,' the exact formula used (including the application of the prior) must be specified precisely for the experiments to be reproducible.","section":"IV.B.1; Algorithm 2"}],"minor_comments":[{"comment":"The Jaccard similarity formula is written as |A ∪ B| / |A ∩ B|, which cannot produce values in the claimed [0,1] range and diverges for disjoint sets; the numerator and denominator appear to be swapped. The reported values (e.g., 0.39) are consistent with the standard |A ∩ B| / |A ∪ B|, so the equation should be corrected.","section":"IV.A; Eq. (10)"},{"comment":"The text reports the uniform-prior average rank as '10.96, or 10.68,' while Table I lists 10.68 ± 0.53, and the table's 'Expert Expert Prior' label appears to be a typo for 'Expert Prior.' Please harmonize the prose and table.","section":"IV.B.2; Table I"},{"comment":"The GitHub URL in the abstract contains a space ('https://github.com/kylag/ttp attribution') and should be 'https://github.com/kylag/ttp_attribution'; the conclusion also contains a typo, 'identfication.'","section":"Abstract; Section VII"},{"comment":"The prompt example mixes formats: 'T1083' is listed without a sub-technique element, while 'T1588','.002' is listed with a separate sub-technique string; the notation for sub-techniques in the example should be consistent.","section":"Fig. 1; Section III.A"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's advertised headline contribution — that LLM-generated TTPs train an above-baseline attribution model — is not the contribution evaluated in the attribution experiments; the gap is a missing experiment rather than a theoretical impossibility. The Jaccard equation error and the inconsistent description of the weight matrix and baseline suggest the paper needs a careful technical revision. The code availability and the concrete pipeline description are strengths that make the required additions straightforward. I saw no evidence of misconduct."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid proof-of-concept for LLM/embedding-based TTP extraction and attribution, but the headline claim that LLM-generated TTPs support above-baseline attribution isn't actually tested. The attribution experiments use only vector-embedding (VE) search TTPs. GPT-4 TTPs are compared against MITRE on Jaccard similarity and frequency correlation, but they are never fed into the weights matrix. So the average rank of 7.55 is evidence for the VE pipeline, not for the GPT-4 TTP claim. That is a fixable gap, not a dead end.\n\nWhat's genuinely new: the end-to-end combination of off-the-shelf GPT-4 and text-embedding-3-large for TTP extraction with a probabilistic weights matrix for attribution. Prior work used GPT-3.5 for TTP extraction and separate ML attribution from manually extracted TTPs. The paper also characterizes the noise fairly: roughly 39% Jaccard similarity for GPT-4, 66% \"in GPT not MITRE,\" a high frequency correlation (R^2=0.81) that they themselves flag may be training-data overlap, and a low hallucination rate (0.76%) with concrete examples. Code and data are public. That is real, reproducible work.\n\nSoft spots, in order of importance. First, the missing GPT-4 attribution experiment. Algorithm 2 is agnostic to how TTPs are generated, so running GPT-4 TTPs through it is straightforward. Without that, the abstract's statement that LLM-generated TTPs \"still prove useful for training a model that performs above baseline\" overreaches. Second, the baselines are weak. Random guessing (mean rank 15) is the only attribution comparator. A prior-only baseline (always predict the most frequent actor) or a standard ML classifier trained on MITRE's human TTPs would put 7.55 in context. Third, the 70/20/10 split over MITRE references needs a deduplication check; near-identical reports for the same actor on both sides of the split would inflate the rank. The paper adds external reports but does not describe deduplication. Finally, minor internal inconsistencies: the Jaccard equation (10) has union over intersection, Figure 2 shows top-5 TTP matches while Algorithm 1 uses argmax, and Section IV.B.1 describes the weights matrix as storing P(t|y) when it is trained as P(y|t). These are typos or confusions, not fatal flaws.\n\nAll that said, the paper is honest about its limitations and the central VE pipeline is plausible. The right fix is to run the GPT-4 TTPs through the same attribution protocol and add stronger baselines. I would send this to serious peer review with a request for major revision; the authors have already shown they can build and release a working end-to-end system, and review could push them to align the claims with the evidence.","headline":"Useful, reproducible pipeline but the headline claim overstates: the 7.55 rank attribution result is for VE-search TTPs only, not the GPT-4 TTPs the abstract credits.","tokens_in":14533,"tokens_out":2344,"would_cite":false,"duration_ms":23721,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that off-the-shelf LLM and embedding tools can extract TTPs noisy enough to train an above-chance threat-actor attribution model, with the best model ranking the true actor 7.55th on average out of 29.","keywords":["cyber threat attribution","large language models","TTP extraction","vector embeddings","MITRE ATT&CK","threat actor profiling","Bayesian classification"],"falsifier":"Compute the maximum pairwise embedding similarity or n-gram overlap between each test document and all training documents; if near-duplicate pairs exist, remove them and rerun the 70/20/10 protocol. If the average rank then degrades toward the random baseline of 15, the reported 7.55 is inflated by memorization rather than generalization.","tokens_in":13554,"feed_emoji":"🕵️","tokens_out":10672,"duration_ms":91593,"temperature":0.7,"pith_summary":"This paper asks whether off-the-shelf LLMs and embedding models can replace manual TTP extraction in cyber-attack attribution. It compares LLM-generated TTPs and embedding-search TTPs to human-labeled MITRE ATT&CK data, finding low per-actor Jaccard similarity (GPT-4 around 0.39, embedding search around 0.18) even though GPT-4's TTP frequency distribution tracks MITRE's with $R^2=0.81$. Using vector-embedding search over raw reports to build per-actor TTP profiles, the resulting model attributes held-out documents with an average rank of 7.55 out of 29 threat actors, versus a random baseline mean of 15. The authors conclude that such pipelines are not ready to automate attribution but could serve as decision-support for analysts.","feed_headline":"LLM-built attack profiles rank true actor 7.55th, beating random 15","feed_subtitle":"Noisy LLM-extracted TTPs still beat guessing at attribution, enough to assist analysts though not replace them.","key_machinery":"The load-bearing object is the TTP (tactic, technique, procedure), MITRE ATT&CK's behavioral indicator vocabulary, with each threat actor represented as a probability vector over TTPs. TTP identification has two routes: prompting GPT-4 to output a comma-separated list of technique IDs, or vector-embedding search in which each document is split into 3-line chunks, each chunk and each TTP definition is embedded with text-embedding-3-large, and the chunk is assigned the TTP with maximum cosine similarity; the HyDE variant replaces bare definitions with hypothetical document embeddings. Attribution then normalizes the TTP counts of a new document into $\\bar{y}$, multiplies by a weight matrix $\\tilde{Y}$ whose rows are per-actor normalized TTP probabilities learned from training reports, and applies Bayes' theorem with either a uniform or empirically fit expert prior to produce a ranked list. This machinery converts raw forensic text into a ranked attribution without any manual label extraction.","core_discovery":"On the paper's own terms, the central discovery is that the noise in LLM-generated TTPs is not random: even though both GPT-4 prompting and vector-embedding search produce sets with low Jaccard similarity to human-generated MITRE labels, GPT-4's per-TTP frequencies track MITRE's frequencies with $R^2=0.81$, and the probabilistic TTP profiles built from embedding search train a model that ranks the correct threat actor 7.55th on average among 29 candidates, above the random baseline mean of 15. The authors read this as evidence that behavioral TTP extraction from raw reports retains enough signal for decision support, with the largest gains for actors with distinctive, low-entropy TTP profiles and notable failures for high-entropy actors such as Lazarus Group and menuPass.","pith_inferences":["A clean testable extension is to rerun the pipeline on reports published after GPT-4's training cutoff; if the $R^2=0.81$ frequency correlation persists, it reflects behavioral regularity rather than memorization of MITRE data.","The high-entropy failure pattern suggests an abstention rule: when the predicted TTP profile has entropy above a learned threshold, the model should return 'insufficient evidence' rather than a ranked list.","The same embedding-search mechanism could be adapted to other behavioral vocabularies, such as ICS/OT techniques or malware API calls, by swapping the TTP definition set, since nothing in the pipeline depends on the specific MITRE taxonomy."],"forward_implications":["An end-to-end attribution pipeline can be assembled from raw CTI reports and off-the-shelf models, reaching an average rank of 7.55 out of 29 threat actors on held-out documents.","The same architecture can rank candidates as a triage aid for analysts, while flagging actors such as Lazarus Group and menuPass as hard to distinguish.","Because GPT-4's TTP output is noisy but frequency-aligned with MITRE, LLM-generated TTP sets are best used to supplement, not replace, human-annotated profiles.","Giving a threat actor more source documents improves downstream attribution rank, so the pipeline's value grows as the reference corpus expands.","A probabilistic, normalized TTP weight matrix provides more granular information to analysts than MITRE's binary presence/absence profiles."],"supporting_citations":[{"why":"Provides the human-labeled TTP sets and per-actor reference documents that serve as ground truth and training corpus.","marker":"[10]"},{"why":"Supplies the TTP taxonomy and definitions used as the candidate set for vector-embedding search.","marker":"[29]"},{"why":"Prior ML-based attribution from manually extracted TTPs that this pipeline automates and extends.","marker":"[2]"},{"why":"Earlier empirical comparison of high-level versus low-level attack patterns for attribution, motivating TTP-level features.","marker":"[18]"},{"why":"Adds hypothetical document embeddings to TTP definitions, used in the best-performing attribution variant.","marker":"[30]"},{"why":"Massive text embedding benchmark used to justify the choice of text-embedding-3-large.","marker":"[28]"},{"why":"Additional open-source reports for 12 threat actors used to reach the full document set.","marker":"[33]"},{"why":"Second external report source used to complete the document set for training and evaluation.","marker":"[34]"}],"fun_headline_variants":["Noisy LLM TTPs still beat random at cyber attribution","LLM-extracted attack profiles rank true actor 7.55th of 29","GPT-4 TTP noise still carries attribution signal","Despite low Jaccard, LLM TTPs train above-random attribution","LLM TTP frequency matches MITRE (R^2=0.81), aids attribution"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The 70/20/10 document split assumes reports about the same threat actor in the test set are not near-duplicates of training reports, and the paper does not describe deduplication or overlap checks before adding external reports for 12 actors.","fun_headline_variants_meta":{"raw":{"variants":["Noisy LLM TTPs still beat random at cyber attribution","LLM-extracted attack profiles rank true actor 7.55th of 29","GPT-4 TTP noise still carries attribution signal","Despite low Jaccard, LLM TTPs train above-random attribution","LLM TTP frequency matches MITRE (R^2=0.81), aids attribution"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001021,"raw_usage":{"total_tokens":4320,"prompt_tokens":973,"completion_tokens":3347,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":589,"completion_tokens_details":{"reasoning_tokens":3247}},"tokens_in":589,"tokens_out":3347,"duration_ms":24701,"temperature":1.0,"reasoning_tokens":3247,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:20:26.421780+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the maximum pairwise embedding similarity or n-gram overlap between each test document and all training documents; if near-duplicate pairs exist, remove them and rerun the 70/20/10 protocol. If the average rank then degrades toward the random baseline of 15, the reported 7.55 is inflated by memorization rather than generalization.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the human-labeled TTP sets and per-actor reference documents that serve as ground truth and training corpus."},{"cited_title":"Win- dows Malware Detection Based on Cuckoo Sandbox Generated Report Using Machine Learning Algorithm,","cited_arxiv_id":null,"evidence_quote":"Supplies the TTP taxonomy and definitions used as the candidate set for vector-embedding search."},{"cited_title":"Cyber Threat Attri- bution Using Unstructured Reports in Cyber Threat Intelligence,","cited_arxiv_id":null,"evidence_quote":"Prior ML-based attribution from manually extracted TTPs that this pipeline automates and extends."},{"cited_title":"A Machine Learning based Empirical Evaluation of Cyber Threat Actors High Level Attack Patterns over Low level Attack Patterns in Attributing Attacks","cited_arxiv_id":"2307.10252","evidence_quote":"Earlier empirical comparison of high-level versus low-level attack patterns for attribution, motivating TTP-level features."},{"cited_title":"MTEB: Massive text embedding benchmark,","cited_arxiv_id":null,"evidence_quote":"Massive text embedding benchmark used to justify the choice of text-embedding-3-large."},{"cited_title":"etda.or.th/cgi-bin/aptgroups.cgi, 2024","cited_arxiv_id":null,"evidence_quote":"Additional open-source reports for 12 threat actors used to reach the full document set."},{"cited_title":"Mazuz, Attack Attribution Dataset , https : / / github","cited_arxiv_id":null,"evidence_quote":"Second external report source used to complete the document set for training and evaluation."}],"review_version":1}