REVIEW 4 major objections 6 minor 36 references
Assessing and Prioritizing Ransomware Risk Based on Historical Victim Data
T0 review · 4 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read A machine learning model trained on historical ransomware victim disclosures and LLM-extracted adversary profiles can predict which specific ransomware variant is most likely to target a given organization, with reported precision…
desk verdict The 99% accuracy is a circular artifact of the synthetic label generator; the paper has a useful LLM extraction component but no real predictive evidence. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing machinery is the synthetic-data generation pair of Algorithm 1 and Algorithm 2, feeding a Random Forest classifier. Algorithm 1 copies each real victim and adds Gaussian noise to revenue, employee count, and the activity measure, preserving country, sectors, and company type, to create additional 'unsafe' samples. Algorithm 2 creates 'safe' samples by probabilistically permuting at least one feature per record—country with probability 0.8, activity set to zero with 0.95, revenue 0.7, employee count 0.3, company type 0.3—so the safe label is defined as 'differs from every documented victim of that variant in at least one feature.' On the adversary side, a large-language-model prompt system using chain-of-thought and multi-shot examples extracts SKRAM (Skills, Knowledge, Resources, Authorities, and Motivation) attributes and attack-technique capability counts from threat reports, supplying the features the classifier combines with victim profiles.
What would settle it
Run the identical Random Forest on a holdout set built from real ransomware victims recorded after the training window plus real companies of similar size and sector that were never attacked; if precision and recall fall well below the reported 99%, or if the model classifies most real non-victims as targets simply because their profiles still resemble some documented victim, the synthetic-label assumption is the reason.
Extended reading notes
Core claim
The paper's central claim is that the combination of LLM-extracted adversary SKRAM profiles, standard company attributes, and a time-sensitive activity measure (the exponentially weighted moving average of monthly attacks) carries enough signal to distinguish entities a given ransomware variant will target from entities it will not. The authors report that a Random Forest trained on 8,200 records—real victims augmented by synthetic unsafe samples plus an equal number of synthetically generated safe samples—achieves 99% precision, recall, and F1. They convert the model's confidence score into a 10-point risk scale from None to Extremely High, and illustrate it with a U.S. automotive and manufacturing company rated Extremely High for Phobos and Low for Rhysida, with the group's recent activity the single most influential feature.
Load-bearing premise
The load-bearing premise is that a company which differs from a documented victim of a ransomware group in any one of the modeled features—location, size, revenue, organization type, or the group's recent activity level—is genuinely safe from that group, and that this manufactured 'safe' label reflects real-world targeting rather than just the permutation rule.
Editorial extensions
If this is right
- Defenders can shrink a global list of ransomware variants to a short set of adversaries whose victim profile matches their organization and whose activity measure is nonzero, making threat intel consumption feasible for small teams.
- For each high-risk variant, the associated attack-technique and capability data can be used to choose mitigations and rehearse incident response, since the model output links a risk score to a specific adversary.
- The 10-point risk scale provides a concrete prioritization signal that can be embedded in existing risk-management workflows without requiring internal telemetry or special equipment.
- The method is designed to transfer across sectors because it uses only public company attributes, so the same pipeline could be run by different organizations with their own profiles.
- The model must be retrained as new victim disclosures appear, because both the activity measure and group victim preferences drift over time; the authors treat continuous observation as part of the approach.
Reading between the lines
- The reported 99% F1 may mostly measure how cleanly the model rediscovers the synthetic 'safe' permutation rule; the decisive test is a holdout set of real future victims and real unvictimized companies, which the authors do not provide.
- Because the activity score is set to zero for most safe samples and is the strongest feature, much of the model's discrimination may reduce to 'is this group currently active?' rather than 'does this group prefer companies like yours?'
- The US-centric bias the authors attribute to their annotation protocol implies the model's risk scores may be trustworthy mainly where victim disclosures are complete; broadening to other regions likely requires correcting for disclosure bias.
- A practical extension would compare this model against a simpler baseline that ranks threats by current activity plus stated sector preferences; if that baseline matches the ranking, the LLM-extracted SKRAM features add little beyond the activity signal.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes an end-to-end pipeline for ransomware risk prioritization: it uses an LLM with chain-of-thought, multi-shot prompts to extract SKRAM (Skills, Knowledge, Resources, Authorities, Motivation) attributes and STIX-compatible features from threat reports; it builds victim profiles from 409 Ransomware.live incidents; it augments the data through two synthetic-data algorithms (noisy copies of victims labeled unsafe, and permuted copies labeled safe); and it trains a random forest to output a 10-point risk score per ransomware group. The authors report 99% precision, recall, and F1 on an 80/20 split of the synthetic dataset and present two illustrative company-level predictions.
Significance. The problem is practically important, and the paper contains useful engineering components: the LLM extraction workflow is described in enough detail to be reproduced, the SKRAM/STIX feature schema is concrete, the synthetic-data algorithms are given as explicit pseudocode, and student validation of LLM outputs is a sensible quality-control step. However, the core empirical claim—that the model predicts which entities a ransomware group will target—is not supported. The 99% metrics are obtained on a synthetic test set whose labels are generated from the same feature-permutation rule the classifier uses, and the paper explicitly defers real-incident validation to future work. As it stands, the contribution is a documented feasibility study of an extraction-and-synthesis pipeline, not a validated predictive model.
major comments (4)
- [Section 4, Algorithm 2 and Figure 2] The evaluation is circular. The 'safe' class is manufactured by copying a known victim and randomly permuting at least one of the exact features the classifier consumes—country (0.8), revenue (0.7), employees (0.3), company type (0.3), and EWMA (0.95)—while 'unsafe' examples are noisy copies of victims with those features preserved (Algorithm 1). The test set used in Section 5.2 is drawn from the same synthetic generation process, so the random forest's 99% precision/recall/F1 measures its ability to detect whether Algorithm 2's mutation rule fired, not its ability to identify entities a ransomware group would actually target. No real non-victim entity or held-out real incident is used to validate the model.
- [Sections 5.4, 6, and Conclusion] The central claim that the model 'predicts the likelihood of an entity being targeted' is unsupported by real-world evaluation. Section 5.4 explicitly states that 'future work will focus on validating model predictions with more extensive datasets of real incidents as they become available,' and Section 6 repeats this deferral. The only quantitative evidence comes from a synthetic test set whose labels are produced by the same algorithms used to train the model, and no baseline comparison (e.g., majority class, logistic regression, or a simple profile-distance rule) is reported. The paper either needs independent outcome data or must restrict its conclusions to a feasibility demonstration of the extraction and synthesis pipeline.
- [Section 5.2 and Table 3] The text and Table 3 contradict each other on the confusion matrix. The text states that the model 'incorrectly classified 13 instances as positive but did not mislabel any instances as negative,' which describes 13 false positives and 0 false negatives, whereas Table 3 places the 13 errors in the actual-positive/predicted-negative cell, i.e., 13 false negatives and 0 false positives. The reported precision and recall values differ depending on which reading is correct, so the inconsistency must be resolved and the metrics recomputed.
- [Section 3.1, Eq. (1)] The EWMA activity feature is described as being 'calculated over the entire dataset for each record' in both training and deployment. For a record at month t, this can incorporate attacks that occur after t, making the feature non-causal for predicting whether an entity is targeted at time t. The authors should clarify whether the EWMA is computed causally (using only months up to the prediction date) and, if not, assess the resulting temporal leakage.
minor comments (6)
- [Section 5.1] The dataset size is internally inconsistent: 409 original victims, n=10 synthetic copies per victim in Algorithm 1, and n=10 safe samples per victim in Algorithm 2 give 8,180 total records, not the stated '8,200 records evenly split between safe and unsafe' (4,100 each).
- [Listing 1.1] The prompt excerpt contains typographical artifacts such as 'du str y', 'I de nti fy', and 'Con si der', which obscure the intended text and should be cleaned before publication.
- [References] Several references contain typographical errors, including 'Anthrooic' (ref. 33) and 'Introducin llama 2' (ref. 34), and should be corrected.
- [Figure 2] Figure 2 is displayed as a code block but is referred to as a table; its formatting and caption should be aligned with its actual presentation.
- [Figure 6 caption] The caption refers to the 'Enumerated Weighted Moving Average' instead of the 'Exponential Weighted Moving Average' used elsewhere in the paper.
- [Section 5.3] The paper does not specify how the classifier's confidence score is converted to the 10-point risk scale (None through Extremely High), which readers would need to reproduce or interpret the example predictions.
Circularity Check
99% accuracy is an artifact of Algorithm 2's synthetic 'safe' labels: safe samples are mutated copies of real victims, so the test set certifies detection of the mutation rule, not real-world targeting prediction.
-
self definitional
[Section 4, Algorithm 2 and Figure 2]
"sample.’safe’ ← 1 ▷ Mark the sample as safe ... until DidPermutate(sample) ... Finally, we ensure at least one field has permuted in order to declare the record safe from any ransomware attacks."
The 'safe' class is manufactured by mutating at least one of the exact features the classifier consumes (country 0.8, revenue 0.7, employees 0.3, company type 0.3, EWMA 0.95), while unsafe samples are untouched copies of real victims. Hence the target label 'safe' is defined as DidPermutate(sample): a deterministic function of the input features. A model trained and tested on data produced by the same Algorithm 2 can achieve near-perfect scores merely by detecting the permutation rule; it is not learning real-world targeting likelihood. No observed non-victim entity is ever used.
-
fitted input called prediction
[Section 5.2 and Section 5.4]
"The resulting model achieves high precision, recall, and F1-score, all measuring at 99%. ... To address this, future work will focus on validating model predictions with more extensive datasets of real incidents as they become available, enhancing our understanding of the model’s applicability across different contexts."
The 99% precision/recall/F1 is computed on an 80/20 split of the same 8,200-record synthetic dataset generated by Algorithms 1 and 2. Section 5.4 explicitly defers validation against real incidents to future work. Thus the headline performance measures how well the Random Forest recovers Algorithm 2's safe/unsafe generation rule on held-out synthetic samples, not predictive skill for real ransomware targeting. The synthetic labeling process is the fitted input, and the reported prediction metrics are its direct consequence.
full rationale
The central circularity is in the construction of the 'safe' class. Algorithm 2 copies a known victim and permutes at least one of the very features the classifier uses, then labels the result 'safe'; Algorithm 1 labels perturbed copies of victims as 'unsafe'. The Random Forest is trained and evaluated on this synthetic dichotomy, so the 99% accuracy in Section 5.2 is largely a measure of how well the model inverts the DidPermutate rule, not how well it predicts real ransomware targeting. The paper itself concedes in Sections 5.4 and 6 that real-incident validation is deferred to future work, confirming that no independent outcome data supports the headline claim. The LLM/SKRAM extraction pipeline and feature engineering are independent contributions and are not themselves circular; the circularity is localized to the synthetic labeling and the evaluation built on it. For this reason the score is 8 rather than 10: the derivation chain is not entirely equivalent to its inputs, but the load-bearing predictive claim reduces, by the paper's own construction, to the synthetic generation rule.
Assumptions & free parameters
free parameters (6)
- EWMA decay lambda =
0.2
- Gaussian noise sigma for revenue, employees, EWMA =
not reported (derived from dataset variability)
- Synthetic samples per victim n =
10
- Permutation probabilities for safe samples =
country 0.8, revenue 0.7, ewma 0.95, employees 0.3, company type 0.3
- Random forest n_estimators =
100
- Random state =
42
assumptions (5)
- domain assumption LLM-extracted SKRAM features are accurate representations of adversary profiles
- ad hoc to paper Synthetic safe samples generated by permuting features with given probabilities represent entities unlikely to be targeted
- domain assumption EWMA of monthly attack counts measures a ransomware group's current activity level
- domain assumption The 409 filtered real victims are representative of ransomware targeting patterns
- ad hoc to paper Random forest generalization on the synthetic test set indicates real-world performance
Cite this review
Pith. "Pith review of Assessing and Prioritizing Ransomware Risk Based on Historical Victim Data." pith.science (2026). https://pith.science/paper/BOOGJAM7
@misc{pith2026250204421,
author = {Pith},
title = {Pith review of: Assessing and Prioritizing Ransomware Risk Based on Historical Victim Data},
year = {2026},
howpublished = {\url{https://pith.science/paper/BOOGJAM7}},
note = {Machine review of arXiv:2502.04421}
}
read the original abstract
We present an approach to identifying which ransomware adversaries are most likely to target specific entities, thereby assisting these entities in formulating better protection strategies. Ransomware poses a formidable cybersecurity threat characterized by profit-driven motives, a complex underlying economy supporting criminal syndicates, and the overt nature of its attacks. This type of malware has consistently ranked among the most prevalent, with a rapid escalation in activity observed. Recent estimates indicate that approximately two-thirds of organizations experienced ransomware attacks in 2023 \cite{Sophos2023Ransomware}. A central tactic in ransomware campaigns is publicizing attacks to coerce victims into paying ransoms. Our study utilizes public disclosures from ransomware victims to predict the likelihood of an entity being targeted by a specific ransomware variant. We employ a Large Language Model (LLM) architecture that uses a unique chain-of-thought, multi-shot prompt methodology to define adversary SKRAM (Skills, Knowledge, Resources, Authorities, and Motivation) profiles from ransomware bulletins, threat reports, and news items. This analysis is enriched with publicly available victim data and is further enhanced by a heuristic for generating synthetic data that reflects victim profiles. Our work culminates in the development of a machine learning model that assists organizations in prioritizing ransomware threats and formulating defenses based on the tactics, techniques, and procedures (TTP) of the most likely attackers.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
Sophos state of ransomware 2023,
Sophos, “Sophos state of ransomware 2023,” Sophos, Tech. Rep., 2023, Accessed: 2023-02-11. [Online]. Available: https://assets.sophos.com/ X24WTUEQ/at/c949g7693gsnjh9rb9gr8/sophos-state-of-ransomware- 2023-wp.pdf. 18 Spencer Massengale and Philip Huff
work page 2023
-
[2]
Ransomware: Recent advances, analysis, challenges and future research directions,
C. Beaman, A. Barkworth, T. D. Akande, S. Hakak, and M. K. Khan, “Ransomware: Recent advances, analysis, challenges and future research directions,” en, Comput. Secur., vol. 111, p. 102 490, Dec. 2021
work page 2021
-
[3]
Ransomware: Evolution, mitigation and prevention,
R. Richardson, “Ransomware: Evolution, mitigation and prevention,” In- ternational Management Review, vol. 13, 1 Jan. 2017
work page 2017
-
[4]
Sophos. “The state of ransomware 2022.” (2022), [Online]. Available: https: / / assets . sophos . com / X24WTUEQ / at / c5234fvn45pvmk5w6nhh4vkh / sophos-state-of-ransomware-2022-infographic.pdf
work page 2022
-
[5]
B. Sherborne, T. Trowbridge, and G. Permana. “Ransomware mitigation: An analytical investigation into the effects and trends of ransomware at- tacks on global business.” Accessed: Year-Month-Day. (Dec. 2022), [On- line]. Available: https://doi.org/10.31234/osf.io/ayc2d
-
[6]
Cyber threat intelligence sharing: Survey and research directions,
T. D. Wagner, K. Mahbub, E. Palomar, and A. E. Abdallah, “Cyber threat intelligence sharing: Survey and research directions,” Comput. Se- cur., vol. 87, p. 101 589, Nov. 2019
work page 2019
-
[7]
J. Ettinger, “The state of cyber intelligence practices in the united states: Cyber intelligence tradecraft report,” Carnegie Mellon University, Soft- ware Engineering Institute, Pittsburgh, PA, Tech. Rep. DM19-0447, 2019, Funded by the Department of Defense under Contract No. F A8702-15-D-
work page 2019
-
[8]
Approved for public release; distribution unlimited
Show all 36 references
-
[9]
Sans 2022 cyber threat intelligence survey,
R. Brown and P. Stirparo, “Sans 2022 cyber threat intelligence survey,” SANS Institute, Feb. 2022, ©2022 SANS™ Institute
2022
-
[10]
An observational assessment of CTI standards for blue teams: Student paper abstract,
J. McHugh, D. Childs, and J. Jenkins, “An observational assessment of CTI standards for blue teams: Student paper abstract,” J. Comput. Sci. Coll., vol. 37, no. 3, p. 168, Oct. 2021
2021
-
[11]
Siracusano, D
G. Siracusano, D. Sanvito, R. Gonzalez, et al., Time for action: Automated analysis of cyber threat intelligence in the wild, 2023. arXiv: 2307.10214 [cs.CR]. [Online]. Available: https://arxiv.org/abs/2307.10214
2023 arXiv
-
[12]
Ttpdrill: Au- tomatic and accurate extraction of threat actions from unstructured text of cti sources,
G. Husari, E. Al-Shaer, M. Ahmed, B. Chu, and X. Niu, “Ttpdrill: Au- tomatic and accurate extraction of threat actions from unstructured text of cti sources,” in Proceedings of the 33rd Annual Computer Security Ap- plications Conference, ser. ACSAC ’17, Orlando, FL, USA: Assoc...
2017
-
[13]
P. Gao, X. Liu, E. Choi, et al., Threatkg: A threat knowledge graph for au- tomated open-source cyber threat intelligence gathering and management,
-
[14]
“Att&ck.” Accessed: October 10, 2023
MITRE Corporation. “Att&ck.” Accessed: October 10, 2023. (2023), [On- line]. Available: https://attack.mitre.org/
2023
-
[15]
Looking beyond iocs: Automatically extracting attack patterns from external cti,
M. T. Alam, D. Bhusal, Y. Park, and N. Rastogi, “Looking beyond iocs: Automatically extracting attack patterns from external cti,” in Proceed- ings of the 26th International Symposium on Research in Attacks, In- trusions and Defenses, ser. RAID ’23, Hong Kong, China: Associati...
2023
-
[16]
Cyber threat attribution using unstruc- tured reports in cyber threat intelligence,
E. Irshad and A. Basit Siddiqui, “Cyber threat attribution using unstruc- tured reports in cyber threat intelligence,” Egyptian Informatics Journal, vol. 24, no. 1, pp. 43–59, Mar. 2023
2023
-
[17]
CTI view: APT threat intelligence analysis system,
Y. Zhou, Y. Tang, M. Yi, C. Xi, and H. Lu, “CTI view: APT threat intelligence analysis system,” en, Security and Communication Networks, vol. 2022, Jan. 2022
2022
-
[18]
Data backup system with integrated active protection against ransomware,
V. Oujezsky, P. Novak, T. Horvath, M. Holik, and M. Jurcik, “Data backup system with integrated active protection against ransomware,” in 2023 46th International Conference on Telecommunications and Signal Process- ing (TSP), IEEE, Jul. 2023, pp. 65–69
2023
-
[19]
Ap- plying staged event-driven access control to combat ransomware,
T. McIntosh, A. S. M. Kayes, Y.-P. P. Chen, A. Ng, and P. Watters, “Ap- plying staged event-driven access control to combat ransomware,”Comput. Secur., vol. 128, p. 103 160, May 2023
2023
-
[20]
A review on spreading and foren- sics analysis of Windows-Based ransomware,
N. M. Chayal, A. Saxena, and R. Khan, “A review on spreading and foren- sics analysis of Windows-Based ransomware,”Annals of Data Science, Jun. 2022
2022
-
[21]
Individual cyber security: Empowering employees to resist spear phishing to prevent identity theft and ransomware attacks,
J. Thomas, “Individual cyber security: Empowering employees to resist spear phishing to prevent identity theft and ransomware attacks,” en, Int. J. Indian Cult. Bus. Manage., vol. 13, no. 6, p. 1, Apr. 2018
2018
-
[22]
Understanding the impact of ransomware: A survey on its evolution, mitigation and prevention techniques,
N. Aldaraani and Z. Begum, “Understanding the impact of ransomware: A survey on its evolution, mitigation and prevention techniques,” in 2018 21st Saudi Computer Society National Computer Conference (NCC), 2018, pp. 1–5. doi: 10.1109/NCG.2018.8593029
2018
-
[23]
Byte frequency based indicators for Crypto-Ransomware detection from empirical analysis,
G. Y. Kim, J.-Y. Paik, Y. Kim, and E.-S. Cho, “Byte frequency based indicators for Crypto-Ransomware detection from empirical analysis,” J. Comput. Sci. Technol., vol. 37, no. 2, pp. 423–442, Apr. 2022
2022
-
[24]
Per- sonality types and ransomware victimisation,
Y. Yilmaz, O. Cetin, C. Grigore, B. Arief, and J. Hernandez-Castro, “Per- sonality types and ransomware victimisation,”Digital Threats, vol. 4, no. 4, pp. 1–25, Oct. 2023
2023
-
[25]
Risk prediction of malware victimization based on user behavior,
F. L. L´ evesque, J. M. Fernandez, and A. Somayaji, “Risk prediction of malware victimization based on user behavior,” in 2014 9th International Conference on Malicious and Unwanted Software: The Americas (MAL- WARE), IEEE, Oct. 2014, pp. 128–134
2014
-
[26]
Chain-of-thought prompting elic- its reasoning in large language models,
J. Wei, X. Wang, D. Schuurmans, et al., “Chain-of-thought prompting elic- its reasoning in large language models,” Advances in Neural Information Processing Systems, vol. 35, pp. 24 824–24 837, 2022
2022
-
[27]
Ransomware live
J. Mousquenton. “Ransomware live.” Licensed under CC BY-NC 4.0. (2023), [Online]. Available: https://ransomware.live/#/
2023
-
[28]
Kojima, S
T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa, Large language models are zero-shot reasoners, 2023. arXiv: 2205.11916 [cs.CL]
2023 arXiv
-
[29]
Structured threat information expression (stix ™ 2.1) - introduc- tion
OASIS. “Structured threat information expression (stix ™ 2.1) - introduc- tion.” [Accessed: October 12, 2023]. (2023), [Online]. Available: https : //oasis-open.github.io/cti-documentation/stix/intro.html. 20 Spencer Massengale and Philip Huff
2023
-
[30]
Measuring and visual- izing cyber threat intelligence quality,
D. Schlette, F. B¨ ohm, M. Caselli, and G. Pernul, “Measuring and visual- izing cyber threat intelligence quality,” Int. J. Inf. Secur., vol. 20, no. 1, pp. 21–38, Feb. 2021
2021
-
[31]
Nist national vulnerability database
“Nist national vulnerability database.” (), [Online]. Available: www.nvd. nist.gov
-
[32]
Announcing grok
X. “Announcing grok.” Accessed on 2023-12-14. (2023), [Online]. Avail- able: https://x.ai/
2023
-
[33]
Welcome to the gemini era
Google. “Welcome to the gemini era.” Accessed on 2023-12-14. (2023), [Online]. Available: https://deepmind.google/technologies/gemini/ #introduction
2023
-
[34]
Introducin llama 2
Meta. “Introducin llama 2.” Accessed on 2023-12-14. (2023), [Online]. Available: https://ai.meta.com/llama/
2023
-
[35]
[Online]
Anthrooic LLM, Anthropc, 2024-05-03, 2024. [Online]. Available: https: //www.anthropic.com/
2024
- [2022]
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.