Evaluating Open-Weight LLMs for Generating Structured Threat Information for Autonomous Vehicle Vulnerabilities

Ahmed Ryan; Md Erfan; Md Kamal Hossain Chowdhury; Md Rayhanur Rahman

REVIEW 3 major objections 5 minor 21 references

Evaluating Open-Weight LLMs for Generating Structured Threat Information for Autonomous Vehicle Vulnerabilities

T0 review · 3 major / 5 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Open-weight LLMs, shown five similar ground-truth examples, turn automotive CVE descriptions into STIX threat objects with F1 near 0.94, CWE mappings near 0.99, and best MITRE ATT&CK technique matching at 0.68.

desk verdict Useful new CAV-to-STIX dataset, but the few-shot evaluation likely leaks ground truth, so the headline scores are not yet trustworthy. read the letter →

arxiv 2607.16175 v1 pith:5PUMBADV submitted 2026-07-17 cs.CR cs.AI

Md Erfan , Ahmed Ryan , Md Kamal Hossain Chowdhury , Md Rayhanur Rahman This is my paper · ORCID

classification cs.CRcs.AI

keywords CVE-to-STIXSTIX2.1open-weightLLMconnectedandautonomousvehiclesCWEmappingMITREATT&CKthreatintelligencegenerationfew-shotprompting

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that open-weight large language models, when given a few similar worked examples, can automate the conversion of plain-text automotive vulnerability reports (CVEs) into structured STIX threat intelligence. The authors built a ground-truth dataset, CAV-STIXGen, mapping 183 connected-and-autonomous-vehicle CVEs to STIX objects, relationships, CWE weakness codes, and MITRE ATT&CK techniques. Across 11 open-weight models, dynamic few-shot prompting lifted object-extraction F1 to 0.94 and CWE mapping to 0.99, while relationship extraction stayed at 0.63 and full ATT&CK matching remained harder. A sympathetic reader would care because this shows that entity- and weakness-level CVE-to-STIX automation is within practical reach for transportation security teams, with graph-level relationship reasoning as the remaining bottleneck.

What carries the argument

The load-bearing artifact is the CAV-STIXGen dataset: 183 manually annotated CVEs with ground-truth STIX 2.1 JSON bundles combining 1,383 domain objects, 1,395 relationship objects, 211 CWE mappings, and 294 MITRE ATT&CK mappings. The other key mechanism is dynamic few-shot prompting, which selects five similar ground-truth STIX examples based on textual description, CWE, and attack-pattern similarity; this example-guided prompting is what moves models from partial contextless output to near-ceiling entity and weakness extraction.

What would settle it

Have independent automotive-security analysts re-annotate a random sample of 30 CAV-STIXGen CVEs and measure their agreement on SDO, SRO, CWE, and ATT&CK labels; if human agreement is substantially below the model F1 scores, the reported numbers reflect annotation leniency rather than extraction quality.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central claim is that CVE-to-STIX generation for the automotive domain reduces to a prompt-engineering problem rather than a model-scaling problem: with five dynamically selected ground-truth examples, several open-weight models achieve near-0.94 F1 for STIX domain objects and near-0.99 F1 for CWE mapping, whereas relationship construction (source-object, target-object, and relationship-type together) and complete MITRE ATT&CK technique identification require richer reasoning and stay below 0.7. The multi-agent variant gives task-specific gains but does not beat the best single-model few-shot configuration. The dataset itself, with its co-occurrence statistics (

Load-bearing premise

The evaluation assumes the manually annotated STIX objects, relationships, CWE codes, and ATT&CK techniques in CAV-STIXGen are correct ground truth, yet inter-annotator agreement is reported only for the relevance-filtering step, not for these structured labels.

Editorial extensions

If this is right

Security analysts can feasibly bootstrap structured threat records from automotive CVE feeds using open-weight LLMs, reserving human review for relationship graphs and attack-technique assignments.
Prompt guidance and domain-specific examples matter more than model size for structured extraction, guiding deployment choices toward small, locally hostable models.
The CWE mapping result near 0.99 suggests weakness-type classification of automotive CVEs can be safely delegated to LLMs, which would speed up CWE-based triage and CWE Top 25 alignment.
The identified ATT&CK co-occurrence patterns (T1499 with T1203 or T1210) give defenders concrete technique combinations to monitor in connected-vehicle environments.
The gap between SDO and SRO performance defines a clear target for follow-up research: graph-consistency validation or relationship-focused fine-tuning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The reported F1 scores are agreements with the annotators' labels, so they are upper bounds on true generation quality; independent re-annotation of a sample would likely yield lower but still practically useful numbers.
CWE mapping near 0.99 may be inflated by the dataset's mostly single-CWE-per-CVE structure; multi-label weakness annotation would probably compress the gap between models.
A testable extension is to apply the same dynamic few-shot recipe to other structured formats (e.g., VEX, CAPEC, or OSV), which would show whether the gains generalize beyond STIX.
The relationship-extraction bottleneck suggests that the next measurable win will come not from larger models but from post-generation validation that checks whether source and target STIX types are valid for the chosen relationship.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take on arXiv:2607.16175. The paper gives the field something it didn't have: a CAV-focused benchmark mapping 183 CVEs to STIX SDO/SRO, CWE, and ATT&CK. That's genuinely useful, and the evaluation across 11 open-weight models with three prompting strategies is thorough. The finding that object extraction and CWE mapping are easy (F1 ~0.94-0.99) while relationships and ATT&CK are hard (F1 ~0.63, Match@1 ~0.68) is plausible and consistent with the tables.

The soft spots are real but fixable. First, inter-annotator agreement is reported only for the relevance filter, not for the structured labels. That matters because all scores are agreement with this ground truth. Second, no confidence intervals, so the comparisons across temperatures and models should be read as descriptive. Third, and most importantly, the dynamic few-shot prompt selection looks like it leaks ground truth. Section IV-B says the five examples are chosen by similarity over description, CWE, and attack-pattern. For a test CVE, those labels are not known in a real use case; if they come from the dataset, the model is being told the very things it is supposed to predict. The paper also never states that the test CVE itself is excluded from the candidate pool. If not, the model literally sees the gold STIX bundle in its own prompt. That would make the headline 0.94/0.63/0.99 numbers a retrieval result, not a generation result. This is not a deep theoretical flaw; it's a methodological ambiguity that the authors can clear up by describing how features are obtained and whether they filter out the target. But it has to be cleared before the central claim is taken at face value.

I don't think this is a desk reject. The dataset and the general evaluation design have value, and the authors are not hiding anything—the description is just underspecified. A serious referee should ask for the leakage fix, annotation reliability statistics, and a non-LLM baseline. After that, the paper would be a solid contribution to the automotive-security/LLM subfield. I'd bring it to a reading group as a case study in few-shot eval pitfalls.

My call: send to peer review, conditionally. The current numbers are not trustworthy until the few-shot selection is clarified, but the underlying resource and question are worth engaging.

Referee Report

3 major / 5 minor

Summary. The paper introduces CAV-STIXGen, a manually constructed dataset of 183 CAV-related CVE records mapped to STIX 2.1 SDOs, SROs, CWE categories, and MITRE ATT&CK techniques. It evaluates 11 open-weight LLMs under three prompting strategies (contextless, STIX-guided, dynamic few-shot) and multiple temperatures, reporting micro-F1 for SDO/SRO/CWE extraction and Match@1/Match@All for ATT&CK mapping. The headline claims are SDO F1=0.94, SRO F1=0.63, CWE F1=0.99, and ATT&CK Match@1=0.68 in single-model settings, with weaker multi-agent results. The paper also analyzes CWE and ATT&CK co-occurrences to identify recurring CAV threat patterns.

Significance. If the evaluation protocol is sound, CAV-STIXGen is a useful domain-specific benchmark and the demonstration that open-weight LLMs can extract SDOs and CWE mappings with high F1 under few-shot prompting is practically relevant for automating CVE-to-STIX pipelines. The dataset, prompts, and replication scripts are promised for release, which is a strength and makes the claims independently checkable. The contribution is incremental but solid: the main value is the CAV-focused ground-truth set, not a new method. However, the headline few-shot results currently rest on an under-specified retrieval step and on structured labels whose reliability is not reported, so the evaluation must be clarified or corrected before the central claim can be accepted.

major comments (3)

[Section IV-B, Table II] The dynamic few-shot evaluation is not fully specified and may leak test-set information. The paper states that 'Dynamic few-shot prompting extends the guided setting with five similar ground-truth STIX examples using similarity score among description, CWE, and attack-pattern.' Computing similarity using CWE and attack-pattern requires those labels for the test CVE; if they come from CAV-STIXGen ground truth, the retrieval uses the very labels the model is asked to predict. In addition, the candidate pool appears to be the same 183-CVE dataset, and the paper does not state that the target CVE is excluded from the five examples. Under either mechanism, the DFS results in Table II could reflect retrieval from a labeled database rather than open-world generation. The authors must specify the retrieval feature set, state explicitly whether the target CVE is filtered out, and ideally rerun w
[Section IV-A, Steps 4-5; Table IV] The validity of CAV-STIXGen as ground truth is load-bearing, but inter-annotator agreement is reported only for relevance filtering (Step 3, Cohen's κ=94%). For the structured labels — SDOs, SROs, CWE mappings, and ATT&CK mappings — the paper describes that 'the first and second authors' annotate and review, but gives no agreement measure, no adjudication protocol, and no audit. Step 5's mapping to MITRE ATT&CK is especially under-specified: the paper says attack-pattern objects are mapped to techniques but does not say whether the mapping is derived from NVD, CWE, manual reasoning, or external knowledge bases. If the structured labels contain inconsistencies or reflect one annotator's preferences, all F1/Match scores in Tables II and III measure agreement with the annotators rather than intrinsic generation quality. Add per-label-type reliability statistics and a detailed annotation pro
[Section IV-B (evaluation matching)] The paper says each generated STIX output is compared with the manually validated ground-truth STIX file, but it never defines the matching algorithm that decides whether a predicted SDO or SRO matches a ground-truth object. Annotator-assigned identifiers (Ei and Ri) cannot be used by models, so matching must be based on object type and properties, and a relationship can match only if its source, target, and type all match. Exact-string matching, semantic matching, and graph alignment would produce very different F1 values, especially for SROs. Without the matching rule or code, the reported numbers are not reproducible and the 0.63 SRO F1 claim is difficult to interpret. Please provide the matching procedure or pseudocode.

minor comments (5)

[Table II] Several table cells are visually malformed, e.g., '0.000.60 0.84 0.91' and '0.25 0.690.88 0.94'. Please add separators or use multi-row formatting so each temperature row is readable.
[Tables V and VI] The tables report one temperature per model-prompt combination with no selection criterion. State whether the chosen temperature is the best, the median, or a representative run, and note that 'best-of-k' selection would inflate scores.
[Section V-B and Table III] The multi-agent configuration is described only as separating SDO, CWE, ATT&CK, SRO, JSON, and validation roles, but no prompt templates, agent communication protocol, or convergence criterion are provided. Include these details in the appendix or supplement.
[Throughout] The text uses inconsistent spacing in 'CA V' (e.g., abstract and body). Also, the note about 'Green box represents the largest value in a column' does not appear in the rendered table; either add the highlight or remove the note.
[Section IV-A, Step 1] The keyword list was generated using ChatGPT, Copilot, Gemini, and Perplexity. The filtering process is described only as 'review, merge, and filter'; reporting the exact number of keywords generated vs. retained would improve reproducibility.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

No new theoretical entities are proposed. The CAV-STIXGen dataset is a constructed artifact rather than a postulated entity, so it is recorded under free parameters/axioms rather than as an invented entity. The main hand-chosen numbers are data-mining thresholds and the few-shot example count, which influence the RQ3 analysis and LLM evaluation but not a theoretical derivation.

free parameters (3)

FP-Growth minimum support threshold = 0.02
Set in Section IV-C for co-occurrence mining; controls which ATT&CK pairs are reported.
Association rule minimum confidence = 0.30
Set in Section IV-C for association rules; determines reported T1548-T1068 associations.
Dynamic few-shot example count k = 5
Set in Section IV-B; directly affects performance gains attributed to DFS prompting.

assumptions (5)

domain assumption STIX 2.1 is the correct/adequate representation for CAV vulnerability information.
Used throughout dataset construction and evaluation; if STIX is not a good fit, the benchmark measures the wrong target.
domain assumption NVD CVE descriptions are accurate and complete enough to annotate affected assets, weaknesses, and attack behaviors.
CVEs from NVD are the source text; all downstream labels derive from them.
domain assumption Manual annotation by the first two authors, with disagreements resolved by review, yields valid ground truth.
Section IV-A Steps 4-5; no independent gold standard or inter-annotator kappa for these labels.
domain assumption Generated STIX objects can be unambiguously matched to ground-truth objects for micro-P/R/F.
Section IV-B says 'compare each generated output with the manually validated ground-truth STIX file' but does not detail the matching rule; ambiguous matches affect all F1 values.
domain assumption MITRE ATT&CK mappings can be derived from attack-pattern SDOs.
Section IV-A Step 5 maps each attack-pattern object to ATT&CK techniques; no explicit disambiguation criteria given.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Evaluating Open-Weight LLMs for Generating Structured Threat Information for Autonomous Vehicle Vulnerabilities." pith.science (2026). https://pith.science/paper/5PUMBADV

@misc{pith2026260716175,
  author       = {Pith},
  title        = {Pith review of: Evaluating Open-Weight LLMs for Generating Structured Threat Information for Autonomous Vehicle Vulnerabilities},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5PUMBADV}},
  note         = {Machine review of arXiv:2607.16175}
}

read the original abstract

Connected and Autonomous Vehicles (CAVs) rely on interconnected software and hardware components, including sensors, Electronic Control Units, in-vehicle infotainment systems, and telematics units, where vulnerabilities can compromise assets, users, and vehicle operations. These vulnerabilities are commonly documented as plain text in the Common Vulnerabilities and Exposures (CVE) database; however, security practitioners require structured information about affected assets, types of weaknesses, and attack behaviors to effectively mitigate the risks from these vulnerabilities. To this end, we evaluate open-weight Large Language Models (LLMs) for generating Structured Threat Information Expression (STIX), a well-known structured format for representing threat information, for CAV-related CVEs. We construct a dataset called CAV-STIXGen that maps CAV vulnerability descriptions to STIX domain objects (SDO), STIX relationship objects (SRO), Common Weakness Enumeration (CWE), and MITRE ATT&CK techniques mappings. Using this dataset, we evaluated 11 open-weight LLMs (4B to 120B parameters) across various prompting strategies and temperatures. Single-model configurations achieve F1 scores of 0.94 for SDO, 0.63 for SRO, and 0.99 for CWE mapping, while complete MITRE ATT&CK mapping remains challenging. In a multi-agent setup, Gemma-4-31B and Codestral-22B achieve F1 scores of 0.91 for SDOs and 0.43 for SROs, respectively. Lastly, we analyze CWE and MITRE ATT&CK co-occurrences to identify recurring threat patterns in the CAV domain, demonstrating how AI-assisted vulnerability-to-STIX translation can automate threat intelligence and prioritize defense in transportation security.

Figures

Figures reproduced from arXiv: 2607.16175 by the authors.

**Figure 2.** Prompting strategies for CVE-to-STIX generation. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 2 linked inside Pith

[1]

Md Tanvirul Alam, Dipkamal Bhusal, Le Nguyen, and Nidhi Rastogi. 2024. Ctibench: A benchmark for evaluat- ing llms in cyber threat intelligence.Advances in Neural Information Processing Systems37 (2024)

2024
[2]

Jacob Cohen. 1960. A coefficient of agreement for nomi- nal scales.Educational and psychological measurement 20, 1 (1960), 37–46

1960
[3]

Sam Curry. 2024. Hacking Kia: Remotely Controlling Cars With Just a License Plate. https://samcurry.net/ hacking-kia. Accessed: 2026-05-26

2024
[4]

Sam Curry and Shubham Shah. 2025. Hacking Subaru: Tracking and Controlling Cars via the STARLINK Admin Panel. https://samcurry.net/hacking-subaru. Accessed: 2026-05-26

2025
[5]

Rikhiya Ghosh, Hans-Martin von Stockhausen, Martin Schmitt, George Marica Vasile, Sanjeev Kumar Karn, and Oladimeji Farri. 2025. Cve-llm: Ontology-assisted automatic vulnerability evaluation using large language models. InProceedings of the AAAI Conference on Artificial Intelligence, V ol. 39. 28757–28765

2025
[6]

Md Wasiul Haque, Md Erfan, Sagar Dasgupta, Md Ray- hanur Rahman, and Mizanur Rahman. 2025. Security Vulnerabilities in Software Supply Chain for Autonomous Vehicles.arXiv:2509.16899(2025)

arXiv 2025
[7]

Ghaith Husari, Ehab Al-Shaer, Mohiuddin Ahmed, Bill Chu, and Xi Niu. 2017. Ttpdrill: Automatic and accurate extraction of threat actions from unstructured text of cti sources. InProceedings of the 33rd annual computer security applications conference. 103–115

2017
[8]

Anooja Joy, Madhav Chandane, Yash Nagare, and Faruk Kazi. 2025. Threat Intelligence Extraction Framework (TIEF) for TTP Extraction.Journal of Cybersecurity and Privacy5, 3 (2025), 63

2025

Show all 21 references

[9]

Francesco Marchiori, Mauro Conti, and Nino Vincenzo Verde. 2023. Stixnet: A novel and modular solution for extracting all stix objects in cti reports. InProceedings of the 18th international conference on availability, reliability and security. 1–11

2023
[10]

MITRE. 2025. 2025 CWE Top 25 Most Dangerous Software Weaknesses. https://cwe.mitre.org/top25/archive/ 2025/2025 cwe top25.html. Accessed: 2026-05-28

2025
[11]

MITRE. 2025. MITRE ATT&CK. https://attack.mitre.org/. Accessed: 2026-05-25

2025
[12]

OASIS Open. 2021. STIX Version 2.1. https://www. oasis-open.org/standard/6426/. Approved: 25 January

2021
[13]

Angelos Papoutsis, Athanasios Dimitriadis, Dimitrios Kavallieros, Theodora Tsikrika, Stefanos Vrochidis, Ioan- nis Kompatsiaris, and Georgios Meditskos. 2025. Cti-gen: A framework for generating stix 2.1 compliant cti using generative ai. In2025 IEEE International Conference o...

2025
[14]

Md Rayhanur Rahman, Rezvan Mahdavi Hezaveh, and Laurie Williams. 2023. What are the attackers doing now? automating cyberthreat intelligence extraction from text on pace with the changing threat landscape: A survey. Comput. Surveys55, 12 (2023), 1–36

2023
[15]

Sayuj Shah and Vijay K Madisetti. 2025. MAD-CTI: Cyber threat intelligence analysis of the dark web using a multi-agent framework.IEEE Access(2025)

2025
[16]

Stefano Simonetto, Thijs Sebastiaan van Ede, Peter Bosch, Willem Jonker, and Ronan Oostveen. 2024. Text2Weak: mapping CVEs to CWEs using description embeddings analysis. In4th Workshop on Artificial Intelligence- Enabled Cybersecurity Analytics

2024
[17]

Giuseppe Siracusano, Davide Sanvito, Roberto Gonzalez, Manikantan Srinivasan, Sivakaman Kamatchi, Wataru Takahashi, Masaru Kawakita, Takahiro Kakumaru, and Roberto Bifulco. 2023. Time for action: Automated analysis of cyber threat intelligence in the wild.arXiv preprint arXiv:...

2023 arXiv
[18]

STIX Project. 2021. STIX. https://stixproject.github.io/. Accessed: 2026-05-15

2021
[19]

Mahzabin Tamanna, Shaswata Mitra, Md Erfan, Ahmed Ryan, Sudip Mittal, Laurie Williams, and Md Rayhanur Rahman. 2026. What Are Adversaries Doing? Automat- ing Tactics, Techniques, and Procedures Extraction: A Systematic Review.arXiv:2604.02377(2026)

2026 arXiv
[20]

Ming Xu, Hongtai Wang, Jiahao Liu, Xinfeng Li, Zheng- min Yu, Weili Han, Hoon Wei Lim, Jin Song Dong, and Jiaheng Zhang. 2024. ThreatPilot: Attack-Driven Threat Intelligence Extraction.arXiv:2412.10872(2024)

2024
[2021]

Accessed: 2026-05-28

2026

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.

[1] [1]

Md Tanvirul Alam, Dipkamal Bhusal, Le Nguyen, and Nidhi Rastogi. 2024. Ctibench: A benchmark for evaluat- ing llms in cyber threat intelligence.Advances in Neural Information Processing Systems37 (2024)

2024

[2] [2]

Jacob Cohen. 1960. A coefficient of agreement for nomi- nal scales.Educational and psychological measurement 20, 1 (1960), 37–46

1960

[3] [3]

Sam Curry. 2024. Hacking Kia: Remotely Controlling Cars With Just a License Plate. https://samcurry.net/ hacking-kia. Accessed: 2026-05-26

2024

[4] [4]

Sam Curry and Shubham Shah. 2025. Hacking Subaru: Tracking and Controlling Cars via the STARLINK Admin Panel. https://samcurry.net/hacking-subaru. Accessed: 2026-05-26

2025

[5] [5]

Rikhiya Ghosh, Hans-Martin von Stockhausen, Martin Schmitt, George Marica Vasile, Sanjeev Kumar Karn, and Oladimeji Farri. 2025. Cve-llm: Ontology-assisted automatic vulnerability evaluation using large language models. InProceedings of the AAAI Conference on Artificial Intelligence, V ol. 39. 28757–28765

2025

[6] [6]

Md Wasiul Haque, Md Erfan, Sagar Dasgupta, Md Ray- hanur Rahman, and Mizanur Rahman. 2025. Security Vulnerabilities in Software Supply Chain for Autonomous Vehicles.arXiv:2509.16899(2025)

arXiv 2025

[7] [7]

Ghaith Husari, Ehab Al-Shaer, Mohiuddin Ahmed, Bill Chu, and Xi Niu. 2017. Ttpdrill: Automatic and accurate extraction of threat actions from unstructured text of cti sources. InProceedings of the 33rd annual computer security applications conference. 103–115

2017

[8] [8]

Anooja Joy, Madhav Chandane, Yash Nagare, and Faruk Kazi. 2025. Threat Intelligence Extraction Framework (TIEF) for TTP Extraction.Journal of Cybersecurity and Privacy5, 3 (2025), 63

2025