Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

VLAI: A RoBERTa-Based Model for Automated Vulnerability Severity Classification

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A RoBERTa model fine-tuned on over 600,000 vulnerability advisories predicts a vulnerability's severity category from its text description, reaching 82.8% test accuracy and matching eventual official severity ratings in about 85% of live…

desk verdict A useful open dataset and deployed triage tool wrapped around a standard RoBERTa fine-tune; the accuracy claims need dedup analysis and a real live-test protocol. read the letter →

arxiv 2507.03607 v1 pith:Q4ZU2FFB submitted 2025-07-04 cs.CR

Cédric Bonhomme , Alexandre Dulaunoy This is my paper · ORCID
classification cs.CR
keywords vulnerabilityseveritypredictionCVSSRoBERTatextclassificationsecuritytriageNLPforcybersecurityopendataset
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a transformer-based text classifier can produce reliable provisional severity estimates for software vulnerabilities before human analysts assign official scores. It fine-tunes a RoBERTa-base model on a dataset of more than 600,000 advisories drawn from several vulnerability feeds, labeling each description by the severity category derived from its CVSS score. On a held-out test set the model reaches 82.8% accuracy, and in a live evaluation on recently disclosed, initially unscored vulnerabilities, its predictions match the eventual official category about 85% of the time. The authors position the model as a triage assistant that gives immediate low-to-critical labels while experts take days to score, not as a replacement for expert judgment.

What carries the argument

The carrying object is a RoBERTa-base transformer with a four-output softmax classification head; the labels (low, medium, high, critical) are derived from CVSS base-score ranges. The mechanism is the pre-trained language model's ability to map the wording of an advisory, such as phrases like remote code execution or privilege escalation, onto an impact category. The surrounding pipeline collects advisories from multiple public and vendor feeds, builds a daily snapshot of descriptions with CVSS-derived labels, fine-tunes for five epochs on two GPUs, and serves the updated checkpoint through a local REST gateway integrated into the lookup service. Including the short title or affected-product identifiers did not add accuracy beyond the description text itself.

What would settle it

Take a random set of newly disclosed advisories before any official score exists, write down the model's predictions, and compare them to the scores assigned later by the relevant official scoring body; the claim weakens if agreement on this pre-registered sample is much lower than 85%, or if accuracy varies sharply by which feed supplied the label.

Watch

Extended reading notes

Core claim

The paper's central claim is that the text of a vulnerability advisory is sufficient, in most cases, to determine whether the issue will be rated low, medium, high, or critical. VLAI encodes the description with RoBERTa, appends a softmax classification head, and is trained with cross-entropy loss on severity labels derived from CVSS base scores. The authors report 82.8% accuracy on a 60k-example test split and roughly 85% agreement with official severity categories in a live experiment on vulnerabilities that had no score at disclosure. Errors that do occur are almost always one category away from the truth, and the model tends to over-predict slightly, which the authors consider acceptable for a security context. The model and dataset are released openly and are retrained daily as new advisories arrive.

Load-bearing premise

The results assume that severity labels built from CVSS scores are consistent across sources and CVSS versions, so that the mapping from score to category is stable; if vendors score the same vulnerability differently, or the category thresholds are arbitrary, the measured accuracy may reflect label construction rather than real text-reading skill.

Editorial extensions

If this is right

  • Defenders can receive a preliminary low-to-critical label for every new advisory in under a second, letting them prioritize patch work while official scoring is still pending.
  • Vulnerabilities from sources that rarely receive official scores still get a severity estimate, reducing the blind spot in third-party or vendor advisories.
  • Because the model is retrained daily on fresh data, its vocabulary and category boundaries can track new vulnerability descriptions as they appear.
  • When the model errs, it usually lands one category away and slightly over-rates severity, so a triage queue built on its labels is unlikely to hide a critical issue at the low end.
  • Open access to the model and dataset allows other teams to reproduce the 82.8% figure and to measure the live 85% agreement on their own feeds.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if the score-to-category thresholds differ between CVSS versions, the 82.8% accuracy could partly measure label consistency; a per-version accuracy breakdown would separate text-reading skill from label artifacts.
  • Editorial inference: the 85% live agreement is a strong but uncalibrated signal; publishing the full confusion matrix and per-class precision and recall for the live set would tell users whether critical predictions deserve immediate action.
  • Editorial inference: adversarial wording is a testable threat model, and replacing precise impact terms with vague synonyms should shift predictions measurably, so the model's robustness to description paraphrases could be quantified and hardened.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents VLAI, a fine-tuned RoBERTa-base model that classifies vulnerability descriptions into four severity categories (low, medium, high, critical). The training data is assembled daily from CVE/NVD, GitHub Security Advisories, PyPI advisories, and CSAF feeds into the CIRCL/vulnerability-scores dataset, currently containing 610k entries with a 550k/60k train/test split. The authors report 82.8% test accuracy and an approximately 85% agreement with later official CVSS categories in a live evaluation of initially unscored vulnerabilities. The model and dataset are open source and deployed in the Vulnerability-Lookup service via an ML gateway. The central claim is that this makes VLAI suitable as a provisional triage tool before official scores are available.

Significance. If the reported accuracies survive a stricter evaluation, VLAI would be a useful practical contribution: it addresses a real operational gap, provides an extensive open dataset, and ships reproducible code, model checkpoints, and a live service integration. The daily retraining cycle is an unusual and valuable artifact, and the use of future official CVSS scores as ground truth in the live test is a sensible direction. However, the paper's current evidence does not yet establish the central claim because the test-set construction and live-test protocol leave plausible leakage paths, and the label mapping and class-level evaluation are underspecified.

major comments (3)
  1. [§4–§5] The 82.8% test accuracy is not supported by a demonstration that the test split is disjoint from the training split at the vulnerability level. The dataset aggregates overlapping sources (CVE/NVD, GHSA, PyPI, CSAF) and the same vulnerability can appear under several identifiers, yet the paper describes no deduplication by canonical CVE ID or by description hash before the 550k/60k split. If near-duplicate rows appear in both splits, RoBERTa can memorise them and the reported accuracy overstates generalisation to unseen vulnerabilities. Please either document a deduplication step or re-run the split so that no vulnerability (or near-duplicate description) appears in both partitions.
  2. [§6] The live-test claim of 'about 85%' agreement is missing the evaluation protocol: no sample size, date range, inclusion criteria, or statement that the model checkpoint used for prediction predated the publication of the official CVSS scores used as ground truth. Because the model is retrained daily on a dataset that absorbs newly scored vulnerabilities, a retrospective comparison using the current checkpoint could have been trained on the very labels it is being tested against. Please specify the exact checkpoint version, the cut-off date of its training snapshot, and the criteria and size of the live sample.
  3. [§4–§5] The label construction is under-specified. Records can contain CVSS v2, v3.0, v3.1 and vendor-assigned scores, and CVSS v2 and v3 use different score ranges, so a single record does not unambiguously map to one of the four severity labels low/medium/high/critical; the paper never states which score is used or how conflicting versions are resolved. In addition, the single 82.8% accuracy figure is not accompanied by class-wise accuracy, macro-F1, or a majority-class baseline, so for a likely imbalanced four-class problem the result cannot be interpreted as strong predictive skill. Please report the deterministic label rule and a full class-level evaluation.
minor comments (5)
  1. [§3] The raw LaTeX fragment '/citeVulnerabilityLookup' appears in §3 and should be replaced with a proper citation.
  2. [§6] The claim that 'rarely did the model completely miss' is unsupported by any quantitative threshold or examples; this qualitative statement should either be made precise or removed.
  3. [§6.2 and Figure 2] The ML-Gateway figure is presented without a caption explaining the components and data flow, so it is not self-contained.
  4. [§5] The hyperparameter description says 'some tuning' but no search procedure or random seed is reported; adding the seed and the number of runs would support reproducibility.
  5. [References] Reference [2] contains a typo ('V alencia') and reference [13] is cited in the body without a page or version note; these should be cleaned up.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the classification accuracy is a standard held-out supervised evaluation and the live test uses later official scores as ground truth.

full rationale

VLAI's central claim is empirical: a RoBERTa model is fine-tuned on vulnerability descriptions labeled with CVSS-derived severity categories and evaluated on a test split of the same corpus (82.8%) plus a separate comparison against later official CVSS scores (85%). Neither evaluation reduces to the training input by construction: the test split is a held-out subset, and the live test uses future expert-assigned scores as ground truth. The severity classes are the target labels of the task, not a fitted parameter renamed as a prediction. The paper's many self-citations ([1],[2],[3],[4],[6],[7],[11],[12]) point to open-source code, datasets, model checkpoints, and a running service; these are verifiable artifacts and infrastructure references, not an unverified uniqueness theorem or ansatz that carries the argument. The reviewer/skeptic concerns about duplicate vulnerabilities across overlapping sources (CVE/NVD, GHSA, PyPI, CSAF) and the absence of a leakage-control protocol for the 85% live-test figure are legitimate evaluation-validity risks, but they are not circularity: no equation in the paper makes the predicted label equal to the training label by construction, and there is no quoted evidence that the live-test vulnerabilities or their eventual scores entered the training set before prediction. Those concerns belong in a correctness/robustness review, not in a circularity finding.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests entirely on the fitted model weights and on unstated assumptions about label consistency and representativeness. No new physical or conceptual entities are introduced.

free parameters (4)
  • Model weights (RoBERTa-base + classification head) = unknown (checkpoint revision 2100b2d)
    The central claim (82.8% accuracy) depends on the learned mapping from description text to severity, which is fitted to 550k training examples.
  • Severity category thresholds = not stated
    The paper does not specify how CVSS base scores are converted into the four severity classes. This mapping is a hand-chosen (or source-specific) parameter that directly determines labels and therefore accuracy.
  • Hyperparameters (learning rate 3e-5, batch size 16, 5 epochs, max seq len 512) = given
    Chosen via common defaults and some tuning; they affect the trained model and reported accuracy.
  • Train/test split ratio = 550k/60k (approximately 90/10)
    The split is taken from a daily snapshot; the exact assignment of entries to train/test is not specified, which can affect the reported accuracy.
assumptions (4)
  • domain assumption CVSS scores assigned by different sources (CVE/NVD, GHSA, PyPI, CSAF) are comparable and can be merged into a single labeled corpus.
    The dataset merges multiple advisory sources; the paper does not describe any normalization or reconciliation of CVSS versions or vendor-specific scoring practices. Section 4.
  • domain assumption The textual description of a vulnerability contains sufficient signal to predict its eventual severity category.
    The entire model rests on this; the paper provides preliminary evidence that title/CPE fields do not add much, but does not quantify the ceiling. Section 5.
  • domain assumption Future official CVSS scores used in the live test are reliable ground truth.
    The 85% agreement claim compares model predictions against later NVD/vendor scores, assuming those scores are correct and unbiased. Section 6.
  • domain assumption The held-out test split and the 'recent vulnerabilities' set are representative of the deployment population.
    No random sampling details are given for the live test; if the selected recent vulnerabilities are cherry-picked, the 85% figure is not generalizable. Section 6.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VLAI: A RoBERTa-Based Model for Automated Vulnerability Severity Classification." pith.science (2026). https://pith.science/paper/Q4ZU2FFB

@misc{pith2026250703607,
  author       = {Pith},
  title        = {Pith review of: VLAI: A RoBERTa-Based Model for Automated Vulnerability Severity Classification},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Q4ZU2FFB}},
  note         = {Machine review of arXiv:2507.03607}
}
read the original abstract

This paper presents VLAI, a transformer-based model that predicts software vulnerability severity levels directly from text descriptions. Built on RoBERTa, VLAI is fine-tuned on over 600,000 real-world vulnerabilities and achieves over 82% accuracy in predicting severity categories, enabling faster and more consistent triage ahead of manual CVSS scoring. The model and dataset are open-source and integrated into the Vulnerability-Lookup service.

Figures

Figures reproduced from arXiv: 2507.03607 by the authors.

Figure 1
Figure 1. Automated pipeline for dataset creation and model training in the Vulnerability-Lookup AI system. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. ML-Gateway. 6 [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Mapping CVEs to MITRE ATT&CK Techniques: A Curated Gold-Set Classifier and the Limits of LLM-Assisted Label Expansion

    cs.CR 2026-07 accept novelty 6.0 of 10

    A classifier trained on 1,207 expert-mapped CVEs roughly doubles recall@5 over a zero-shot baseline, while LLM-generated labels at ~0.39 expert agreement provide no reliable gain and degrade rare-technique coverage at...

Reference graph

Works this paper leans on

16 extracted references · 14 canonical work pages · cited by 1 Pith paper

  1. [1]

    vulnerability-lookup/VulnTrain

    C ´edric Bonhomme. vulnerability-lookup/VulnTrain. https://github.com/ vulnerability-lookup/VulnTrain, jul 2025

    work page 2025

  2. [2]

    Vulnerability-lookup - an open source tool to support cvd processes

    Bonhomme C ´edric. Vulnerability-lookup - an open source tool to support cvd processes. In V alencia 2024 UNDP/UNICC/FIRST Technical Colloquium, October 2024

    work page 2024

  3. [3]

    Vulnerability-lookup official website

    CIRCL. Vulnerability-lookup official website. https://vulnerability-lookup.org/,

    work page

  4. [4]

    Ai datasets and vlai model — discourse.ossbase.org

    CIRCL. Ai datasets and vlai model — discourse.ossbase.org. https://discourse.ossbase. org/t/ai-datasets-and-vlai-model/105 , 2025. [Accessed 04-07-2025]. 8 VLAI: A RoBERTa-Based Model for Automated Vulnerability Severity Classification

    work page 2025

  5. [5]

    Circl.lu - ai strategy

    CIRCL. Circl.lu - ai strategy. https://www.circl.lu/pub/ai-strategy/, 2025. [Ac- cessed 04-07-2025]

    work page 2025

  6. [6]

    Circl/vulnerability-severity-classification-roberta-base · hug- ging face — huggingface.co

    CIRCL. Circl/vulnerability-severity-classification-roberta-base · hug- ging face — huggingface.co. https://huggingface.co/CIRCL/ vulnerability-severity-classification-roberta-base , 2025. [Accessed 04-07-2025]

    work page 2025

  7. [7]

    Recent vulnerabilities - vulnerability-lookup - vulnerability.circl.lu - fkie source

    CIRCL. Recent vulnerabilities - vulnerability-lookup - vulnerability.circl.lu - fkie source. https: //vulnerability.circl.lu/recent#fkie_nvd, 2025. [Accessed 04-07-2025]

    work page 2025

  8. [8]

    Revisiting pre- trained models for Chinese natural language processing

    Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Shijin Wang, and Guoping Hu. Revisiting pre- trained models for Chinese natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings , pages 657–668, Online, November

    work page 2020

Show all 16 references
  1. [9]

    Cwe prediction using cve description - the semantic similarity approach

    Kethan Kota, Manjunatha A, and Sree Vivek S. Cwe prediction using cve description - the semantic similarity approach. volume 235, pages 1167–1178, 2024. International Conference on Machine Learning and Data Engineering (ICMLDE 2023)

    2024

  2. [10]

    Roberta: A robustly optimized BERT pretraining approach

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692, 2019

    1907 arXiv

  3. [11]

    vulnerability-scores (revision 6764823)

    Computer Incident Response Center Luxembourg. vulnerability-scores (revision 6764823). https: //huggingface.co/datasets/CIRCL/vulnerability-scores, 2025

    2025

  4. [12]

    vulnerability-severity-classification- roberta-base (revision 2100b2d)

    Computer Incident Response Center Luxembourg. vulnerability-severity-classification- roberta-base (revision 2100b2d). https://huggingface.co/CIRCL/ vulnerability-severity-classification-roberta-base , 2025

    2025

  5. [13]

    Cvss-bert: Explainable natural language processing to determine the severity of a computer security vulnerability from its description, 2021

    Mustafizur Shahid and Herv ´e Debar. Cvss-bert: Explainable natural language processing to determine the severity of a computer security vulnerability from its description, 2021

    2021

  6. [14]

    Misp: The design and implementation of a collaborative threat intelligence sharing platform

    Cynthia Wagner, Alexandre Dulaunoy, G ´erard Wagener, and Andras Iklody. Misp: The design and implementation of a collaborative threat intelligence sharing platform. In Proceedings of the 2016 ACM on Workshop on Information Sharing and Collaborative Security , pages 49–56. ACM...

    2016

  7. [2020]

    Association for Computational Linguistics

  8. [2024]

    [Accessed 04-07-2025]

    2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.