Pith. sign in

REVIEW 8 cited by

CyberMetric: A Benchmark Dataset based on Retrieval-Augmented Generation for Evaluating LLMs in Cybersecurity Knowledge

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2402.07688 v2 pith:YE4KOCWE submitted 2024-02-12 cs.AI cs.CR

Norbert Tihanyi , Mohamed Amine Ferrag , Ridhi Jain , Tamas Bisztray , Merouane Debbah This is my paper · ORCID
classification cs.AIcs.CR
keywords llmscybersecurityhumanquestionscybermetric-80expertsknowledgemodels
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Large Language Models (LLMs) are increasingly used across various domains, from software development to cyber threat intelligence. Understanding all the different fields of cybersecurity, which includes topics such as cryptography, reverse engineering, and risk assessment, poses a challenge even for human experts. To accurately test the general knowledge of LLMs in cybersecurity, the research community needs a diverse, accurate, and up-to-date dataset. To address this gap, we present CyberMetric-80, CyberMetric-500, CyberMetric-2000, and CyberMetric-10000, which are multiple-choice Q&A benchmark datasets comprising 80, 500, 2000, and 10,000 questions respectively. By utilizing GPT-3.5 and Retrieval-Augmented Generation (RAG), we collected documents, including NIST standards, research papers, publicly accessible books, RFCs, and other publications in the cybersecurity domain, to generate questions, each with four possible answers. The results underwent several rounds of error checking and refinement. Human experts invested over 200 hours validating the questions and solutions to ensure their accuracy and relevance, and to filter out any questions unrelated to cybersecurity. We have evaluated and compared 25 state-of-the-art LLM models on the CyberMetric datasets. In addition to our primary goal of evaluating LLMs, we involved 30 human participants to solve CyberMetric-80 in a closed-book scenario. The results can serve as a reference for comparing the general cybersecurity knowledge of humans and LLMs. The findings revealed that GPT-4o, GPT-4-turbo, Mixtral-8x7B-Instruct, Falcon-180B-Chat, and GEMINI-pro 1.0 were the best-performing LLMs. Additionally, the top LLMs were more accurate than humans on CyberMetric-80, although highly experienced human experts still outperformed small models such as Llama-3-8B, Phi-2 or Gemma-7b.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Agentic and Generative AI for Open-Source Intelligence and Cyber Investigations: Taxonomy, Evaluation, Challenges, and Future Directions

    cs.CR 2026-07 conditional novelty 6.0 of 10

    Across 74 OSINT/CTI AI studies, hallucination is widely named but end-to-end measured in only one non-reproducible system, so a human–AI co-pilot is the most defensible near-term architecture.

  2. Scale-free congestion clusters in large-scale traffic networks: a continuum modeling study

    physics.soc-ph 2026-04 unverdicted novelty 6.0 of 10

    The Aw–Rascle–Zhang continuum model on directed lattice networks yields power-law spatiotemporal congestion clusters with finite-size scaling by linear system size.

  3. Measuring and Augmenting Large Language Models for Solving Capture-the-Flag Challenges

    cs.AI 2025-06 reject novelty 6.0 of 10

    A benchmark and agent for CTF solving, but the agent's retrieval database appears to contain the answers to the test challenges, undermining the reported improvements.

  4. SV-TrustEval-C: Evaluating Structure and Semantic Reasoning in Large Language Models for Source Code Vulnerability Analysis

    cs.SE 2025-05 conditional novelty 6.0 of 10

    Large language models perform poorly on a new C-code vulnerability benchmark, indicating they rely on pattern matching rather than genuine reasoning.

  5. Open Security Benchmark: Towards Autonomous Enterprise Cyber Defense

    cs.CR 2026-07 conditional novelty 5.0 of 10

    OSB proposes frozen synthetic-enterprise snapshots with gold posture answers so AI agents can be benchmarked on security investigation via SQL or native vendor APIs.

  6. Domyn-Small: A European 10B Reasoning Language Model

    cs.CL 2026-05 conditional novelty 5.0 of 10

    Domyn-Small is a 10B reasoning LLM that claims to deliver roughly one-third the inference tokens of Qwen3.5-9B at competitive accuracy, though results are marked as preliminary.

  7. Toward Cybersecurity-Expert Small Language Models

    cs.CL 2025-10 conditional novelty 5.0 of 10

    A family of 4B–20B cybersecurity models fine-tuned on an enriched, expert-steered reasoning dataset matches or beats larger frontier models on core CTI benchmarks.

  8. A Call for Collaborative Intelligence: Why Human-Agent Systems Should Precede AI Autonomy

    cs.AI 2025-06 conditional novelty 4.0 of 10

    A position paper arguing that LLM-based human-agent systems, not fully autonomous agents, should be the immediate goal for AI development.

Pith tools