REVIEW 7 cited by
SecQA: A Concise Question-Answering Dataset for Evaluating Large Language Models in Computer Security
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
In this paper, we introduce SecQA, a novel dataset tailored for evaluating the performance of Large Language Models (LLMs) in the domain of computer security. Utilizing multiple-choice questions generated by GPT-4 based on the "Computer Systems Security: Planning for Success" textbook, SecQA aims to assess LLMs' understanding and application of security principles. We detail the structure and intent of SecQA, which includes two versions of increasing complexity, to provide a concise evaluation across various difficulty levels. Additionally, we present an extensive evaluation of prominent LLMs, including GPT-3.5-Turbo, GPT-4, Llama-2, Vicuna, Mistral, and Zephyr models, using both 0-shot and 5-shot learning settings. Our results, encapsulated in the SecQA v1 and v2 datasets, highlight the varying capabilities and limitations of these models in the computer security context. This study not only offers insights into the current state of LLMs in understanding security-related content but also establishes SecQA as a benchmark for future advancements in this critical research area.
Forward citations
Cited by 7 Pith papers
-
SecBench: A Comprehensive Multi-Dimensional Benchmarking Dataset for LLMs in Cybersecurity
A large, bilingual, multi-domain cybersecurity benchmark with both multiple-choice and short-answer questions, plus LLM-based automatic grading and results for 16 models.
-
Domyn-Small: A European 10B Reasoning Language Model
Domyn-Small is a 10B reasoning LLM that claims to deliver roughly one-third the inference tokens of Qwen3.5-9B at competitive accuracy, though results are marked as preliminary.
-
MEQA: A Meta-Evaluation Framework for Question & Answer LLM Benchmarks
MEQA scores eight cybersecurity QA benchmarks against a 44-sub-criteria rubric, finding strengths in reproducibility and comparability and weaknesses in prompt robustness and reliability.
-
Chinese SafetyQA: A Safety Short-form Factuality Benchmark for Large Language Models
A 2,000-question Chinese safety factuality benchmark shows most LLMs are inaccurate on safety knowledge, with retrieval helping more than self-reflection.
-
Llama-3.1-FoundationAI-SecurityLLM-8B-Instruct Technical Report
Foundation-Sec-8B-Instruct, an instruction-tuned 8B cybersecurity LLM, is released and claimed to beat Llama 3.1-8B-Instruct on CTIBench-RCM and CTIBench-MCQA while remaining competitive on general instruction-following.
-
Forewarned is Forearmed: A Survey on Large Language Model-based Agents in Autonomous Cyberattacks
A review of LLM-based agents as autonomous cyberattackers, arguing that they lower attack costs, scale up threats, and outpace existing defenses.
-
Exploring the Role of Large Language Models in Cybersecurity: A Systematic Survey
A survey that organizes LLM-based cybersecurity defense by attack-phase, threat-intelligence, and deployment categories, and identifies post-intrusion defense as the main understudied area.
Discussion (0). Continue with ORCID to comment.