REVIEW 6 cited by
EnIGMA: Interactive Tools Substantially Assist LM Agents in Finding Security Vulnerabilities
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Although language model (LM) agents have demonstrated increased performance in multiple domains, including coding and web-browsing, their success in cybersecurity has been limited. We present EnIGMA, an LM agent for autonomously solving Capture The Flag (CTF) challenges. We introduce new tools and interfaces to improve the agent's ability to find and exploit security vulnerabilities, focusing on interactive terminal programs. These novel Interactive Agent Tools enable LM agents, for the first time, to run interactive utilities, such as a debugger and a server connection tool, which are essential for solving these challenges. Empirical analysis on 390 CTF challenges across four benchmarks demonstrate that these new tools and interfaces substantially improve our agent's performance, achieving state-of-the-art results on NYU CTF, Intercode-CTF, and CyBench. Finally, we analyze data leakage, developing new methods to quantify it and identifying a new phenomenon we term soliloquizing, where the model self-generates hallucinated observations without interacting with the environment. Our code and development dataset are available at https://github.com/SWE-agent/SWE-agent/tree/v0.7 and https://github.com/NYU-LLM-CTF/NYU_CTF_Bench/tree/main/development respectively.
Forward citations
Cited by 6 Pith papers
-
RECEIPT: Deterministic, Reward-Hacking-Resistant Verification for White-Box Agentic XSS Discovery
A replay-based verifier with environment isolation, role separation, and a browser-execution sentinel lets white-box LLM agents report XSS exploits that are real, reproducible, attacker-to-victim vulnerabilities.
-
Veritas: Grounding LLM Agents for Reliable Vulnerability Reasoning over Stripped Binaries
Veritas detects out-of-bounds vulnerabilities in stripped binaries at 90% recall by grounding LLM reasoning in static witness-backed flows and runtime validation.
-
FirmPilot: Evidence-Guided Multi-Agent Environment Recovery for IoT Firmware Rehosting
An evidence-guided multi-agent rehosting loop raises web-service reachability on the LFwC corpus from 25.49% to 52.39% and network reachability from 39.30% to 71.93%.
-
VEXAIoT: Autonomous IoT Vulnerability EXploitation using AI Agents
A multi-agent LLM framework autonomously discovers and exploits OWASP-mapped IoT vulnerabilities with 95% success across 260 trials in IoTGoat and Metasploitable2.
-
Running in CIRCLE? A Simple Benchmark for LLM Code Interpreter Security
CIRCLE is a 1,260-prompt benchmark that measures how often commercial LLM code interpreters refuse, execute, or time out on resource-exhaustion tasks, revealing large and inconsistent safety gaps.
-
From Promise to Peril: Rethinking Cybersecurity Red and Blue Teaming in the Age of LLMs
LLMs can assist both attackers and defenders in cybersecurity, but context limits, hallucinations, and weak reasoning make them unsafe to deploy without human oversight and real-world evaluation.
Discussion (0). Sign in to comment.