Towards Log Analysis with AI Agents: Cowrie Case Study

Cagatay Coban; Efe Emir Y\"uce; Enis Karaarslan; Esin G\"uler

REVIEW 3 major objections 5 minor 7 references

Towards Log Analysis with AI Agents: Cowrie Case Study

T0 review · 3 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A lightweight rule-based pipeline turns over 300,000 Cowrie honeypot logs into structured attacker-intent reports in minutes.

desk verdict A transparent, modest rule-based Cowrie log summarizer whose abstract overclaims AI agents; the pipeline part is fine, the headline claim is not yet built. read the letter →

arxiv 2509.05306 v1 pith:KGADYP5P submitted 2025-08-22 cs.CR cs.AIcs.MA

Enis Karaarslan , Esin Güler , Efe Emir Yüce , Cagatay Coban This is my paper · ORCID

classification cs.CRcs.AIcs.MA

keywords CowriehoneypotloganalysisAIagentsrule-basedclassificationattackintentlabelingthreatintelligenceSSHautomatedtriage

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper aims to show that a lightweight, fully automated pipeline can make sense of the flood of logs produced by a Cowrie honeypot without expensive security infrastructure. It builds a small Python system that reads raw JSON session logs, groups them into attacker sessions, and uses a hand-written keyword dictionary and scoring rules to label each session's intent and attacker skill. On a dataset of 313,412 log events, the system identified 26,368 command-bearing sessions and produced CSV, HTML, and chart summaries within minutes on a 4-core machine. The authors report that the overwhelming majority of sessions are shallow probes by low-skill automated actors, followed by malware-deployment attempts. The paper frames this as a first step toward fuller AI-agent-driven analysis, with the current engine deliberately transparent and rule-based.

What carries the argument

The central object is the analyze_session_with_rules function, a rule-based classifier that assigns each attacker session a score across predefined categories by matching commands against a keyword dictionary (e.g., wget, curl for malware deployment; ls, rm for reconnaissance). It then applies heuristic thresholds to output a primary intent and an estimated skill level. The surrounding pipeline does the data shaping: converting thousands of JSON events into coherent sessions grouped by Cowrie's session ID, then exporting the labeled results to CSV, HTML, and two summary charts.

What would settle it

Take a random sample of, say, 100 sessions that the pipeline labeled, have an independent security analyst label the same sessions by reading the raw logs, and measure agreement; if agreement on intent or skill is low, the headline distribution collapses as an artifact of the rule set. A cheaper check: feed the pipeline synthetic sessions with known ground-truth labels and see whether the heuristic thresholds recover them.

Watch

Extended reading notes

Core claim

In the paper's own terms, the discovery is that a simple, explainable rule-based engine can transform high-volume Cowrie honeypot logs into structured, human-readable threat intelligence. The implemented pipeline—reading cowrie JSON files, filtering command-input events, grouping them by session ID, and scoring each session against keyword categories—successfully handled over 300,000 events and assigned intent and skill labels to the large majority of 26,368 sessions. The dominant labels were 'Shallow Probe' and 'Malware Deployment', with skill almost always 'Low (Script Kiddie)' or 'Medium (Automated Script)', indicating that exposed SSH honeypots attract opportunistic, automated attacks ra

Load-bearing premise

The reported attack landscape rests entirely on the assumption that the curated keyword list and scoring thresholds in analyze_session_with_rules are a faithful stand-in for what attackers actually intend and how skilled they are.

Editorial extensions

If this is right

Honeypot operators without a SIEM can run the pipeline on a commodity machine and get a same-day read on what their honeypot is attracting.
The reported label distribution—shallow probes dominating, malware deployment second—implies that exposed SSH honeypots are mostly hit by automated, opportunistic attack tools.
Because the engine is rule-based and transparent, its classifications can be inspected and adjusted when new attacker command patterns appear.
The generated CSV and HTML reports let a researcher drill from aggregate charts to individual sessions and suspicious URLs, supporting deeper manual investigation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper leaves implicit that the same session-grouping and scoring skeleton could be ported to other honeypot log formats by swapping the keyword dictionary and event filters; that would be a direct test of whether the approach is generic or Cowrie-specific.
A reader should not infer that the reported attack landscape is ground truth: the labels inherit the biases of the hand-chosen keywords and thresholds, so agreement with manual analyst labels on a sample would be the natural validation step.
The 'AI agent' framing currently outruns the implementation—the deployed engine is deterministic scoring, not an autonomous planner; the true agentic behavior described in the model remains future work.
If the rule set is updated periodically, running the pipeline across time could reveal shifts in attacker automation, e.g., new malware-deployment command families appearing as older download commands are replaced.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a small engineering report about a Python/pandas pipeline that parses Cowrie JSON logs, groups events into sessions, scores sessions with a hand-written keyword dictionary, and emits CSV/HTML/PNG summaries. That part is real, runs on 313k events in minutes, and the GitHub repo is a plus. If you want a lightweight log summarizer for a small honeypot deployment, this is a reasonable starting point.

The paper's actual novelty is thin. The related work already lists HoneyAnalyzer and Kyoto 2006+, and the implemented components are standard pandas filtering, keyword scoring, and matplotlib charts. The "AI agents" framing is the problem: the abstract and Section IV describe an agent with NLP, planning, memory, and tool calling, but Section V and the repository implement none of that. Section VII explicitly defers LLM integration to future work. So the central advertised claim is unsupported by the artifact. That is an internal inconsistency, not an external dispute. If you strip the agent language, the remaining claim is honest: a fast, transparent, rule-based log summarizer.

The soft spots beyond the overclaim: the classification labels (Shallow Probe, Malware Deployment, etc.) come from keyword score thresholds with no ground truth or baseline, so the reported "attack landscape" in Section VI is partly definitional. Sessions with 1-2 commands are labeled Shallow Probe, and then the paper reports most sessions are Shallow Probe. That is circular. The skill-level labels are similarly heuristic. No baseline or comparison against a real analyst's labeling is provided. However, the parsing and grouping functionality is independent of these heuristics and appears sound. The paper is transparent about the rule-based nature in Section V, which counts in its favor.

For whom: students or small teams setting up a Cowrie honeypot who want a free, simple log summarizer without SIEM infrastructure. Not for researchers looking for validated detection or new threat-intelligence capability. The honest core deserves a serious referee if revised to remove the agent overclaim, add a baseline or manual validation sample, and reposition the contribution as a rule-based tool. As it stands, conditional acceptance at best; the abstract needs rewriting and the evaluation needs strengthening. I would not cite the agent claim, but I might cite the tool description if I needed a reproducible Cowrie parsing example.

Recommendation: send to peer review after major revision, mainly to enforce honesty in the claims and add a minimal validation step. The pipeline itself is coherent and useful enough to merit referee time once the framing is fixed.

Referee Report

3 major / 5 minor

Summary. The paper presents a Python/pandas pipeline for Cowrie honeypot log analysis. It ingests JSON logs, filters command-input events, groups them into sessions by session ID, scores each session with a hand-coded keyword dictionary, and labels each session by intent and attacker skill. On 313,412 events it produces 26,368 sessions and generates CSV/HTML reports and bar/pie charts. The authors claim this is an AI-agent-based system that intelligently parses and summarizes logs, and use it to conclude that the SSH threat landscape is dominated by low-skill automated Shallow Probe and Malware Deployment activity. Section VII defers actual LLM integration to future work.

Significance. Taken at face value as a lightweight log-analysis tool, the paper has useful properties: the pipeline is simple, transparent, fully reproducible through the public GitHub repository, and demonstrated on 313,412 events processed in minutes on commodity hardware. The grouping of raw events into sessions and the CSV/HTML/PNG outputs are practical contributions for small honeypot deployments. However, the advertised novelty is 'AI agents', and that is not present in the implementation; the actual engine is a rule-based classifier. Because the classifier labels are unvalidated and the main finding is partly definitional, the scientific contribution is currently a proof-of-concept tool description rather than a validated analysis of attacker behavior. The manuscript is a reasonable engineering report if reframed, but it does not yet support the abstract's claims.

major comments (3)

[Abstract, §IV, §V, §VII] The central claim is internally inconsistent with the artifact. The abstract says the approach 'leverages AI agents to intelligently parse, summarize, and extract insights,' and §IV describes an agentic architecture with NLP, planning, memory, and tool-calling. However, §V implements only a deterministic Python/pandas script with a hand-coded keyword dictionary in analyze_session_with_rules, and §VII lists LLM integration via LangChain as future work. No AI agent, LLM, planning module, memory, or tool-call loop is implemented. The title, abstract, and model description must be revised to describe a rule-based pipeline, or the agent must actually be implemented, before the claimed contribution is supported.
[§IV Analysis and §VI] The reported attack landscape is generated by unvalidated keyword-scoring heuristics. analyze_session_with_rules assigns labels such as 'Malware Deployment' and 'Reconnaissance' from a predefined dictionary (wget, curl, ls, rm) and threshold-based skill levels, but no ground truth, baseline, expert-labeled validation set, or manual audit is presented. Consequently, the distribution in Fig. 3 is a statement about the rule engine's output, not evidence about actual attacker intent. The paper should report precision/recall on a labeled subset (e.g., 100-200 sessions annotated by analysts) or explicitly label the findings as exploratory outputs of the rule engine.
[§IV/§VI Shallow Probe definition] The headline result that most sessions are 'Shallow Probe' is partly definitional. The heuristic classifies sessions with 1-2 commands as Shallow Probe (stated in §IV and §VI), so the observation that the majority fall into that category is by construction. This does not invalidate the pipeline, but the paper should not present the label distribution as an independent empirical discovery about the threat landscape without separating the construction of the category from the finding.

minor comments (5)

[§III Related Works] Typo: 'Song and firends' should be 'Song and friends'.
[Author affiliation] Typo: 'Deparment' should be 'Department'.
[§VI] The text refers to 'see Figure 3', but the manuscript excerpt does not include the actual chart image. Ensure the final submission embeds the bar chart and pie chart.
[§V/§VI Dataset description] The dataset provenance is underspecified: collection dates, honeypot location, network environment, and log file time spans are not given. Adding these details would improve reproducibility and interpretation.
[Abstract] The abstract promises 'while also considering the security implications of deploying such an autonomous system,' but the paper only mentions generic honeypot risks in §II. Either add a dedicated security-implications discussion or remove this claim.

Circularity Check

1 steps flagged · score 5.0 of 10

Attack-intent 'findings' are outputs of the same keyword/heuristic rules that define the categories; the AI-agent claim is deferred, not circular.

self definitional [Section IV (Analysis step) and Section VI (Results)]
"It uses a predefined dictionary of keywords (e.g., wget, ls, rm) to score each session across several categories (Reconnaissance, Malware Deployment, etc.). Based on these scores, it applies a set of heuristics to classify the session's primary intent and the attacker's estimated skill level. ... The primary finding ... is that the overwhelming majority of attacker sessions were classified with the intent of "Shallow Probe" representing bots or attackers who perform minimal reconnaissance (1-2 commands) before disconnecting."

The categories reported as empirical attack patterns are constructed by the analyzer itself: the keyword dictionary and heuristics assign each session an intent label, and Section VI then reports the distribution of those assigned labels as the discovered threat landscape. In particular, 'Shallow Probe' is characterized as minimal reconnaissance of 1-2 commands, which is exactly the kind of session-count heuristic the analysis step uses; so the headline result that most sessions are shallow probes is a restatement of the labeling rule, not an independent measurement of attacker behavior. No ground-truth labels, baseline, or external validation establish that the keyword scores correspond to real attacker intent, so the 'attack pattern' conclusion reduces to the rule set by construction.

full rationale

The paper contains no fitted-parameter prediction, no self-citation chain, and no imported uniqueness theorem; the core pipeline (JSON ingestion, pandas session grouping, rule-based classification, CSV/HTML/PNG reporting) is a self-contained implementation whose functionality stands independently of any claim about agents. The only circularity is in the interpretation of the classifier's output: Section IV defines attack-intent categories through a hand-coded keyword dictionary and heuristics, and Section VI presents the resulting label frequencies (e.g., dominance of 'Shallow Probe', 'Malware Deployment') as the paper's main threat-intelligence finding. That finding is definitionally tied to the rules that generate the labels. In addition, there is a serious internal inconsistency that is not circularity: the abstract claims the approach 'leverages AI agents', but Section V describes only a Python/pandas script with analyze_session_with_rules, and Section VII explicitly defers LLM integration and agent tool-use to future work. This unsupported agent claim lowers the paper's evidentiary quality but is a correctness/validity issue rather than a circular-derivation issue. Overall, the attack-pattern result is partially circular by construction, while the log-summarization tool itself is independent, giving a middle score.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim rests on hand-chosen classification rules and the assumption that keyword and session-size heuristics reflect real attacker behavior. No new physical entity is introduced. The paper's AI agent is a framing device, not an implemented component, and therefore is not listed as an invented entity.

free parameters (3)

Intent keyword dictionary and per-category scores
Section IV Analysis: analyze_session_with_rules scores sessions using a predefined dictionary of keywords (wget, curl, ls, rm) across categories. The keywords and weights are hand-chosen and directly determine the reported intent distribution.
Intent and skill heuristic thresholds
Section IV Analysis: heuristics classify primary intent and attacker skill level, with sessions of 1-2 commands treated as shallow probes. These thresholds are not derived from data and define the headline findings.
Command-input event filter
Section IV Processing: the pipeline filters to command-input events only, excluding other Cowrie event types. This choice shapes what counts as an attack session and therefore the reported distributions.

assumptions (3)

domain assumption Keyword presence is a valid proxy for attacker intent and skill
Section IV Analysis relies on keyword scoring to label sessions as Reconnaissance, Malware Deployment, and so on. No validation against ground truth is provided.
domain assumption Cowrie command-input events capture the meaningful attack behavior
Section IV Processing drops non-command-input events before grouping sessions, assuming those events are not needed for intent classification.
domain assumption Honeypots capture only malicious traffic with zero false positives
Section II states this as a key advantage of honeypots. The analysis treats all recorded sessions as attacks without independent confirmation.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Towards Log Analysis with AI Agents: Cowrie Case Study." pith.science (2026). https://pith.science/paper/KGADYP5P

@misc{pith2026250905306,
  author       = {Pith},
  title        = {Pith review of: Towards Log Analysis with AI Agents: Cowrie Case Study},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KGADYP5P}},
  note         = {Machine review of arXiv:2509.05306}
}

read the original abstract

The scarcity of real-world attack data significantly hinders progress in cybersecurity research and education. Although honeypots like Cowrie effectively collect live threat intelligence, they generate overwhelming volumes of unstructured and heterogeneous logs, rendering manual analysis impractical. As a first step in our project on secure and efficient AI automation, this study explores the use of AI agents for automated log analysis. We present a lightweight and automated approach to process Cowrie honeypot logs. Our approach leverages AI agents to intelligently parse, summarize, and extract insights from raw data, while also considering the security implications of deploying such an autonomous system. Preliminary results demonstrate the pipeline's effectiveness in reducing manual effort and identifying attack patterns, paving the way for more advanced autonomous cybersecurity analysis in future work.

Figures

Figures reproduced from arXiv: 2509.05306 by the authors.

**Figure 2.** Honeypot Architecture integrated memory module continuously logs all interaction sequences in a structured format, creating a rich, contextual dataset of the attack. Furthermore, the agent autonomously executes its core function: pattern extraction. With its toolcalling capabilities, it invokes external analytics modules to process the captured data, clustering similar behaviors, and determining the underlying tact… view at source ↗

**Figure 1.** Cowrie Honeypot Elements In the context of our project, agentic architecture is deployed with two primary objectives: autonomous system hardening and automated attack pattern (TTP) extraction. The agent utilizes advanced NLP capabilities to not only comprehend the attacker’s intent in real-time but to engage them in a dynamic, multi-step dialogue. The agent autonomously and strategically deploys decoy services and c… view at source ↗

**Figure 3.** Attack Intent Distribution honeypot data. The system effectively transforms highvolume, low-level log data into structured reports and easy-to-understand visualizations. The key contribution is a lightweight, transparent, and extensible process that can be used to derive meaningful threat intelligence. This can be easily setup at honeypot deployments without the need for complex infrastructure. For a future work, s… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

7 extracted references · 6 canonical work pages

[1]

(2011, April)

Song, J., Takakura, H., Okabe, Y., Eto, M., Inoue, D., & Nakao, K. (2011, April). Statistical analysis of honeypot data and building of Kyoto 2006+ dataset for NIDS evaluation. In Proceedings of the first workshop on building analysis datasets and gathering experience returns for security (pp. 29-36)

work page 2011
[2]

Thakar, U., Varma, S., & Ramani, A. K. (2005, September). Honeyanalyzer–analysis and extraction of intrusion detection patterns & signatures using honeypot. In Proceedings of the Second International Conference on Innovations in Information Technology (pp. 1-7)

work page 2005
[3]

(2015, September)

Moore, C., & Al-Nemrat, A. (2015, September). An analysis of honeypot programs and the attack data collected. In International Conference on Global Security, Safety, and Sustainability (pp. 228-238). Cham: Springer International Publishing

work page 2015
[4]

Krishnan, N. (2025). Ai agents: Evolution, architecture, and real-world applications. arXiv preprint arXiv:2503.12687

arXiv 2025
[5]

IBM. (2025). What are AI agents? IBM Think Topics. https://www.ibm.com/think/topics/ai-agents

work page 2025
[6]

(2023, October 15)

GitHub Resources. (2023, October 15). What are AI agents? GitHub. https://github.com/resources/articles/ai/what-are-ai-agents

work page 2023
[7]

Kopka and P

H. Kopka and P. W. Daly, A Guide to , 3rd ed. 1em plus 0.5em minus 0.4em Harlow, England: Addison-Wesley, 1999

work page 1999

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [1]

(2011, April)

Song, J., Takakura, H., Okabe, Y., Eto, M., Inoue, D., & Nakao, K. (2011, April). Statistical analysis of honeypot data and building of Kyoto 2006+ dataset for NIDS evaluation. In Proceedings of the first workshop on building analysis datasets and gathering experience returns for security (pp. 29-36)

work page 2011

[2] [2]

Thakar, U., Varma, S., & Ramani, A. K. (2005, September). Honeyanalyzer–analysis and extraction of intrusion detection patterns & signatures using honeypot. In Proceedings of the Second International Conference on Innovations in Information Technology (pp. 1-7)

work page 2005

[3] [3]

(2015, September)

Moore, C., & Al-Nemrat, A. (2015, September). An analysis of honeypot programs and the attack data collected. In International Conference on Global Security, Safety, and Sustainability (pp. 228-238). Cham: Springer International Publishing

work page 2015

[4] [4]

Krishnan, N. (2025). Ai agents: Evolution, architecture, and real-world applications. arXiv preprint arXiv:2503.12687

arXiv 2025

[5] [5]

IBM. (2025). What are AI agents? IBM Think Topics. https://www.ibm.com/think/topics/ai-agents

work page 2025

[6] [6]

(2023, October 15)

GitHub Resources. (2023, October 15). What are AI agents? GitHub. https://github.com/resources/articles/ai/what-are-ai-agents

work page 2023

[7] [7]

Kopka and P

H. Kopka and P. W. Daly, A Guide to , 3rd ed. 1em plus 0.5em minus 0.4em Harlow, England: Addison-Wesley, 1999

work page 1999