{"id":"34b064cd-0427-4db8-aace-9b2ea18856a7","arxiv_id":"2507.09762","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":8,"one_line_summary":"EventHunter automatically clusters fragmented hacker-forum posts into security events and ranks them by timeliness, relevance, credibility, and completeness.","lead":"This paper presents EventHunter, a system that groups and ranks cybersecurity discussions from hacker forums into distinct threat events. It combines contrastive text embeddings, clustering, and a four-factor priority score to help analysts focus on the most urgent postings.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No explicit holdout between contrastive fine-tuning and the 21-event clustering test set; if absent, NMI 0.712/ARI 0.347 reflect training on test events rather than detection of new events.","rationale":"The reader's weakest_assumption identifies the most load-bearing gap. The entire contribution rests on showing that fine-tuned embeddings generalize to unseen event discussions. Section III-D trains on ground-truth event pairs, Section IV-D evaluates on 21 events drawn from the same 70-event pool, and no holdout statement appears. The authors may have performed a proper split but omitted it; the concern is not an accusation, just an unstated necessary condition. I agree with the reader that this is the key uncertainty. My independent reading did not surface a different flaw that would be more fundamental: the ranking module is clearly under-validated, but ranking only matters after clustering works; the entity-aware mechanism's mixed results are honestly reported and the paper does not overstate that contribution. The classification and NER evaluations provide useful supporting evidence and the released code is a genuine asset, but neither rescues the central clustering claim if the split is missing. The proposed test—re-running fine-tuning with the 21 test events explicitly withheld—would settle the question empirically. Since the reader's verdict is CONDITIONAL pending this clarification, I recommend keeping the verdict unchanged rather than escalating, because the flaw is an omitted assumption that the authors can likely repair or confirm.","tokens_in":19696,"tokens_out":3354,"duration_ms":38877,"concrete_test":"Download the public EventHunter repository and inspect the fine-tuning data-loading code for any split between the 70-event ground truth and the 21-event evaluation subset. Then rerun the DarkBERT pairwise-loss configuration with an explicit leave-out: remove all posts belonging to the 21 test events from the contrastive training set, retrain, and cluster the held-out posts. Compare NMI/ARI on the 21 events to the published 0.712/0.347. If the retrained values are substantially lower (e.g., NMI < 0.5), the published numbers are contaminated by training on test clusters; if they remain similar, the holdout concern is resolved. Report the exact event IDs used in each split.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section III-D defines the contrastive objective using positive pairs drawn from the same ground-truth event and negative pairs from different events. Section IV-D then evaluates clustering on 21 events \"randomly selected\" from the same 70-event ground-truth set described in Section IV-A. The paper never states that these 21 events were excluded from the triplet/pair construction during fine-tuning. If they were not excluded, the embedding model has already seen same-event positive pairs for the exact test clusters, so the reported DarkBERT NMI 0.712 / ARI 0.347 primarily measure memorization of training clusters, not the ability to group previously unseen event discussions. That would directly invalidate the central claim that EventHunter detects and clusters emerging security events without prior knowledge. A secondary but related issue is that the ground-truth clusters themselves are built using \"unique and unambiguous markers\" (CVEs, malware names, entity names); this makes the clustering task easier, but the unstated split is the more consequential problem. The ranking case studies are qualitative and the weights in Eq. 9 are self-defined, but those would only matter if clustering generalization is established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"EventHunter is presented as an unsupervised pipeline that processes CrimeBB hacker-forum posts through five stages: keyword-based pre-filtering and multi-class post classification (DarkBERT, average F1 0.739 over 14,386 manually annotated posts), zero-shot LLM-based entity extraction (mistral-nemo, mention-level F1 52.13 on the CyNER benchmark), contrastively fine-tuned Transformer post embeddings (triplet and pairwise SimCSE-style losses, with an optional entity-aware input mechanism), HDBSCAN clustering, and a weighted daily ranking over timeliness, relevance, credibility, and completeness. The clustering evaluation uses 21 events randomly selected from a curated 70-event ground truth and reports DarkBERT (Pairwise) with NMI 0.712 and ARI 0.347 and CySecBERT (Pairwise) with ARI 0.377, against TF-IDF and Word2Vec baselines of NMI 0.638 and 0.222. Ranking is demonstrated with three qualitative case studies and a top-3 comparison against recency and activity heuristics, and the authors conclude that EventHunter reduces forum noise and surfaces high-priority threats. The paper explicitly acknowledges the static (snapshot) nature of the clustering evaluation, sensitivity of the Priority Score to its weights, and the absence of external ground truth for ranking as limitations.","tokens_in":19974,"tokens_out":16634,"duration_ms":171864,"significance":"If the clustering numbers survive a correctly held-out evaluation, this is a valuable end-to-end contribution: it is one of the few large-scale studies of forum event detection with an explicit analyst-facing prioritization stage, and it ships public code, evaluates classification on 14,386 labeled posts, compares five Transformer architectures and three contrastive objectives, and benchmarks zero-shot NER on the standard CyNER corpus. The decisive weakness is that the reported NMI/ARI values may measure memorization of events seen during contrastive fine-tuning rather than generalization to new events; because the system's stated purpose is early detection of emerging events, the value of the contribution depends on resolving this. The ranking contribution is plausible but is currently supported only by qualitative evidence, and the paper's own limitation list stops short of acknowledging the train/test leakage risk.","major_comments":[{"comment":"On reading the paper, the holdout concern is confirmed: §III-D, §IV-A, and §IV-D contain no statement that the 21 events used as the clustering test set in §IV-D were excluded from the contrastive fine-tuning of §III-D, in which positive pairs are drawn from posts of the same ground-truth event and negative pairs from different ground-truth events. Because §IV-D selects the test events from the same 70-event ground-truth set described in §IV-A, the reported NMI 0.712 and ARI 0.347 (Table IV) may reflect the embedding model having been trained on pairs drawn from the exact events on which it is later evaluated. This is load-bearing: if the 21 events were not held out, the numbers measure memorization of the ground-truth clusters, and the central claim that EventHunter detects new, previously unseen events is unsupported. The authors must state the split explicitly and, unless the 21 events were genuinely excluded from pair construction, re-run the evaluation on events held out from fine-tuning and report those results. The same absence of a stated train/test split also affects the classification evaluation in §IV-B/Table III for the 14,386 annotated posts. The ground-truth curation in §IV-A (clusters defined by unique shared markers such as CVEs, malware names, and target names) makes the holdout essential rather than cosmetic, since the contrastive objective is trained to reproduce exactly the entity-sharing structure that defines the target clusters.","section":"§III-D, §IV-A, §IV-D"},{"comment":"All clustering results are single-run values with no variance estimate, significance test, or stated random seed, and the paper does not specify how HDBSCAN noise points (posts labeled as noise) are treated when computing ARI and NMI with scikit-learn; options such as dropping noise points, labeling them as one additional cluster, or giving each noise point its own label materially change both metrics. The 'Clusters Found' column in Tables IV and V (e.g., 14 clusters found by DarkBERT against 21 true events, 9 by BERT) shows substantial under-segmentation, which the paper does not interpret. Reporting mean and standard deviation over multiple random subsets of the 70 events, a seed, and explicit noise handling is necessary before the quantitative ranking of configurations (e.g., 0.712 vs. 0.714) can be taken at face value.","section":"§IV-D, Tables IV–VII"},{"comment":"Table IV reports RoBERTa (Triplet+Pairwise) with ARI 0.402 and NMI 0.714, while the Overall Performance Summary in the same section states that DarkBERT (Pairwise Loss) achieved the highest NMI (0.712); both statements cannot be true as written. No row in Tables V or VI corresponds to the Table IV RoBERTa configuration, so the reader cannot reconcile the conflict or determine which configuration is actually best. The manuscript's headline result is therefore ambiguous, and the tables, findings, and summary must be made mutually consistent.","section":"§IV-D, Table IV vs. Overall Performance Summary"},{"comment":"The prioritization claim relies on three qualitative case studies (Table VIII) and a top-3 comparison against recency and activity baselines (Table IX). A distinct ordering is not evidence of a better ordering: the weights in Eq. (9) are fixed by the authors (wT=0.35, wV=0.25, wR=0.20, wC=0.20), the relevance term is disabled (V=0) during evaluation, and no quantitative link between the Priority Score and any external measure of event importance (verified incident severity, public disclosure timing, or analyst judgment) is provided. The weight-sensitivity concern is acknowledged in §V-A, but that does not supply the missing evidence for the ranking contribution; at minimum, a sensitivity analysis over the Eq. (9) weights and a small external-validity check (e.g., correlation with later public reporting dates) are needed.","section":"§IV-E, Eq. (9)"}],"minor_comments":[{"comment":"The paper calls CyNER 'the standard CyNER dataset [2]', but reference [2] is a Python library; please specify the underlying corpus (e.g., SAM, CNN, DNS, or MITRE) and the entity schema used in the zero-shot evaluation.","section":"§IV-C"},{"comment":"The row labeled 'Word2Vec (GloVe)' conflates two distinct embedding families; please state which method actually produced the reported ARI -0.024 and NMI 0.222.","section":"Table IV"},{"comment":"The pairwise-loss comparison values quoted in Finding 5 (RoBERTa NMI 0.702, DarkBERT NMI 0.592) conflict with Table VI (RoBERTa Pairwise 0.664, DarkBERT Pairwise 0.712), and the detailed results are omitted; the discrepancy and the omitted numbers must be resolved.","section":"§IV-D, Finding 5"},{"comment":"The column header 'DosAttack' differs from the 'DoS/DDoS' category name used in Table I; please harmonize the terminology.","section":"Table III"},{"comment":"The abstract and problem statement call the framework 'unsupervised' even though §III-D fine-tunes the embedding model using ground-truth event clusters; please clarify that the clustering stage is unsupervised while the representation-learning stage is supervised by the curated event labels.","section":"§III-A and §I"},{"comment":"The claim of identifying incidents 'without relying on predefined keywords' is in tension with the keyword-based filtering used to build the classification dataset; please clarify whether keyword filtering is applied at inference time in the deployed pipeline or only during dataset construction.","section":"Abstract and §III-C"},{"comment":"To make the evaluation reproducible from the public code alone (since CrimeBB raw posts are not shareable), please release or describe the ground-truth post-ID sets for the 70 events, the classification labels, and the random seed used to select the 21 test events.","section":"§IV-A and §IV-D"},{"comment":"min_cluster_size=5 for HDBSCAN directly controls the number and granularity of event clusters and is a free parameter; a short sensitivity statement for at least the best configuration would show that the headline NMI/ARI numbers are not artifacts of this choice.","section":"§IV-A"}],"recommendation":"major_revision","confidential_remarks":"The reader's conditional verdict is essentially correct: the decisive question is whether the 21 test events were included in the contrastive fine-tuning pairs. Since the code is public, this can be checked quickly, and I recommend asking the authors to confirm the split and, if necessary, to supply results on genuinely held-out events before the paper proceeds. Independent of that, Table IV and the Overall Performance Summary contradict each other about the best configuration, and Finding 5's quoted baselines disagree with Table VI; these must be fixed regardless. The manuscript sits at the applied NLP/security-systems boundary rather than core systems security, and whether that fits this journal's scope is for the editor to weigh."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Thanks for the report. I went through the paper and largely agree with your read, with one caveat: the train/test split issue is real and load-bearing, not just a clean-room nit.\n\nWhat's actually new: the assembled pipeline—classification, LLM-based NER, entity-aware contrastive embeddings, HDBSCAN, multi-criteria ranking—evaluated on CrimeBB at this scale. That integrated setup, with the specific ranking formula, isn't in the cited prior work. The classification evaluation on 14,386 labeled posts is legitimate, and the zero-shot NER comparison against CyNER is a useful data point. The ranking framework is thoughtfully designed around CTI quality dimensions.\n\nThe soft spots: first, and most important, the paper never states that the 21 test events were excluded from the contrastive fine-tuning. Section III-D defines positive pairs from ground-truth events; Section IV-D samples the test set from the same 70-event ground truth. If the test events were used to build training pairs, then the DarkBERT NMI 0.712/ARI 0.347 largely measure how well the model memorized those specific clusters, not whether it can group unseen event discussions. The authors must clarify the split or rerun with a clean holdout. Without that, the central generalization claim is unsupported. Second, the abstract's entity-aware contribution is overstated: their own Finding 4 shows the input-level entity mechanism helps only some models, and the output-level loss hurt. That's an honest negative result, but the framing should match it. Third, the ranking evaluation is applied to the curated ground-truth clusters, not to the pipeline's discovered clusters, and the weights in Eq. 9 are self-defined; the case studies are illustrative, not validation. The authors acknowledge this in the limitations.\n\nOverall, the paper is coherent and methodical, and the limitations section is candid. It deserves a serious referee, but the referee should demand a clean train/test split and an end-to-end evaluation. If the authors can confirm the split, the operational claims become credible. I would not cite the clustering numbers until that's resolved.\n\nRecommendation: send to peer review with a request for major revision focused on the holdout question.","headline":"Solid system paper with an unsupported clustering generalization claim until the train/test split is clarified.","tokens_in":20529,"tokens_out":2551,"would_cite":false,"duration_ms":27217,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Hacker-forum posts become ranked security events via clustering","keywords":["cyber threat intelligence","hacker forums","security event detection","contrastive learning","HDBSCAN clustering","event prioritization","named entity recognition","transformer embeddings"],"falsifier":"Re-run the pipeline with a strict holdout: remove every post belonging to the 21 ground-truth events from the contrastive fine-tuning set, train on the remaining posts, then cluster the held-out events. If NMI falls well below 0.712 and ARI below 0.347, the published numbers depended on training on the test events.","tokens_in":19490,"feed_emoji":"📡","tokens_out":6136,"duration_ms":65525,"temperature":0.7,"pith_summary":"EventHunter is an unsupervised pipeline that takes noisy, fragmented posts from underground forums and groups them into distinct security events, then ranks those events so an analyst knows what to investigate first. The paper's claim is that contrastively fine-tuned Transformer embeddings, built with an extra signal that marks named entities such as CVEs, malware names, and targeted organizations, make posts about the same incident cluster together far better than keyword, TF-IDF, or word-vector baselines. On 21 curated real-world events, the best configuration reaches NMI 0.712 and ARI 0.347, and a case-study ranking puts the Optus breach first while demoting older, thinner discussions. The payoff, if true, is moving from a firehose of forum chatter to a short, prioritized list of emerging threats.","feed_headline":"Hacker-forum posts become ranked security events via clustering","feed_subtitle":"A density-based system groups scattered underground discussions into one incident list, and ranks the Optus breach first.","key_machinery":"The mechanism is a three-stage representation-to-cluster-to-score chain. Posts are first encoded by a Transformer whose input layer adds an entity-presence embedding to the token, segment, and position embeddings, so named entities act as anchors; the model is then fine-tuned with a combined contrastive objective made of a triplet margin loss and a SimCSE-style pairwise cosine loss over in-batch negatives. HDBSCAN groups the resulting embeddings into clusters of at least five posts and labels the rest as noise. Each cluster receives a daily priority score $s_i = w_T T + w_V V + w_R R + w_C C$, where timeliness mixes exponential recency decay with log-scaled activity, relevance is semantic entity overlap with an optional analyst query, credibility averages the reputation of contributing authors, and completeness counts unique entities and entity types.","core_discovery":"The central discovery is that a fully unsupervised pipeline can turn fragmented forum posts into event clusters that align with known incidents. The best configuration uses a cybersecurity-domain Transformer fine-tuned with a pairwise contrastive loss: the embedding space groups posts by shared event, and HDBSCAN extracts clusters while marking off-topic posts as noise. The authors also show that a weighted priority score combining timeliness, relevance, credibility, and completeness produces a plausible ranking, with the Optus breach ranked first and an older LockBit infrastructure discussion ranked near the bottom. They conclude that the framework reduces noise and surfaces high-priority threats for security analysts.","pith_inferences":["Beyond the paper's evaluation, a sliding-window version of the clustering could test whether the framework detects when a new event begins and when an old one fades, which the static snapshot does not measure.","The credibility and timeliness metrics rely only on internal forum metadata; correlating the ranked clusters against an external incident database would show whether the top-ranked events correspond to major real-world breaches.","The failure of the output-level entity loss suggests that forcing post embeddings to align with individual entity embeddings distracts from event-level structure; a plausible alternative is entity-weighted attention rather than an auxiliary loss.","If actor scores were accumulated from contributions to high-ranked clusters, the framework would yield an event-centric measure of forum influence that could be checked against known threat-actor profiles."],"forward_implications":["An analyst could monitor a hacker forum daily and receive a ranked event list without waiting for public vulnerability disclosures or vendor reports.","Events discussed across multiple threads or forums with different vocabulary can be merged by embedding proximity, even when no keyword is shared.","The priority score gives a transparent order: recent, active, author-credible, information-rich clusters rise, while older or sparse discussions fall, so triage effort concentrates on the top of the list.","The zero-shot LLM entity extraction makes the pipeline adaptable to new forum slang and entity types without retraining a domain-specific NER model.","The reported clustering metrics, NMI 0.712 and ARI 0.347 on 21 curated events, provide a testable benchmark for future event-detection work on underground-forum data."],"supporting_citations":[{"why":"Supplies the multi-forum post corpus with member, thread, and timestamp metadata used throughout the framework and evaluation.","marker":"[44]"},{"why":"Supplies the HDBSCAN density-based clustering algorithm that converts post embeddings into event clusters and labels noise points.","marker":"[10]"},{"why":"Provides the SimCSE pairwise contrastive loss that structures embeddings by pulling positive pairs together and pushing in-batch negatives apart.","marker":"[25]"},{"why":"Supplies DarkBERT, the domain-pretrained Transformer whose pairwise-loss configuration achieves the highest reported NMI.","marker":"[30]"},{"why":"Defines the security-event detection framing and provides the supervised NER baseline used for comparison with the zero-shot LLM extraction.","marker":"[15]"},{"why":"Provides the large-scale measurement showing that forum discussions often precede official reports, motivating the operational early-warning goal.","marker":"[42]"},{"why":"Supplies the threat-intelligence quality dimensions used to justify the ranking components.","marker":"[26]"},{"why":"Defines the timeliness, credibility, completeness, and relevance quality dimensions that shape the priority score.","marker":"[63]"}],"fun_headline_variants":["Unsupervised clustering turns hacker forum posts into ranked threat events","EventHunter clusters and ranks cyber threats without predefined keywords","Hacker forum chatter becomes prioritized security events via contrastive learning","Dynamic clustering surfaces and prioritizes hacker forum security events","Hacker forum posts are clustered and ranked without labeled data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The clustering numbers assume the 21 test events were held out of the contrastive fine-tuning; the paper never states that split, so if the model was trained on posts from those same events, the reported NMI and ARI would overstate how well it detects new events.","fun_headline_variants_meta":{"raw":{"variants":["Unsupervised clustering turns hacker forum posts into ranked threat events","EventHunter clusters and ranks cyber threats without predefined keywords","Hacker forum chatter becomes prioritized security events via contrastive learning","Dynamic clustering surfaces and prioritizes hacker forum security events","Hacker forum posts are clustered and ranked without labeled data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001364,"raw_usage":{"total_tokens":5465,"prompt_tokens":810,"completion_tokens":4655,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":426,"completion_tokens_details":{"reasoning_tokens":4574}},"tokens_in":426,"tokens_out":4655,"duration_ms":34977,"temperature":1.0,"reasoning_tokens":4574,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:48:18.044179+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the pipeline with a strict holdout: remove every post belonging to the 21 ground-truth events from the contrastive fine-tuning set, train on the remaining posts, then cluster the held-out events. If NMI falls well below 0.712 and ARI below 0.347, the published numbers depended on training on the test events.","supporting_citations":[{"cited_title":"Thomas, Alice Hutchings, and Richard Clayton","cited_arxiv_id":null,"evidence_quote":"Supplies the multi-forum post corpus with member, thread, and timestamp metadata used throughout the framework and evaluation."},{"cited_title":"Density- based clustering based on hierarchical density estimates","cited_arxiv_id":null,"evidence_quote":"Supplies the HDBSCAN density-based clustering algorithm that converts post embeddings into event clusters and labels noise points."},{"cited_title":"Darkbert: A language model for the dark side of the internet","cited_arxiv_id":null,"evidence_quote":"Supplies DarkBERT, the domain-pretrained Transformer whose pairwise-loss configuration achieves the highest reported NMI."},{"cited_title":"Tweezers: A Framework for Security Event Detection via Event Attribution-centric Tweet Embedding","cited_arxiv_id":"2409.08221","evidence_quote":"Defines the security-event detection framing and provides the supervised NER baseline used for comparison with the zero-shot LLM extraction."},{"cited_title":"You might have known it earlier: Analyzing the role of underground forums in threat intelligence","cited_arxiv_id":null,"evidence_quote":"Provides the large-scale measurement showing that forum discussions often precede official reports, motivating the operational early-warning goal."},{"cited_title":"The ‘big beast to tackle’: Practices in quality assurance for cyber threat intelligence","cited_arxiv_id":null,"evidence_quote":"Supplies the threat-intelligence quality dimensions used to justify the ranking components."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the timeliness, credibility, completeness, and relevance quality dimensions that shape the priority score."}],"review_version":1}