{"id":"e88d7ec2-27d7-4d08-a99b-5e7186a3426a","arxiv_id":"2505.06701","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"RuleGenie combines CodeT5 rule embeddings, top-5 cosine similarity, and chain-of-thought LLM analysis to flag redundant SIEM rules across Sigma, Splunk, and AQL formats.","lead":"The paper presents RuleGenie, a system that uses language model embeddings and an LLM to find redundant or overlapping SIEM security rules and suggest which to keep, merge, or remove. A generalist reader might care because reducing duplicate alerts can lower analyst alert fatigue and improve security operations.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The CodeT5 top-k prefilter sets an unmeasured recall ceiling, and the ground truth appears close to syntactically near-identical rules, so the LLM-stage metrics may not demonstrate semantic redundancy detection.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the CodeT5 top-k retrieval prefilter determines what the LLM is allowed to judge, so any redundant pair that is not syntactically close is invisible to the system. My stress-test agrees and sharpens it: the ground truth, as described, appears dominated by rules with identical query structures or clear structural overlap, which means the reported recall may be an upper bound on what the pipeline can achieve on more semantically divergent redundancy. The paper provides useful supporting evidence: k-selection and threshold ablations, CoT comparison, and cost analysis. However, without a measurement of retrieval recall on a semantically diverse test set, the central claim that LLM semantic review is what enables effective cross-format SIEM deduplication is not fully established. The reader's CONDITIONAL verdict already reflects this uncertainty, so the verdict is unchanged. The proposed concrete test would settle whether the concern lands by separating retrieval-stage misses from LLM-stage errors.","tokens_in":11126,"tokens_out":4980,"duration_ms":50395,"concrete_test":"Construct a held-out redundancy test set by taking 50 existing ground-truth redundant Sigma pairs and applying meaning-preserving perturbations: rename fields to synonyms, replace query operators with structurally different equivalents, reorder conditions, and split one rule into two chained conditions. Run RuleGenie end-to-end with k=5 and with k=N (brute-force, no prefilter) on the same partition. If the k=5 recall falls below the brute-force recall, or if the fraction of redundant pairs in the CodeT5 top-5 drops below 0.966, the prefilter is the binding constraint and the claims in Section 5 should be re-scoped to syntactically near-identical redundancy. Repeat on AQL-converted rules to test platform-agnosticism.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The end-to-end recall in Tables 3 and 4 is bounded above by the retrieval stage described in Section 3.3, where k=5 candidates are selected by CodeT5 cosine similarity. Retrieval recall is reported only for the Sigma set (0.966 at k=5, Section 4.3.2); the paper does not separately state how many ground-truth redundant pairs are discarded before the LLM stage for AQL or Splunk. More importantly, the Introduction describes the redundant rules as 'demonstrating overlapping functionalities expressed via identical query structures or detection objectives,' and Table 1 does not characterize the diversity of the manual ground truth. If most annotated redundant pairs are near-duplicates with identical query structures, CodeT5 will rank them highly, and the high recall would be an artifact of the evaluation set rather than evidence of the LLM's semantic equivalence capability. The pipeline will silently miss redundant rules that express the same detection logic with different field names, operator styles, or SIEM schemas, because those pairs are pruned before the LLM sees them. Section 6.1 compares brute-force LLM analysis with top-k LLM analysis, but it does not compare against a simple syntactic deduplication baseline, so the marginal contribution of the LLM stage is not isolated. This is not an internal contradiction, but it makes the central claim less secure than the headline precision 0.941 and recall 0.910 suggest.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"RuleGenie proposes a three-phase pipeline for optimizing SIEM detection rule sets: (1) generating rule embeddings with the CodeT5 transformer, (2) retrieving the top-k most similar rules via cosine similarity, and (3) using an LLM with chain-of-thought prompting to assess semantic redundancy, performance trade-offs, and generate recommendations. The system is evaluated on Sigma, Splunk, and AQL rule sets, with ground truth formed by manual analyst curation. The paper reports high precision and recall (e.g., 0.941 precision and 0.910 recall for Qwen-2.5-14B-Instruct on Sigma/AQL) and an 80x speedup over brute-force LLM pairwise analysis. The central claim is that LLM-based semantic analysis after embedding-based retrieval is a practical way to identify redundant rules across SIEM formats.","tokens_in":11346,"tokens_out":4823,"duration_ms":44721,"significance":"If the reported results are reliable, RuleGenie addresses a real operational pain point—SIEM rule redundancy causing alert fatigue and inefficiency—with a pipeline that combines code-aware embeddings and LLM reasoning. The scarcity of prior work on SIEM rule optimization makes the application contribution potentially valuable. The paper also provides a useful decomposition of the pipeline (retrieval vs. LLM) and reports cost and latency data for GPT-4o versus local models. However, the current evaluation has several methodological gaps—test-set leakage in parameter selection, an uncharacterized ground truth, and a missing retrieval-recall ceiling analysis—that prevent the results from fully supporting the central claim as stated.","major_comments":[{"comment":"The end-to-end recall reported in Tables 3 and 4 is bounded above by the recall of the CodeT5 top-k retrieval, but retrieval recall is reported only for the Sigma set (0.966 at k=5). The paper does not state retrieval recall for the Splunk and AQL sets, nor does it quantify how many ground-truth redundant pairs are pruned before the LLM stage. Without this decomposition, the reported recall cannot be attributed to the LLM's semantic capabilities; the headline recall values may simply reflect the retrieval stage. Please report per-dataset retrieval recall and an error analysis of pairs missed before the LLM.","section":"Sections 3.3 and 4.3.2"},{"comment":"The values of k=5 and the similarity threshold of 75 are selected using the same Sigma rule set on which the headline Sigma and AQL results in Table 3 are reported. Similarly, the embedding model and the LLM are chosen based on performance on the same data (Sections 4.3.1 and 4.3.3). This creates test-set leakage: the reported numbers are tuned on the same data, so they do not estimate performance on unseen rule sets. The authors should use a held-out validation split or nested cross-validation for parameter selection, or explicitly report performance on a held-out set.","section":"Sections 4.3.2 and 4.3.4"},{"comment":"The ground truth is described only as 'established through expert analyst review' with no annotation protocol, no definition of redundancy, no inter-annotator agreement, and no examples of the annotated pairs. Given that the Introduction characterizes redundant rules as 'identical query structures or detection objectives,' it is unclear whether the ground truth contains any semantically redundant but syntactically diverse pairs. If it consists mostly of near-duplicates, then the reported recall does not demonstrate the system's ability to detect semantic redundancy. Please provide an annotation guideline, agreement statistics, and a breakdown of the redundancy types in the ground truth.","section":"Section 4.4 and Table 1"},{"comment":"The AQL rows are not independent evidence of platform-agnostic behavior, because all AQL rules are programmatic conversions of the same 2,347 Sigma rules (Section 4.2) and thus share identical semantics and ground truth. The near-identical results for Sigma and AQL are therefore expected and do not show generalization to a genuinely different rule-set corpus; only the Splunk results provide such evidence. Additionally, Section 6.1 compares the top-k pipeline against a brute-force LLM workflow but not against a purely syntactic baseline (e.g., exact match or high token-overlap deduplication), so the marginal contribution of the LLM stage is not isolated. Please add a syntactic baseline and clearly frame the AQL results as a syntax-conversion check rather than a new-platform validation.","section":"Section 5.1, Table 3"}],"minor_comments":[{"comment":"Several typos appear, including 'Beg-Gurion' (should be Ben-Gurion), 'address theses challenges' (these), 'syntatic' (syntactic), and 'matrices' (criteria) in Section 4.4; please proofread carefully.","section":"Throughout"},{"comment":"The sentence describing the three pre-classification categories is duplicated; one copy should be removed.","section":"Section 4.2"},{"comment":"The CoT ablation reports precision 0.818 and recall 0.733, which differ from the main Qwen results in Table 3 (0.941 and 0.910 on Sigma). Please clarify whether the ablation used a subset of the data and specify the configuration, so the discrepancy is not confusing.","section":"Section 6.2 and Table 6"},{"comment":"Table 2 reports precision 0.818 and recall 0.733 at threshold 75, while Table 3 reports different Sigma numbers for the same threshold; please reconcile these values or explicitly state that they correspond to different configurations or evaluation subsets.","section":"Section 4.3.4 and Table 2"},{"comment":"The statement 'Code will be made available once the paper is accepted' makes it impossible to reproduce the experiments at review time; please make the code and the exact prompts available with the submission or give a detailed pseudocode of the LLM prompt structure.","section":"Section 4.1"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely operational problem, and the pipeline is sensible, but the evaluation has a recurring leakage problem: nearly all design choices and the reported numbers come from the same Sigma-derived data. The AQL results are essentially a re-formatting of the Sigma set, so the platform-agnostic claim rests largely on the Splunk results alone. I would encourage the editor to require the authors to separate tuning and evaluation sets, report retrieval-stage recall per format, and make the ground-truth protocol and code available. With those changes the contribution could become solid."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take on RuleGenie. It's a sensible engineering contribution: first direct application I know of that uses an embedding retriever plus an LLM to recommend redundant SIEM rules across Sigma, Splunk, and AQL. The paper does some things right: it uses public rule repositories (2,347 Sigma, 1,640 Splunk), gives a concrete pipeline, and includes ablations for embedding model, k, similarity threshold, and LLM choice, plus a cost analysis that makes the open-source Qwen choice credible. The efficiency comparison against brute-force pairwise LLM analysis (80x speedup) is useful, and the chain-of-thought vs single-prompt table is honest evidence that the reasoning stage matters.\n\nThe soft spots are real and mostly in the evaluation. The ground truth is described only as 'manual analyst curation' with no protocol, no inter-annotator agreement, and no release of the annotations. That alone caps how much I trust the precision/recall numbers. More concerning, k=5 and the similarity threshold of 75 are selected on the same Sigma set that produces the headline Sigma and AQL results, so those numbers are tuned, not predicted. The AQL set is a programmatic conversion of the Sigma rules, so it is not an independent platform test; the Splunk set is the only independent one and the precision there is noticeably lower (0.673-0.795). The stress-test concern is on point: the CodeT5 top-k prefilter sets an unmeasured recall ceiling, and the paper does not report retrieval-stage recall for AQL or Splunk. Given that the ground truth seems to favor rules with 'identical query structures,' the high end-to-end recall may partly reflect retrieval of near-duplicates rather than semantic equivalence. A syntactic-dedup baseline (e.g., normalized text equality or Jaccard on tokens) would have isolated the LLM's contribution; its absence is a real omission.\n\nNone of this is fatal, and the central mechanism is plausible. But the evaluation as reported supports a conditional 'works in a carefully set-up setting' claim, not an accept-the-numbers claim. Code release is promised only upon acceptance; for a reproducibility-conscious referee, that should be a precondition, not an afterthought.\n\nBottom line: worth a serious referee, especially a security or applied-LLM venue, but the authors should be pushed to publish annotations, add a non-LLM baseline, and report retrieval recall for every platform.\n\nBest.","headline":"A plausible first stab at LLM-based SIEM rule dedup with a real evaluation corpus, but the evaluation's ground truth and parameter tuning make the headline numbers softer than they look.","tokens_in":11938,"tokens_out":1808,"would_cite":false,"duration_ms":16873,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RuleGenie claims that an LLM reviewing the top five embedding-neighbor pairs of each SIEM rule can flag redundant detection rules across Sigma, Splunk, and AQL, with precision up to 0.941 and recall up to 1.000.","keywords":["SIEM","rule redundancy detection","rule set optimization","large language models","chain-of-thought","code embeddings","cosine similarity","Sigma rules"],"falsifier":"Measure the recall of the retrieval stage alone: count how many of the known ground-truth redundant target/candidate pairs appear in the top-5 cosine-similarity list before any LLM runs. Then rewrite some redundant rules to be semantically identical but syntactically distant (different field names and query structure) and repeat; if retrieval-stage recall falls materially below the reported end-to-end recall, the embedding pre-filter is the binding ceiling and the headline numbers overstate what the system can catch.","tokens_in":10882,"feed_emoji":"🛡️","tokens_out":9122,"duration_ms":75157,"temperature":0.7,"pith_summary":"RuleGenie is a recommender system that targets a practical pain point: security operations teams get overwhelmed by alerts because SIEM rule sets accumulate overlapping and redundant detection logic. The paper argues that a two-stage pipeline—code-embedding retrieval followed by large-language-model semantic analysis—can flag redundant rules across three real-world rule formats (Sigma, Splunk, and AQL) with high precision and recall. If true, this would automate part of a task currently done by hand, reduce false positives, and free analysts for more strategic work. The best reported configuration reaches 0.941 precision and 0.910 recall on Sigma/AQL with a locally run open-source model, and recall up to 1.000 on Splunk with a proprietary model.","feed_headline":"LLM review flags duplicate SIEM rules with 94% precision","feed_subtitle":"Code-embedding search plus chain-of-thought review catches overlapping detection rules, easing alert fatigue.","key_machinery":"The load-bearing mechanism is a three-phase pipeline. First, CodeT5, a code-oriented encoder-decoder transformer, converts each SIEM rule into an embedding that captures its syntax and structure. Second, cosine similarity with k=5 retrieves the nearest neighbors of a target rule, pruning the candidate space before any expensive reasoning. Third, an LLM (Qwen-2.5-14B-Instruct or GPT-4o) runs a four-stage chain-of-thought analysis on each candidate pair: semantic and functional overlap, hierarchical dependence, performance comparison across coverage, efficiency, and false positives, and recommendation generation. A similarity score threshold of 75 decides whether a pair advances to the later stages. The retrieval step is what makes the system scalable, while the LLM step is what makes the recommendations trustworthy.","core_discovery":"The central claim is that redundant SIEM rules can be reliably detected by combining syntactic embedding similarity with semantic LLM review. Each rule is embedded with CodeT5, the target rule's five closest neighbors are retrieved by cosine similarity, and a chain-of-thought LLM then judges each candidate pair for functional overlap, hierarchical dependence, coverage, false-positive behavior, and efficiency. On the paper's curated ground truth, this detects truly redundant rules and produces recommendations that human analysts validate as correct: Qwen-2.5-14B-Instruct reaches 0.941 precision and 0.910 recall on Sigma and AQL, and GPT-4o reaches 1.000 recall on Splunk with 0.673 precision. The authors also show that chain-of-thought prompting raises precision from 0.250 to 0.818 compared with a single prompt, and the embedding pre-filter gives an 80x speedup over brute-force pairwise analysis.","pith_inferences":["The paper does not report retrieval-stage recall in isolation, so the end-to-end numbers leave open how many redundant pairs were already lost before the LLM saw them; a fair benchmark would measure top-k recall on the ground-truth pairs directly.","Because the top-k pre-filter fixes the recall ceiling, the framework's upper bound is set by CodeT5's embedding space; replacing or fine-tuning the embedder on SIEM rule pairs could raise that ceiling without changing the downstream LLM stage.","The same retrieval-plus-LLM architecture could be applied to other rule-based security artifacts, such as firewall rules, YARA signatures, or cloud detection policies, wherever syntactic and semantic redundancy coexist.","The similarity threshold of 75 was tuned on Sigma rules, and reported precision differs across formats (for example 0.941 on Sigma versus 0.795 on Splunk for the best model), so deployment to a new format may require recalibrating that threshold."],"forward_implications":["Security teams could route newly written SIEM rules through this triage loop to catch likely duplicates before deployment, reducing alert noise without a rule-by-rule manual audit.","Because a locally deployed 14B open-source model achieves the best precision on Sigma and AQL, rule-set optimization can run on-premises, which addresses data-privacy constraints of proprietary cloud APIs.","The 80x speedup over brute-force pairwise comparison makes whole-rule-set cleanup feasible at enterprise scale, converting a periodic manual audit into a routine batch process.","The large jump from single-prompt to chain-of-thought analysis (precision 0.250 to 0.818) indicates that the prompting format is a core part of the contribution, not just the choice of model.","The identical performance on Sigma and its AQL conversion suggests the method transfers across SIEM query languages without retraining the embedder or the LLM."],"supporting_citations":[{"why":"Supplies CodeT5, the code-oriented transformer whose embeddings carry the syntactic similarity signal for all retrieval.","marker":"[28]"},{"why":"Supplies chain-of-thought prompting, which the paper shows raises precision from 0.250 to 0.818.","marker":"[29]"},{"why":"Sigma detection rules are one of the two primary real-world datasets used for evaluation.","marker":"[4]"},{"why":"Splunk Security Content repository is the other primary dataset, contributing 1,640 rules.","marker":"[26]"},{"why":"pySigma converts Sigma rules into AQL and Splunk formats, enabling the cross-platform evaluation.","marker":"[1]"},{"why":"GPT-4o is one of the two LLMs evaluated, achieving recall 1.000 on Splunk.","marker":"[14]"},{"why":"Qwen-2.5-14B-Instruct is the locally deployed open-source LLM that delivers the best precision results.","marker":"[30]"},{"why":"Provides the cosine-similarity and nearest-neighbor retrieval basis for the top-k pre-filter.","marker":"[23]"}],"fun_headline_variants":["AI duo finds redundant SIEM rules, cuts alert noise","Embeddings plus LLM catch overlapping detection rules","RuleGenie: 94% precision spotting duplicate SIEM rules","Chain-of-thought LLM sniffs out redundant SIEM rules"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pipeline depends on the assumption that every truly redundant rule pair sits close enough in CodeT5 embedding space to appear among each other's five nearest neighbors, because pairs that the retrieval stage misses never reach the LLM and can never be recommended for removal.","fun_headline_variants_meta":{"raw":{"variants":["AI duo finds redundant SIEM rules, cuts alert noise","Embeddings plus LLM catch overlapping detection rules","RuleGenie: 94% precision spotting duplicate SIEM rules","Chain-of-thought LLM sniffs out redundant SIEM rules"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000208,"raw_usage":{"total_tokens":1445,"prompt_tokens":1028,"completion_tokens":417,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":644,"completion_tokens_details":{"reasoning_tokens":348}},"tokens_in":644,"tokens_out":417,"duration_ms":4049,"temperature":1.0,"reasoning_tokens":348,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:35:36.450237+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the recall of the retrieval stage alone: count how many of the known ground-truth redundant target/candidate pairs appear in the top-5 cosine-similarity list before any LLM runs. Then rewrite some redundant rules to be semantically identical but syntactically distant (different field names and query structure) and repeat; if retrieval-stage recall falls materially below the reported end-to-end recall, the embedding pre-filter is the binding ceiling and the headline numbers overstate what the system can catch.","supporting_citations":[{"cited_title":"contributors","cited_arxiv_id":null,"evidence_quote":"Sigma detection rules are one of the two primary real-world datasets used for evaluation."},{"cited_title":"Splunk security content homepage, n.d","cited_arxiv_id":null,"evidence_quote":"Splunk Security Content repository is the other primary dataset, contributing 1,640 rules."},{"cited_title":"URL https:// sigmahq-pysigma.readthedocs.io/en/latest/","cited_arxiv_id":null,"evidence_quote":"pySigma converts Sigma rules into AQL and Splunk formats, enabling the cross-platform evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the cosine-similarity and nearest-neighbor retrieval basis for the top-k pre-filter."}],"review_version":1}