{"id":"7e38d830-da83-434d-8442-1fad175764a6","arxiv_id":"2507.09133","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"CLIProv detects APT behaviors by embedding provenance log sequences and MITRE ATT&CK text into one semantic space via contrastive learning, then retrieving the closest technique description.","lead":"CLIProv is a threat-hunting system that aligns system audit logs with natural-language threat intelligence using contrastive learning, then detects attacks by searching for the most similar attack-pattern description. The paper reports strong precision and speed on public APT datasets, but evaluation gaps and missing artifacts temper the claims.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Positive log-text pairs share exact entity names (processes, files, IPs); the claimed behavior-semantic alignment is untested, and the CICAPT-IIoT precision drop to 52% is the expected signature of entity-name memorization. An anonymized retraining check is required.","rationale":"The reader's weakest_assumption is the most load-bearing: the semantic alignment claim is the paper's central contribution. I agree with that identification. The concern is concrete: because positive pairs are generated from the same attack scenario, entity names are shared between modalities; the log encoder sees only triples of entity names and a small set of edge labels, so token-level overlap is the most direct cue available to the InfoNCE objective. The observed drop on CICAPT-IIoT is consistent with this shortcut, though not conclusive: the model still achieves 84.36% recall, suggesting some transfer. An anonymization ablation is the single experiment that separates the two explanations. If the model fails under anonymized training, then the efficiency and precision advantages over query-graph methods are mostly inherited from RoBERTa's token matching rather than from a novel multimodal semantic alignment, and the paper's claims need substantial recalibration. If it succeeds, the central contribution is credible. I therefore maintain the reader's CONDITIONAL verdict: the paper should not be rejected outright because the architecture is plausible and the cross-dataset recall shows some generalization, but it cannot be accepted as-is without the anonymization test and the additional reproducibility items the reader noted (code, error bars, validation-based hyperparameter selection). My concern is the same as the reader's weakest assumption, so agreement is 'agree' and the verdict is unchanged.","tokens_in":22785,"tokens_out":5335,"duration_ms":64595,"concrete_test":"Retrain CLIProv on the same training splits and with the same hyperparameters, but replace every process, file, and socket node name in every log sequence with a type-consistent placeholder (e.g., PROC0001, FILE0002, SOCK0003) before sequence construction; keep edge types, graph structure, partition boundaries, and threat-intelligence texts unchanged. Evaluate on the original CADETS/THEIA/ATLAS test sets and on CICAPT-IIoT using the identical protocol that produced Table 3. If node-level precision falls substantially below the reported 76.67%/82.14%/62.09% (or the model cannot learn to align), the encoder was exploiting entity-name overlap rather than behavior semantics. If precision stays within a few points, the semantic-alignment claim survives this concern.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing premise is that the RoBERTa-based dual encoders in §5.3 learn an abstract mapping from provenance behavior to natural-language TTP descriptions. But in §5.2.4 each positive pair is the attack's ground-truth intelligence text and the log sequence extracted from that same attack, so the two modalities share the exact process names, file paths, socket addresses and IPs (e.g., 'nginx', '/tmp/vUgefal', '78.205.235.65'). The log sequence is just the concatenation of typed triples (src, edge, dst), whose only tokens outside a small set of edge verbs are these entity names. InfoNCE (Eqs. 5–7) can therefore be minimized by aligning the embeddings of shared entity tokens, with no need to learn that 'connect to shellcode server' corresponds to (process, connect, IP). The within-dataset tests in Table 3 use held-out scenarios from the same hosts in the same engagement, so many entity names and file-system paths are reused; they do not control for this shortcut. The cross-dataset test, CICAPT-IIoT, uses a different environment and shows node-level precision dropping to 52.38%, exactly the signature of entity-overlap dependence. Section 7.1 attributes failures to knowledge gaps and Wi-Fi discovery false positives, but never considers the entity-memorization confound. Until an anonymization experiment is run, the central claim that CLIProv captures behavior semantics rather than surface-form overlap is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"CLIProv proposes a multimodal contrastive-learning framework that aligns provenance log sequences with natural-language threat intelligence in a shared embedding space, then treats threat detection as a nearest-neighbor semantic search over MITRE ATT&CK technique descriptions, outputting TTP labels and reconstructed attack scenarios. The paper details preprocessing (graph reduction, subgraph partitioning, sequence construction), a dual RoBERTa encoder with a projection layer and bidirectional InfoNCE loss, and evaluates on CADETS, THEIA, ATLAS, and CICAPT-IIoT, comparing against query-graph-based (Poirot, ProvG-Searcher) and anomaly-based (Threatrace, KAIROS) methods. The central claims are that CLIProv achieves higher precision and significantly improved detection efficiency, and that it generalizes across attack scenarios, vulnerabilities, and datasets.","tokens_in":23129,"tokens_out":5686,"duration_ms":63411,"significance":"If the semantic-alignment claim holds, CLIProv is a valuable step toward replacing hand-crafted query graphs with flexible natural-language semantic search over threat intelligence, while also producing interpretable TTP-labeled attack scenarios. The paper includes a full prototype, experiments on four public datasets, a cross-dataset zero-shot evaluation on CICAPT-IIoT, and low per-subgraph query latency (8.27 ms), which are strengths. However, the central claim that the alignment is semantic rather than surface-form based is currently not supported: the training pairs are manually constructed from ground-truth text and the corresponding log sequences, which share exact entity identifiers, and no experiment controls for entity-name memorization. The within-dataset results may therefore overstate generalization, and the lower CICAPT-IIoT precision (52.38% node-level in Table 3) is consistent with the confound. A second protocol issue is the apparent selection of the augmentation hyperparameter n_aug using the CICAPT-IIoT test set. These issues are load-bearing for the paper's main claims and require additional experiments and protocol corrections.","major_comments":[{"comment":"The positive pairs are created by manually splitting ground-truth threat intelligence to match specific attack log sequences (Section 5.2.4), and the log sequences are serializations of typed triples that contain the exact process names, file paths, and IP addresses also present in the paired text (e.g., 'nginx', '/tmp/vUgefal', '78.205.235.65'). Because the InfoNCE loss (Eqs. 5–7) only requires the two embeddings to be close, the dual encoders can minimize the loss by aligning these shared surface tokens without learning behavior-level semantics. This confound is not addressed anywhere in the paper, and the drop in node-level precision from 76.67% (CADETS) to 52.38% (CICAPT-IIoT) in Table 3 is the expected signature of entity-overlap dependence. Please run an experiment in which all entity identifiers (process names, file paths, IP addresses) are anonymized or replaced with generic placeholders before both training and testing; if detection precision and alignment scores remain comparable, the semantic-alignment claim is supported, and if they collapse, the claim must be substantially revised.","section":"§5.2.4, §5.3.3"},{"comment":"The CICAPT-IIoT dataset is described as used solely to test generalization, with no ground truth for attack processes (Section 6.1.1), yet Figure 5 reports TPR and FPR on CICAPT-IIoT under different values of n_aug, and the text states that the TPR of the CICAPT-IIoT dataset rises with more augmented samples. This indicates that n_aug was selected using the held-out test set, which leaks test information into hyperparameter selection and invalidates the cross-dataset generalization claim. The authors should select n_aug (and all other hyperparameters) using a validation split from the training datasets only, and then report the CICAPT-IIoT result for the chosen value.","section":"§6.1.1, §6.4.1"},{"comment":"No error bars or variance are reported for any detection metric, even though Section 6.4.2 describes random sampling of benign sequences with averaging over five experiments. Without a measure of variance it is impossible to assess whether the differences observed between datasets or between CLIProv and the baselines are statistically meaningful. Additionally, since CICAPT-IIoT is stated to lack ground truth for attack processes, the computation of node-level and graph-level precision/recall on this dataset needs to be clarified: what is used as ground truth for the 58 attack graphs and 6,222 benign graphs, and how were the corresponding intelligence labels (e.g., T1546, T1055 in Table 4) assigned?","section":"Table 3, §6.2"},{"comment":"The comparison with Threatrace is reported under both the original DARPA ground truth and a '2-hop' labeling protocol, and the paper emphasizes that Threatrace produces many false positives under the original labels. Reporting both protocols is fair, but the narrative that CLIProv is superior to anomaly detection is weakened by the fact that Threatrace's design explicitly treats 2-hop neighbors as part of the attack; the original-label protocol is not the one Threatrace was designed for. Furthermore, Threatrace is trained on 11 days of CADETS data while CLIProv uses 3 days, and no error bars are given for any method. The comparison should be presented as indicative rather than conclusive, or the methods should be evaluated under comparable conditions (e.g., same training data split and same labeling protocol with variance estimates).","section":"§6.3.2, Table 6"},{"comment":"Treating log sequences as sentences and encoding them with RoBERTa raises an untested load-bearing premise: that the similarity between log-sequence embeddings and text embeddings reflects shared behavior semantics rather than overlap of entity names and paths. The manual construction of positive pairs in Section 5.2.4 provides supervision for the specific entity combinations in the training scenarios, so the model is a supervised aligner rather than an unsupervised semantic discovery mechanism. The paper should explicitly acknowledge this and provide the anonymization experiment from the first major comment to justify the 'semantic' terminology, as well as a brief discussion of how RoBERTa's subword tokenization handles domain-specific tokens such as IP addresses and file paths.","section":"§5.3.1"}],"minor_comments":[{"comment":"The section heading 'Runtime Performanc' is missing the final 'e'.","section":"§6.6"},{"comment":"The text refers to the 'DRAPA TC' dataset; the correct acronym is DARPA TC, and this should be consistent throughout the paper (Section 6.1.1 already uses DARPA).","section":"§1, Figure 1"},{"comment":"Table 3 reports CADETS node-level precision of 76.67%, while Table 7 reports graph-level precision of 100% for CLIProv on CADETS; the relationship between node-level and graph-level metrics should be explained explicitly so readers do not perceive a contradiction.","section":"Tables 3 and 7"},{"comment":"The prompt template for GPT-3 augmentation is given as a single sentence ('Please rewrite the following sentence, keeping the original meaning intact.'); more detail on how semantic preservation was verified, and whether any augmented outputs were manually filtered, would improve reproducibility.","section":"§5.2.4"},{"comment":"The 'Alignment Score' is taken from Poirot's methodology, but the paper does not define it or state whether it is computed automatically or with human annotation; please add a definition or a reference to the original scoring procedure.","section":"§6.3.1, Table 5"},{"comment":"The paper does not state whether the source code, trained models, or processed datasets will be released; providing these would substantially strengthen the reproducibility of the results and the ability of other researchers to apply the anonymization test suggested above.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The manuscript addresses a timely and relevant problem, and the proposed framework is elegant and well-positioned relative to prior work. However, the load-bearing semantic-alignment claim is currently confounded by entity-name overlap between the paired modalities, and the hyperparameter selection on the CICAPT-IIoT test set is a serious protocol flaw. I recommend major revision: the anonymization experiment is essential, the hyperparameter protocol must be corrected, and variance estimates and ground-truth determination for CICAPT-IIoT should be reported. With these additions, the paper could become a solid contribution; without them, the central claims are not sufficiently supported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take on CLIProv (arXiv:2507.09133). The central idea is worth your attention: it replaces query-graph construction with a contrastive dual-encoder that aligns raw provenance log sequences to natural-language threat intelligence, then hunts by semantic search against MITRE ATT&CK. As far as I can tell from the cited literature, that combination is new—Poirot, DeepHunter, and ProvG-Searcher all still rely on query templates or graph structures. The individual components (RoBERTa, InfoNCE) are standard, but the system-level framing is a real contribution.\n\nIt also does some things well. The subgraph partitioning and virtual-node-based scenario reconstruction are pragmatic. The ablation study is honest—it shows that single-modal training and contrastive-plus-classifier both do worse, which supports the semantic-search design. The efficiency numbers compared to Poirot's 19+ hours are plausible, and the authors clearly state limitations around knowledge scope and fake threat intel.\n\nBut there is a load-bearing soft spot. In Section 5.2.4, positive training pairs are created by taking the ground-truth attack's intelligence text and the log sequence from that same attack. Those pairs share exact entity names: process names like 'nginx', file paths like '/tmp/vUgefal', IP addresses like '78.205.235.65'. Since a log sequence is just a concatenation of (src, edge, dst) triples, RoBERTa can minimize the InfoNCE loss by aligning those surface tokens without learning any behavioral semantics. The paper never runs an entity-anonymization test. The cross-dataset result on CICAPT-IIoT—node-level precision dropping to 52%—is exactly the signature you'd predict for entity-memorization. Section 7.1 blames knowledge gaps, but it never considers this confound. That is the main thing I want a revision to address.\n\nA second, smaller issue: Section 6.4.1 varies n_aug and reports TPR on CICAPT-IIoT, which looks like test-set-based parameter selection. There are also no error bars, no code or data release, and the perfect 100% graph-level precision on small held-out sets should be read cautiously. The comparison with Threatrace is roughly fair (they do adjust for the 2-hop label protocol), so I don't want to overstate that.\n\nOverall: if anonymized retraining preserves precision, the paper's claim stands and this is a solid systems contribution. If it collapses, the method needs a different representation to actually encode behavior. Either way, it deserves a serious referee. My recommendation: accept for review, but require an entity-anonymization experiment, variance estimates, and artifact release before final acceptance.\n\nI'd bring it to a reading group as a case study of shortcut learning in multimodal security ML, but I wouldn't cite its core claim until the confound is resolved.","headline":"A genuinely new idea for linking threat intel text to provenance logs, but the evaluation doesn't yet prove the alignment is semantic rather than entity-name memorization; an anonymization test is essential.","tokens_in":23679,"tokens_out":3244,"would_cite":false,"duration_ms":39540,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CLIProv claims that APT threat hunting can be reduced to semantic search: contrastively aligning provenance logs with natural-language threat intelligence so that each attack segment is labeled by its nearest TTP description.","keywords":["threat intelligence","provenance logs","contrastive learning","threat hunting","semantic search","TTP identification","attack scenario reconstruction","multimodal learning"],"falsifier":"Take a trained CLIProv model and anonymize every process name, file path, and IP address in the CADETS and THEIA test scenarios, replacing them with random tokens, then re-run node-level detection. If precision drops far below the reported 76.67% and 82.14%, the model was matching surface identifiers rather than behavior semantics.","tokens_in":22520,"feed_emoji":"🔎","tokens_out":10104,"duration_ms":97800,"temperature":0.7,"pith_summary":"CLIProv tries to establish that threat hunting in system provenance logs can be reduced to a semantic search: instead of hand-converting threat intelligence into query graphs, an analyst can match raw log segments against natural-language descriptions of attack techniques. The system trains a log encoder and a text encoder together with contrastive learning so that a log sequence and the threat-intelligence sentence describing the same attack pattern land close together in one vector space. Detection then becomes nearest-neighbor retrieval from a MITRE ATT&CK database, with each retrieved description supplying a TTP (Tactics, Techniques, and Procedures) label and a temporal-causal stitching step reconstructing the full attack scenario. If this works as reported, it removes the most labor-intensive part of provenance-based threat hunting and makes coverage of new attacks a matter of adding or updating intelligence text. The evaluation across four datasets reports high precision and query times of seconds per scenario.","feed_headline":"Semantic search replaces hand-built query graphs for APT hunting","feed_subtitle":"Contrastive alignment labels APT attacks with TTPs, no query graphs needed.","key_machinery":"The load-bearing mechanism is a dual-encoder contrastive alignment: a RoBERTa text encoder and a RoBERTa log encoder, with a two-layer residual projection network, trained by a bidirectional InfoNCE loss (equations (5)-(7)) to map matching log sequences and intelligence sentences to nearby points in a shared space. Detection is then a nearest-neighbor lookup against threat-intelligence descriptions from MITRE ATT&CK (equation (8)), where the retrieved description's TTP label is the detection result. Supporting machinery includes graph reduction (merging redundant network connections, cascade directory events, and similar file objects via Levenshtein distance), DFS-based subgraph partitioning with a time-density threshold $\\theta_{\\max} = 20$ minutes, and a temporal causal reasoning step that uses virtual nodes and a time-dependent Dijkstra algorithm to stitch isolated attack subgraphs into a single attack graph.","core_discovery":"CLIProv's central claim is that the semantic gap between high-level threat intelligence and low-level provenance logs can be bridged by multimodal contrastive learning, making attack detection a retrieval problem. Log sequences are formed from system behavior subgraphs, each a series of process-file-socket operation triples, and encoded by RoBERTa as if they were sentences. A bidirectional InfoNCE loss pulls matched log-text pairs together and pushes unmatched pairs apart, while a two-layer residual projection maps both modalities into a shared space. At query time the same encoders embed every log subgraph and the ATT&CK text database, and each subgraph receives the TTP label whose description is closest in embedding space. The paper reports that this identifies attacks across scenarios, vulnerabilities, and datasets; labels them with technique IDs such as T1071 and T1189; and reconstructs complete attack scenarios through temporal causal reasoning over virtual nodes.","pith_inferences":["A test the paper does not run is entity anonymization: replace process names, file paths, and IP addresses in the test logs with random tokens and re-measure precision. If precision collapses, the model is partly memorizing identifiers from training scenarios rather than learning behavior semantics.","The same contrastive alignment could be applied to other audit sources, such as Windows ETW and Linux Audit, and to alert triage, where natural-language descriptions of past incidents would serve as the query database.","If the alignment is genuinely semantic, threat-intelligence reports could be fed into the query database as soon as they are written, turning publication into detection capability without engineering effort.","The paper notes but does not evaluate the risk of fake or poisoned threat intelligence; adversarial training with fake samples as negatives is suggested as future work, not tested."],"forward_implications":["If the reported results hold, security analysts can hunt for threats by writing or importing natural-language descriptions of attack techniques, with no query graphs to maintain.","Every detected behavior receives an ATT&CK TTP label, so the output is interpretable at the level of tactics and techniques rather than raw system events.","Adding coverage of a new attack requires adding or updating threat-intelligence text, rather than hand-authoring new graph patterns.","Because detection is a nearest-neighbor search over precomputed embeddings, the paper reports per-subgraph query time of 8.27 ms, which would make interactive investigation of large provenance graphs practical.","The method remains knowledge-based: techniques far outside the training intelligence, such as Wi-Fi Discovery in the CICAPT-IIoT test, are missed or confused with normal behavior, a limitation the paper acknowledges."],"supporting_citations":[{"why":"Supplies the contrastive-learning paradigm that aligns two modalities into one embedding space.","marker":"[14]"},{"why":"RoBERTa is the pretrained encoder used for both log sequences and threat-intelligence text.","marker":"[38]"},{"why":"InfoNCE is the contrastive loss that pulls matched log-text pairs together and pushes negatives apart.","marker":"[45]"},{"why":"Poirot introduced the query-graph threat-hunting approach that CLIProv is designed to replace, and its alignment scoring is reused to evaluate scenario quality.","marker":"[12]"},{"why":"ProvG-Searcher is the query-graph baseline with the most similar architecture; CLIProv reports accuracy and timing comparisons against it.","marker":"[9]"},{"why":"DARPA TC datasets CADETS and THEIA provide the provenance logs and ground-truth attack scenarios used in training and testing.","marker":"[13]"},{"why":"MITRE ATT&CK supplies the threat-intelligence text database that CLIProv searches against at detection time.","marker":"[39]"},{"why":"GPT-3 is used to augment the limited threat-intelligence texts, creating more positive training pairs.","marker":"[44]"},{"why":"KAIROS is an anomaly-based comparison baseline and the source of some attack-scenario descriptions for CADETS and THEIA.","marker":"[10]"},{"why":"Threatrace is the other anomaly-based baseline; its node-level results are compared with CLIProv's.","marker":"[18]"}],"fun_headline_variants":["Contrastive learning turns APT detection into a log-to-intel search","APT detection via semantic search over provenance logs","No hand-built query graphs: contrastive learning aligns logs and TTPs","CLIProv aligns logs with threat intel via contrastive learning","Semantic search labels APT attacks with TTPs from logs alone"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole method rests on the assumption that a log segment and a threat-intelligence sentence land close together in the learned space because they describe the same behavior, not because the log happens to contain the same process names, file paths, or IP addresses that appeared in the training text.","fun_headline_variants_meta":{"raw":{"variants":["Contrastive learning turns APT detection into a log-to-intel search","APT detection via semantic search over provenance logs","No hand-built query graphs: contrastive learning aligns logs and TTPs","CLIProv aligns logs with threat intel via contrastive learning","Semantic search labels APT attacks with TTPs from logs alone"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000796,"raw_usage":{"total_tokens":3497,"prompt_tokens":933,"completion_tokens":2564,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":549,"completion_tokens_details":{"reasoning_tokens":2473}},"tokens_in":549,"tokens_out":2564,"duration_ms":20130,"temperature":1.0,"reasoning_tokens":2473,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:03:23.014820+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a trained CLIProv model and anonymize every process name, file path, and IP address in the CADETS and THEIA test scenarios, replacing them with random tokens, then re-run node-level detection. If precision drops far below the reported 76.67% and 82.14%, the model was matching surface identifiers rather than behavior semantics.","supporting_citations":[{"cited_title":"Radford, J","cited_arxiv_id":null,"evidence_quote":"Supplies the contrastive-learning paradigm that aligns two modalities into one embedding space."},{"cited_title":"1795–1812","cited_arxiv_id":null,"evidence_quote":"Poirot introduced the query-graph threat-hunting approach that CLIProv is designed to replace, and its alignment scoring is reused to evaluate scenario quality."},{"cited_title":"Altinisik, F","cited_arxiv_id":null,"evidence_quote":"ProvG-Searcher is the query-graph baseline with the most similar architecture; CLIProv reports accuracy and timing comparisons against it."},{"cited_title":"URL: https://github.com/darpa-i2o/Transparent-Computing/blob/master/ README-E3.md, accessed 2024.7.15","cited_arxiv_id":null,"evidence_quote":"DARPA TC datasets CADETS and THEIA provide the provenance logs and ground-truth attack scenarios used in training and testing."},{"cited_title":"URL:https://attack.mitre.org/, ac- cessed 2024.7.15","cited_arxiv_id":null,"evidence_quote":"MITRE ATT&CK supplies the threat-intelligence text database that CLIProv searches against at detection time."},{"cited_title":"Brown, B","cited_arxiv_id":null,"evidence_quote":"GPT-3 is used to augment the limited threat-intelligence texts, creating more positive training pairs."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Threatrace is the other anomaly-based baseline; its node-level results are compared with CLIProv's."}],"review_version":1}