{"id":"d79e4292-a55b-48c1-8f34-bf46df44ed76","arxiv_id":"2506.10175","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"AURA combines retrieval-augmented agents and large language models to attribute APT campaigns to threat groups and nations, with natural language justifications.","lead":"This paper presents AURA, a multi-agent system that uses large language models and retrieval to attribute cyber attacks to known threat groups while explaining its reasoning. It is relevant to security analysts who want automated, interpretable attribution from threat reports.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The held-out split isolates test reports from the LLM pretraining corpus, but not from AURA's own retrieval KB: related reports in the 2,199-document corpus can explicitly name the ground-truth actor, so reported accuracy may reflect lookup rather than reasoning.","rationale":"The reader's weakest assumption was the unverified post-cutoff pretraining split. I agree that is a real problem, but I think the more immediately damaging and testable flaw is internal: the retrieval corpus itself likely contains the answer. The paper's own case study shows the attribution justification citing a public report that directly links the campaign to APT36, which is exactly the kind of content that retrieval could surface from the 2,199-document KB. Because the test and KB reports are drawn from the same repositories without campaign-level deduplication, a cosine-similarity search over indicator-derived queries will often retrieve the same campaign's analysis, including its named actor. If this occurs on a large share of the 30 reports, the reported accuracies measure retrieval of an explicit label rather than inference from heterogeneous indicators. This does not rule out the framework being useful, but it means the evaluation as presented cannot support the central claim. The proposed check, logging the retrieved chunks and checking for embedded ground-truth labels, is low-cost and would settle the question definitively. My verdict therefore remains conditional: the paper's framework is plausible, but the accuracy numbers need to be shown to survive retrieval-corpus contamination before they can be taken as evidence for the RAG-based approach.","tokens_in":14225,"tokens_out":11940,"duration_ms":141441,"concrete_test":"For each of the 30 test reports, rerun the retrieval step exactly as in the paper and log the top-k chunks supplied to the attribution agent. Then check, for each chunk, whether it contains an explicit attribution statement naming the ground-truth actor or one of its aliases (e.g., \"APT28\", \"Fancy Bear\", \"this campaign is attributed to...\"). Report the fraction of correct top-1/top-2 predictions for which at least one retrieved chunk contains the ground-truth label. If the fraction is high (e.g., >50%), the reported accuracy is contaminated by label leakage and the core evaluation should be redone with a KB scrubbed of all documents mentioning the test actors/campaigns.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Section 5) is that AURA attributes held-out APT reports from their behavioral indicators. The 30-test/2,199-KB split in Section 4.1 removes only the exact report files from the vector database; it does not remove other reports describing the same campaign or actor. Both splits come from the same GitHub collections (APTNotes, CyberMonitor), which contain multiple vendor reports per campaign. The Semantic Retriever (Section 3.3) returns top-k chunks by cosine similarity, and the Decision Agent (Section 3.4) filters only for relevance, not for whether the chunk already contains the answer. Consequently, a test report's indicators can retrieve a KB chunk with an explicit sentence such as \"this campaign is attributed to APT28,\" which lets the Attribution Agent (Section 3.7) output the ground truth without reasoning over TTPs or IoCs. The Section 6.2.1 case study demonstrates the mechanism: AURA's justification says \"Public threat intelligence by Cyfirma [24] directly links this campaign to APT36,\" i.e., the deciding context contains the label. Disabling web search (Section 4.3) does not address this internal leakage. This concern is load-bearing because it attacks the validity of the headline accuracy numbers themselves, independent of whether the test reports are inside or outside any LLM's pretraining data. It also explains why a group-level top-2 accuracy of 73.33% on 30 reports is not, by itself, evidence of the multi-agent RAG design's contribution.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces AURA, a multi-agent retrieval-augmented framework for attributing APT campaigns to threat groups and nation-states. The system preprocesses analyst queries, rewrites them, retrieves chunks from a 2,199-report vector database, uses a decision agent to filter irrelevant context, optionally performs web search (disabled in evaluation), and generates an attribution decision with a natural-language justification. The authors evaluate four proprietary LLMs on 30 held-out reports, reporting up to 63.33% top-1 and 73.33% top-2 group-level accuracy for GPT-4o, and up to 100% top-2 nation-level accuracy for Claude 3.5 Sonnet. Justification quality is assessed with readability, lexical richness, coherence, perplexity, and an LLM-as-judge protocol; two case studies are also presented. A limitation section acknowledges the small test set.","tokens_in":14560,"tokens_out":6909,"duration_ms":79197,"significance":"If the accuracy results survive a leakage-controlled re-evaluation, AURA would be a useful step toward transparent, interpretable APT attribution: the modular agent design, dual group/nation granularity, and explicit justifications are valuable for analyst-facing decision support, and using a held-out set of 30 recent reports is a reasonable start given the difficulty of obtaining post-training data. The decision to disable web search during evaluation and the paper's candid limitation statement are strengths. However, the contribution currently rests on a narrow empirical base: there are no quantitative baselines, no uncertainty quantification, and no human validation of the justifications. The framework itself is an engineering combination of existing RAG and LLM components, so its value depends on the empirical claims, which are not yet established.","major_comments":[{"comment":"The held-out split only removes the 30 test report files from the vector database, not the many other reports in the same GitHub collections (APTNotes, CyberMonitor) that describe the same campaigns or actors. The Semantic Retriever returns top-k chunks by cosine similarity, and the Decision Agent filters for generic relevance, not for whether a chunk already contains the ground-truth actor name. The Section 6.2.1 case study is direct evidence: the generated justification says \"Public threat intelligence by Cyfirma [24] directly links this campaign to APT36,\" so the retrieved context includes a document with the label. Because both the training corpus and the test set are drawn from the same collections, other vendor write-ups for the same campaign are likely in the 2,199-document corpus. This makes the headline 63.33%/73.33% numbers potentially a lookup effect rather than evidence of reasoning over TTPs and IoCs. The authors should report, for each of the 30 test reports, whether any top-k chunk returned before the Decision Agent contains the ground-truth group or nation name, and should re-run the evaluation with a knowledge base pruned of all reports that mention those labels or describe the same campaign.","section":"§3.3–3.4, §4.1, §5, §6.2.1"},{"comment":"The central accuracy claims rest on only 30 test reports, with no confidence intervals, significance tests, or baselines. With n=30, the difference between GPT-4o's 63.33% and Claude 3.5 Sonnet's 53.33% top-1 group accuracy is about three reports, so the reported ranking of models is not statistically supported. There is also no quantitative comparison: no no-retrieval LLM prompt, no single-agent or naive-RAG ablation, and no comparison with NO-DOUBT, DeepAPT, or the prior TTP-based methods cited in Table 1. The sentence in Section 6 that \"the integration of semantic retrieval with task-specific knowledge effectively grounds the attribution process\" is therefore not demonstrated by the presented experiments. In addition, the pretraining-cutoff argument in Section 4.1 is asserted but not verified for each of the four proprietary models; the paper's own limitation section acknowledges the small test size. A minimal fix is to provide per-report predictions with bootstrapped confidence intervals, a retrieval-free baseline, and a rerun with a pruned knowledge base.","section":"§5, §4.1"},{"comment":"The claim of \"expert-aligned justifications\" is unsupported. The LLM-as-judge is GPT-4o, which is the same model family used for the attribution agent in most experiments and for all non-attribution agents in every experiment (per Section 3.9); this is a self-referential evaluation, not an independent expert assessment. The automated metrics in Section 6.1.1 (readability, lexical richness, embedding coherence, perplexity) measure textual surface properties, not attribution correctness or alignment with expert reasoning. The abstract and Section 6.1.2 use the phrases \"expert-aligned\" and \"human-aligned evaluation,\" but no human annotators are involved. The authors should either add a human evaluation with multiple expert annotators, or reframe the claim as an LLM-as-judge quality check and remove \"expert-aligned\" from the abstract.","section":"§6.1.2, Abstract"}],"minor_comments":[{"comment":"The text says the framework comprises \"six key components\" but then enumerates seven items (i)-(vii); please correct the count or the list.","section":"§3.1"},{"comment":"The setup section says AURA is evaluated using the pass@3 metric, but Section 5 reports only top-1 and top-2 accuracy; please report the pass@3 results or remove the sentence.","section":"§4.3, §5"},{"comment":"The notation is inconsistent: L is defined as the space of natural language queries, yet Eq. (2) uses J ∈ L for justifications, and Section 3.8 refers to E as \"retrieved evidence\" even though E was defined as extracted entities in Section 3.2 and C denotes retrieved chunks.","section":"Table 2, §3.8"},{"comment":"The embedding model is described only as \"OpenAI text embedding model\"; please specify the exact model name and version for reproducibility.","section":"§3.3"},{"comment":"The perplexity score of 57.05 is reported without specifying which language model computed it; please state the model and tokenizer used.","section":"§6.1.1"},{"comment":"The double check marks (\"✓✓\") in the AURA row are unexplained; use single check marks or add a note explaining the convention.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"For the editor: the internal-leakage concern in Major Comment 1 is serious enough that I would like to see the leakage audit before accepting the paper. If the audit shows that a large fraction of correct top-2 attributions come from retrieved chunks that already contain the ground-truth actor or nation label, then the paper should be rejected or substantially reframed as a study of retrieval fidelity rather than attribution reasoning. The public GitHub sourcing of both training and test collections makes this a concrete risk, not a hypothetical one."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe genuinely new thing here is the agent orchestration: query rewriting, relevance gating, conversational memory, and justification synthesis combined for APT attribution. I haven't seen that combination in prior work, and the system is described clearly enough to be reproduced. The authors are also honest that the test set is small and that they plan open-source evaluation later.\n\nThe problem is the evaluation, and it's load-bearing. The headline accuracy numbers—63.33% group top-1, 73.33% top-2, up to 100% nation top-2 on 30 reports—do not demonstrate that AURA is reasoning over TTPs and IoCs. The held-out split removes the 30 test reports from the vector DB, but the residual 2,199-document corpus is drawn from the same GitHub collections (APTNotes, CyberMonitor) and includes multiple vendor reports on the same campaigns. Retrieval can therefore return a chunk that explicitly names the ground-truth actor. The case study makes this concrete: AURA's justification cites a Cyfirma report that directly links the campaign to APT36. Disabling web search does not address this internal leakage. So the measured accuracy may be lookup, not inference.\n\nThere are also standard-but-real issues: no baseline comparison to prior attribution systems (Guru et al., NO-DOUBT, the authors' own TTP-based method), no confidence intervals on 30 samples, and an LLM-as-Judge using gpt-4o, same model family as the generator. The paper acknowledges the small test set, but not the internal leakage.\n\nThat said, the framework is plausible and the modular design is a reasonable step toward interpretable attribution, which is a real need. The fix is straightforward: build a test set of campaigns not represented in the knowledge base at all, compare against a simpler RAG baseline and a single-pass LLM, and get human expert ratings on justifications. With that, the contribution becomes credible.\n\nI would send this to review because the idea deserves referee time, but I'd expect major revision. The stress-test is correct; raise it front and center.\n\n— [Your name]","headline":"The agent architecture is a real combination, but the evaluation lets the system retrieve the answer from related reports rather than reason toward it, so the accuracy claims are unproven.","tokens_in":15049,"tokens_out":3460,"would_cite":false,"duration_ms":36801,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AURA shows that a retrieval-augmented multi-agent framework can attribute APT campaigns to known threat groups, reaching 63.33% top-1 group-level accuracy and 100% top-2 nation-level accuracy on 30 held-out reports.","keywords":["Threat Attribution","Advanced Persistent Threats","Retrieval-Augmented Generation","Multi-Agent Systems","Large Language Models","Cyber Threat Intelligence","Tactics Techniques and Procedures","Interpretability"],"falsifier":"Run a membership test on the 30 test reports against each of the four models, or prompt the models to complete or summarize a test report with retrieval disabled; if any model reproduces report-specific details or attains comparable attribution accuracy without the retrieval context, the measured accuracy does not isolate the framework's contribution.","tokens_in":14006,"feed_emoji":"🛡️","tokens_out":8077,"duration_ms":81401,"temperature":0.7,"pith_summary":"Attributing a cyber attack to the responsible state-sponsored group is usually a slow, manual job that mixes malware artifacts, infrastructure clues, and judgment calls. This paper claims that much of that reasoning can be delegated to a team of cooperating LLM agents that rewrite the analyst's query, retrieve relevant threat reports from a vector database, filter out irrelevant context, and then produce both an attributed actor and a plain-language justification. The authors evaluate the framework on 30 APT reports deliberately selected to postdate the LLMs' training cutoffs and report up to 63.33% top-1 and 73.33% top-2 group-level accuracy, with 100% top-2 accuracy at nation level for one model. If these numbers hold, AURA would give threat analysts a transparent triage tool that surfaces plausible actors and the evidence behind them, rather than a black-box label.","feed_headline":"AI agents attribute APT groups with 63% top-1 accuracy","feed_subtitle":"Retrieval-augmented multi-agent system also hits 100% top-2 nation-level attribution on 30 held-out threat reports.","key_machinery":"The load-bearing mechanism is the agent pipeline itself, expressed as the transformation $\\mathrm{AURA}(Q)=(A,J)$. Six components do the work: a preprocessing module that extracts TTPs, IoCs, malware names, and timelines; a semantic retriever that ranks chunks of the threat-report corpus by cosine similarity of dense embeddings; a decision agent that discards irrelevant retrieved context; a query-rewriting agent that resolves pronouns and vague references before retrieval; an attribution-generation agent that invokes the LLM over the rewritten query and retrieved evidence to pick the actor; and a conversational-memory module that carries context across turns. A web-search module exists but is disabled during evaluation to prevent leakage, so the reported numbers test only internal knowledge-base retrieval.","core_discovery":"On the paper's own terms, the central discovery is that a retrieval-augmented multi-agent pipeline can turn unstructured threat intelligence into an interpretable attribution decision: the framework maps a natural-language query $Q$ to a predicted threat actor $A$ and a justification $J$. The pipeline decomposes the task into query rewriting, semantic retrieval over a corpus of 2,199 threat reports, a relevance decision gate, attribution generation, and justification synthesis, with the attribution step carried out by one of four proprietary LLMs. Across 30 held-out test reports, the best configuration reaches 63.33% top-1 and 73.33% top-2 group-level accuracy, and one model reaches 83.33% top-1 and 100% top-2 nation-level accuracy, with justifications rated fluent and coherent by an LLM acting as judge. The authors take these results as evidence that agent modularity plus retrieval grounding improves both accuracy and explainability compared with single-pass LLM extraction or ML classifiers used in prior work.","pith_inferences":["A direct ablation the paper does not run — attribution with retrieval disabled — would separate the contribution of the RAG pipeline from the parametric knowledge of the LLM; without it, the 63% figure mixes both sources.","Because the test set has only 30 reports, the gap between the best model (63.33%, 19/30) and the second-best (53.33%, 16/30) is within a few correct answers of being noise; larger testbeds are needed before treating model rankings as stable.","The same agent architecture should transfer to intrusion-set-level or campaign-level attribution, since the retriever and justification modules do not depend on group-level labels.","One can test the value of the decision agent by feeding the attribution agent unfiltered top-k chunks; if accuracy does not drop, the relevance gate is not load-bearing."],"forward_implications":["Nation-level attribution is consistently easier than group-level attribution in these experiments, so the framework is most reliable as a geopolitical triage filter rather than a definitive actor identifier.","Surfacing two candidates instead of one raises accuracy by roughly ten points at group level, which matters because real APT groups often share tooling and TTPs.","The framework is provider-agnostic: swapping the final attribution LLM changes accuracy but leaves the pipeline intact, making the design portable to newer or open-weight models.","With web search disabled, the reported numbers are a conservative floor; deployment with live retrieval is expected to perform at least as well on previously unseen intelligence.","Generated justifications give analysts a traceable path from evidence to actor, which is the property earlier ML-based attribution systems lacked."],"supporting_citations":[{"why":"Supplies the 4C attribution-granularity framing used to set group-level and nation-level evaluation targets.","marker":"[31]"},{"why":"Publicly collected threat reports used to build the knowledge-base corpus.","marker":"[2]"},{"why":"Second public collection of campaign reports used to build the knowledge-base corpus.","marker":"[6]"},{"why":"Defines the pass@k sampling metric that the evaluation uses to handle LLM output variability.","marker":"[4]"},{"why":"Ground-truth report linking the case-study campaign to its attributed actor.","marker":"[24]"},{"why":"Ground-truth report for one of the ambiguous attribution test cases.","marker":"[32]"},{"why":"Ground-truth report for the second ambiguous attribution test case.","marker":"[36]"},{"why":"Provides the actor-profile documentation used in the case study to confirm historical TTP and targeting alignment.","marker":"[1]"},{"why":"Earlier feature-based attribution method that AURA positions against in related work.","marker":"[17]"},{"why":"Earlier report-based attribution classifier whose lack of interpretability motivates AURA's justification module.","marker":"[18]"}],"fun_headline_variants":["Agent framework nails APT groups via retrieval-augmented reasoning","AURA: agent-driven threat attribution with RAG and LLM justifications","Retrieval-augmented agents hit 100% nation-level APT attribution","Interpretable APT attribution from multi-agent LLM pipeline"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results stand on the unverified assumption that none of the 30 test reports appeared in the training data of the four evaluated LLMs; the paper relies on publication dates versus knowledge cutoffs but never checks this per model.","fun_headline_variants_meta":{"raw":{"variants":["Agent framework nails APT groups via retrieval-augmented reasoning","AURA: agent-driven threat attribution with RAG and LLM justifications","Retrieval-augmented agents hit 100% nation-level APT attribution","Interpretable APT attribution from multi-agent LLM pipeline"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000206,"raw_usage":{"total_tokens":1403,"prompt_tokens":957,"completion_tokens":446,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":573,"completion_tokens_details":{"reasoning_tokens":369}},"tokens_in":573,"tokens_out":446,"duration_ms":5698,"temperature":1.0,"reasoning_tokens":369,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:32:25.689092+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a membership test on the 30 test reports against each of the four models, or prompt the models to complete or summarize a test report with retrieval disabled; if any model reproduces report-specific details or attains comparable attribution accuracy without the retrieval context, the measured accuracy does not isolate the framework's contribution.","supporting_citations":[{"cited_title":"B., Alptekin, H., and C etin, O","cited_arxiv_id":null,"evidence_quote":"Second public collection of campaign reports used to build the knowledge-base corpus."},{"cited_title":"https://attack.mitre.org/groups/G0037/","cited_arxiv_id":null,"evidence_quote":"Defines the pass@k sampling metric that the evaluation uses to handle LLM output variability."},{"cited_title":"Chasing the Shadows: TTPs in Action to Attribute Advanced Persistent Threats","cited_arxiv_id":"2409.16400","evidence_quote":"Ground-truth report linking the case-study campaign to its attributed actor."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Ground-truth report for one of the ambiguous attribution test cases."},{"cited_title":"Flesch–kincaid readability tests","cited_arxiv_id":null,"evidence_quote":"Ground-truth report for the second ambiguous attribution test case."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Earlier feature-based attribution method that AURA positions against in related work."},{"cited_title":"A review of attribution technical for apt attacks","cited_arxiv_id":null,"evidence_quote":"Earlier report-based attribution classifier whose lack of interpretability motivates AURA's justification module."}],"review_version":1}