Pith. sign in

REVIEW 4 major objections 5 minor 98 references

ContextBuddy: AI-Enhanced Contextual Insights for Security Alert Investigation (Applied to Intrusion Detection)

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An AI assistant that learns which context cues past analysts used can raise alert classification accuracy by 21.1% and trim validation time by 24%.

desk verdict A real new application of IL to context selection for alert triage, but the headline 'learns from analysts' claim is not yet supported because the traces are from RL agents and the user study lacks a feature-subset control. read the letter →

arxiv 2506.09365 v1 pith:4623BM4B submitted 2025-06-11 cs.CR

Ronal Singh , Mohan Baruwal Chhetri , Surya Nepal , Cecile Paris This is my paper · ORCID
classification cs.CR
keywords securityoperationscentrehuman-machinecollaborationalertcontextimitationlearningintrusiondetectionexplainableAItriage
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ContextBuddy's central claim is that the bottleneck in security alert triage is not a shortage of contextual information but the difficulty of knowing which cues matter for a given alert, and that an AI can learn that skill by imitating how analysts have investigated alerts before. The paper formulates context selection as a sequential decision problem, generates investigation traces with reinforcement-learning 'analysts', and trains an imitation-learning policy (generative adversarial imitation learning on multi-analyst traces works best) that suggests which context categories to consult for a new alert. In simulation, ContextBuddy raised the F1 of simulated analysts by 2.5% on the HIKARI-2021 dataset and 9% on UNSW-NB15, cut false negatives, kept false positives below 1%, and improved decision confidence. In a within-subject user study with 13 non-experts, a one-time context suggestion improved classification accuracy from 65.4% to 86.5% (a 21.1% increase, p=0.008) and reduced alert validation time by 24% (p=0.01). If the result transfers to real security operations centres, it offers a way to fight alert fatigue and speed up investigations using data already produced by past investigations.

What carries the argument

The load-bearing object is the context-selection policy: a mapping from the analyst's current state (features already seen, context categories already requested, prediction confidence) to the next context category to request, learned by generative adversarial imitation learning (GAIL, an imitation method that trains the policy against a discriminator trying to tell its behaviour apart from the demonstrations) on investigation traces. The traces come from reinforcement-learning 'analysts' (A2C, PPO, DQN) that request context subsets and classify with a library of XGBoost classifiers, one per context combination, guided by a hand-shaped reward that favours correct, confident, low-cost investigations. That same classifier library generates SHAP-based explanations in the user interface, letting a user weigh evidence for and against each attack class. ContextBuddy delivers its learned suggestions either one-time (a full plan of context requests up front) or iteratively (suggesting the next context given the interaction history), with the one-time mode producing the reported gains.

What would settle it

Record how professional SOC analysts actually investigate alerts, meaning which context sources they open, in what order, and what they conclude, across a few hundred real alerts, then train ContextBuddy on those traces and measure whether its suggestions improve the analysts' own accuracy or time. If real analysts request context very differently from the simulated ones, such as making far fewer requests, following tool-driven ordering, or acting on policy-driven exemptions, the imitation-learning foundation would not transfer and the simulated gains would not reproduce with experts.

Watch

Extended reading notes

Core claim

ContextBuddy's central claim is that the right response to context overload is not to generate more enrichments but to model how analysts have actually chosen context in the past, and to reuse that behaviour to guide new investigations. The paper formulates an analyst's investigation as a Markov decision process whose actions are requests for context categories (packet counts, payload information, timing, and so on), then trains imitation-learning policies on traces produced by reinforcement-learning analysts (A2C, PPO, DQN). A multi-source GAIL policy trained on traces from all three RL models learned the most transferable context-selection strategy, matching or exceeding its teachers' F1 and producing the suggestions used in the teaming experiments. When simulated analysts adopted ContextBuddy's suggestions under a confidence-threshold strategy, F1 rose by 2.5% (HIKARI) and 9% (UNSW) relative to working alone, false negatives fell by 1.5% and 10% respectively, and false positives stayed below 1%. The user study then showed that non-expert participants given one-time context suggestions reached 86.5% classification accuracy versus 65.4% without assistance, in 24% less time, with SHAP-based evidence presented for and against each class rather than a recommended verdict.

Load-bearing premise

Everything rests on the premise that reinforcement-learning agents trained with a hand-shaped reward approximate how real security analysts choose context closely enough that imitating their traces captures human investigation strategy; the paper itself states in its limitations that real-world SOC data are still needed to confirm this.

Editorial extensions

If this is right

  • Training ContextBuddy on traces from multiple analyst behaviours (the GAIL_ALL variant) outperforms training on any single analyst model, so diversity in investigation histories improves the learned context-selection policy.
  • Simulated analysts who accept ContextBuddy's context suggestions only when their own confidence is low perform better than working alone, with the gain holding across both datasets and large effect sizes.
  • ContextBuddy recovers specific attack classes that analysts miss, such as probing, brute-force, and crypto-miner events in HIKARI and exploits and backdoors in UNSW, raising per-class F1 from as low as 0.46 to 0.78–0.91.
  • For non-expert users, one-time context suggestions beat both unaided work and an iterative interaction mode (86.5% versus 65.4% accuracy), suggesting that curating the relevant subset up front is the core benefit.
  • Because suggestions are occasionally wrong (around 15 reversed decisions across both datasets), the design pairs every suggestion with SHAP evidence so the analyst can decide when to override it.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Nothing in the method is specific to intrusion detection: the same MDP-plus-imitation-learning recipe could be applied to other evidence-gathering tasks in a SOC, such as phishing triage, malware sandbox review, or cloud misconfiguration tickets, wherever request logs can be captured.
  • The large gap between one-time and iterative assistance in the user study suggests a general design lesson for human-AI teaming: when users lack domain expertise, a single curated information set may beat a dialogue, because every additional round asks the user to integrate evidence they cannot yet weigh.
  • A direct test of the paper's central premise would be to instrument a real SOC's co-pilot sessions, log which context sources analysts open and when, and check whether GAIL trained on those logs shifts suggestion quality relative to RL-generated traces; the paper's architecture anticipates exactly this data source.
  • The occasional reversed decisions under confidence-based adoption imply that the assistant should expose not only feature importance but calibration information, such as how often similar suggestions led to correct outcomes, so analysts can adjust reliance dynamically.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces ContextBuddy, an imitation-learning assistant that models how security analysts select contextual information while triaging intrusion alerts, and evaluates it through staged simulation and a user study. Context selection is formulated as an MDP; simulated analysts are RL agents (A2C, PPO, DQN) trained with a hand-shaped reward; ContextBuddy is trained on their traces with BC, AIRL, and GAIL, and the best variant (GAIL_ALL) is then teamed with the same agents. The simulation results claim F1 improvements of 2.5% (HIKARI-2021) and 9% (UNSW-NB15), reduced false negatives, and improved decision confidence when simulated analysts adopt ContextBuddy suggestions. The user study, with N=13 non-experts in a within-subject design, reports that one-time assistance improves classification accuracy by 21.1% (p=0.008) and reduces alert validation time by 24% relative to the no-assistance baseline. The paper's central claim is that ContextBuddy learns context-selection patterns from analysts, but the primary demonstration uses synthetic RL analysts, and the human-study design does not isolate the learned selection policy from a simple reduction in the number of features shown to participants.

Significance. The problem addressed is important: SOC analysts face alert overload, and a system that learns which contextual cues matter for particular alerts could improve both accuracy and efficiency. The paper has several strengths: it gives an explicit MDP/IL formulation, compares three IL approaches (BC, AIRL, GAIL), uses two public intrusion-detection datasets, reports effect sizes and post-hoc power, and provides extensive appendices on feature groupings, alert selection, and questionnaire items. The user study was ethics-approved and used counterbalancing for the C1/C2 comparison. However, the load-bearing evidence for the headline claim is currently insufficient. The simulation arm is substantially self-referential: ContextBuddy is trained on trajectories from the same RL agents with which it is later teamed, and the simulated analysts reuse the same XGBoost classifiers. The user-study arm confounds the learned feature selection with feature-subset restriction, since C2 presents fewer features than C1 and no random-subset control is included.

major comments (4)
  1. [Sections V-F and VI-C] The simulation evaluation is circular with respect to the claim that ContextBuddy 'learns from analysts.' ContextBuddy's GAIL_ALL policy is trained on trajectories collected from A2C, PPO, and DQN agents, and it is later evaluated by teaming it with those same three agent types, which also reuse the same pre-trained XGBoost classifiers described in Section V-D. Reported F1 gains, false-negative reductions, and confidence improvements may therefore reflect the imitation policy's ability to reproduce the training agents' own context-request behavior rather than a capability that transfers to independent or unseen analysts. A stronger design would hold out one analyst type during training (e.g., train on A2C and PPO, evaluate with DQN) or use real analyst traces; Section X acknowledges the use of simulated analysts but does not address this specific circularity in the teaming evaluation.
  2. [Sections VII-B, VII-D, and VIII] The user-study comparison between C1 (all features visible) and C2 (only ContextBuddy-selected features) confounds the learned context-selection policy with the number of features shown to participants. Section VII-B states that participants in C2 are 'restricted to only the most relevant features suggested by it,' while C1 gives access to all features. The reported 21.1% accuracy gain and 24% time reduction could be driven by reduced information overload for non-experts rather than by the specific learned selections. The paper reports no control condition with a randomly selected feature subset of the same size, and no condition in which all features remain visible but ContextBuddy's suggestions are highlighted. Without such a control, the user study does not isolate the contribution of the learned policy, which is the central claim of the paper.
  3. [Section VII-E1 and Section VIII] The Iterative condition (C3) is always placed last and was not counterbalanced, and participants in C3 could toggle between all features and ContextBuddy-selected ones. As the authors note, placing C3 last may affect performance through fatigue, strategy changes, or learning, and the qualitative feedback in Section VIII indicates that participants perceived C3 as offering 'two versions' of ContextBuddy. This order and interface confound undermines the comparison between One-time and Iterative assistance reported in Table III and discussed in Sections VIII and IX, including the interpretation that iterative interactions 'may require higher analyst expertise.' The C3 results should either be counterbalanced in a future study, analyzed with an order-aware model, or explicitly reported as exploratory rather than as evidence about the interaction mode.
  4. [Section VI-C] The 'Random' baseline in the simulation randomizes whether the analyst adopts ContextBuddy's suggestion (50% probability), not the content of the suggestion. It therefore does not test whether the specific context categories selected by ContextBuddy's learned policy are responsible for the observed F1 and confidence gains, as opposed to any additional context features being appended to the analyst's feature set. A more informative control would present randomly selected context categories of the same cardinality as ContextBuddy's suggestions under the same confidence-based adoption rule. Without such a control, the simulation cannot distinguish 'ContextBuddy selects useful contexts' from 'any additional context helps these RL agents,' and the paper's discussion that 'ContextBuddy contributes meaningful contextual information' (Section VI-C) is not fully supported.
minor comments (5)
  1. [Figures 3-8] Several figure captions in the provided manuscript contain garbled '/uni...' tokens (e.g., Figures 3, 4, 5, 6, 7, and 8) that make them unreadable; the captions should be regenerated so that they clearly describe the plotted quantities.
  2. [Abstract and Section VIII] The abstract reports p=0.01 for the 24% time reduction, while Section VIII reports p=0.014 with an unadjusted p=0.005; the discrepancy in the corrected p-value should be resolved, and the multiple-comparison procedure (which tests, which correction, how many comparisons) should be stated precisely.
  3. [Table III] The first two numeric columns of Table III are labeled only 'CB' and 'Participant'; the 'CB' column appears to be the classifier-only baseline, but this is not defined in the table or in the surrounding text, making the comparison ambiguous for readers.
  4. [Section V-F] The text says '200,000 timestamps' where 'timesteps' is meant, both in the training-process description and in the following sentence; this typo should be corrected.
  5. [Section VII-D] The four alerts per condition are deliberately selected to represent one easy TP, one easy TN, one hard FP, and one hard FN. The paper should state explicitly whether participants were informed of this composition, because the small per-condition alert count and the deliberate mix of easy and hard cases affect how the accuracy and time results should be interpreted.

Circularity Check

1 steps flagged · score 6.0 of 10

Simulation evaluation is a closed loop: ContextBuddy is trained on RL-analyst traces and then evaluated by teaming with those same RL analysts and the same XGBoost classifiers; the user study is external but lacks a random-subset control.

  1. fitted input called prediction [Section V (Evaluation Setup, Simulation-Based), first paragraph; see also Sections V-F and VI-C]
    "To test ContextBuddy's effectiveness at helping these analysts improve their performance, we then tested ContextBuddy by teaming it with the same simulated analysts on a new set of events."

    The simulated analysts whose traces trained ContextBuddy (A2C, PPO, DQN; Section V-F collects 100 trajectories from each model) are exactly the agents used in the evaluation dyad (Section VI-C). The 'analyst' is an XGBoost classifier over context subsets, and a suggestion is accepted only if the same classifier's softmax confidence increases on the extended feature set. Reported gains such as F1 +2.5% (HIKARI) and +9% (UNSW) therefore largely measure how well the imitator re-proposes the imitated agents' own context requests to the shared classifier, not whether learned selection transfers to independent analysts.

full rationale

The strongest circularity is in the simulation arm. ContextBuddy (GAIL_ALL) is trained on trajectories from the same A2C/PPO/DQN simulated analysts it is later teamed with, and the acceptance rule is based on the same pre-trained XGBoost classifiers those analysts used. The resulting F1, accuracy, and confidence improvements therefore partly reduce to self-consistency between the imitator and the imitated agents plus the shared classifier's confidence behavior, rather than to a transferable learned context-selection policy. The 'Random' baseline randomizes adoption of ContextBuddy's suggestions, not the suggested contexts themselves, so it does not test whether the specific learned selections drive the gains. This makes the simulation's predictive claim partially circular. The user study supplies genuinely external evidence with non-expert humans, which prevents the entire paper from being fully circular. However, the user-study comparison of C1 (all features) versus C2 (only ContextBuddy-selected features) confounds learned context selection with feature-subset size; no random-subset control is reported. That is a validity concern rather than a definitional reduction, so it is not listed as an additional circular step. Self-citations appear in the paper, but none is load-bearing for the derivation: author-group works are used for framing human-AI teaming and augmentation, not as the source of the main quantitative claims. The paper itself acknowledges the simulated-analyst limitation in Section X ('We used simulated analysts; naturally, we need real-world SOC data'). Overall, the simulation-based demonstration is substantially self-referential, giving a partial circularity score of 6.

Assumptions & free parameters 3 free parameters · 4 assumptions · 1 invented entities

The central claim rests on the assumption that synthetic RL analysts can stand in for real SOC analysts, on a hand-shaped reward that defines good investigation behaviour, and on the reliability of XGBoost confidence and SHAP explanations. The free parameters are the reward coefficients and RL hyperparameters that shape the demonstrations; without them the imitation policy would be different. No new physical entities are postulated, but the RL-simulated analysts are an invented proxy with no independent evidence that they match human behaviour.

free parameters (3)
  • Analyst reward shaping constants (lambda1, lambda2, eta1, eta2, phi, psi, omega) = lambda1=-0.02, lambda2=-0.5, eta1=0.2, phi=10, psi=5, omega=-5; eta2 unspecified
    Hand-chosen constants defining simulated analyst behaviour; the quality of the synthetic demonstrations, and hence ContextBuddy's learned policy, depends on these values (Section V-D).
  • RL hyperparameters (gamma, entropy coefficient) = PPO/A2C/DQN gamma=0.99; ent_coef 0.01/0.001/0.12
    Selected via Optuna over 2000 trials (Section V-E); they shape the simulated analysts whose traces train ContextBuddy.
  • Confidence acceptance thresholds = 0.90, 0.80, 0.70, 0.60
    Independent variables in the teaming strategies (Section VI-C); the reported accuracy and confidence gains depend on these threshold choices.
assumptions (4)
  • domain assumption Analysts behave rationally and maximize expected reward in a Markov decision process.
    Section III-C states: 'We assume that analysts behave rationally, aiming to maximise expected rewards, the environment transitions are deterministic, and the set of possible contextual information types is finite and known.'
  • ad hoc to paper Reinforcement-learning agents trained with the hand-shaped reward are valid stand-ins for real SOC analysts.
    Introduced to generate training data when no analyst-annotated dataset exists (Section V); the validity of the entire simulation arm rests on this proxy assumption.
  • domain assumption XGBoost classifiers trained on feature subsets provide reliable class probabilities for confidence-based suggestion acceptance.
    Used throughout the simulation and SHAP explanations; no calibration analysis is provided.
  • domain assumption SHAP values accurately explain classifier decisions for non-expert users.
    Section II-C chooses SHAP based on prior literature; no user comprehension measurement beyond self-report is provided.
invented entities (1)
  • Simulated RL analysts (A2C, PPO, DQN agents)
    purpose: Generate investigation traces used as demonstrations for ContextBuddy and as teammates in the simulation evaluation.
    No evidence that their context-request behaviour matches real SOC analysts; the paper itself lists real-world SOC data as future work (Section X).

how reviews work

0 comments
Cite this review

Pith. "Pith review of ContextBuddy: AI-Enhanced Contextual Insights for Security Alert Investigation (Applied to Intrusion Detection)." pith.science (2026). https://pith.science/paper/4623BM4B

@misc{pith2026250609365,
  author       = {Pith},
  title        = {Pith review of: ContextBuddy: AI-Enhanced Contextual Insights for Security Alert Investigation (Applied to Intrusion Detection)},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4623BM4B}},
  note         = {Machine review of arXiv:2506.09365}
}
read the original abstract

Modern Security Operations Centres (SOCs) integrate diverse tools, such as SIEM, IDS, and XDR systems, offering rich contextual data, including alert enrichments, flow features, and similar case histories. Yet, analysts must still manually determine which of these contextual cues are most relevant when validating specific alerts. We introduce ContextBuddy, an AI assistant that learns from analysts' prior investigations to help them identify the most relevant context for new alerts. Rather than providing enrichments, ContextBuddy models how analysts have previously selected context and suggests tailored cues based on the characteristics of each alert. We formulate context selection as a sequential decision-making problem and apply imitation learning (IL) to capture analysts' strategies, evaluating multiple IL approaches. Through staged evaluation, we validate ContextBuddy using two intrusion detection datasets (HIKARI-2021, UNSW-NB15). In simulation-based experiments, ContextBuddy helped simulated reinforcement learning analysts improve classification accuracy (p < 0.001) (increasing F1 by 2.5% for HIKARI and 9% for UNSW), reducing false negatives (1.5% for HIKARI and 10% for UNSW), and keeping false positives below 1%. Decision confidence among agents also improved by 2-3% (p < 0.001). In a within-subject user study (N=13; power = 0.8), non-experts using ContextBuddy improved classification accuracy by 21.1% (p = 0.008) and reduced alert validation time by 24% (p = 0.01). These results demonstrate that by learning context-selection patterns from analysts, ContextBuddy can yield notable improvements in investigation effectiveness and efficiency.

Figures

Figures reproduced from arXiv: 2506.09365 by the authors.

Figure 1
Figure 1. ContextBuddy: an imitation learning assistant trained on prior investigation history to help analysts identify the relevant [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Main components of ContextBuddy and an example of a user’s interaction with it. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 4
Figure 4. Comparing F1 scores across RL and IL models for [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figures from the paper (25 more)
Figure 3
Figure 3. Figure 3: Comparing F1 scores across RL and IL models for [PITH_FULL_IMAGE:figures/full_fig_p007_3.png]
Figure 5
Figure 5. Figure 5: The performance gain when simulated analysts team [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: The performance gain when simulated analysts team [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 8
Figure 8. Figure 8: Improvement in FP and FN when simulated analysts [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 9
Figure 9. Figure 9: The figure shows how ContextBuddy’s suggestions [PITH_FULL_IMAGE:figures/full_fig_p009_9.png]
Figure 10
Figure 10. Figure 10: Class Distribution of Selected Alerts [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 11
Figure 11. Figure 11: Class distribution of selected alerts for the UNSW [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]
Figure 10
Figure 10. Figure 10: Class Distribution of selected alerts matching the [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]
Figure 12
Figure 12. Figure 12: High-level overview of participant task. [PITH_FULL_IMAGE:figures/full_fig_p019_12.png]
Figure 13
Figure 13. Figure 13: The alerts are displayed in a table [PITH_FULL_IMAGE:figures/full_fig_p021_13.png]
Figure 14
Figure 14. Figure 14: Example of feature visualisation showing key statistics (mean, median, and mode) for an alert. [PITH_FULL_IMAGE:figures/full_fig_p021_14.png]
Figure 15
Figure 15. Figure 15: Explanation of feature visualisation [PITH_FULL_IMAGE:figures/full_fig_p021_15.png]
Figure 16
Figure 16. Figure 16: Iterative Panel: Allows participants to filter features. This panel is present in C3: Iterative assistance condition only. [PITH_FULL_IMAGE:figures/full_fig_p022_16.png]
Figure 17
Figure 17. Figure 17: AI-generated explanations displaying feature importance values for different classes. [PITH_FULL_IMAGE:figures/full_fig_p022_17.png]
Figure 18
Figure 18. Figure 18: Discussion of how to interpret explainability information. [PITH_FULL_IMAGE:figures/full_fig_p023_18.png]
Figure 19
Figure 19. Figure 19: Final classification decision panel where participants submit their classification and report confidence. [PITH_FULL_IMAGE:figures/full_fig_p023_19.png]
Figure 20
Figure 20. Figure 20: User accuracies. 0 200 400 600 Analyst One−time Iterative Condition Completion Time ± SD (s) Completion Time by Condition [PITH_FULL_IMAGE:figures/full_fig_p024_20.png]
Figure 21
Figure 21. Figure 21: Completion times. 40 60 80 100 Analyst One−time Iterative Condition Confidence ± SD (%) Confidence by Condition [PITH_FULL_IMAGE:figures/full_fig_p024_21.png]
Figure 25
Figure 25. Figure 25: Simulated Analyst Training Process streamlining the classification process. J. Testing the Simulated Analysts and ContextBuddy Dyad Once the simulated analysts and ContextBuddy are trained, the next step is testing their collaboration. In this phase, the simulated ana…
Figure 26
Figure 26. Figure 26: ContextBuddy (AIRL Assistant) Training Process [PITH_FULL_IMAGE:figures/full_fig_p026_26.png]
Figure 27
Figure 27. Figure 27: Testing the Simulated Analyst-ContextBuddy Dyad in simulation-based experiments. [PITH_FULL_IMAGE:figures/full_fig_p026_27.png]
Figure 28
Figure 28. Figure 28: Breakdown of performance by individual analysts [PITH_FULL_IMAGE:figures/full_fig_p027_28.png]
Figure 32
Figure 32. Figure 32: The analysts’ prediction confidence (Hikari) [PITH_FULL_IMAGE:figures/full_fig_p027_32.png]
Figure 30
Figure 30. Figure 30: Breakdown of performance by individual analysts [PITH_FULL_IMAGE:figures/full_fig_p027_30.png]
Figure 33
Figure 33. Figure 33: The analysts’ prediction confidence (UNSW) [PITH_FULL_IMAGE:figures/full_fig_p027_33.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

98 extracted references · 77 canonical work pages

  1. [1]

    Sans soc survey 2024,

    C. Crowley, “Sans soc survey 2024,” tech. rep., 2024

    2024

  2. [2]

    99% false positives: A qualitative study of{soc}analysts’ perspectives on security alarms,

    B. A. Alahmadi, L. Axon, and I. Martinovic, “99% false positives: A qualitative study of{soc}analysts’ perspectives on security alarms,” in31st USENIX Security Symposium (USENIX Security 22), pp. 2783– 2800, usenix.org, 2022

    2022

  3. [3]

    Knerler, I

    K. Knerler, I. Parker, and C. Zimmerman,11 Strategies of a World-Class Cybersecurity Operations Center. MITRE, 2023

    2023

  4. [4]

    Towards human-AI teaming to mitigate alert fatigue in security operations centres,

    M. Baruwal Chhetri, S. Tariq, R. Singh, F. Jalalvand, C. Paris, and S. Nepal, “Towards human-AI teaming to mitigate alert fatigue in security operations centres,”ACM Trans. Internet Technol., vol. 24, pp. 1–22, Aug. 2024

    2024

  5. [5]

    Are we there yet? an industrial viewpoint on provenance-based endpoint detection and response tools,

    F. Dong, S. Li, P. Jiang, D. Li, H. Wang, L. Huang, X. Xiao, J. Chen, X. Luo, Y . Guo, and X. Chen, “Are we there yet? an industrial viewpoint on provenance-based endpoint detection and response tools,” inProceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, CCS ’23, (New York, NY , USA), pp. 2396– 2410, Association for Co...

    2023

  6. [6]

    Tactical provenance analysis for endpoint detection and response systems,

    W. U. Hassan, A. Bates, and D. Marino, “Tactical provenance analysis for endpoint detection and response systems,” in2020 IEEE Symposium on Security and Privacy (SP), pp. 1172–1189, IEEE, May 2020

    2020

  7. [7]

    DEEPCASE: Semi-Supervised contextual analysis of security events,

    T. van Ede, H. Aghakhani, N. Spahn, R. Bortolameotti, M. Cova, A. Continella, M. van Steen, A. Peter, C. Kruegel, and G. Vigna, “DEEPCASE: Semi-Supervised contextual analysis of security events,” in2022 IEEE Symposium on Security and Privacy (SP), pp. 522–539, IEEE, May 2022

    2022

  8. [8]

    ATTACK2VEC: Leveraging Temporal Word Embeddings to Understand the Evolution of Cyberattacks

    Y . Shen and G. Stringhini, “ATTACK2VEC: Leveraging temporal word embeddings to understand the evolution of cyberattacks,”USENIX Secur Symp, vol. abs/1905.12590, May 2019

    work page Pith review arXiv 1905

Show all 98 references
  1. [9]

    SeqMask: Behavior extraction over cyber threat intelligence via multi-instance learning,

    W. Ge and J. Wang, “SeqMask: Behavior extraction over cyber threat intelligence via multi-instance learning,”Comput. J., vol. 67, pp. 253– 273, Nov. 2022

    2022

  2. [10]

    Trident: A universal framework for fine-grained and class-incremental unknown traffic detec- tion,

    Z. Zhao, Z. Li, Z. Song, W. Li, and F. Zhang, “Trident: A universal framework for fine-grained and class-incremental unknown traffic detec- tion,” inProceedings of the ACM Web Conference 2024, vol. 3, (New York, NY , USA), pp. 1608–1619, ACM, May 2024

    2024

  3. [11]

    Effect of automation transparency in the management of multiple unmanned vehicles,

    A. Bhaskara, L. Duong, J. Brooks, R. Li, R. McInerney, M. Skinner, H. Pongracic, and S. Loft, “Effect of automation transparency in the management of multiple unmanned vehicles,”Appl. Ergon., vol. 90, p. 103243, Jan. 2021

    2021

  4. [12]

    AI-driven guided response for security operation centers with microsoft copilot for security,

    S. Freitas, J. Kalajdjieski, A. Gharib, and R. McCann, “AI-driven guided response for security operation centers with microsoft copilot for security,”arXiv [cs.LG], July 2024

    2024

  5. [13]

    Impact of team collaboration on cybersecurity situational awareness,

    P. Rajivan and N. Cooke, “Impact of team collaboration on cybersecurity situational awareness,” inTHEORY AND MODELS FOR CYBER SITU- ATION AWARENESS(P. Liu, S. Jajodia, and C. Wang, eds.), vol. 10030 ofLecture Notes in Computer Science, pp. 203 – 226, Springer Verlag, 2017

    2017

  6. [14]

    Opportunities and challenges for Human-Machine teaming in cybersecurity operations,

    C. Lyn Paul, L. M. Blaha, C. K. Fallon, C. Gonzalez, and R. S. Gutzwiller, “Opportunities and challenges for Human-Machine teaming in cybersecurity operations,”Proc. Hum. Fact. Ergon. Soc. Annu. Meet., vol. 63, pp. 442–446, Nov. 2019

    2019

  7. [15]

    Considerations for Human- Machine teaming in cybersecurity,

    S. R. Gomez, V . Mancuso, and D. Staheli, “Considerations for Human- Machine teaming in cybersecurity,” inAugmented Cognition: 13th International Conference, AC 2019, Held as Part of the 21st HCI International Conference, HCII 2019, Orlando, FL, USA, July 26–31, 2019, Proceedi...

    2019

  8. [16]

    Adapt and overcome: Perceptions of adaptive autonomous agents for Human-AI teaming,

    A. I. Hauptman, B. G. Schelble, N. J. McNeese, and K. C. Madathil, “Adapt and overcome: Perceptions of adaptive autonomous agents for Human-AI teaming,”Comput. Human Behav., vol. 138, p. 107451, Jan. 2023

    2023

  9. [17]

    Machines as teammates: A research agenda on AI in team collaboration,

    I. Seeber, E. Bittner, R. O. Briggs, T. de Vreede, G.-J. de Vreede, A. Elkins, R. Maier, A. B. Merz, S. Oeste-Reiß, N. Randrup, G. Schwabe, and M. S ¨ollner, “Machines as teammates: A research agenda on AI in team collaboration,”Inf. Manag., vol. 57, p. 103174, Mar. 2020

    2020

  10. [18]

    Human-machine teaming is key to AI adoption: clinicians’ experiences with a deployed machine learning system,

    K. E. Henry, R. Kornfield, A. Sridharan, R. C. Linton, C. Groh, T. Wang, A. Wu, B. Mutlu, and S. Saria, “Human-machine teaming is key to AI adoption: clinicians’ experiences with a deployed machine learning system,”NPJ Digit. Med., vol. 5, p. 97, July 2022

    2022

  11. [19]

    A2C: A modular multi- stage collaborative decision framework for human-AI teams,

    S. Tariq, M. B. Chhetri, S. Nepal, and C. Paris, “A2C: A modular multi- stage collaborative decision framework for human-AI teams,”ArXiv, vol. abs/2401.14432, Jan. 2024

    2024 arXiv

  12. [20]

    Com- bating alert fatigue with AlertPro: Context-aware alert prioritization using reinforcement learning for multi-step attack detection,

    X. Wang, X. Yang, X. Liang, X. Zhang, W. Zhang, and X. Gong, “Com- bating alert fatigue with AlertPro: Context-aware alert prioritization using reinforcement learning for multi-step attack detection,”Comput. Secur., vol. 137, p. 103583, Feb. 2024

    2024

  13. [21]

    Trust my IDS: An explainable AI integrated deep learning-based transparent threat detection system for industrial networks,

    S. Shoukat, T. Gao, D. Javeed, M. S. Saeed, and M. Adil, “Trust my IDS: An explainable AI integrated deep learning-based transparent threat detection system for industrial networks,”Computers & Security, Nov. 2024

    2024

  14. [22]

    Human-in-the-loop cyber intrusion detection using active learning,

    Y . Kim, G. D ´an, and Q. Zhu, “Human-in-the-loop cyber intrusion detection using active learning,”IEEE Trans. Inf. Forensics Secur., vol. PP, no. 99, pp. 1–1, 2024

    2024

  15. [23]

    Everybody’s got ML, tell me what else you have: Practitioners’ perception of ML-Based security tools and explanations,

    J. Mink, H. Benkraouda, L. Yang, A. Ciptadi, A. Ahmadzadeh, D. V otipka, and G. Wang, “Everybody’s got ML, tell me what else you have: Practitioners’ perception of ML-Based security tools and explanations,” in2023 IEEE Symposium on Security and Privacy (SP), pp. 2068–2085, lim...

    2023

  16. [24]

    An assessment of the usability of machine learning based tools for the security operations center,

    S. Oesch, R. Bridges, J. Smith, J. Beaver, J. Goodall, K. Huffer, C. Miles, and D. Scofield, “An assessment of the usability of machine learning based tools for the security operations center,” in2020 International Conferences on Internet of Things (iThings) and IEEE Green Com...

    2020

  17. [25]

    Asynchronous methods for deep reinforcement learning,

    V . Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu, “Asynchronous methods for deep reinforcement learning,” inProceedings of The 33rd International Con- ference on Machine Learning(M. F. Balcan and K. Q. Weinberger, eds.), vol. 4...

    1928

  18. [26]

    Proximal policy optimization algorithms,

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,”arXiv [cs.LG], July 2017

    2017

  19. [27]

    Human-level control through deep reinforcement learning,

    V . Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis, “Human-level control through deep...

    2015

  20. [28]

    Generating network intrusion detection dataset based on real and encrypted synthetic attack traffic,

    A. Ferriyan, A. H. Thamrin, K. Takeda, and J. Murai, “Generating network intrusion detection dataset based on real and encrypted synthetic attack traffic,”NATO Adv. Sci. Inst. Ser. E Appl. Sci., vol. 11, p. 7868, Aug. 2021

    2021

  21. [29]

    UNSW-NB15: a comprehensive data set for network intrusion detection systems (UNSW-NB15 network data set),

    N. Moustafa and J. Slay, “UNSW-NB15: a comprehensive data set for network intrusion detection systems (UNSW-NB15 network data set),” in2015 Military Communications and Information Systems Conference (MilCIS), pp. 1–6, IEEE, Nov. 2015

    2015

  22. [30]

    Learning robust rewards with adversarial inverse reinforcement learning,

    J. Fu, K. Luo, and S. Levine, “Learning robust rewards with adversarial inverse reinforcement learning,” Oct. 2017

    2017

  23. [31]

    Generative adversarial imitation learning,

    J. Ho and S. Ermon, “Generative adversarial imitation learning,”Adv. Neural Inf. Process. Syst., 2016

    2016

  24. [32]

    Behavioral cloning from observa- tion,

    F. Torabi, G. Warnell, and P. Stone, “Behavioral cloning from observa- tion,”arXiv [cs.AI], May 2018

    2018

  25. [33]

    Dos and don’ts of machine learning in computer security,

    D. Arp, E. Quiring, F. Pendlebury, A. Warnecke, F. Pierazzi, C. Wress- negger, L. Cavallaro, and K. Rieck, “Dos and don’ts of machine learning in computer security,” in31st USENIX Security Symposium (USENIX Security 22), pp. 3971–3988, usenix.org, 2022

    2022

  26. [34]

    Deep learning for cyber security intrusion detection: Approaches, datasets, and comparative study,

    M. A. Ferrag, L. Maglaras, S. Moschoyiannis, and H. Janicke, “Deep learning for cyber security intrusion detection: Approaches, datasets, and comparative study,”Journal of Information Security and Applications, vol. 50, p. 102419, Feb. 2020

    2020

  27. [35]

    UNICORN: Runtime provenance-based detector for advanced persistent threats,

    X. Han, T. Pasquier, A. Bates, J. Mickens, and M. Seltzer, “UNICORN: Runtime provenance-based detector for advanced persistent threats,” in 27th Annual Network and Distributed System Security Symposium, NDSS 2020, Jan. 2020

    2020

  28. [36]

    MAlign: Explainable static raw- byte based malware family classification using sequence alignment,

    S. Saha, S. Afroz, and A. H. Rahman, “MAlign: Explainable static raw- byte based malware family classification using sequence alignment,” Comput. Secur., vol. 139, p. 103714, Apr. 2024

    2024

  29. [37]

    An overview of the strategic advan- tages of AI-Powered threat intelligence in the cloud,

    V . Mallikarjunaradhya and others, “An overview of the strategic advan- tages of AI-Powered threat intelligence in the cloud,”J. Sci., 2023

    2023

  30. [38]

    Kovacevic and N

    B. Kovacevic and N. DiCola,Security Orchestration, Automation, and Response for Security Analysts: Learn the secrets of SOAR to improve MTTA and MTTR and strengthen your organization’s security posture. Packt Publishing Ltd, July 2023

    2023

  31. [39]

    Context2Vector: Accelerating security event triage via context representation learning,

    J. Liu, R. Zhang, W. Liu, Y . Zhang, D. Gu, M. Tong, X. Wang, J. Xue, and H. Wang, “Context2Vector: Accelerating security event triage via context representation learning,”Information and Software Technology, vol. 146, p. 106856, June 2022

    2022

  32. [40]

    RAPID: Real-Time alert investigation with context-aware prioritization for efficient threat discovery,

    Y . Liu, X. Shu, Y . Sun, J. Jang, and P. Mittal, “RAPID: Real-Time alert investigation with context-aware prioritization for efficient threat discovery,” inProceedings of the 38th Annual Computer Security Applications Conference, ACSAC ’22, (New York, NY , USA), pp. 827– 840,...

    2022

  33. [41]

    DrSec: Flexible distributed representations for efficient endpoint security,

    M. Sharif, P. Datta, A. Riddle, K. Westfall, A. Bates, V . Ganti, M. Lentz, and D. Ott, “DrSec: Flexible distributed representations for efficient endpoint security,” in2024 IEEE Symposium on Security and Privacy (SP), pp. 145–145, IEEE Computer Society, 2024

    2024

  34. [42]

    MStream: Fast anomaly detection in multi-aspect streams,

    S. Bhatia, A. Jain, P. Li, R. Kumar, and B. Hooi, “MStream: Fast anomaly detection in multi-aspect streams,” inProceedings of the Web Conference 2021, (New York, NY , USA), ACM, Apr. 2021

    2021

  35. [43]

    Net-track: Generic web tracking detection using packet metadata,

    D. Lee, M. Joo, and W. Lee, “Net-track: Generic web tracking detection using packet metadata,” inProceedings of the ACM Web Conference 2023, (New York, NY , USA), ACM, Apr. 2023

    2023

  36. [44]

    Flow-MAE: Leveraging masked AutoEncoder for accurate, efficient and robust malicious traffic classifi- cation,

    Z. Hang, Y . Lu, Y . Wang, and Y . Xie, “Flow-MAE: Leveraging masked AutoEncoder for accurate, efficient and robust malicious traffic classifi- cation,” inProceedings of the 26th International Symposium on Research in Attacks, Intrusions and Defenses, (New York, NY , USA), ACM...

    2023

  37. [45]

    FlowTransformer: A transformer framework for flow-based network intrusion detection systems,

    L. D. Manocchio, S. Layeghy, W. W. Lo, G. K. Kulatilleke, M. Sarhan, and M. Portmann, “FlowTransformer: A transformer framework for flow-based network intrusion detection systems,”Expert Syst. Appl., vol. 241, p. 122564, May 2024

    2024

  38. [46]

    Who goes first? influences of Human-AI workflow on decision making in clinical imaging,

    R. Fogliato, S. Chappidi, M. Lungren, P. Fisher, D. Wilson, M. Fitzke, M. Parkinson, E. Horvitz, K. Inkpen, and B. Nushi, “Who goes first? influences of Human-AI workflow on decision making in clinical imaging,” inProceedings of the 2022 ACM Conference on Fairness, Accountabil...

    2022

  39. [47]

    To trust or to think: Cognitive forcing functions can reduce overreliance on AI in AI-assisted decision-making,

    Z. Buc ¸inca, M. B. Malaya, and K. Z. Gajos, “To trust or to think: Cognitive forcing functions can reduce overreliance on AI in AI-assisted decision-making,”Proc. ACM Hum.-Comput. Interact., vol. 5, pp. 1–21, Apr. 2021

    2021

  40. [48]

    Do People Engage Cognitively with AI? Impact of AI Assistance on Incidental Learning,

    K. Z. Gajos and L. Mamykina, “Do People Engage Cognitively with AI? Impact of AI Assistance on Incidental Learning,” in27th International Conference on Intelligent User Interfaces, IUI ’22, (New York, NY , USA), pp. 794–806, Association for Computing Machinery, Mar. 2022

    2022

  41. [49]

    Explainable AI is dead, long live explainable AI! hypothesis- driven decision support using evaluative AI,

    T. Miller, “Explainable AI is dead, long live explainable AI! hypothesis- driven decision support using evaluative AI,” inProceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’23, (New York, NY , USA), pp. 333–342, Association for Computi...

    2023

  42. [50]

    Towards the new XAI: A Hypothesis-Driven approach to decision support using evidence,

    T. Le, T. Miller, R. Singh, and L. Sonenberg, “Towards the new XAI: A Hypothesis-Driven approach to decision support using evidence,” Feb. 2024

    2024

  43. [51]

    The effects of explanations on automation bias,

    M. Vered, T. Livni, P. D. L. Howe, T. Miller, and L. Sonenberg, “The effects of explanations on automation bias,”Artif. Intell., vol. 322, p. 103952, Sept. 2023

    2023

  44. [52]

    Shneiderman,Human-Centered AI

    B. Shneiderman,Human-Centered AI. Oxford University Press, Jan. 2022

    2022

  45. [53]

    A unified approach to interpreting model predictions,

    S. M. Lundberg and S.-I. Lee, “A unified approach to interpreting model predictions,”Adv. Neural Inf. Process. Syst., pp. 4765–4774, May 2017

    2017

  46. [54]

    A value for n-person games,

    L. S. Shapley, “A value for n-person games,”Contributions to the Theory of Games, 1953

    1953

  47. [55]

    A systematic literature review for network intrusion detection system (IDS),

    O. H. Abdulganiyu, T. Ait Tchakoucht, and Y . K. Saheed, “A systematic literature review for network intrusion detection system (IDS),”Int. J. Inf. Secur., vol. 22, pp. 1125–1162, Oct. 2023

    2023

  48. [56]

    Explainable artificial intelligence (XAI) for intrusion detection and mitigation in intelligent connected vehicles: A review,

    C. I. Nwakanma, L. A. C. Ahakonye, J. N. Njoku, J. C. Odirichukwu, S. A. Okolie, C. Uzondu, C. C. Ndubuisi Nweke, and D.-S. Kim, “Explainable artificial intelligence (XAI) for intrusion detection and mitigation in intelligent connected vehicles: A review,”NATO Adv. Sci. Inst. ...

    2023

  49. [57]

    Explainable AI for intrusion detection systems: A model development and experts’ evaluation,

    H. Durojaye and M. Naiseh, “Explainable AI for intrusion detection systems: A model development and experts’ evaluation,” inLecture Notes in Networks and Systems, Lecture notes in networks and systems, pp. 301–318, Cham: Springer Nature Switzerland, 2024

    2024

  50. [58]

    Explainable AI for comparative analysis of intrusion detection models,

    P. M. Corea, Y . Liu, J. Wang, S. Niu, and H. Song, “Explainable AI for comparative analysis of intrusion detection models,” June 2024

    2024

  51. [59]

    XAI-IDS: Toward proposing an explainable artificial intelligence framework for enhancing network intrusion detection systems,

    O. Arreche, T. Guntur, and M. Abdallah, “XAI-IDS: Toward proposing an explainable artificial intelligence framework for enhancing network intrusion detection systems,”NATO Adv. Sci. Inst. Ser. E Appl. Sci., vol. 14, p. 4170, May 2024

    2024

  52. [60]

    Towards understanding alerts raised by unsupervised network intrusion detection systems,

    M. Lanvin, P.-F. Gimenez, Y . Han, F. Majorczyk, L. M ´e, and E. Totel, “Towards understanding alerts raised by unsupervised network intrusion detection systems,” inProceedings of the 26th International Symposium on Research in Attacks, Intrusions and Defenses, RAID ’23, (New ...

    2023

  53. [61]

    Robust network intru- sion detection through explainable artificial intelligence (XAI),

    P. Barnard, N. Marchetti, and L. A. DaSilva, “Robust network intru- sion detection through explainable artificial intelligence (XAI),”IEEE Networking Letters, vol. 4, pp. 167–171, Sept. 2022

    2022

  54. [62]

    Evaluating feature relevance XAI in network intrusion detection,

    J. Tritscher, M. Wolf, A. Hotho, and D. Schl ¨or, “Evaluating feature relevance XAI in network intrusion detection,” inExplainable Artificial Intelligence, pp. 483–497, Springer Nature Switzerland, 2023

    2023

  55. [63]

    M. L. Puterman,Markov Decision Processes: Discrete Stochastic Dy- namic Programming. John Wiley & Sons, Aug. 2014

    2014

  56. [64]

    A survey of imitation learning: Algorithms, recent developments, and challenges,

    M. Zare, P. M. Kebria, A. Khosravi, and S. Nahavandi, “A survey of imitation learning: Algorithms, recent developments, and challenges,” Sept. 2023

    2023

  57. [65]

    Imitation learning: Progress, taxonomies and challenges,

    B. Zheng, S. Verma, J. Zhou, I. W. Tsang, and F. Chen, “Imitation learning: Progress, taxonomies and challenges,”IEEE Trans Neural Netw Learn Syst, vol. PP, Oct. 2022

    2022

  58. [66]

    Contextual information fusion for in- trusion detection: a survey and taxonomy,

    A. Aleroud and G. Karabatis, “Contextual information fusion for in- trusion detection: a survey and taxonomy,”Knowl. Inf. Syst., vol. 52, pp. 563–619, Sept. 2017

    2017

  59. [67]

    Network traffic anomaly detection model based on feature grouping and multi-autoencoders integration,

    Y . Zhou, H. Zeng, Z. Zheng, and W. Zhang, “Network traffic anomaly detection model based on feature grouping and multi-autoencoders integration,”Electron. Lett., vol. 60, Dec. 2024

    2024

  60. [68]

    FSS-PART: Feature grouping subset model for predicting network attacks,

    R. Shanker, V . Madaan, and P. Agrawal, “FSS-PART: Feature grouping subset model for predicting network attacks,”SN Comput. Sci., vol. 5, Dec. 2023

    2023

  61. [69]

    A lightweight and efficient IoT intrusion detection method based on feature grouping,

    M. He, Y . Huang, X. Wang, P. Wei, and X. Wang, “A lightweight and efficient IoT intrusion detection method based on feature grouping,” IEEE Internet Things J., vol. 11, pp. 2935–2949, Jan. 2024

    2024

  62. [70]

    Muniz,The Modern Security Operations Center

    J. Muniz,The Modern Security Operations Center. Addison-Wesley Professional, Apr. 2021

    2021

  63. [71]

    Learning from class-imbalanced data: Review of methods and applications,

    G. Haixiang, L. Yijing, J. Shang, G. Mingyun, H. Yuanyue, and G. Bing, “Learning from class-imbalanced data: Review of methods and applications,”Expert Syst. Appl., vol. 73, pp. 220–239, May 2017

    2017

  64. [72]

    Machine learning and large language models-based techniques for cyber threat detection: A comparative study,

    A. Abdennebi, R. Morsli, N. Kara, and H. Ould-Slimane, “Machine learning and large language models-based techniques for cyber threat detection: A comparative study,” in2024 7th Conference on Cloud and Internet of Things (CIoT), pp. 1–9, IEEE, 2024

    2024

  65. [73]

    Efficient anomaly detection in tabular cybersecurity data using large language models,

    X. Zhao, X. Leng, L. Wang, N. Wang, and Y . Liu, “Efficient anomaly detection in tabular cybersecurity data using large language models,” Scientific Reports, vol. 15, no. 1, p. 3344, 2025

    2025

  66. [74]

    Anomaly detection in fog computing architectures using custom tab transformer for internet of things,

    A. I. Alzahrani, A. Al-Rasheed, A. Ksibi, M. Ayadi, M. M. Asiri, and M. Zakariah, “Anomaly detection in fog computing architectures using custom tab transformer for internet of things,”Electronics, vol. 11, no. 23, p. 4017, 2022

    2022

  67. [75]

    Foundations for an empirically determined scale of trust in automated systems,

    J.-Y . Jian, A. M. Bisantz, and C. G. Drury, “Foundations for an empirically determined scale of trust in automated systems,”Int. J. Cogn. Ergon., vol. 4, pp. 53–71, Mar. 2000

    2000

  68. [76]

    Development and validation of two instruments measuring intrinsic, extraneous, and germane cognitive load,

    M. Klepsch, F. Schmitz, and T. Seufert, “Development and validation of two instruments measuring intrinsic, extraneous, and germane cognitive load,”Front. Psychol., vol. 8, p. 1997, Nov. 2017

    1997

  69. [77]

    Designing for appropriate reliance: The roles of ai uncertainty presentation, initial user decision, and user demographics in ai-assisted decision-making,

    S. Cao, A. Liu, and C.-M. Huang, “Designing for appropriate reliance: The roles of ai uncertainty presentation, initial user decision, and user demographics in ai-assisted decision-making,”Proceedings of the ACM on Human-Computer Interaction, vol. 8, no. CSCW1, pp. 1–32, 2024

    2024

  70. [78]

    Mix and match: Charac- terizing heterogeneous human behavior in ai-assisted decision making,

    Z. Lu, S. H. A. Mahmoo, Z. Li, and M. Yin, “Mix and match: Charac- terizing heterogeneous human behavior in ai-assisted decision making,” inProceedings of the AAAI Conference on Human Computation and Crowdsourcing, vol. 12, pp. 95–104, 2024

    2024

  71. [79]

    Fostering appropriate reliance on large language models: The role of explanations, sources, and inconsistencies,

    S. S. Kim, J. W. Vaughan, Q. V . Liao, T. Lombrozo, and O. Rus- sakovsky, “Fostering appropriate reliance on large language models: The role of explanations, sources, and inconsistencies,”arXiv preprint arXiv:2502.08554, 2025. APPENDIX APPENDIX: FIGURES ANDTABLESSUMMARY •Figur...

    2025 arXiv

  72. [80]

    Alerts:Participants begin by reviewing the raw alert data, which presents key features such asflow duration,packet counts, andflag counts(Figure 13)

  73. [81]

    See Figure 15 to see how participants were told they could use it

    Feature Visualisation:The visualisation compares the alert’s feature values with statistical summaries computed from past alerts (Figure 14). See Figure 15 to see how participants were told they could use it

  74. [82]

    (Figure 16)

    Interaction Panel:This panel provides iterative assistance (only in Iterative condition) by allowing participants to selectively filter and focus on all, ContextBuddy selected, or participant-selected features, which are then visualised and explained in a more tailored manner....

  75. [83]

    These explanations show each feature’s positive or negative contribution towards the alert being classified into each class (Figure 17)

    AI-Generated Explanations:Alongside the feature visual- isations, the AI provides explanations in the form of feature im- portance values, computed using Shapley values. These explanations show each feature’s positive or negative contribution towards the alert being classified...

  76. [84]

    They will also report their confidence level in the decision and indicate to what extent they relied on the visualisations, explanations, and prior knowledge (Figure 19)

    Final Classification Decision:After reviewing the feature visualisations and AI explanations, participants are asked to classify the alert into one of six potential classes. They will also report their confidence level in the decision and indicate to what extent they relied on...

  77. [85]

    We explicitly instructed them that they were rating the AI and the explanations that it generates relative to the task and not their trust in AI in general

    Trust in AI Explanations:Participants were asked to express their trust in the AI and its provided explanations. We explicitly instructed them that they were rating the AI and the explanations that it generates relative to the task and not their trust in AI in general. They ra...

  78. [86]

    Perceived Cognitive Load:Participants were also asked to express their perceived cognitive load while performing the task of classifying intrusion alerts. The cognitive load was rated on a 7- point scale, with 1 being ”Very low” and 7 being ”Very high.” The items included in t...

  79. [87]

    The analyst is allocated an alert and provided with an initial subset of features

  80. [88]

    Based on this initial information, the analyst requests additional context (e.g., packet count) as needed. Fig. 13: The alerts are displayed in a table. Fig. 14: Example of feature visualisation showing key statistics (mean, median, and mode) for an alert. Fig. 15: Explanation...

  81. [89]

    The requested subset of features is returned to the analyst

  82. [90]

    This iterative process allows the simulated analysts to learn how to classify alerts efficiently by requesting the most relevant context features

    The analyst appends the additional features to the initial subset and uses a machine learning model (e.g., XGBoost classifier) to predict the alert type. This iterative process allows the simulated analysts to learn how to classify alerts efficiently by requesting the most rel...

  83. [91]

    The system collects the alerts and context features that the analysts explored during the training phase

  84. [92]

    18: Discussion of how to interpret explainability information

    These data are used to train ContextBuddy, which em- ploys AIRL to learn a policy that mimics the analysts’ Fig. 18: Discussion of how to interpret explainability information. Fig. 19: Final classification decision panel where participants submit their classification and repor...

  85. [93]

    After training, ContextBuddy can assist analysts by sug- gesting relevant context features for new alerts, effectively Fig

    ContextBuddy generalises from this training data and develops the ability to identify the most relevant context features for future alerts. After training, ContextBuddy can assist analysts by sug- gesting relevant context features for new alerts, effectively Fig. 25: Simulated...

  86. [94]

    The SOC agent allocates an alert and provides the simu- lated analyst with an initial subset of features

  87. [95]

    The analyst can request further contextual information as needed, while ContextBuddy assists by suggesting the most relevant context based on its learned policy

  88. [96]

    This collaboration ensures that the analyst and Con- textBuddy contribute to the alert classification process, leveraging each party’s strengths

    The analyst makes a final decision about the alert classi- fication using a machine learning classifier. This collaboration ensures that the analyst and Con- textBuddy contribute to the alert classification process, leveraging each party’s strengths. ContextBuddy assists by re...

  89. [97]

    Re- call that the results reported in the main paper are aggregated results

    Breakdown of Simulated Analysts’ Performance by Indi- vidual Analysts:The following section provides a breakdown of analysts’ performance by individual analyst instances. Re- call that the results reported in the main paper are aggregated results

  90. [98]

    26: ContextBuddy (AIRL Assistant) Training Process Fig

    Analysts’ Confidence When Working Alone vs With Con- textBuddy:The following plots show the effect on prediction confidence when simulated analysts were assisted by Con- textBuddy Fig. 26: ContextBuddy (AIRL Assistant) Training Process Fig. 27: Testing the Simulated Analyst-Co...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.