Pith. sign in

REVIEW 3 major objections 4 minor 22 references

Guarding Against Malicious Biased Threats (GAMBiT) Experiments: Revealing Cognitive Bias in Human-Subjects Red-Team Cyber Range Operations

T0 review · 3 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Three new datasets link real attackers' keystrokes and traffic to planted cognitive-bias traps.

desk verdict GAMBiT is a genuinely substantial multi-modal red-team telemetry release, but the bias triggers and EKM labels are unvalidated, so the title's 'revealing' goes beyond what the paper demonstrates. read the letter →

arxiv 2508.20963 v1 pith:BGU4PEEV submitted 2025-08-28 cs.CR cs.GT

Brandon Beltz , Jim Doty , Yvonne Fonken , Nikolos Gurney , Brett Israelsen , Nathan Lau , Stacy Marsella , Rachelle Thomas
show 4 more authors
Stoney Trent Peggy Wu Ya-Ting Yang Quanyan Zhu
This is my paper · ORCID
classification cs.CRcs.GT
keywords cognitivebiasredteamcyberrangeattackerbehaviormodelinghumansubjectsnetworktelemetryhostdataset
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper releases three large-scale datasets of skilled human attackers conducting red-team operations in a realistic simulated enterprise network, with every session instrumented from three angles at once: host telemetry, network telemetry, and human-factors data. Across the three experiments, roughly 20 attackers per experiment each spent two eight-hour days on self-directed attack objectives, starting from a standardized Linux workstation inside a mid-size enterprise network. The central idea is that the scenarios embed controlled cognitive-bias triggers—a notes file offering risky high-value credentials, misleadingly named admin accounts, conspicuous decoy files, and dead-end artifacts—designed to elicit loss aversion, base-rate neglect, availability bias, confirmation bias, and sunk-cost fallacy. By synchronizing trigger encounters with everything the attacker did before and after, the release aims to be the first dataset that lets researchers connect observable intrusion behavior to the cognitive biases that shape it, and to serve as a benchmark for bias-aware defenses. If the dataset works as intended, it closes a gap between psychological theories of judgment and operational attacker behavior.

What carries the argument

The central mechanism is the cognitive-bias trigger: a plausible, scenario-embedded artifact—a notes file dangling credentials with higher detection risk, misleadingly named admin accounts, salient decoy files, or aliased commands that dead-end—designed to elicit a specific bias (loss aversion, base-rate neglect, availability, confirmation, sunk cost) and placed along common attack paths so attackers encounter it naturally. Each trigger carries a structured code (bias type, class, instance), and in Experiments 2 and 3 the release adds timestamped encounter labels plus Expert Knowledge Model sensors that classify subsequent actions as 'rational' or 'biased.' The second load-bearing element is

What would settle it

A reader could test the central claim by taking the released trigger-encounter annotations and comparing behavior in trigger-present experiments with the trigger-free control: if encountering a planted artifact (say, the risky-credentials notes file) produces no measurable shift in subsequent commands, dwell time, or path persistence relative to control sessions, or if independent coders cannot reproduce the rational-versus-biased labels at better than chance, the bias-elicitation claim is falsified.

Watch

Extended reading notes

Core claim

On its own terms, the discovery reported here is a dataset, not a finding about attackers. The release covers three red-team experiments run from July 2024 to March 2025, each with 19–20 skilled attackers who spent two eight-hour days operating from a standardized Linux attack workstation inside a realistic, isolated clone of an enterprise network of roughly 40 virtual machines with live services and synthetic user traffic. For every session the archive bundles synchronized host telemetry (keylogger streams, clipboard text, shell histories), network telemetry (full packet captures and intrusion-detection alerts), and human-factors data (psychometric batteries, hourly and end-of-day surveys,

Load-bearing premise

The load-bearing premise is that the planted artifacts actually induce the intended cognitive biases in participants and that the expert labels correctly identify which actions were biased; the release reports no manipulation check, pilot data, or label-agreement validation.

Editorial extensions

If this is right

  • Because Experiment 2 removes all triggers, trigger-present runs can be compared against a clean control to isolate whether planted artifacts change command choices, dwell time, and mission progress.
  • The single-clock alignment of keylogs, shell history, and network captures lets an analyst reconstruct exact command sequences around each trigger encounter, turning each bias into a checkable behavioral signature rather than relying on self-report alone.
  • Network-side captures and intrusion alerts mean biased choices can be studied from the defender's vantage point, which is what a real-time bias-aware defense would need.
  • The standardized start box and isolated per-participant ranges support reproducible benchmarking: different detection or behavior-prediction pipelines can be scored on identical attack sessions.
  • The psychometric battery opens individual-difference questions, such as whether cognitive-reflection or risk-propensity scores predict who falls for which trigger.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Even if the triggers fail to elicit the intended biases, the release still stands as a large synchronized corpus of skilled human attackers, so general attacker modeling, tool-use statistics, and detection benchmarking do not depend on the bias story.
  • A useful table the paper does not provide is per-trigger encounter rates; if most participants never meet most triggers, cross-condition comparisons will be underpowered and the dataset's bias value shifts to qualitative case studies.
  • The expert 'rational versus biased' labels are best treated as model outputs until independent coders can reproduce them from the same logs; a simple agreement study would settle whether they are ground truth or hypotheses.
  • The three-experiment structure enables a label-free bias test: train an action-sequence model on trigger-free sessions, then look for anomalous deviations after triggers in the other experiments.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This manuscript is a data descriptor for three human-subjects red-team (HSR) experiments conducted on the SimSpace Cyber Force Platform, released through IEEE DataPort. Each experiment involved 19–20 skilled attackers performing two 8-hour days of self-paced operations in a simulated enterprise network. The dataset synchronizes host telemetry (keylogs, shell histories), network data (PCAP, Suricata alerts), and human-factors instruments (psychometrics, hourly surveys, operational notes). The authors' central claim is that the dataset supports bias-aware analytics: cognitive-bias triggers were embedded in the network (loss aversion, base-rate neglect, availability heuristic, confirmation bias, sunk-cost fallacy) to elicit and measure biases, with derivative annotations including trigger encounters and Expert Knowledge Model (EKM) labels classifying actions as 'rational' or 'biased'. The paper documents the experimental design, data organization, file formats, and usage scenarios.

Significance. If the bias construct is valid, this is a potentially valuable resource: it is one of the few public datasets combining full network packet captures, host-level keystroke logs, and psychometric assessments from realistic, repeatable red-team exercises. The authors have included good documentation practices—standardized directory structures, SHA-256 checksums, schema definitions, and synchronized timestamps—which are concrete strengths. However, the scientific payload of the paper depends on two unvalidated constructs: (i) the embedded triggers actually elicit the intended cognitive biases, and (ii) the EKM labels provide ground-truth 'rational' vs. 'biased' classifications. The paper provides no manipulation checks, no comparison between trigger-present and trigger-absent conditions, no inter-rater reliability for trigger-encounter annotations, and no validation of EKM labels against expert consensus or any independent ground truth. Without this evidence, the dataset's most distinctive contribution—enabling bias-aware modeling—is not supported, even though the multi-modal telemetry alone may still support general attacker-behavior research.

major comments (3)
  1. [§3.2 and §3.6] The central claim that the embedded triggers 'elicit' and 'reveal' cognitive biases is not backed by any manipulation check or validation. The paper states that triggers were 'designed' and 'iteratively tested' but provides no pilot data, no encounter-rate statistics, no comparison of behavioral outcomes between the trigger-present conditions (HSR1, HSR3) and the trigger-absent control (HSR2), and no baseline rates of the target behaviors. As the Limitations section (§6) itself concedes, triggers 'may shape action sequences'—but no evidence quantifies that shaping. Without a demonstrated manipulation effect, the headline claim of 'revealing cognitive bias' is unsupported. I recommend adding an explicit cross-condition analysis or, at minimum, reporting trigger encounter rates and outcome comparisons for relevant actions.
  2. [§3.2 and §3.5] The Expert Knowledge Model (EKM) is introduced as producing annotations indicating whether subsequent actions are 'rational' or 'biased'. The paper does not define the model's inputs, decision rule, thresholds, or validation procedure. No inter-rater reliability, agreement with expert panels, or comparison against any independent ground truth is reported. Because these EKM labels are released as derivative annotations, any downstream machine-learning benchmark using them will inherit unvalidated labels as ground truth. The authors should either provide the EKM implementation and validation results or clearly reframe these labels as model outputs rather than ground-truth annotations.
  3. [§3.5 and Table 2] Trigger encounter annotations with timestamps and 'co-occurring bias codes' are described as present only in HSR2 and HSR3, while the baseline-with-triggers condition (HSR1) apparently lacks these timestamped annotations. This asymmetry is not discussed when the paper proposes cross-condition comparisons (HSR1/HSR3 vs. HSR2). Moreover, no inter-rater reliability is reported for the trigger-encounter annotations or the associated bias codes. This is a load-bearing gap for any analysis that relies on precise trigger encounters to link stimuli to behavior.
minor comments (4)
  1. [§2.2 vs. §4.1] The directory organization is described inconsistently: §2.2 places 'derived/' inside participant bundles and the figure shows participant subfolders, while §4.1 describes modality-specific subdirectories at the top-level archive. Clarify the exact hierarchy.
  2. [Abstract and §2.4] The abstract lists 'NetFlow summaries' as a captured modality, but §2.4 and the modal lists under 'network/' mention only PCAP, Suricata eve.json, and 'flows summary' in the figure. If NetFlow is included, it should appear in the modality description; if not, remove the term.
  3. [Table 2] Data sizes vary widely (722.8 GB vs. 2.1 TB and 2.8 TB), but the paper does not explain this difference. Since §1.2 states only Experiment 1 has large PCAP files, this should be stated explicitly in Table 2 or its caption to avoid confusion.
  4. [Throughout] There are minor grammatical issues, e.g., 'under operationally realistic' in §3.2 is incomplete, and the reference to 'CTF' in §5 should be 'Cry Wolf' based on context. A language edit is recommended.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity found; the dataset paper is self-contained and does not reduce any claim to its own inputs.

full rationale

GAMBiT is a data descriptor, not a derivation: the central claim is that three multi-modal human-subjects red-team datasets exist, were captured in a realistic cyber range, and are released with raw telemetry and derivative annotations. That claim is supported by the described capture infrastructure, release archives, timestamps, and file organization, not by any equation fitted to the data. The paper reports no predictive results that could be forced by construction, and none of the cited prior work is load-bearing in the sense of substituting for evidence of the dataset's existence. The bias-trigger design and Expert Knowledge Model labels are described as annotations, but the paper does not present a statistical claim that the triggers caused specific biases; it states that the data 'supports' such analysis. Therefore, concerns about the validity of the triggers or the EKM labels are empirical validation risks, not circularity: there is no quoted equation or construction showing that the labels are defined in terms of the very outcome they are meant to explain. Self-citations appear in the motivation and related-work discussion, but they do not carry the central claim. Under the hard rule that circularity must be exhibited by quoting a specific reduction, no such reduction is present in this manuscript.

Assumptions & free parameters 1 free parameters · 4 assumptions · 1 invented entities

The central claim is the existence and scientific utility of the dataset. That claim rests on several unverified premises: the psychological instruments measure what they claim, the simulated range is a valid proxy for a real enterprise network, the embedded triggers elicit the intended biases, and the EKM labels for rational versus biased actions are correct. The first two are standard domain assumptions; the last two are ad hoc to this project and carry most of the risk.

free parameters (1)
  • EKM rational/biased classification threshold = unknown
    The Expert Knowledge Model sensor classifies actions as 'rational' or 'biased' (Sections 3.2 and 3.5), but the paper does not specify the model, training data, or decision threshold. Any downstream analysis relying on these labels depends on an uncalibrated parameter.
assumptions (4)
  • domain assumption The psychometric instruments (CRT, BFI-2-XS, GRiPS, A-DMC) measure the cognitive traits they are standardly used to measure.
    Used in Section 3.4 to characterize participant profiles; no validation within this paper or citation to validation studies beyond standard use.
  • domain assumption The SimSpace cyber range with ~40 VMs and synthetic user traffic is a sufficiently realistic proxy for a mid-sized enterprise network to support generalization of attacker behavior findings.
    The entire dataset's external validity rests on this premise; the paper acknowledges limitations in Section 6 but provides no comparison to real enterprise traffic.
  • ad hoc to paper The embedded artifacts (e.g., the loss-aversion notes file, misleading 'adm' account names, deceptive aliased commands) induce the intended cognitive biases in participants.
    Designed by SMEs in Section 3.2; no manipulation check, pilot data, or behavioral validation is reported.
  • ad hoc to paper The Expert Knowledge Model (EKM) annotations correctly classify actions as 'rational' or 'biased'.
    Mentioned in Sections 3.2 and 3.5 as a derivative artifact; no ground truth, inter-rater reliability, or model specification is provided, so the labels are unvalidated.
invented entities (1)
  • Expert Knowledge Model (EKM) bias sensors
    purpose: Provide near-real-time classification of attacker actions as 'rational' or 'biased' and annotate trigger encounters (Sections 3.2, 3.5).
    The model is not described, trained, or externally validated; its outputs are used as labels in the dataset, so evidence for its correctness is circular to the authors' own design.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Guarding Against Malicious Biased Threats (GAMBiT) Experiments: Revealing Cognitive Bias in Human-Subjects Red-Team Cyber Range Operations." pith.science (2026). https://pith.science/paper/BGU4PEEV

@misc{pith2026250820963,
  author       = {Pith},
  title        = {Pith review of: Guarding Against Malicious Biased Threats (GAMBiT) Experiments: Revealing Cognitive Bias in Human-Subjects Red-Team Cyber Range Operations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BGU4PEEV}},
  note         = {Machine review of arXiv:2508.20963}
}
read the original abstract

We present three large-scale human-subjects red-team cyber range datasets from the Guarding Against Malicious Biased Threats (GAMBiT) project. Across Experiments 1-3 (July 2024-March 2025), 19-20 skilled attackers per experiment conducted two 8-hour days of self-paced operations in a simulated enterprise network (SimSpace Cyber Force Platform) while we captured multi-modal data: self-reports (background, demographics, psychometrics), operational notes, terminal histories, keylogs, network packet captures (PCAP), and NIDS alerts (Suricata). Each participant began from a standardized Kali Linux VM and pursued realistic objectives (e.g., target discovery and data exfiltration) under controlled constraints. Derivative curated logs and labels are included. The combined release supports research on attacker behavior modeling, bias-aware analytics, and method benchmarking. Data are available via IEEE Dataport entries for Experiments 1-3.

Figures

Figures reproduced from arXiv: 2508.20963 by the authors.

Figure 1
Figure 1. Directory layout: experiment-level files and participant-indexed bundles; each participant folder contains modality-specific [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Representative GAMBiT cyber range topology showing core services, segmented zones, and participant start node. [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

22 extracted references · 18 canonical work pages

  1. [1]

    Security investment under cognitive constraints: A gestalt nash equilibrium approach

    Juntao Chen and Quanyan Zhu. Security investment under cognitive constraints: A gestalt nash equilibrium approach. In 2018 52nd Annual Conference on Information Sciences and Systems (CISS) , pages 1–6. IEEE, 2018

    work page 2018

  2. [2]

    Stuck on a phishing lure: differential use of base rates in self and social judgments of susceptibility to cyber risk

    E Blair Cox, Quanyan Zhu, and Emily Balcetis. Stuck on a phishing lure: differential use of base rates in self and social judgments of susceptibility to cyber risk. Comprehensive Results in Social Psychology, 4(1):25–52, 2020

    work page 2020

  3. [3]

    Cognitive reflection and decision making

    Shane Frederick. Cognitive reflection and decision making. Journal of Economic perspectives , 19(4):25–42, 2005. 11

    work page 2005

  4. [4]

    Gonzalez, P

    C. Gonzalez, P. Aggarwal, P. Rajivan, S. Venkatesan, A. Aggarwal, and M. Jos´ e Ferreira. Situated cyber cogvuln study (psyccdef). IARPA ReSCIND Data Set, 2025. Dataset Title: Situated Cyber CogVuln Study (PsyCCDef); 111 cyber-expert participants; February–December 2025; CSV and XLSX formats; available via OSF

    work page 2025

  5. [5]

    Game of Travesty: Decoy-based Psychological Cyber Deception for Proactive Human Agents

    Yinan Hu and Quanyan Zhu. Game of travesty: Decoy-based psychological cyber deception for proactive human agents. arXiv preprint arXiv:2309.13403 , 2023

    work page Pith review arXiv 2023

  6. [6]

    Farsighted risk mitigation of lateral movement using dynamic cognitive honeypots

    Linan Huang and Quanyan Zhu. Farsighted risk mitigation of lateral movement using dynamic cognitive honeypots. In International conference on decision and game theory for security , pages 125–146. Springer, 2020

    work page 2020

  7. [7]

    Cognitive security: a system-scientific approach

    Linan Huang and Quanyan Zhu. Cognitive security: a system-scientific approach . Springer Nature, 2023

    work page 2023

  8. [8]

    Psyborg+: Modeling and simulation for detecting cognitive biases in advanced persistent threats

    Shuo Huang, Fred Jones, Nikolos Gurney, David Pynadath, Kunal Srivastava, Stoney Trent, Peggy Wu, and Quanyan Zhu. Psyborg+: Modeling and simulation for detecting cognitive biases in advanced persistent threats. In MILCOM 2024-2024 IEEE Military Communications Conference (MILCOM) , pages 1–6. IEEE, 2024

    work page 2024

Show all 22 references
  1. [9]

    Game theory and machine learning for cyber security

    Charles A Kamhoua, Christopher D Kiekintveld, Fei Fang, and Quanyan Zhu. Game theory and machine learning for cyber security . John Wiley & Sons, 2021

    2021

  2. [10]

    Kamhoua, Alexander Kott, Quanyan Zhu, and Nandi Leslie, editors

    Charles A. Kamhoua, Alexander Kott, Quanyan Zhu, and Nandi Leslie, editors. Autonomous Cyber Re- silience. Wiley–IEEE Computer Society Press, 2026. Hardcover; expected March 23, 2026

    2026

  3. [11]

    From texts to shields: Convergence of large language models and cybersecurity

    Tao Li, Ya-Ting Yang, Yunian Pan, and Quanyan Zhu. From texts to shields: Convergence of large language models and cybersecurity. arXiv preprint arXiv:2505.00841 , 2025

    2025 arXiv

  4. [12]

    Symbiotic game and foundation models for cyber deception operations in strategic cyber warfare

    Tao Li and Quanyan Zhu. Symbiotic game and foundation models for cyber deception operations in strategic cyber warfare. arXiv preprint arXiv:2403.10570 , 2024

    2024 arXiv

  5. [13]

    Psyscam: A benchmark for psychological techniques in real-world scams

    Shang Ma, Tianyi Ma, Jiahao Liu, Wei Song, Zhenkai Liang, Xusheng Xiao, and Yanfang Ye. Psyscam: A benchmark for psychological techniques in real-world scams. arXiv preprint arXiv:2505.15017, 2025. Published May 21, 2025; dataset and code available at anonymous.4open.science link

    2025

  6. [14]

    Bilateral cognitive security games in networked control systems under stealthy injection attacks

    Anh Tung Nguyen, Quanyan Zhu, and Andr´ e Teixeira. Bilateral cognitive security games in networked control systems under stealthy injection attacks. arXiv preprint arXiv:2505.01232 , 2025

    2025 arXiv

  7. [15]

    Game theory for cyber deception

    Jeffrey Pawlick and Quanyan Zhu. Game theory for cyber deception . Springer, 2021

    2021

  8. [16]

    Leveraging behavioral science to mitigate cyber security risk

    Shari Lawrence Pfleeger and Deanna D Caputo. Leveraging behavioral science to mitigate cyber security risk. Computers & security , 31(4):597–611, 2012

    2012

  9. [17]

    Cry wolf: Toward an experimentation platform and dataset for human factors in cyber security analysis

    William Roden and Lucas Layman. Cry wolf: Toward an experimentation platform and dataset for human factors in cyber security analysis. In Proceedings of the 2020 ACM Southeast Conference (ACMSE 2020) , pages 264–267, Tampa, FL, USA, April 2020. ACM. Also available at https://a...

    2020 arXiv

  10. [18]

    Modeling cognitive biases in decision-theoretic planning for active cyber deception

    Aditya Shinde and Prashant Doshi. Modeling cognitive biases in decision-theoretic planning for active cyber deception. In Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems, pages 1718–1726, 2024

    2024

  11. [19]

    A multi-resolution dynamic game framework for cross-echelon decision- making in cyber warfare

    Ya-Ting Yang and Quanyan Zhu. A multi-resolution dynamic game framework for cross-echelon decision- making in cyber warfare. arXiv preprint arXiv:2507.03021 , 2025

    2025 arXiv

  12. [20]

    When to deceive: A cross-layer stackelberg game framework for strategic timing of cyber deception

    Ya-Ting Yang and Quanyan Zhu. When to deceive: A cross-layer stackelberg game framework for strategic timing of cyber deception. arXiv preprint arXiv:2505.21244 , 2025

    2025 arXiv

  13. [21]

    Game theory meets llm and agentic ai: Reimagining cybersecurity for the age of intelligent threats

    Quanyan Zhu. Game theory meets llm and agentic ai: Reimagining cybersecurity for the age of intelligent threats. arXiv preprint arXiv:2507.10621 , 2025

    2025 arXiv

  14. [22]

    Yu, and Cliff Wang, editors

    Quanyan Zhu, Zhuo Lu, Paul L. Yu, and Cliff Wang, editors. Foundations of Cyber Deception: Modeling, Analysis, Design, Human Factors, and Their Convergence , volume 90 of Advances in Information Security . Springer, Cham, 2025. Hardcover expected August 19, 2025. 12

    2025

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.