Pith. sign in

REVIEW 4 major objections 6 minor 17 references

From nuclear safety to LLM security: Applying non-probabilistic risk management strategies to build safe and secure LLM-powered systems

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that 100+ qualitative, non-probabilistic risk-management strategies developed in nuclear engineering, medicine, and other fields can be adapted to secure LLM-powered systems, especially where event spaces are open-ended…

desk verdict A useful, honest position paper that maps a 100+ engineering risk strategy catalogue into LLM security; the transfer is a proposal backed by examples, not by security-specific data, and the paper is appropriately modest about that. read the letter →

arxiv 2505.17084 v1 pith:BBXAUAHI submitted 2025-05-20 cs.CR cs.AI

Alexander Gutfraind , Vicki Bier This is my paper
classification cs.CRcs.AI
keywords largelanguagemodelsecuritynon-probabilisticriskmanagementRDOTcatalogueadaptiveadversariesqualitativeanalysisAIsafetyretrieval-augmentedgenerationstrategiestransfer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large language models fail in ways that standard probabilistic risk analysis cannot easily track, because the space of possible failures is open-ended and adversaries adapt. This paper argues that a catalogue of 100+ non-probabilistic, field-agnostic risk-reducing strategies, developed in nuclear engineering, medicine, and other high-risk fields, can be carried over to LLM-powered systems. The strategies are grouped into five categories—structural, reactive, formal, counter-adversarial, and multi-stage—and mapped to concrete LLM security measures, including defenses against adaptive adversaries. A pilot study on retrieval-augmented generation hallucinations reported that 15% of the strategies were highly promising and previously unconsidered, which the paper takes as evidence the catalogue can also help in adversarial security settings. If the transfer holds, managers and engineers would gain a practical risk-management toolkit that works before enough incident data exists for probabilistic estimates.

What carries the argument

The central object is RDOT, the Risk-reducing design and operation toolkit: a catalogue of 100+ generic strategies, each defined in one line and grouped into structural, reactive, formal, counter-adversarial, and multi-stage categories. The catalogue carries the argument by providing a common vocabulary for risks that resist enumeration, and the paper's Appendix A1 condenses it. The second mechanism is a matching workflow: a reasoning-capable LLM is given the strategy list plus a project description and asked to pair components with candidate strategies, then an engineer filters the suggestions for feasibility. A third element is the project triage criteria (impact, resources, adversarial relevance), which reduce the 100+ options to a few dozen.

What would settle it

Give the full RDOT catalogue to one set of LLM security engineers and a standard LLM security checklist to another, apply both to the same set of realistic attack scenarios, and count how many successful mitigations each group produces; the transfer claim would fail if the catalogue group finds no new effective strategies beyond the checklist, or if a review of documented LLM security incidents shows that most successful mitigations correspond to strategies already standard in the field rather than to novel RDOT entries.

Watch

Extended reading notes

Core claim

The paper's central claim is that qualitative risk management can be generalized across fields, so the Risk-reducing design and operation toolkit (RDOT) is a useful starting point for LLM security. It demonstrates this by reviewing each of five RDOT categories and showing familiar and unfamiliar examples: robust and fail-safe design, multi-layer defense, intrusion detection, FMECA and HAZOP-style hazard identification, game-theoretic and deception strategies against adaptive adversaries, and long-term steps like safety culture, sequential prototyping, and knowledge dissemination. The paper also gives selection criteria for projects (impact, resources, adversarial relevance) and an LLM-assisted workflow in which a reasoning model proposes candidate strategies from the catalogue. The evidence offered is a generalization from a single pilot in which 15% of strategies were judged highly promising and not previously considered and 31% were already considered, and the paper frames the result as a complement to, not replacement for, probabilistic risk analysis as data accumulates.

Load-bearing premise

The load-bearing premise is that the RDOT catalogue is a complete and valid enumeration of generic risk-reducing strategies, and that the single-project pilot finding on retrieval-augmented generation hallucinations transfers to adversarial LLM security settings.

Editorial extensions

If this is right

  • LLM security teams can mine the full RDOT catalogue for mitigations that are uncommon in software engineering, such as FMECA, operator checkrides, and sacrificial subsystems, rather than relying only on known LLM defenses.
  • Architects can use the three triage dimensions to narrow the catalogue to a manageable set for a specific system, making the approach usable in ordinary product development.
  • The LLM-assisted matching workflow gives a repeatable way to surface unfamiliar strategies, though the raw output needs engineer vetting before adoption.
  • As incident data accumulates, teams should gradually shift from qualitative strategies to probabilistic methods, which can then refine or replace the non-probabilistic choices.
  • Because overlapping safeguards can introduce new failure modes, iterative prototyping and progressive refinement are expected to be part of any serious adoption of the toolkit.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the catalogue's transferability holds, an organization's safety culture may matter more than its AI tooling: many of the strategies are organizational practices such as training, checkrides, and independent audits, and teams in safety-critical industries already have the management processes to absorb them.
  • The same mapping exercise could be run for other responsible-AI dimensions, such as fairness or privacy, by pairing the RDOT catalogue with a domain-specific list of harms and a few real incidents; the paper does not do this.
  • A controlled comparison of incident-response teams, one using the full catalogue and one using standard LLM security checklists, would separate the value of the catalogue itself from the value of having any structured prompt; the paper reports only a single-project pilot.
  • The counter-adversarial strategies such as deception, randomization, and secrecy are described as protections against adversaries, but the same tools could plausibly be used defensively against LLM-powered attackers in ways the paper only gestures at.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper argues that qualitative, non-probabilistic risk-management strategies drawn from engineering and other fields, collected in the RDOT catalogue of 100+ strategies, can be applied to improve the security and safety of LLM-powered systems, particularly against adaptive adversaries. It positions the argument against probabilistic risk analysis, describes five categories of strategies, gives illustrative examples from nuclear and other safety-critical domains, and presents an LLM-assisted workflow for selecting strategies. The main evidence offered for the catalogue's value is a prior same-group pilot that applied RDOT to reduce hallucinations in a RAG system, with 15% of strategies judged highly promising and 31% already considered.

Significance. If the transfer claim were established, the paper would provide a practically useful complement to probabilistic risk analysis for AI risk, especially in open-ended settings where event spaces cannot be enumerated. The paper's strength is its cross-disciplinary synthesis: the RDOT catalogue is a valuable resource, and the discussion of safety culture, FMECA, HAZOP, and similar methods is a useful reminder that qualitative risk management in high-stakes engineering deserves attention in AI security. However, the current manuscript does not validate the transfer to adversarial settings: the evidence is a hallucination pilot, and the promised mapping to LLM security is not actually presented. The paper is a position/conceptual piece, not an empirical study, and its contribution is accordingly limited to a plausible but unsubstantiated proposal.

major comments (4)
  1. [§2 and §3] The only quantitative evidence cited for RDOT is a same-group pilot on reducing hallucinations in a RAG system (Shi and Gutfraind 2025), with 15% of strategies judged highly promising and 31% already considered. Section 2 explicitly notes that this pilot 'did not consider the potential of RDOT to improve security in adversarial settings.' Section 3 then generalizes to 'LLM security' and the abstract claims coverage 'including risks from adaptive adversaries.' This is a load-bearing leap: the paper needs either evidence or a structured argument that hallucination-reduction findings transfer to security threats such as prompt injection, jailbreaks, or data exfiltration, or it must substantially weaken the claim to non-adversarial safety.
  2. [§3 and Table A1] The abstract states that the strategies 'are mapped to LLM security,' but no such mapping is presented. Table A1 is an abbreviated catalogue of strategies with definitions and categories; it does not map strategies to specific LLM security risks, threat models, or attack stages. Section 3's examples (multi-layer defense, canary detection, red teaming) are already standard in software security and are not a systematic mapping. The paper should either include the promised mapping (for example, a table listing each strategy and the applicable LLM threats or system components) or explicitly reframe the contribution as an illustrative selection rather than a complete mapping.
  3. [§3.6] The proposed LLM-powered workflow for matching strategies to projects is described with 'We have also had success' but no evaluation methodology, no criteria for success, and no comparison against a baseline or alternative selection process are provided. Since this workflow is listed as one of the paper's contributions, it needs at least a case study with a qualitative assessment, or the claim must be reduced to a suggestion for future work.
  4. [§2 and Table A1] The paper relies on the RDOT catalogue (Gutfraind 2023) as a comprehensive enumeration of risk-reducing strategies, but it does not summarize the catalogue's construction, inclusion criteria, or validation. Without this, it is difficult to assess whether the catalogue is complete or whether omissions could undermine the proposed transfer. Additionally, the five categories overlap (e.g., 'Adversarial' and 'harnessing' are combined in some rows), and some entries such as 'Wait and see' and 'Use the default action' may be trivial or unhelpful for LLM security. The paper should at minimum describe Gutfraind 2023's methodology and note any known limitations of the catalogue.
minor comments (6)
  1. [Abstract and §3] The paper repeatedly refers to '100+' strategies, but Table A1 contains fewer than 100 entries and is explicitly abbreviated; the full dataset is only available online. The reader cannot verify the count or the completeness from the manuscript, so a note describing the full dataset's size and composition would be helpful.
  2. [References] In the reference list, 'Anazon Web Services' should be 'Amazon Web Services.'
  3. [Table A1] There are several typos and inconsistencies in the table: 'escaling' should be 'escalating'; one row uses 'format - workflow' where others use 'formal - workflow'; and category labels are inconsistent (e.g., 'adversarial' alone vs. 'adversarial; harnessing'). These should be corrected for readability.
  4. [§3.1] The sentence 'Another important class of techniques for risk reduction is technical standards and regulation' is grammatically incomplete and should be revised.
  5. [§3.3 and §3.4] The pointers 'see Table A1 in the Appendix' appear in sections that discuss applications, but Table A1 is only a catalogue, not a mapping to LLM security. These pointers are misleading and should be removed or replaced with a reference to the actual mapping, if one is added.
  6. [§5] The Limitations section is brief and does not explicitly acknowledge the lack of empirical validation of the transfer to adversarial security. Adding a sentence noting that the security claims are proposals supported only by anecdotal examples and an unrelated pilot would improve the paper's honesty.

Circularity Check

1 steps flagged · score 4.0 of 10

The framework's only cited empirical warrant is a same-group pilot, but the mapping itself is not derived from it; no by-construction circularity.

  1. self citation load bearing [Section 2 (Risk management frameworks) and Section 3 (Adapting the framework)]
    "As evidence of the potential of RDOT, a recent study has applied it to the problem of reducing hallucinations in a LLM-powered system. This system was a previously deployed solution that utilized LLM and a knowledge base in a retrieval-augmented generation (RAG) architecture (Lewis et al., 2020). On reviewing the 100+ strategies in RDOT, 15% were judged as highly promising and not previously considered in this context. Another 31% were previously considered or already utilized in the project (Shi and Gutfraind, 2025)."

    The paper's only quantitative evidence that the RDOT catalogue is valuable comes from Shi and Gutfraind (2025), a pilot whose author overlaps with this paper's first author and which is neither machine-checked nor independently replicated. The pilot tested hallucinations in a RAG system, not adversarial LLM security, as the paper itself concedes. Section 3 then says 'Generalizing from a single project' and extends the catalogue to LLM security 'including risks from adaptive adversaries.' So the load-bearing support for the central transfer claim is a self-citation that did not test the target domain. This is not a by-construction reduction, but it is the kind of self-referential evidence chain the rubric targets.

full rationale

No self-definitional or equation-level circularity was found: the paper does not fit parameters or define quantities in terms of outcomes, and its mapping in Table A1 is a qualitative catalogue, not a predictive model. The individual strategies (FMECA, HAZOP, canary detection, etc.) have external grounding in the cited engineering literature. The central weakness is evidentiary rather than logical: the only empirical support for the RDOT catalogue's utility is a same-group hallucination pilot that explicitly excluded adversarial settings, and the security-specific generalization is asserted rather than derived. Because the mapping and the LLM-based matching workflow have independent content and do not reduce to the pilot's numbers, the appropriate circularity score is moderate (4), not near-total.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

No numerical free parameters are fitted. The paper's central argument rests on assumptions about the impracticality of PRA, the transferability of generic risk strategies, the completeness of the self-authored RDOT catalogue, the validity of a same-group pilot, and the usefulness of an LLM matching workflow. These are domain and self-citation assumptions rather than mathematical axioms.

assumptions (5)
  • domain assumption Probabilistic risk analysis is impractical for LLM-powered systems because the event space is open-ended and adversaries adapt, so qualitative strategies are needed.
    Stated in the abstract and Section 2 as the motivating premise for the entire transfer; it is plausible but not empirically established.
  • domain assumption Risk-reducing strategies are generic across fields and can be transferred from nuclear and civil engineering to LLM security without substantive adaptation.
    Invoked in Section 3 and throughout the paper; the paper gives examples but no systematic demonstration of transferability.
  • ad hoc to paper The RDOT catalogue (Gutfraind 2023) is a comprehensive and valid enumeration of qualitative risk-reducing strategies.
    The 100+ strategies are drawn from the first author's own prior catalogue; the paper does not independently validate its completeness or representativeness.
  • ad hoc to paper The prior same-group study on RAG hallucinations (Shi and Gutfraind 2025) provides evidence that RDOT strategies are novel and promising when applied to LLM systems.
    Used as the main quantitative evidence (15% highly promising, 31% already considered) but the study is not described in enough detail to assess, and it did not cover adversarial security settings.
  • domain assumption An LLM with reasoning capabilities can produce useful candidate strategy matches when given the strategy list and a project description.
    The proposed LLM-powered workflow in Section 3.6 is asserted from 'our experience' without prompt details, outputs, or evaluation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of From nuclear safety to LLM security: Applying non-probabilistic risk management strategies to build safe and secure LLM-powered systems." pith.science (2026). https://pith.science/paper/BBXAUAHI

@misc{pith2026250517084,
  author       = {Pith},
  title        = {Pith review of: From nuclear safety to LLM security: Applying non-probabilistic risk management strategies to build safe and secure LLM-powered systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BBXAUAHI}},
  note         = {Machine review of arXiv:2505.17084}
}
read the original abstract

Large language models (LLMs) offer unprecedented and growing capabilities, but also introduce complex safety and security challenges that resist conventional risk management. While conventional probabilistic risk analysis (PRA) requires exhaustive risk enumeration and quantification, the novelty and complexity of these systems make PRA impractical, particularly against adaptive adversaries. Previous research found that risk management in various fields of engineering such as nuclear or civil engineering is often solved by generic (i.e. field-agnostic) strategies such as event tree analysis or robust designs. Here we show how emerging risks in LLM-powered systems could be met with 100+ of these non-probabilistic strategies to risk management, including risks from adaptive adversaries. The strategies are divided into five categories and are mapped to LLM security (and AI safety more broadly). We also present an LLM-powered workflow for applying these strategies and other workflows suitable for solution architects. Overall, these strategies could contribute (despite some limitations) to security, safety and other dimensions of responsible AI.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

17 extracted references · 13 canonical work pages

  1. [1]

    3University of Wisconsin at Madison

    1Amazon Web Services and 2Loyola University of Chicago. 3University of Wisconsin at Madison. *Send correspondence to sgfriend@amazon.com Abstract Large language models (LLMs) offer unprecedented and growing capabilities, but also introduce complex safety and security challenges that resist conventional risk management. While conventional probabilistic ris...

    work page 2022

  2. [5]

    Strategies involving multi-stage or long-term risk management Structural strategies: Briefly, these strategies involve designing and developing systems to be robust and resilient to uncertainties. Representative examples of this class of strategies are enhancing resilience (Ganin et al., 2016), spatial separation (Todinov, 2015), multi-layered defenses (N...

    work page 2016

  3. [9]

    arXiv:2310.17688 [cs]

    Managing AI Risks in an Era of Rapid Progress. arXiv:2310.17688 [cs]. Vicki M Bier and M. Naceur Azaiez

    arXiv

  4. [11]

    On strategies for risk management and decision making under uncertainty shared across multiple fields

    On strategies for risk management and decision making under uncertainty shared across multiple fields arXiv:2309.03133 [cs, math, q-fin]. Philipp Hacker

    work page Pith review arXiv

  5. [12]

    In 2023 ACM Conference on Fairness, Accountability, and Transparency, pages 1112–1123, Chicago IL USA

    Regulating ChatGPT and other Large Generative AI Models. In 2023 ACM Conference on Fairness, Accountability, and Transparency, pages 1112–1123, Chicago IL USA. ACM. Shane G. Henderson and Barry L. Nelson

    work page 2023

  6. [13]

    arXiv:2312.02119 [cs, stat]

    Tree of Attacks: Jailbreaking Black-Box LLMs Automatically. arXiv:2312.02119 [cs, stat]. Andrea Miotti and Akash Wasil

    arXiv

  7. [14]

    arXiv:2310.20563 [cs]

    Taking control: Policies to address extinction risks from advanced AI. arXiv:2310.20563 [cs]. Maximilian Mozes, Xuanli He, Bennett Kleinberg, and Lewis D. Griffin

    work page arXiv

  8. [15]

    arXiv:2308.12833 [cs]

    Use of LLMs for Illicit Purposes: Threats, Prevention Measures, and Vulnerabilities. arXiv:2308.12833 [cs]. Rick Nunes-Vaz, Steven Lord, and Jolanta Ciuk

    arXiv

Show all 17 references
  1. [16]

    Hong Zheng, L.G

    Jailbroken: How Does LLM Safety Training Fail? arXiv:2307.02483 [cs]. Hong Zheng, L.G. Tham, and Defu Liu

    arXiv

  2. [1996]

    has been found to be useful in cases where the undesirable outcomes of interest are clearly identified and the biggest challenge is estimating their probabilities. Once the largest contributors to risk have been estimated, the decision of what to do about them may be straightf...

    1996

  3. [1998]

    In Annual Reliability and Maintainability Symposium

    The new SAE FMECA standard. In Annual Reliability and Maintainability Symposium. 1998 Proceedings. International Symposium on Product Quality and Integrity, pages 48–53, Anaheim, CA, USA. IEEE. Marion A. Brach

    1998

  4. [2009]

    Other strategies are inspired directly or indirectly by game theory; e.g., minimax, randomization (Jain et al., 2010), and others

    could also be applied for securing LLMs. Other strategies are inspired directly or indirectly by game theory; e.g., minimax, randomization (Jain et al., 2010), and others. Applying game theory requires fairly sophisticated analysis, but can offer guaranteed security even again...

    2010

  5. [2010]

    can then examine alternative risk mitigations, trade-offs, nonlinearities, and dependencies (Keeney and Raiffa, 1993). From nuclear safety to LLM security: Applying non-probabilistic risk management strategies to build safe and secure LLM-powered systems Alexander Gutfraind1,2...

    1993

  6. [2011]

    Defence and Peace Economics, 22(1):43–61

    Secrecy and Deception at Equilibrium, with Applications to Anti‐Terrorism Resource Allocation. Defence and Peace Economics, 22(1):43–61. Gutfraind & Bier - From nuclear safety to LLM security 7 Appendix A Table A1: An abbreviated version of RDOT catalog. The full dataset (cont...

  7. [2012]

    has proposed a system-theoretic accident model and processes, which is specifically designed to identify potential risks in the face of the complexities, interactions, and nonlinearities that can be associated with software. Gutfraind & Bier - From nuclear safety to LLM securi...

    2011

  8. [2020]

    In 2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST), pages 96–107, Porto, Portugal

    A Study on Challenges of Testing Robotic Systems. In 2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST), pages 96–107, Porto, Portugal. IEEE. Ajay Agrawal, Joshua Gans, and Avi Goldfarb

    2020

  9. [2023]

    Scenario analysis, even if not comprehensive, can give system designers and other decision makers a feel for the range of possible outcomes from a given design decision or hazard

    and references contained therein. Scenario analysis, even if not comprehensive, can give system designers and other decision makers a feel for the range of possible outcomes from a given design decision or hazard. While commonly used as part of PRA, scenario analysis can also ...

    1999

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.