Pith. sign in

REVIEW 4 major objections 6 minor 2 cited by

An alignment safety case sketch based on debate

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper argues that training a model by debate can put a hard upper bound on the fraction of its outputs that are wrong or misaligned, provided the model does not deliberately underperform during training, and that this bound can be…

desk verdict A self-aware safety-case sketch whose value is in its structure and honest gap-identification, not in any completed guarantee; the missing exploration guarantee is exactly as load-bearing as the stress-test says. read the letter →

arxiv 2505.03989 v3 pith:TXR76YVU submitted 2025-05-06 cs.AI

Marie Davidsen Buhl , Jacob Pfau , Benjamin Hilton , Geoffrey Irving This is my paper · ORCID
classification cs.AI
keywords AIsafetydebatescalableoversightcaseexplorationhackinglow-stakesalignmentonlinetraininghonesty
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that debate training can put a hard upper bound on the fraction of a model's outputs that are misaligned, and that this bound can be kept during deployment through continuous online retraining. Concretely, if a model reaches an approximate global equilibrium of a debate game whose equilibrium strategy is honest play, and the deployment context is low-stakes, then the model cannot take enough bad actions to produce an unacceptable outcome. The argument is built as a four-part safety case for an internal AI research agent, where the relevant catastrophe is producing false or sabotaged research. Instead of checking outputs against ground truth, the paper checks a debate protocol against a prohibitively expensive computation $M$ that is hypothesised to be correct, so that good play in the game becomes evidence of correctness. The sketch is explicit that the strongest version of the argument depends on research that does not yet exist, chiefly exploration guarantees ruling out deliberate underperformance during training.

What carries the argument

The central mechanism is the debate game, specifically a prover-estimator protocol in which one model (Alice) produces subclaims and another (Bob) assigns probabilities to them, with a simulated human judge deciding the winner. Equilibrium behaviour in this game is linked to truth through the big computation $M$: the exhaustive tree of every possible subclaim and counterclaim, adjudicated by simulated human judges at every leaf, which is hypothesised to be correct. The debate protocol is meant to accelerate $M$, and the safety case argues that approximate global equilibria of the game produce approximately the same outputs as $M$. The argument also relies on a relative efficiency notion $F$ (Bob needs only polynomial more compute than Alice), continuous online training every $k$ steps during deployment, a stability condition on $M$ (small variations in input probabilities do not change outputs), and a threat model parameter $Z$ giving the number of bad actions needed to cause unacceptable outcomes.

What would settle it

Run a controlled debate-training experiment in which one debater is seeded with knowledge of a highly rewarded counterargument and instructed to withhold it; if the training signal does not punish the resulting loss of reward, the exploration guarantee fails. Separately, take a small verifiable domain, implement $M$ as an exhaustive recursive debate tree, and test whether random small perturbations of leaf-judge probabilities flip many outcomes; if they do, the stability assumption fails for that domain.

Watch

Extended reading notes

Core claim

The central claim is that an asymptotic guarantee of honesty is achievable in principle: given a debate protocol whose approximate global equilibrium is honest play, training that converges to that equilibrium yields a system with error rate at most $\epsilon'$, and online training during deployment keeps the error rate below $\epsilon$ for $\epsilon > \epsilon'$. The paper formalizes this as four key claims: training reaches an approximate global equilibrium (C2.1); such equilibria produce correct answers in a $(1-\epsilon')$-fraction of cases (C2.2); the error rate does not drift past $\epsilon$ during deployment (C2.3); and the deployment context is tolerant of an $\epsilon$ error rate (C1.2). The concrete object is a sandboxed AI R&D agent working in an offline datacentre, with unacceptable outcomes enumerated as research sabotage, malicious code exfiltration, and self-exfiltration. The introduction states the core claim directly: debate training can upper bound the proportion of model outputs that are misaligned, as long as the model does not deliberately underperform during training.

Load-bearing premise

The load-bearing premise is an exploration guarantee: evidence that the model is not forgoing highly rewarded strategies during training, since without it an apparent equilibrium may be a false one; separately, the debate-accelerated computation $M$ must be stable, so that small variations in input probabilities do not change its outputs.

Editorial extensions

If this is right

  • If the four key claims hold, a developer can deploy an internal AI research agent with a quantified error budget and argue that catastrophic research sabotage is impossible, because reaching it would require more than $Z$ coordinated bad actions while the error rate caps the number of bad actions per iteration.
  • The safety case converts the broad question "is this model aligned?" into concrete researchable subproblems: proving protocol properties, demonstrating training convergence, detecting exploration hacking, and bounding systematic human-judge error.
  • Because it is an asymptotic-incentives argument, it can be combined in parallel with behavioural evaluations, interpretability-based arguments, and control measures to produce a stronger overall safety case than any single argument alone.
  • The argument is restricted to low-stakes settings: in high-stakes contexts, key claims 3 and 4 fail, and the paper argues that the evidence gathered for claims 1 and 2, especially exploration guarantees, may be the main route to extending the approach.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct way to test the exploration-guarantee requirement is to build a debate environment with a known high-reward withheld strategy, train normally, and check whether any proposed detection method (perturbation analysis, adversarial RL, best-of-n sampling) flags the loss of reward; failure to flag would invalidate a practical implementation of the sketch.
  • The stability assumption could be probed on a small verifiable domain by implementing a restricted version of $M$ as an exhaustive recursive debate tree and measuring whether random small perturbations of leaf-judge probabilities frequently flip outcomes; frequent flips would show the prover-estimator protocol's usefulness is limited for those queries.
  • The low-stakes framework implies a concrete trade-off between training latency $k$ and the error bound, so a simulation benchmark varying update frequency and comparing measured error rates to the regret bound $O(\rho k \sqrt{T})$ would test whether the deployment-time maintenance claim is plausible in practice.
  • The paper hints that residual errors may be anticorrelated with harm severity; if that holds, a worst-case error budget could be replaced by a lower expected-harm budget, but this would require new evidence beyond the present sketch.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper presents a 'safety case sketch' arguing that debate training could, in a low-stakes internal deployment, upper bound the fraction of misaligned outputs produced by an AI research agent. The argument is decomposed into four key claims: (C2.1) training reaches an approximate global equilibrium of a debate game, (C2.2) equilibria of the game produce correct answers in a (1-ε')-fraction of cases, (C2.3) online training prevents the error rate from drifting past ε during deployment, and (C1.2) the deployment context is low-stakes, so an ε error rate cannot cause unacceptable outcomes. The paper uses Claims-Arguments-Evidence (CAE) notation to build the safety case, explicitly marking several evidence boxes as blank or requiring further research, most notably the exploration guarantee (E5.3) and the stability of the big computation M (E5.5). It concludes that the argument is not strong with current techniques and lists open research problems.

Significance. The paper's main contribution is an unusually honest and structured mapping of the assumptions underlying debate-based alignment. It correctly identifies exploration hacking and the stability of the proposed prover-estimator debate protocol as the critical open problems that would determine whether such a safety case can ever be realized. The CAE decomposition is internally coherent, and the use of defeaters (D5.1, D5.3, D5.4, D5.5, D6.1-D6.3, D4.1-D4.4) is a clear improvement over informal position papers. The related-work survey is balanced, and the proposed research directions (adversarial RL, singular learning theory, model organisms) are plausible. If the missing evidence boxes could be filled, the result would be a rigorous low-stakes alignment guarantee; as it stands, the paper is a valuable research agenda rather than a working safety case, and it should be evaluated as such.

major comments (4)
  1. [§3.2, §4.4.1] The entire safety case is built on 'prover-estimator debate (forthcoming)', yet the protocol is never defined anywhere in the manuscript. Key claim C2.2 depends on C3.4 ('E approximates big computation M'), whose evidence box E4.3 is labelled 'Debate protocol proof' but no proof is provided. Because the reader cannot check whether this specific protocol avoids the obfuscated-arguments problem or approximates M, the decomposition of key claim 2 is not evaluable. The authors should either include at least a formal specification of the protocol (even in an appendix) or explicitly mark C3.4 as a placeholder to be filled by future work, rather than listing a proof as if it already exists.
  2. [§3.3, §4.3.3] The most load-bearing subclaim in the safety case, C4.3 ('A is not exploration hacking'), has an empty evidence box (E5.3), and the text states 'We do not currently know how to get exploration guarantees.' Since C2.1 (training reaches equilibrium E) depends on C4.3, and C1.1 (error rate at most ε) depends on C2.1, the entire epsilon-bound argument is currently unsupported. The paper does acknowledge this gap later (e.g. 'this is a major gap in our current sketch'), but the abstract and introduction present the conditional claim as the paper's core claim without a prominent caveat that the condition is not merely unattested but presently unsolved. The authors should add an explicit statement in the abstract and introduction that the claimed upper bound is hypothetical and currently lacks evidence, and they should consider adding a summary table of which evidence boxes are filled, blank, or partially supported.
  3. [§4.4.2, §5.2] The stability assumption (C4.6, evidence E5.5) is left blank, and the paper argues that instability is not directly a safety problem because unstable queries 'will simply return no answer' in prover-estimator debate. However, the mechanism by which the protocol detects instability and abstains is not described. If the protocol cannot reliably identify unstable queries, instability could produce false outputs rather than abstention, which would make C3.4 and hence C2.2 unsound. The authors should either specify how the protocol decides to return 'no answer' or soften the claim that instability does not affect safety.
  4. [§4.1, §3.1, footnote 12] The paper uses several formally loaded terms without definitions or derivations: 'approximate global equilibrium', 'approximate M-execution', 'bounded systematic error', and the continuous version 'O(ρk√T + ρε′T)' given in footnote 12. For the safety case to be even a candidate for future rigor, the authors should provide at least informal definitions of these terms and a sketch of how the regret bound is obtained. As written, the quantitative claims are unfalsifiable, and a reader cannot determine whether the decomposition of C1.1 into C2.1-C2.3 is logically sufficient. Adding a short formalization appendix would strengthen the paper considerably.
minor comments (6)
  1. [§4.4.2] There is a typo: 'this may non generalise to non-verifiable domains' should read 'this may not generalise to non-verifiable domains'.
  2. [Figure 6 / D5.3] The defeater label 'Training is struck in a local minimum' should be 'Training is stuck in a local minimum'.
  3. [Figure 11] The full CAE diagram is extremely dense and the text is very small; it would benefit from a higher-resolution version or a more legible layout, perhaps splitting the diagram across multiple figures.
  4. [Table 1 / Figure 3] In Figure 3 and Table 1, the phrase 'that are satisfy' should be 'that satisfy'.
  5. [§3.1] The term 'action' is used to define Z and the online-training latency k, but the granularity of an action in the AI R&D deployment is never defined (e.g. is a single token an action, or is a tool call an action?). This ambiguity affects the interpretability of E4.5 and the low-stakes argument.
  6. [§2, references] The paper relies on 'prover-estimator debate (forthcoming)' without giving a citation or preprint identifier; if such a document already exists, it should be referenced, and if not, the manuscript should say that it is work in progress.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the argument is explicitly conditional and marks its unproven premises (exploration guarantees, judge reliability, stability) as open evidence boxes rather than deriving them.

full rationale

The paper is a safety-case sketch, not a derivation of a result from fitted inputs. Its central claim is explicitly conditional: 'debate training can upper bound the proportion of model outputs that are misaligned, as long as the model does not deliberately underperform during training' (Section 1). Each condition is tracked as a separate claim with its own evidence box, and where evidence is missing the paper says so: Section 3.3 states 'We do not currently know how to get exploration guarantees', E5.3 is left with 'Further research required - possible evidence sources include supervised fine-tuning, adversarial RL, singular learning theory, and formal heuristic explanations', and E5.5 (stability) is similarly blank. These are unresolved premises, not conclusions smuggled from premises. The roundabout route through M ('E approximates big computation M') is not a circular identification: M is defined as a prohibitively expensive computation that calls (simulated) human judges at every leaf, and the claim that M is approximately correct is itself a separate subclaim (C3.5) resting on an explicit assumption about judge reliability (C4.5) with proposed empirical evidence (E6.1). The same judge appears in the game and in M, but the paper does not derive judge reliability from the game; it lists it as one of the most challenging open assumptions ('We expect that getting good human inputs will be one of the most challenging parts of this safety case'). The references to 'prover-estimator debate (forthcoming)' and to Irving et al. (2018) are not used to forbid alternatives: the paper states the argument structure is general ('the argument structure should be applicable to other protocols as well') and that a proof of M-approximation must still be supplied (E4.3). No fitted parameter is renamed as a prediction, and no uniqueness theorem is imported from the authors' prior work. The main limitations are incompleteness rather than circularity.

Assumptions & free parameters 5 free parameters · 5 assumptions · 3 invented entities

The safety case is almost entirely built on assumptions that the paper itself flags as unproven. The most significant are the existence of an efficient debate protocol with an equilibrium-honesty proof, the stability of the reference computation M, the reliability of simulated human judges, and the availability of exploration guarantees. None of these are demonstrated, and the paper explicitly says so. The free parameters (epsilon, epsilon', Z, k, n) are left unquantified, so the argument is structural rather than quantitative.

free parameters (5)
  • epsilon' (training-time error bound) = not specified
    The acceptable error rate at equilibrium is left abstract; no method for computing or bounding it is given.
  • epsilon (deployment-time error bound) = not specified
    The deployment error bound is assumed to be maintainable above epsilon', but no quantitative derivation is provided.
  • Z (minimal number of bad actions for unacceptable outcome) = not specified
    The safety case depends on there being a large Z, but no method for computing it in the deployment context is provided beyond threat modelling.
  • k (online training latency) = not specified
    The regret bound involves latency k; the paper notes it must be 'high enough' but gives no quantitative guidance.
  • n (number of debate rounds for checker) = not specified
    The checker runs n rounds of debate; no value or method for choosing n is provided.
assumptions (5)
  • domain assumption A simulated human judge can provide a sufficiently reliable reward signal over the relevant distribution of queries.
    Used in Step 2 and Key Claim 2 (C4.5, C5.1, C5.2). The entire debate-training argument depends on the judge's reliability.
  • domain assumption The computation M, expanding the full debate tree with simulated human judges, is mostly correct and stable under small perturbations of input probabilities.
    Introduced in Key Claim 2 (C3.4, C3.5, C4.6). The debate protocol is claimed to approximate M; without M's correctness and stability, equilibrium play does not imply honesty.
  • ad hoc to paper There exists an efficient pair of models achieving an approximate global equilibrium of the debate game.
    Claim C3.1, evidenced by E4.1 'Debate protocol proof', which is stated as forthcoming and not included. The entire argument depends on this existence.
  • domain assumption The model is not extremely superhuman and therefore cannot gradient hack.
    Used in Section 3.3 and defeater D5.2. The authors explicitly assume the system is not capable of gradient hacking, without independent evidence.
  • standard math Standard game-theoretic and RL background: training converges to local equilibrium, and approximate local equilibria relate to approximate global equilibria.
    Used implicitly in Key Claim 1 (C3.2, C3.3). The paper itself notes the tabular Stackelberg result does not transfer to non-tabular games.
invented entities (3)
  • Prover-estimator debate protocol
    purpose: A debate protocol claimed to avoid the obfuscated-arguments problem, enabling equilibrium play to imply honesty.
    The protocol is cited as 'forthcoming' and is not described or proven in this paper; no independent evidence is provided.
  • Big computation M
    purpose: A prohibitively expensive hypothetical computation that expands the full debate tree; the debate protocol is claimed to approximate M.
    M is a conceptual device to connect equilibrium play to correctness. Its correctness and stability are assumed.
  • Exploration-predictor model / whitebox exploration-guarantee techniques
    purpose: Proposed evidence sources to rule out exploration hacking.
    Listed as 'early ideas' in Section 3.3; none are implemented or demonstrated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of An alignment safety case sketch based on debate." pith.science (2026). https://pith.science/paper/TXR76YVU

@misc{pith2026250503989,
  author       = {Pith},
  title        = {Pith review of: An alignment safety case sketch based on debate},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TXR76YVU}},
  note         = {Machine review of arXiv:2505.03989}
}
read the original abstract

If AI systems match or exceed human capabilities on a wide range of tasks, it may become difficult for humans to efficiently judge their actions -- making it hard to use human feedback to steer them towards desirable traits. One proposed solution is to leverage another superhuman system to point out flaws in the system's outputs via a debate. This paper outlines the value of debate for AI safety, as well as the assumptions and further research required to make debate work. It does so by sketching an ``alignment safety case'' -- an argument that an AI system will not autonomously take actions which could lead to egregious harm, despite being able to do so. The sketch focuses on the risk of an AI R\&D agent inside an AI company sabotaging research, for example by producing false results. To prevent this, the agent is trained via debate, subject to exploration guarantees, to teach the system to be honest. Honesty is maintained throughout deployment via online training. The safety case rests on four key claims: (1) the agent has become good at the debate game, (2) good performance in the debate game implies that the system is mostly honest, (3) the system will not become significantly less honest during deployment, and (4) the deployment context is tolerant of some errors. We identify open research problems that, if solved, could render this a compelling argument that an AI system is safe.

Figures

Figures reproduced from arXiv: 2505.03989 by the authors.

Figure 1
Figure 1. The alignment strategy covered by this safety case sketch consists of four steps. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. A diagram of the deployment context to which this safety case sketch applies. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. A simplified version of the safety case sketch. The four key claims are labelled in red. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: The initial claims and argument required [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: The main objective O ′ and how it breaks down into four key claims (C2.1, C2.2, C2.3 and C1.2), in CAE notation [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: The first key claim (C2.1) and the argument required to support it, in CAE notation. [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 8
Figure 8. Figure 8: An analogy between debate and go, represented visually: in both cases a single game with strong [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: The third key claim (C2.3) and the argument [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: The fourth key claim (C1.2) and the argument required to support it, in CAE notation. [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Avoiding Obfuscation with Prover-Estimator Debate

    cs.AI 2025-06 reject novelty 8.0 of 10

    Prover-estimator debate replaces the opponent's choice of subclaim with probability estimates, and the paper claims this avoids obfuscation, though the proof of the core indistinguishability lemma is flawed.

  2. How to Avoid Debate: Scalable AI Safety via Doubly-Efficient Interactive Proofs

    cs.AI 2026-07 accept novelty 7.5 of 10

    Doubly-efficient single-prover interactive proofs and arguments exist for robust oracle circuits and for low-degree oracles, enabling relativizing verification without debate.

Reference graph

Works this paper leans on

54 extracted references · 23 canonical work pages · cited by 2 Pith papers

  1. [1]

    AI Security Institute research agenda

    AI Security Institute . AI Security Institute research agenda. Technical report, AI Security Institute, Department for Science, Innovation and Technology, May 2025. URL https://www.aisi.gov.uk/research-agenda. Accessed 6 May 2025

    work page 2025

  2. [2]

    Concrete problems in AI safety, 2016

    Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Man \'e . Concrete problems in AI safety, 2016. URL https://arxiv.org/abs/1606.06565

    arXiv 2016

  3. [3]

    Responsible scaling policy

    Anthropic . Responsible scaling policy. https://assets.anthropic.com/m/24a47b00f10301cd/original/Anthropic-Responsible-Scaling-Policy-2024-10-15.pdf, October 2024. Effective October 15, 2024

    work page 2024

  4. [4]

    Training language models to win debates with self-play improves judge accuracy, 2024

    Samuel Arnesen, David Rein, and Julian Michael. Training language models to win debates with self-play improves judge accuracy, 2024. URL https://arxiv.org/abs/2409.16636

    arXiv 2024

  5. [5]

    Studying small language models with susceptibilities, 2025

    Garrett Baker, George Wang, Jesse Hoogland, and Daniel Murfet. Studying small language models with susceptibilities, 2025. URL https://arxiv.org/abs/2504.18274

    arXiv 2025

  6. [6]

    Towards evaluations-based safety cases for AI scheming, 2024

    Mikita Balesni, Marius Hobbhahn, David Lindner, Alexander Meinke, Tomek Korbak, Joshua Clymer, Buck Shlegeris, J \'e r \'e my Scheurer, Charlotte Stix, Rusheb Shah, Nicholas Goldowsky-Dill, Dan Braun, Bilal Chughtai, Owain Evans, Daniel Kokotajlo, and Lucius Bushnaq. Towards evaluations-based safety cases for AI scheming, 2024. URL https://arxiv.org/abs/2...

    arXiv 2024

  7. [7]

    Debate update: Obfuscated arguments problem

    Beth Barnes. Debate update: Obfuscated arguments problem. AI Alignment Forum, December 2020. URL https://www.alignmentforum.org/posts/PJLABqQ962hZEqhdB/debate-update-obfuscated-arguments-problem. Accessed 6 May 2025

    work page 2020

  8. [8]

    Writeup: P rogress on AI safety via debate

    Beth Barnes and Paul Christiano. Writeup: P rogress on AI safety via debate. AI Alignment Forum, February 2020. URL https://www.alignmentforum.org/posts/Br4xDbYu4Frwrb64a/writeup-progress-on-ai-safety-via-debate-1. Accessed 6 May 2025

    work page 2020

Show all 54 references
  1. [9]

    une s Baydin, Sheila McIlraith, Qiqi Gao, Ashwin Acharya, David Krueger, Anca Dragan, Philip Torr, Stuart Russell, Daniel Kahneman, Jan Brauner, and S \

    Yoshua Bengio, Geoffrey Hinton, Andrew Yao, Dawn Song, Pieter Abbeel, Trevor Darrell, Yuval Noah Harari, Ya-Qin Zhang, Lan Xue, Shai Shalev-Shwartz, Gillian Hadfield, Jeff Clune, Tegan Maharaj, Frank Hutter, At l m G\"une s Baydin, Sheila McIlraith, Qiqi Gao, Ashwin Acharya, D...

    2024

  2. [10]

    o lkopf, Dawn Song, Alvaro Soto, Lee Tiedrich, Ga \

    Yoshua Bengio, S \"o ren Mindermann, Daniel Privitera, Tamay Besiroglu, Rishi Bommasani, Stephen Casper, Yejin Choi, Philip Fox, Ben Garfinkel, Danielle Goldfarb, Hoda Heidari, Anson Ho, Sayash Kapoor, Leila Khalatbari, Shayne Longpre, Sam Manning, Vasilios Mavroudis, Mantas M...

    2025

  3. [11]

    Building blocks for assurance cases

    Robin Bloomfield and Kateryna Netkachova. Building blocks for assurance cases. In 2014 IEEE International Symposium on Software Reliability Engineering Workshops, pp.\ 186--191, 2014. doi:10.1109/ISSREW.2014.72

    2014 doi

  4. [12]

    Superintelligence: Paths, Dangers, Strategies

    Nick Bostrom. Superintelligence: Paths, Dangers, Strategies. Oxford University Press, Oxford, UK, 2014. ISBN 9780199678112

    2014

  5. [13]

    Samuel R. Bowman, Jeeyoon Hyun, Ethan Perez, Edwin Chen, Craig Pettit, Scott Heiner, Kamil\.e Luko s i\=ut\.e, Amanda Askell, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Christopher Olah, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tra...

    2022 arXiv

  6. [14]

    Scalable AI safety via doubly-efficient debate, 2023

    Jonah Brown-Cohen, Geoffrey Irving, and Georgios Piliouras. Scalable AI safety via doubly-efficient debate, 2023. URL https://arxiv.org/abs/2311.14125

    2023 arXiv

  7. [15]

    Safety cases for frontier AI , 2024

    Marie Davidsen Buhl, Gaurav Sett, Leonie Koessler, Jonas Schuett, and Markus Anderljung. Safety cases for frontier AI , 2024

    2024

  8. [16]

    The Alignment Problem: Machine Learning and Human Values

    Brian Christian. The Alignment Problem: Machine Learning and Human Values. W. W. Norton & Company, New York, 2020

    2020

  9. [17]

    Low- S takes alignment

    Paul Christiano. Low- S takes alignment. AI Alignment (Medium article), April 2021. URL https://ai-alignment.com/low-stakes-alignment-f3c36606937f. Accessed 6 May 2025

    2021

  10. [18]

    Supervising strong learners by amplifying weak experts, 2018

    Paul Christiano, Buck Shlegeris, and Dario Amodei. Supervising strong learners by amplifying weak experts, 2018. URL https://arxiv.org/abs/1810.08575

    2018 arXiv

  11. [19]

    Safety cases: H ow to justify the safety of advanced AI systems, 2024

    Joshua Clymer, Nick Gabrieli, David Krueger, and Thomas Larsen. Safety cases: H ow to justify the safety of advanced AI systems, 2024. URL https://arxiv.org/abs/2403.10462

    2024 arXiv

  12. [20]

    Safety case template for frontier AI : A cyber inability argument, 2024

    Arthur Goemans, Marie Davidsen Buhl, Jonas Schuett, Tomek Korbak, Jessica Wang, Benjamin Hilton, and Geoffrey Irving. Safety case template for frontier AI : A cyber inability argument, 2024

    2024

  13. [21]

    Frontier Safety Framework

    Google DeepMind . Frontier Safety Framework . https://storage.googleapis.com/deepmind-media/DeepMind.com/Blog/updating-the-frontier-safety-framework/Frontier

  14. [22]

    Notes on countermeasures for exploration hacking (aka sandbagging)

    Ryan Greenblatt. Notes on countermeasures for exploration hacking (aka sandbagging). AI Alignment Forum, March 2025. URL https://www.alignmentforum.org/posts/abmzgwfJA9acBoFEX/notes-on-countermeasures-for-exploration-hacking-aka. Accessed 6 May 2025

    2025

  15. [23]

    Bowman, and Evan Hubinger

    Ryan Greenblatt, Carson Denison, Benjamin Wright, Fabien Roger, Monte MacDiarmid, Sam Marks, Johannes Treutlein, Tim Belonax, Jack Chen, David Duvenaud, Akbir Khan, Julian Michael, S \"o ren Mindermann, Ethan Perez, Linda Petrini, Jonathan Uesato, Jared Kaplan, Buck Shlegeris,...

    2024 arXiv

  16. [24]

    Safety cases: A scalable approach to frontier AI safety, 2025

    Benjamin Hilton, Marie Davidsen Buhl, Tomek Korbak, and Geoffrey Irving. Safety cases: A scalable approach to frontier AI safety, 2025. URL https://arxiv.org/abs/2503.04744

    2025 arXiv

  17. [25]

    Formal verification, heuristic explanations and surprise accounting, June 2024

    Jacob Hilton. Formal verification, heuristic explanations and surprise accounting, June 2024. URL https://www.alignment.org/blog/formal-verification-heuristic-explanations-and-surprise-accounting/. Alignment Research Center Blog. Accessed 06 May 2025

    2024

  18. [26]

    Risks from learned optimization in advanced machine learning systems, 2021

    Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse, and Scott Garrabrant. Risks from learned optimization in advanced machine learning systems, 2021. URL https://arxiv.org/abs/1906.01820

    2021 arXiv

  19. [27]

    Safety cases at AISI

    Geoffrey Irving. Safety cases at AISI . https://www.aisi.gov.uk/work/safety-cases-at-aisi, March 2024

    2024

  20. [28]

    AI safety via debate, 2018

    Geoffrey Irving, Paul Christiano, and Dario Amodei. AI safety via debate, 2018. URL https://arxiv.org/abs/1805.00899

    2018 arXiv

  21. [29]

    Eliciting bad contexts

    Geoffrey Irving, Joseph Isaac Bloom, and Tomek Korbak. Eliciting bad contexts. AI Alignment Forum, January 2025. URL https://www.alignmentforum.org/posts/inkzPmpTFBdXoKLqC/eliciting-bad-contexts. Accessed 6 May 2025

    2025

  22. [30]

    AI alignment: A comprehensive survey, 2025

    Jiaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He, Lukas Vierling, Donghai Hong, Jiayi Zhou, Zhaowei Zhang, Fanzhi Zeng, Juntao Dai, Xuehai Pan, Kwan Yee Ng, Aidan O'Gara, Hua Xu, Brian Tse, Jie Fu, Stephen McAleer, Yaodong Yang...

    2025 arXiv

  23. [31]

    Siegel, J \'a nos Kram \'a r, Jonah Brown-Cohen, Samuel Albanie, Jannis Bulian, Rishabh Agarwal, David Lindner, Yunhao Tang, Noah D

    Zachary Kenton, Noah Y. Siegel, J \'a nos Kram \'a r, Jonah Brown-Cohen, Samuel Albanie, Jannis Bulian, Rishabh Agarwal, David Lindner, Yunhao Tang, Noah D. Goodman, and Rohin Shah. On scalable oversight with weak llms judging strong llms, 2024. URL https://arxiv.org/abs/2407.04622

    2024 arXiv

  24. [32]

    Bowman, Tim Rockt \"a schel, and Ethan Perez

    Akbir Khan, John Hughes, Dan Valentine, Laura Ruis, Kshitij Sachan, Ansh Radhakrishnan, Edward Grefenstette, Samuel R. Bowman, Tim Rockt \"a schel, and Ethan Perez. Debating with more persuasive llms leads to more truthful answers, 2024. URL https://arxiv.org/abs/2402.06782

    2024 arXiv

  25. [33]

    A sketch of an AI control safety case, 2025

    Tomek Korbak, Joshua Clymer, Benjamin Hilton, Buck Shlegeris, and Geoffrey Irving. A sketch of an AI control safety case, 2025. URL https://arxiv.org/abs/2501.17315

    2025 arXiv

  26. [34]

    Power-seeking can be probable and predictive for trained agents, 2023

    Victoria Krakovna and Janos Kramar. Power-seeking can be probable and predictive for trained agents, 2023. URL https://arxiv.org/abs/2304.06528

    2023 arXiv

  27. [35]

    Goal misgeneralization in deep reinforcement learning

    Lauro Langosco Di Langosco, Jack Koch, Lee D Sharkey, Jacob Pfau, and David Krueger. Goal misgeneralization in deep reinforcement learning. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th Internatio...

    2022

  28. [36]

    The local learning coefficient: A singularity-aware complexity measure, 2024

    Edmund Lau, Zach Furman, George Wang, Daniel Murfet, and Susan Wei. The local learning coefficient: A singularity-aware complexity measure, 2024. URL https://arxiv.org/abs/2308.12108

    2024 arXiv

  29. [37]

    Scalable agent alignment via reward modeling: a research direction, 2018

    Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable agent alignment via reward modeling: a research direction, 2018. URL https://arxiv.org/abs/1811.07871

    2018 arXiv

  30. [38]

    Julian Michael, Salsabila Mahdi, David Rein, Jackson Petty, Julien Dirani, Vishakh Padmakumar, and Samuel R. Bowman. Debate helps supervise unreliable experts, 2023. URL https://arxiv.org/abs/2311.08702

    2023 arXiv

  31. [39]

    The alignment problem from a deep learning perspective, 2025

    Richard Ngo, Lawrence Chan, and S \"o ren Mindermann. The alignment problem from a deep learning perspective, 2025. URL https://arxiv.org/abs/2209.00626

    2025 arXiv

  32. [40]

    Preparedness framework

    OpenAI . Preparedness framework. https://cdn.openai.com/pdf/18a02b5d-6b67-4cec-ab64-68cdfbddebcd/preparedness-framework-v2.pdf, April 2025

    2025

  33. [41]

    The effects of reward misspecification: Mapping and mitigating misaligned models

    Alexander Pan, Kush Bhatia, and Jacob Steinhardt. The effects of reward misspecification: Mapping and mitigating misaligned models. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=JYtwGwIL7ye

    2022

  34. [42]

    Richard Yuanzhe Pang, Alicia Parrish, Nitish Joshi, Nikita Nangia, Jason Phang, Angelica Chen, Vishakh Padmakumar, Johnny Ma, Jana Thompson, He He, and Samuel Bowman. Q u ALITY : Question answering with long input texts, yes! In Marine Carpuat, Marie-Catherine de Marneffe, and...

    2022

  35. [43]

    Park, Simon Goldstein, Aidan O'Gara, Michael Chen, and Dan Hendrycks

    Peter S. Park, Simon Goldstein, Aidan O'Gara, Michael Chen, and Dan Hendrycks. AI deception: A survey of examples, risks, and potential solutions. Patterns, 5 0 (5): 0 100988, may 2024. ISSN 2666-3899. doi:10.1016/j.patter.2024.100988

    2024

  36. [44]

    Alicia Parrish, Harsh Trivedi, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Amanpreet Singh Saimbhi, and Samuel R. Bowman. Two-turn debate doesn't help humans answer hard reading comprehension questions, 2022 a . URL https://arxiv.org/abs/2210.10860

    2022 arXiv

  37. [45]

    Alicia Parrish, Harsh Trivedi, Ethan Perez, Angelica Chen, Nikita Nangia, Jason Phang, and Samuel R. Bowman. Single-turn debate does not help humans answer hard reading-comprehension questions, 2022 b . URL https://arxiv.org/abs/2204.05212

    2022 arXiv

  38. [46]

    Human Compatible: Artificial Intelligence and the Problem of Control

    Stuart Russell. Human Compatible: Artificial Intelligence and the Problem of Control. Penguin Books, New York, 2019. ISBN 9780525558637

    2019

  39. [47]

    Goal misgeneralization: Why correct specifications aren't enough for correct goals, 2022

    Rohin Shah, Vikrant Varma, Ramana Kumar, Mary Phuong, Victoria Krakovna, Jonathan Uesato, and Zac Kenton. Goal misgeneralization: Why correct specifications aren't enough for correct goals, 2022. URL https://arxiv.org/abs/2210.01790

    2022 arXiv

  40. [48]

    An approach to technical AGI safety and security

    Rohin Shah, Alex Irpan, Alexander Matt Turner, Anna Wang, Arthur Conmy, David Lindner, Jonah Brown-Cohen, Lewis Ho, Neel Nanda, Raluca Ada Popa, Rishub Jain, Rory Greig, Samuel Albanie, Scott Emmons, Sebastian Farquhar, S \'e bastien Krier, Senthooran Rajamanoharan, Sophie Bri...

    2025

  41. [49]

    Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R. Bowman, Esin Durmus, Zac Hatfield-Dodds, Scott R Johnston, Shauna M Kravec, Timothy Maxwell, Sam McCandlish, Kamal Ndousse, Oliver Rausch, Nicholas Schiefer, Da Yan, Miranda Zhang, and Ethan Pere...

    2024

  42. [50]

    Joar Skalse, Nikolaus H. R. Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing reward hacking, 2025. URL https://arxiv.org/abs/2209.13085

    2025 arXiv

  43. [51]

    Frontier AI safety commitments, AI seoul summit 2024

    UK and Republic of Korea . Frontier AI safety commitments, AI seoul summit 2024. Government publication, Department for Science, Innovation and Technology, May 2024. URL https://www.gov.uk/government/publications/frontier-ai-safety-commitments-ai-seoul-summit-2024/frontier-ai-...

    2024

  44. [52]

    Defence standard 00-56 issue 4: S afety management requirements for defence systems

    UK Ministry of Defence . Defence standard 00-56 issue 4: S afety management requirements for defence systems. Technical report, UK Ministry of Defence, 2007

    2007

  45. [53]

    Bowman, He He, and Shi Feng

    Jiaxin Wen, Ruiqi Zhong, Akbir Khan, Ethan Perez, Jacob Steinhardt, Minlie Huang, Samuel R. Bowman, He He, and Shi Feng. Language models learn to mislead humans via rlhf, 2024. URL https://arxiv.org/abs/2409.12822

    2024 arXiv

  46. [54]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.