Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

Out of Control -- Why Alignment Needs Formal Control Theory (and an Alignment Control Stack)

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that formal optimal control theory should be central to AI alignment, and proposes a ten-layer Alignment Control Stack to make alignment methods formal, interoperable, and amenable to assurance.

desk verdict Useful taxonomy, honest position piece, but the control-theory core is a textbook toy model and the jump to frontier LLMs is analogy, not argument. read the letter →

arxiv 2506.17846 v1 pith:FRPWR5LP submitted 2025-06-21 cs.AI

Elija Perrier This is my paper
classification cs.AI
keywords AIalignmentcontroltheoryoptimalStackLQGstochasticsafetymulti-agentsystems
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This position paper argues that AI alignment research suffers from two systematic gaps: results are often empirical and hard to generalise, and there is no shared taxonomy for where and how controls are applied. The proposed fix is to put formal optimal control theory at the centre of alignment and to organise the field around an Alignment Control Stack, a ten-layer hierarchy from physical hardware up to societal governance. Each layer exposes state variables, measurements, and control handles, and the paper shows with a toy linear-Gaussian model how two layers (training and behavioural output) can be jointly optimised as a single LQG control problem via the separation principle. The intended payoff is that alignment methods become comparable, composable, and able to carry formal guarantees, which the paper argues is also what regulators need. The position is that this bridges empirically validated control methods with practical deployment of frontier AI.

What carries the argument

The central object is the Alignment Control Stack (ACS): a ten-layer hierarchy—physical hardware, system software, AI framework, model architecture, training, behavioural output, interpretability, preference/reward, multi-agent, and socio-technical—in which each layer has its own state variables, measurable signals, and control handles, and layers are coupled vertically (lower layers shape upper ones) and horizontally (parallel agents or controls interact). The mathematical machinery that carries the vertical example is the Linear-Quadratic-Gaussian (LQG) framework: with a linear state update $w_{t+1}=w_t-u_t+\xi_t$, a linear observation $y_t=c^\top w_t+v_t$, and quadratic costs, the separation principle says the optimal strategy is a Kalman filter for the unobserved weights plus a Riccati-derived linear feedback law on the estimate. The horizontal example is carried by Hamilton–Jacobi–Isaacs (HJI) equations from dynamic game theory, which characterise Nash equilibria for coupled stochastic systems and give a formal handle for mechanism design. These tools do the argument's work by showing that even a minimal, noisy, partially observed model of LLM training and behaviour admits provably optimal control.

What would settle it

Run a real or realistic training run with minibatch noise and measure the residuals of the equations $w_{t+1}=w_t-u_t+\xi_t$, $y_t=c^\top w_t+v_t$; if the residuals are strongly non-Gaussian, state-dependent, or high-dimensional in a way that biases the Kalman estimate, the separation principle no longer applies and controllers built on it will miss their safety targets.

Watch

Extended reading notes

Core claim

Stated on the paper's own terms, the central claim is that alignment should be recast as a formal optimal control problem over a layered AI stack. Prevailing safety work—red teaming, RLHF, mechanistic interpretability, protocol evaluations—provides valuable but largely empirical, method-specific insights; formal control theory supplies the missing mathematical language: state-space models, controller synthesis through Riccati equations and dynamic programming, robust and game-theoretic analysis for adversarial settings, and stochastic estimation for noisy, partially observed systems. The Alignment Control Stack is the coordinating object: ten layers, each with its own measurement signals and admissible control actions, with vertical coupling between layers and horizontal coupling between parallel agents or controls. The paper's worked example takes a training-process layer with dynamics $w_{t+1} = w_t - u_t + \xi_t$ and a behavioural layer with readout $y_t = c^\top w_t + v_t$, combines them into an augmented LQG system, and shows that the separation principle makes the Kalman filter plus feedback law jointly optimal. The conclusion is that this kind of formal treatment should underpin alignment research and deployment assurance.

Load-bearing premise

The load-bearing premise is that the linear-Gaussian surrogate—linear weight updates plus Gaussian noise and a linear readout—resembles real LLM training and behaviour closely enough that conclusions from the LQG example carry over to actual systems.

Editorial extensions

If this is right

  • Alignment proposals could be compared by shared formal criteria—stability, reachability, robustness, and expected cost under uncertainty—rather than by isolated empirical demonstrations.
  • Under the LQG surrogate, training-layer state regulation and behavioural-layer output estimation can be designed separately and still be jointly optimal, which greatly simplifies stacked safety mechanisms.
  • The Alignment Control Stack gives regulators and auditors a structured way to ask where a control is applied, what it measures, and what assurance it provides, supporting impact assessments and ongoing monitoring.
  • In multi-agent settings, modifying reward functions, communication channels, or allowed action spaces becomes a formal mechanism-design problem with game-theoretic equilibria as the target.
  • Stochastic and robust control can supply high-probability safety bounds for noisy systems like LLMs, complementing purely deductive formal verification.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's LQG example is explicitly a toy; a natural extension is to test on real models whether a Kalman-style estimate over activation or parameter states improves downstream safety control, and where the linear-Gaussian residual breaks down.
  • The ACS could be operationalized as a concrete compliance artifact—each layer with named indicators, control methods, and failure modes—though the paper leaves that engineering step implicit.
  • The horizontal game-theoretic framing suggests a specific testable intervention: restricting covert communication channels between agents should shift equilibria away from collusion, which connects directly to steganography results the paper cites.
  • Because the LQG optimality is exact only under linear-Gaussian assumptions, the next natural layer is robust control: if real dynamics deviate, an H-infinity formulation would preserve worst-case bounds, a direction the paper motivates but does not prove.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This position paper argues that formal optimal control theory should be central to AI alignment research, and it introduces a ten-layer Alignment Control Stack (ACS) that spans physical hardware, software, training, behavior, interpretability, rewards, multi-agent dynamics, and societal governance. The paper identifies two perceived gaps in alignment research—a lack of formalization and a lack of coordination/interoperability between alignment techniques—and proposes the ACS plus control-theoretic synthesis as remedies. It provides a mathematical primer on optimal control, a toy LQG model intended to illustrate vertical alignment between a training-process layer and a behavioral-output layer, a game-theoretic sketch of horizontal alignment between multiple AI stacks, and a section responding to objections about complexity, interpretability, specification gaming, and empiricism.

Significance. If the programmatic claims were substantiated, the ACS could provide a useful shared vocabulary for comparing alignment interventions and for reasoning about where controls are applied, and the formal-control framing could bring genuinely rigorous tools to bear on concrete safety problems. The paper has real strengths: it is honest about being a position paper, it engages with standard objections, it provides a detailed and well-organized taxonomy in the appendices, and the LQG derivation for the stated linear-Gaussian surrogate is correct textbook material. However, the central generalization from the toy model to real large language models is unanalyzed, and the interoperability claims are asserted rather than demonstrated. The significance of the paper is therefore primarily programmatic: it identifies a promising direction and a useful organizational scheme, but it does not yet establish that formal control theory can deliver the promised guarantees for frontier AI systems.

major comments (3)
  1. [Sections 4.1–4.2 and 5] The load-bearing claim that formal control theory can provide a rigorous foundation for advanced AI systems is not supported by the toy LQG example. The paper itself calls Eq. (2) 'the simplest linear-Gaussian surrogate' and Section 4.1 a 'toy model,' yet the separation-principle result is exact only for that linear-Gaussian plant. No argument is supplied that real LLM training or deployment dynamics admit a state-space model with known read-out vector c, Gaussian noise, and stable linear structure. Section 5's response—that stochastic and robust control are 'specifically designed for uncertain, high-dimensional systems'—is a statement of intent, not a demonstration that LLMs lie in the applicable regime. To make the central claim load-bearing, the paper needs either a faithfulness argument for a state-space representation of at least one concrete stage of training or inference, or an explicit statement of the conditions under which the LQG guarantees degrade gracefully under model mismatch.
  2. [Section 3.1] The interoperability claim is central to the proposed Alignment Control Stack, but it is only asserted. The claim that 'layer-local controllers can expose contractable interface variables whose behaviour can be used by the next layer up as a well-modelled plant' is not backed by a composition theorem, an interface contract, or a worked example of two controllers composed across layers. The LQG example in Section 4.1 is a single plant with a two-part output; it does not implement the hierarchical composition promised in Section 3.1. Without such a result or a concrete specification, the assertion that the ACS 'can be used to facilitate interoperability analysis' remains a taxonomy-level aspiration rather than a technical proposal.
  3. [Section 4.2, Eq. (3)] The augmented system in Eq. (3) is not a standard state-space model: it places the observation y_t as a component of the 'next state' vector while also using y_t as the measured output. A correct formulation would be the pair w_{t+1}=w_t-u_t+ξ_t and y_t=c^T w_t+v_t, with the LQG estimator and controller acting on the state estimate. As written, the claimed separation-principle solution does not follow from Eq. (3). The intended textbook result is recoverable from Eqs. (1)–(2), but the presentation in Eq. (3) is a genuine technical flaw in the paper's main worked example.
minor comments (4)
  1. [Section 4.1] The label 'deterministic control' for Layer 5 is inconsistent with the stochastic dynamics in Eq. (1), which include the Gaussian noise term ξ_t; clarify that the plant is stochastic and that the controller is deterministic only given the current state estimate.
  2. [Table 1] In Layer 1, the rows 'Specialized Processors' and 'Accelerator Architecture' list identical focus, measurements, and control methods; consolidate or differentiate these entries.
  3. [References] References [34] and [35] are duplicate listings of the same paper 'AI safety via debate'; retain the correct author list in a single entry.
  4. [Section 4.2] The claim of 'provable minimal expected cost' should be qualified explicitly by the LQG assumptions (linear dynamics, Gaussian noise, known covariances, quadratic cost); otherwise the statement overreaches the conditions under which the separation principle applies.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the LQG example uses an independent, textbook separation theorem under an explicitly stated surrogate model, and the Alignment Control Stack is a taxonomy, not a derivation.

full rationale

The paper makes no fitted predictions and contains no load-bearing self-citations or imported uniqueness claims. The only formal derivation, Sections 4.1 and 4.2, starts with an explicitly labeled surrogate ("We adopt the simplest linear-Gaussian surrogate", Eq. (2)) and then applies textbook LQG separation results (Appendix A.2) to that model; the resulting "provable minimal expected cost" is a standard theorem conditional on the stated linear-Gaussian assumptions, not a conclusion that re-imports its own inputs. The Alignment Control Stack (Fig. 1, Table 1) is a hierarchical taxonomy classifying known control mechanisms; it organizes and names layers rather than deriving a prediction from them, so its self-referential framing as a proposed standard is not equation-level circularity. Section 5 acknowledges the complexity objection and offers programmatic responses; those are arguments about scope and applicability, not circular reductions. The paper's acknowledged limitations (e.g., "The approach is not complete and the use of one formalism over another is highly context-dependent") further indicate that no result is being presented as forced or self-proving. The absence of a layer-composition theorem in Section 3.1 is a completeness gap, not a circularity. No circular step meeting the quote-and-reduction standard was found.

Assumptions & free parameters 0 free parameters · 4 assumptions · 1 invented entities

The central argument rests on three families of assumptions: that AI systems are usefully modeled as stochastic dynamical systems; that the ten-layer ACS decomposition yields meaningful interfaces; and that control-theoretic guarantees transfer from toy surrogates to frontier models. None of these is empirically established in the paper, and together they constitute the non-trivial content that the position needs to carry.

assumptions (4)
  • domain assumption AI systems, including frontier LLMs, can be usefully modeled as stochastic dynamical systems amenable to control-theoretic analysis.
    Invoked throughout, especially in Section 3 and the LQG example, but never empirically established. Real LLMs have enormous state spaces and nonlinear dynamics.
  • ad hoc to paper The linear-Gaussian surrogate of Section 4.1 (w_{t+1}=w_t-u_t+ξ_t, y_t=c^T w_t+v_t) is representative enough to transfer LQG conclusions to real training and behavior.
    The paper explicitly adopts this surrogate 'for simplicity' and provides no evidence of fidelity to real systems.
  • ad hoc to paper The ten-layer Alignment Control Stack decomposition yields meaningful, contractable interfaces between layers.
    The stack is introduced in Section 3 and used to claim vertical and horizontal interoperability, but no formal interface definition or validation is given.
  • domain assumption Classical control guarantees such as stability and robustness transfer from engineered physical systems to socio-technical and governance layers.
    The paper repeatedly extends control concepts to Layers 9 and 10 (multi-agent and societal) without a model of how economic or governance dynamics obey controllability or observability assumptions.
invented entities (1)
  • Alignment Control Stack (ACS): a ten-layer hierarchical taxonomy from physical hardware to societal impact.
    purpose: To organize measurement and control interventions across the AI stack and to enable analysis of vertical and horizontal interoperability.
    The ACS is introduced as a conceptual framework. No falsifiable predictions are derived from it, and no empirical validation is attempted. It functions as an organizing metaphor rather than a measurable entity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Out of Control -- Why Alignment Needs Formal Control Theory (and an Alignment Control Stack)." pith.science (2026). https://pith.science/paper/FRPWR5LP

@misc{pith2026250617846,
  author       = {Pith},
  title        = {Pith review of: Out of Control -- Why Alignment Needs Formal Control Theory (and an Alignment Control Stack)},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FRPWR5LP}},
  note         = {Machine review of arXiv:2506.17846}
}
read the original abstract

This position paper argues that formal optimal control theory should be central to AI alignment research, offering a distinct perspective from prevailing AI safety and security approaches. While recent work in AI safety and mechanistic interpretability has advanced formal methods for alignment, they often fall short of the generalisation required of control frameworks for other technologies. There is also a lack of research into how to render different alignment/control protocols interoperable. We argue that by recasting alignment through principles of formal optimal control and framing alignment in terms of hierarchical stack from physical to socio-technical layers according to which controls may be applied we can develop a better understanding of the potential and limitations for controlling frontier models and agentic AI systems. To this end, we introduce an Alignment Control Stack which sets out a hierarchical layered alignment stack, identifying measurement and control characteristics at each layer and how different layers are formally interoperable. We argue that such analysis is also key to the assurances that will be needed by governments and regulators in order to see AI technologies sustainably benefit the community. Our position is that doing so will bridge the well-established and empirically validated methods of optimal control with practical deployment considerations to create a more comprehensive alignment framework, enhancing how we approach safety and reliability for advanced AI systems.

Figures

Figures reproduced from arXiv: 2506.17846 by the authors.

Figure 1
Figure 1. Alignment Control Stack (ACS). The Alignment Control Stack details a vertically hierarchi￾cal set of layers at which various forms of control may be utilised, ranging from the physical circuitry level (e.g. compute control) through to control via model hyperparameters, model behavioural testing (e.g. adversarial methods) and socio-technical measures (governance). from learned optimisation [21] or self-supervised lea… view at source ↗
Figure 1
Figure 1. Each layer presents a means of measurement and control of integrated AI systems. This [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Towards Measurement Theory for Artificial Intelligence

    cs.AI 2025-07 conditional novelty 4.0 of 10

    A formal measurement theory for AI, built from representational measurement theory, measure theory, metrology, and psychometrics, would make evaluations of AI systems commensurable and scientifically grounded.

Reference graph

Works this paper leans on

97 extracted references · 55 canonical work pages · cited by 1 Pith paper

  1. [1]

    AI control: Improv- ing safety despite intentional subversion

    Ryan Greenblatt, Buck Shlegeris, Kshitij Sachan, and Fabien Roger. AI control: Improv- ing safety despite intentional subversion. arXiv preprint arXiv:2312.06942 (Published in Proceedings of the 41st International Conference on Machine Learning), 2023

    arXiv 2023

  2. [2]

    A sketch of an AI control safety case

    Tomek Korbak, Joshua Clymer, Benjamin Hilton, Buck Shlegeris, and Geoffrey Irving. A sketch of an AI control safety case. arXiv preprint arXiv:2501.17315, 2025

    arXiv 2025

  3. [3]

    Subversion strategy eval: Can language models statelessly strategize to subvert control protocols? arXiv preprint arXiv:2412.12480, 2024

    Alex Mallen, Charlie Griffin, Misha Wagner, Alessandro Abate, and Buck Shlegeris. Subversion strategy eval: Can language models statelessly strategize to subvert control protocols? arXiv preprint arXiv:2412.12480, 2024

    arXiv 2024

  4. [4]

    Adaptive deployment of untrusted LLMs reduces distributed threats

    Jiaxin Wen, Vivek Hebbar, Caleb Larson, Aryan Bhatt, Ansh Radhakrishnan, Mrinank Sharma, Henry Sleight, Shi Feng, He He, Ethan Perez, Buck Shlegeris, and Akbir Khan. Adaptive deployment of untrusted LLMs reduces distributed threats. arXiv preprint arXiv:2411.17693, 2024

    arXiv 2024

  5. [5]

    Hidden in plain text: Emergence & mitigation of steganographic collusion in LLMs

    Yohan Mathew, Ollie Matthews, Robert McCarthy, Joan Velja, Christian Schroeder de Witt, Dy- lan Cope, and Nandi Schoots. Hidden in plain text: Emergence & mitigation of steganographic collusion in LLMs. arXiv preprint arXiv:2410.03768, 2024

    arXiv

  6. [6]

    Mechanistic anomaly detection and elk

    Paul Christiano. Mechanistic anomaly detection and elk. https://ai-alignment.com/ mechanistic-anomaly-detection-and-elk-fb84f4c6d0dc , 2022

    2022

  7. [7]

    Representation engineering: A top-down approach to ai transparency

    Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al. Representation engineering: A top-down approach to ai transparency. ArXiv preprint, abs/2310.01405, 2023

    arXiv

  8. [8]

    Michaud, Stephen Casper, Max Tegmark, William Saunders, David Bau, Eric Todd, Atticus Geiger, Mor Geva, Jesse Hoogland, Daniel Murfet, and Tom McGrath

    Lee Sharkey, Bilal Chughtai, Joshua Batson, Jack Lindsey, Jeff Wu, Lucius Bushnaq, Nicholas Goldowsky-Dill, Stefan Heimersheim, Alejandro Ortega, Joseph Bloom, Stella Biderman, Adria Garriga-Alonso, Arthur Conmy, Neel Nanda, Jessica Rumbelow, Martin Wattenberg, Nandi Schoots, Joseph Miller, Eric J. Michaud, Stephen Casper, Max Tegmark, William Saunders, D...

    arXiv 2025

Show all 97 references
  1. [9]

    Berkovitz and Negash G

    Leonard D. Berkovitz and Negash G. Medhin. Nonlinear Optimal Control Theory. Chapman and Hall/CRC, 1995

    1995

  2. [10]

    A research agenda: Dynamic models to defend against correlated attacks

    Ian Goodfellow. A research agenda: Dynamic models to defend against correlated attacks. arXiv preprint arXiv:1903.06293, 2019

    1903 arXiv

  3. [11]

    Learning to summarize from human feedback

    Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M Ziegler, Ryan Lowe, Chelsea V oss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback. Advances in Neural Information Processing Systems, 33:3008–3021, 2020

    2020

  4. [12]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730...

    2022

  5. [13]

    Fine-tuning language models from human preferences

    Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019

    1909 arXiv

  6. [14]

    A general language assistant as a laboratory for alignment

    Amanda Askell, Yuntao Bai, Anna Chen, Sourced Dathathri, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Jackson Kernion, et al. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861, 2021

    2021 arXiv

  7. [15]

    Constitutional ai: Harmlessness from ai feedback

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022. 11

    2022 arXiv

  8. [16]

    Cooperative inverse reinforcement learning

    Dylan Hadfield-Menell, Stuart J Russell, Pieter Abbeel, and Anca Dragan. Cooperative inverse reinforcement learning. In Advances in neural information processing systems, pages 3909– 3917, 2016

    2016

  9. [17]

    Scalable agent alignment via reward modeling: A research direction

    Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable agent alignment via reward modeling: A research direction. arXiv preprint arXiv:1811.07871, 2018

    2018 arXiv

  10. [18]

    Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned

    Ethan Perez, Saffron Huang, Yinan He, Chunting Song, Jared Hilton, Mónica Co-Reyes, Chelsea V oss, Hieu Cat Tran, Thorir Brynjolfsson, Shawn Sherred, et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209....

    2022 arXiv

  11. [19]

    Goal mis- generalization: Why correct specifications aren’t enough

    Lauro Langosco, Lee Sharkey, Jan Pfau, David Krueger, and Laurent Orseau. Goal mis- generalization: Why correct specifications aren’t enough. arXiv preprint arXiv:2310.07907, 2023

    2023 arXiv

  12. [20]

    Consequences of misaligned AI

    Simon Zhuang and Dylan Hadfield-Menell. Consequences of misaligned AI. In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Ad- vances in Neural Information Processing Systems 33: Annual Conference on Neural Information Pro...

    2020

  13. [21]

    Risks from learned optimization in advanced machine learning systems

    Evan Hubinger. Risks from learned optimization in advanced machine learning systems. arXiv preprint arXiv:1906.01820 (updated 2020), 2020

    1906 arXiv

  14. [22]

    The dangers of sutra: A roadmap to catastrophic risk from ai trained with self-supervised learning

    Lee Sharkey. The dangers of sutra: A roadmap to catastrophic risk from ai trained with self-supervised learning. arXiv preprint arXiv:2401.01909, 2024

    2024 arXiv

  15. [23]

    Steganography in chain of thought reasoning

    A Ray. Steganography in chain of thought reasoning. https://www.alignmentforum.org/ posts/yDcMDJeSck7SuBs24, 2022

    2022

  16. [24]

    Neural linguistic steganography

    Zachary Ziegler, Yuntian Deng, and Alexander Rush. Neural linguistic steganography. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1210–12...

    2019

  17. [25]

    Information hiding-a survey

    Fabien AP Petitcolas, Ross J Anderson, and Markus G Kuhn. Information hiding-a survey. Proceedings of the IEEE, 87(7):1062–1078, 1999

    1999

  18. [26]

    Cyclegan, a master of steganography.ArXiv preprint, abs/1712.02950, 2017

    Casey Chu, Andrey Zhmoginov, and Mark Sandler. Cyclegan, a master of steganography.ArXiv preprint, abs/1712.02950, 2017

    2017 arXiv

  19. [27]

    A review on text steganography techniques

    Mohammed Abdul Majeed, Rossilawati Sulaiman, Zarina Shukur, and Mohammad Kamrul Hasan. A review on text steganography techniques. Mathematics, 9(21):2829, 2021

    2021

  20. [28]

    Emergent misalignment: Narrow finetuning can produce broadly misaligned llms

    Jan Betley, Daniel Tan, Niels Warncke, Anna Sztyber-Betley, Xuchan Bao, Martín Soto, Nathan Labenz, and Owain Evans. Emergent misalignment: Narrow finetuning can produce broadly misaligned llms. arXiv preprint arXiv:2502.17424, 2025

    2025

  21. [29]

    Discovering latent knowledge in language models without supervision

    Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in language models without supervision. Advances in Neural Information Processing Systems, 36, 2023

    2023

  22. [30]

    Arc’s first technical report: Eliciting latent knowledge

    Paul Christiano, Mark Xu, and Ajeya Cotra. Arc’s first technical report: Eliciting latent knowledge. 2021

    2021

  23. [31]

    Is power-seeking ai an existential risk? arXiv preprint arXiv:2206.13353, 2022

    Joseph Carlsmith. Is power-seeking ai an existential risk? arXiv preprint arXiv:2206.13353, 2022

    2022 arXiv

  24. [32]

    Optimal policies tend to seek power

    Alexander Matt Turner, Logan Smith, Rohin Shah, Andrew Critch, and Prasad Tadepalli. Optimal policies tend to seek power. In Marc’Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors,Advances in Neural Information Processing S...

    2021

  25. [33]

    Corrigibility

    Nate Soares, Benja Fallenstein, Eliezer Yudkowsky, and Stuart Armstrong. Corrigibility. AAAI Workshop: AI and Ethics, 2015

    2015

  26. [35]

    Ai safety via debate

    Geoffrey Irving, Paul Christiano, and Dario Amodei. Ai safety via debate. ArXiv preprint, abs/1805.00899, 2018

    2018 arXiv

  27. [36]

    Debate helps supervise unreliable experts

    Julian Michael, Salsabila Mahdi, David Rein, Jackson Petty, Julien Dirani, Vishakh Padmaku- mar, and Samuel R Bowman. Debate helps supervise unreliable experts. ArXiv preprint, abs/2311.08702, 2023

    2023 arXiv

  28. [37]

    Towards guar- anteed safe ai: A framework for ensuring robust and reliable ai systems

    David Dalrymple, Joar Skalse, Yoshua Bengio, Stuart Russell, Max Tegmark, Sanjit Seshia, Steve Omohundro, Christian Szegedy, Ben Goldhaber, Nora Ammann, et al. Towards guar- anteed safe ai: A framework for ensuring robust and reliable ai systems. arXiv preprint arXiv:2405.06624, 2024

    2024 arXiv

  29. [38]

    Evaluating evasive jailbreaks and safety-washing in llms

    Ziqing Yao, Hongxin Wang, Wenbo Wang, Ruyi Wang, Jing Wu, and Run Jia. Evaluating evasive jailbreaks and safety-washing in llms. arXiv preprint arXiv:2402.02867, 2024

    2024 arXiv

  30. [39]

    The effects of reward misspecification: Mapping and mitigating misaligned models

    Alexander Pan, Kushal Bhatia, and Jacob Steinhardt. The effects of reward misspecification: Mapping and mitigating misaligned models. arXiv preprint arXiv:2201.03544, 2022

    2022 arXiv

  31. [40]

    The alignment problem from a deep learning perspective

    Richard Ngo, Lawrence Chan, and Sören Mindermann. The alignment problem from a deep learning perspective. arXiv preprint arXiv:2209.00626, 2022

    2022 arXiv

  32. [41]

    Deep reinforcement learning from human preferences

    Paul F Christiano, Jan Leike, Tom B Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in neural information processing systems, volume 30, 2017

    2017

  33. [42]

    Pre- training language models with human preferences for few-shot learning

    Ramana Kumar Ramamurthy, Sandhini Agarwal, Long Ouyang, Jeff Wu, Nisan Stiennon, Daniel M Ziegler, Ryan Lowe, Chelsea V oss, Alec Radford, Dario Amodei, et al. Pre- training language models with human preferences for few-shot learning. arXiv preprint arXiv:2202.03202, 2022

    2022 arXiv

  34. [43]

    Measuring progress on scalable oversight for large language models

    Samuel R Bowman, Jeeyoon Hyun, Ethan Perez, Edwin Chen, Craig Pettit, Scott Heiner, Kamil˙e Lukoši¯ut˙e, Amanda Askell, Andy Jones, Anna Chen, et al. Measuring progress on scalable oversight for large language models. arXiv preprint arXiv:2211.03540, 2022

    2022 arXiv

  35. [44]

    Discovering language model behaviors with model-written evaluations

    Ethan Perez, Sam Ringer, Karina Lukoši¯ut˙e, Krystal Nguyen, Edwin Chen, Yinan He, Newton Howland, Chip Huyen, Lazaro LTorch, Mantas Miranda, et al. Discovering language model behaviors with model-written evaluations. arXiv preprint arXiv:2212.09251, 2022

    2022 arXiv

  36. [45]

    Steering llama 2 via contrastive activation addition

    Alexander Matt Turner, Lisa Anne Smith, Amphilochia Variengien, S K Ghasemipour, Jacob Steinhardt, and David Krueger. Steering llama 2 via contrastive activation addition. arXiv preprint arXiv:2312.06681, 2023

    2023 arXiv

  37. [46]

    Truthful ai: Developing and governing ai that does not lie

    Owain Evans, Andreas Stuhlmüller, and CDE. Truthful ai: Developing and governing ai that does not lie. arXiv preprint arXiv:2110.06674, 2021

    2021 arXiv

  38. [47]

    Safe reinforcement learning via shielding

    Mohammed Alshiekh, Roderick Bloem, Rüdiger Ehlers, Bettina Könighofer, Scott Niekum, and Ufuk Topcu. Safe reinforcement learning via shielding. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018

    2018

  39. [48]

    Robust adversarial reinforcement learning

    Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforcement learning. In International conference on machine learning, pages 2817–2826. PMLR, 2017

    2017

  40. [49]

    Dynamic programming and optimal control, volume 1

    Dimitri P Bertsekas. Dynamic programming and optimal control, volume 1. Athena scientific, 3rd edition, 2005

    2005

  41. [50]

    Robust and optimal control

    Kemin Zhou, John C Doyle, and Keith Glover. Robust and optimal control. Prentice Hall, 1996

    1996

  42. [51]

    Optimal control theory: an introduction

    Donald E Kirk. Optimal control theory: an introduction. Prentice-Hall, 1970. 13

    1970

  43. [52]

    Optimal control

    Frank L Lewis, Draguna Vrabie, and Vassilis L Syrmos. Optimal control. John Wiley & Sons, 3rd edition, 2012

    2012

  44. [53]

    Adaptive control

    Karl Johan Åström and Björn Wittenmark. Adaptive control. Courier Corporation (originally Dover Publications, 2008), 2nd edition, 2013

    2008

  45. [54]

    Model predictive control

    Eduardo F Camacho and Carlos Bordons. Model predictive control. Springer Science & Business Media, 2nd edition, 2013

    2013

  46. [55]

    Stochastic optimal control: the discrete-time case , volume 5

    Dimitri Bertsekas and Steven E Shreve. Stochastic optimal control: the discrete-time case , volume 5. Athena Scientific, 1996

    1996

  47. [56]

    Nonlinear control systems

    Alberto Isidori. Nonlinear control systems. Springer Science & Business Media, 3rd edition, 1995

    1995

  48. [57]

    Geometric control of mechanical systems: modeling, analysis, and design for simple mechanical control systems

    Francesco Bullo and Andrew D Lewis. Geometric control of mechanical systems: modeling, analysis, and design for simple mechanical control systems . Springer Science & Business Media, 2004

    2004

  49. [58]

    Hybrid dynamical systems: modeling, stability, and robustness

    Rafal Goebel, Ricardo G Sanfelice, and Andrew R Teel. Hybrid dynamical systems: modeling, stability, and robustness. Princeton University Press, 2012

    2012

  50. [59]

    Reinforcement learning: An introduction

    Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2nd edition, 2018. 14 A Primer on Formal Optimal Control Theory This appendix provides a brief, mathematically detailed overview of key concepts from optimal control theory relevant to AI ...

    2018

  51. [60]

    ˙x∗(t) = ∂H ∂λ (t, x∗(t), u∗(t), λ(t)) (12) ˙λ(t) = − ∂H ∂x (t, x∗(t), u∗(t), λ(t)) (13) where the Hamiltonian H(t, x, u, λ) = λ0f 0(t, x, u) + Pn i=1 λi(t)f i(t, x, u)

    Adjoint Dynamics (Costate Equations). ˙x∗(t) = ∂H ∂λ (t, x∗(t), u∗(t), λ(t)) (12) ˙λ(t) = − ∂H ∂x (t, x∗(t), u∗(t), λ(t)) (13) where the Hamiltonian H(t, x, u, λ) = λ0f 0(t, x, u) + Pn i=1 λi(t)f i(t, x, u)

  52. [61]

    For almost all t ∈ [t0, t1], H(t, x∗(t), u∗(t), λ(t)) = sup z∈Ω(t,x∗(t)) H(t, x∗(t), z, λ(t)) (14)

    Maximality Condition (for λ0 = −1, minimizing J). For almost all t ∈ [t0, t1], H(t, x∗(t), u∗(t), λ(t)) = sup z∈Ω(t,x∗(t)) H(t, x∗(t), z, λ(t)) (14)

  53. [62]

    At the endpoints (t0, x∗(t0)) and (t1, x∗(t1)), the vector (−H(t0), λ(t0), H(t1), −λ(t1)) (where H(t) is H evaluated along the extremal) is orthog- onal to the end-manifold B

    Transversality Conditions. At the endpoints (t0, x∗(t0)) and (t1, x∗(t1)), the vector (−H(t0), λ(t0), H(t1), −λ(t1)) (where H(t) is H evaluated along the extremal) is orthog- onal to the end-manifold B. The PMP converts an infinite-dimensional optimisation problem into a probl...

  54. [63]

    Estimate the state ˆxt = E[xt|Yt] (where Yt is history of observations ys, s≤ t) using a Kalman Filter: dˆxt = (A(t)ˆxt + B(t)ut)dt + K(t)(dyt − H(t)ˆxtdt) (21) K(t) = P (t)H(t)T (D(t)D(t)T )−1 (22) ˙P (t) = A(t)P (t) + P (t)A(t)T − P (t)H(t)T (D(t)D(t)T )−1H(t)P (t) + C(t)C(t...

  55. [64]

    Apply the optimal deterministic LQR control law ut = −L(t)ˆxt, where L(t) = R(t)−1B(t)T S(t) and S(t) is the solution to a matrix Riccati differential equation: − ˙S(t) = A(t)T S(t) + S(t)A(t) − S(t)B(t)R(t)−1B(t)T S(t) + Q(t) with S(T ) = QT . A.3 Robust and Game-Theoretic Co...

  56. [65]

    This involves adjusting operational parameters of physical components

    Hardware Parameter Regulation . This involves adjusting operational parameters of physical components. For instance, Dynamic V oltage and Frequency Scaling (DVFS) optimizes the power-performance trade-off. • Purpose. To manage energy consumption while meeting performance targe...

  57. [66]

    This concerns distributing computational tasks across available processing units (e.g., GPU kernels)

    Workload Scheduling and Resource Allocation. This concerns distributing computational tasks across available processing units (e.g., GPU kernels). • Purpose. To maximise resource utilization, ensure fairness, or meet quality-of-service (QoS) for AI tasks. • Implementation. Sch...

  58. [67]

    This includes mechanisms like Error Correction Codes (ECC) in memory or redundant array of inexpensive disks (RAID) for storage

    Error Detection and Correction. This includes mechanisms like Error Correction Codes (ECC) in memory or redundant array of inexpensive disks (RAID) for storage. • Purpose. To ensure data integrity and system resilience against hardware faults. • Implementation. Hardware-level ...

  59. [68]

    OS and containerization tools allow setting limits on resource consumption per process or container (e.g., Linux cgroups)

    Resource Quotas and Limits. OS and containerization tools allow setting limits on resource consumption per process or container (e.g., Linux cgroups). • Purpose. To prevent resource exhaustion by any single AI component, ensure fair sharing, and isolate processes. • Implementa...

  60. [69]

    The OS scheduler manages the execution order of processes and threads

    Process Scheduling and Prioritization. The OS scheduler manages the execution order of processes and threads. • Purpose. To optimize system responsiveness and throughput according to defined policies, ensuring critical AI processes receive adequate CPU time. • Implementation. ...

  61. [70]

    Using containers, virtual machines, or specific OS mechanisms (e.g., seccomp, AppArmor) to restrict the privileges and visibility of AI processes

    Sandboxing and Isolation. Using containers, virtual machines, or specific OS mechanisms (e.g., seccomp, AppArmor) to restrict the privileges and visibility of AI processes. • Purpose. To enhance security by limiting the potential impact of a compromised AI component or untrust...

  62. [71]

    Frameworks often perform optimisations like op- erator fusion, constant folding, and memory planning on the AI model’s computational graph

    Computational Graph Optimisation. Frameworks often perform optimisations like op- erator fusion, constant folding, and memory planning on the AI model’s computational graph. • Purpose. To accelerate model execution and reduce memory usage during training and inference. • Imple...

  63. [72]

    Techniques applied during computation to maintain numeri- cal precision and prevent issues like vanishing/exploding gradients

    Numerical Stability Controls. Techniques applied during computation to maintain numeri- cal precision and prevent issues like vanishing/exploding gradients. • Purpose. To ensure stable and effective model training. 19 • Implementation. Gradient clipping (if ∥∇L∥2 > Cclip, then...

  64. [73]

    Mechanisms for managing data and model parallelism, synchronizing gradients, and handling parameter updates across multiple devices or nodes

    Distributed Training Coordination. Mechanisms for managing data and model parallelism, synchronizing gradients, and handling parameter updates across multiple devices or nodes. • Purpose. To scale training to large models and datasets. • Implementation. Parameter server archit...

  65. [74]

    Its efficiency and correctness are crucial for successful model development

    and during the training process (Layer 5). Its efficiency and correctness are crucial for successful model development. Controls within the framework can ensure reproducible computations, manage resource usage at a finer grain than the OS, and implement fundamental stability c...

  66. [75]

    The deliberate selection of network structures, layer types, and connectivity

    Architectural Design Choices. The deliberate selection of network structures, layer types, and connectivity. • Purpose. To incorporate domain knowledge, manage model capacity, and influence learnability and generalisation. For example, convolutional layers for spatial equivari...

  67. [76]

    Incorporating architectural elements that intrinsi- cally promote better generalisation or robustness, such as dropout layers or batch normalisa- tion

    Regularization through Architecture. Incorporating architectural elements that intrinsi- cally promote better generalisation or robustness, such as dropout layers or batch normalisa- tion. • Purpose. To prevent overfitting and improve model performance on unseen data. • Implem...

  68. [77]

    Methods like pruning (removing less important weights/neurons) and quantization (reducing the precision of weights and activations)

    Model Compression Techniques . Methods like pruning (removing less important weights/neurons) and quantization (reducing the precision of weights and activations). • Purpose. To reduce model size, inference latency, and computational cost, making models more deployable, especi...

  69. [78]

    Selecting, cleaning, and transforming the training data

    Data Curation, Augmentation, and Filtering. Selecting, cleaning, and transforming the training data. • Purpose. To improve the quality, diversity, and representativeness of the training set, reduce biases, and enhance model generalisation. • Implementation. Applying filters ba...

  70. [79]

    Defining the loss function Ltask that the model aims to minimise, and potentially adding regularization terms Lregi or constraints

    Objective Function Design . Defining the loss function Ltask that the model aims to minimise, and potentially adding regularization terms Lregi or constraints. • Purpose. To guide the learning process towards desired behaviours and encode prefer- ences or constraints beyond si...

  71. [80]

    Choosing the optimizer (e.g., SGD, Adam) and its parameters (e.g., learning rate ηt, batch size Bt, momentum βt)

    Optimisation Algorithm and Hyperparameter Management. Choosing the optimizer (e.g., SGD, Adam) and its parameters (e.g., learning rate ηt, batch size Bt, momentum βt). • Purpose. To efficiently and effectively navigate the loss landscape and find model parameters that yield go...

  72. [81]

    Monitoring performance on a validation set and stop- ping training when performance no longer improves, saving model checkpoints periodically

    Early Stopping and Checkpointing. Monitoring performance on a validation set and stop- ping training when performance no longer improves, saving model checkpoints periodically. • Purpose. To prevent overfitting to the training data and to save the model state that generalises ...

  73. [82]

    Post-processing model outputs to detect and remove or modify undesirable content (e.g., harmful language, private information)

    Output Filtering and Sanitization. Post-processing model outputs to detect and remove or modify undesirable content (e.g., harmful language, private information). • Purpose. To act as a safety net, preventing the model from causing immediate harm through its outputs. • Impleme...

  74. [83]

    Abstaining from providing an output or escalating to a human supervisor if the model’s confidence in its output is below a certain threshold

    Confidence-Based Rejection or Escalation . Abstaining from providing an output or escalating to a human supervisor if the model’s confidence in its output is below a certain threshold. • Purpose. To reduce risks associated with incorrect or unreliable AI outputs, especially in...

  75. [84]

    Runtime Monitoring and Anomaly Detection. Observing model outputs and behaviour over time to detect deviations from expected patterns, which might indicate issues like model drift, adversarial attacks, or emergent misbehaviour. • Purpose. To provide an ongoing assessment of mo...

  76. [85]

    Altering or ablating spe- cific internal parts of a model (e.g., neurons, attention heads, identified circuits) that are understood to contribute to certain behaviours

    Mechanistic Interventions based on Identified Components. Altering or ablating spe- cific internal parts of a model (e.g., neurons, attention heads, identified circuits) that are understood to contribute to certain behaviours. • Purpose. To test causal hypotheses about model f...

  77. [86]

    Using explanations of model behaviour (e.g., why a particular output was generated) to identify and diagnose issues

    Explanation-Guided Monitoring and Debugging. Using explanations of model behaviour (e.g., why a particular output was generated) to identify and diagnose issues. • Purpose. To gain insights into failure modes, biases, or spurious correlations learned by the model. • Implementa...

  78. [87]

    Directly modifying model weights or structure based on interpretability insights to correct specific knowledge or behaviours without full retraining

    Model Editing and Refinement. Directly modifying model weights or structure based on interpretability insights to correct specific knowledge or behaviours without full retraining. • Purpose. To efficiently patch model flaws or update its knowledge in a targeted manner. • Imple...

  79. [88]

    Techniques for capturing human preferences and values to guide AI behaviour, often by training a reward model RM (y) based on human feedback

    Value Elicitation and Reward Modelling. Techniques for capturing human preferences and values to guide AI behaviour, often by training a reward model RM (y) based on human feedback. 23 • Purpose. To create a differentiable proxy for human values that can be used to fine-tune A...

  80. [89]

    Using a learned reward model or direct human feedback to fine-tune a base AI model towards desired behaviours

    Reinforcement Learning from Human Feedback (RLHF) and Similar Methods. Using a learned reward model or direct human feedback to fine-tune a base AI model towards desired behaviours. • Purpose. To steer model behaviour to be more helpful, harmless, and honest, beyond what was l...

  81. [90]

    Systematically probing AI systems to discover vulnerabilities, failure modes, and misalignments before deployment

    Red Teaming and Adversarial Evaluation. Systematically probing AI systems to discover vulnerabilities, failure modes, and misalignments before deployment. • Purpose. To proactively identify and understand potential risks under challenging or adversarial conditions. • Implement...

  82. [91]

    Defining explicit principles, rules, or constitutions that the AI must adhere to, and using AI itself to critique and revise its outputs to comply with these rules

    Constitutional AI and Rule-Based Safeguards . Defining explicit principles, rules, or constitutions that the AI must adhere to, and using AI itself to critique and revise its outputs to comply with these rules. • Purpose. To embed high-level ethical guidelines or safety constr...

  83. [92]

    Designing the rules of interaction, com- munication protocols, and reward/utility functions for individual agents within a multi-agent system

    Mechanism Design and Incentive Engineering. Designing the rules of interaction, com- munication protocols, and reward/utility functions for individual agents within a multi-agent system. • Purpose. To shape agent incentives such that self-interested behaviour leads to desirabl...

  84. [93]

    Implementing systems for establish- ing, communicating, monitoring, and enforcing social norms or rules within an AI agent population

    Normative Control and Governance Structures. Implementing systems for establish- ing, communicating, monitoring, and enforcing social norms or rules within an AI agent population. • Purpose. To regulate agent behaviour, resolve conflicts, and maintain stability in artificial s...

  85. [94]

    Establishing standardized ways for agents to exchange information, signal intent, and coordinate actions

    Communication and Coordination Protocols. Establishing standardized ways for agents to exchange information, signal intent, and coordinate actions. • Purpose. To enable effective collaboration and reduce misunderstandings or conflicts arising from poor communication. • Impleme...

  86. [95]

    Governments and international bodies establishing laws, standards, and policies governing the development, deployment, and use of AI

    Legislation, Regulation, and Policy. Governments and international bodies establishing laws, standards, and policies governing the development, deployment, and use of AI. • Purpose. To set legal boundaries, ensure accountability, protect fundamental rights, and mitigate large-...

  87. [96]

    Professional organiza- tions, research institutions, and industry consortia developing and promoting principles for responsible AI

    Ethical Guidelines, Codes of Conduct, and Industry Standards. Professional organiza- tions, research institutions, and industry consortia developing and promoting principles for responsible AI. • Purpose. To foster a culture of responsibility and guide AI practitioners in maki...

  88. [97]

    Efforts to inform the public about AI, foster critical discussion, and involve diverse stakeholders in shaping AI’s future

    Public Discourse, Education, and Stakeholder Engagement. Efforts to inform the public about AI, foster critical discussion, and involve diverse stakeholders in shaping AI’s future. • Purpose. To ensure that AI development is democratically accountable and reflects a broad rang...

  89. [98]

    Formal processes for evaluating the potential societal impacts of AI systems before and during deployment, and for independently verifying claims about their safety or fairness

    Impact Assessments and Auditing Mechanisms . Formal processes for evaluating the potential societal impacts of AI systems before and during deployment, and for independently verifying claims about their safety or fairness. • Purpose. To proactively identify, assess, and mitiga...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.