REVIEW 8 cited by
Is Power-Seeking AI an Existential Risk?
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
This report examines what I see as the core argument for concern about existential risk from misaligned artificial intelligence. I proceed in two stages. First, I lay out a backdrop picture that informs such concern. On this picture, intelligent agency is an extremely powerful force, and creating agents much more intelligent than us is playing with fire -- especially given that if their objectives are problematic, such agents would plausibly have instrumental incentives to seek power over humans. Second, I formulate and evaluate a more specific six-premise argument that creating agents of this kind will lead to existential catastrophe by 2070. On this argument, by 2070: (1) it will become possible and financially feasible to build relevantly powerful and agentic AI systems; (2) there will be strong incentives to do so; (3) it will be much harder to build aligned (and relevantly powerful/agentic) AI systems than to build misaligned (and relevantly powerful/agentic) AI systems that are still superficially attractive to deploy; (4) some such misaligned systems will seek power over humans in high-impact ways; (5) this problem will scale to the full disempowerment of humanity; and (6) such disempowerment will constitute an existential catastrophe. I assign rough subjective credences to the premises in this argument, and I end up with an overall estimate of ~5% that an existential catastrophe of this kind will occur by 2070. (May 2022 update: since making this report public in April 2021, my estimate here has gone up, and is now at >10%.)
Forward citations
Cited by 8 Pith papers
-
Risky Business: Measuring The Faithfulness-Safety Tension
Faithful reasoning and safety pull in opposite directions in current reasoning models, and the two behaviors are controlled by anti-correlated internal vectors that can be steered independently.
-
Adversarial Attacks on Robotic Vision Language Action Models
Text-based adversarial suffixes can make OpenVLA robot policies elicit chosen target actions with over 90% success on one-hot targets and persist across rollout steps.
-
Will artificial agents pursue power by default?
Power is a convergent instrumental goal only in an incomplete sense; no full ranking of situations by power can predict agent behavior under minimal assumptions.
-
Will AI Tell Lies to Save Sick Children? Litmus-Testing AI Values Prioritization with AIRiskDilemmas
A new benchmark elicits AI models' value priorities from choices in 3,000 AI-risk dilemmas and reports correlations between those priorities and risky behaviors, including on the external HarmBench.
-
Intelligent Cause Prioritisation? An Analysis of AI Policy Priorities and Governance in Africa
African governments' AI policies prioritize development and opportunity over AI safety, with structural risks almost entirely absent from public discourse.
-
NEST: Nascent Encoded Steganographic Thoughts
Frontier LLMs can embed short digit sequences in sentence acrostics (Claude Opus 4.5: 92% per-digit at D=4) but fail to jointly solve hidden reasoning tasks and encode the solution.
-
Out of Control -- Why Alignment Needs Formal Control Theory (and an Alignment Control Stack)
A position paper proposing that AI alignment adopt formal optimal control and a ten-layer Alignment Control Stack for organizing and interoperating control interventions.
- Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIs
Discussion (0). Continue with ORCID to comment.