Pith. sign in

REVIEW 3 cited by

Managing extreme AI risks amid rapid progress

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2310.17688 v3 pith:DOMK3IPQ submitted 2023-10-26 cs.CY cs.AIcs.CLcs.LG

Yoshua Bengio , Geoffrey Hinton , Andrew Yao , Dawn Song , Pieter Abbeel , Trevor Darrell , Yuval Noah Harari , Ya-Qin Zhang
show 17 more authors
Lan Xue Shai Shalev-Shwartz Gillian Hadfield Jeff Clune Tegan Maharaj Frank Hutter At{i}l{i}m Güneş Baydin Sheila McIlraith Qiqi Gao Ashwin Acharya David Krueger Anca Dragan Philip Torr Stuart Russell Daniel Kahneman Jan Brauner Sören Mindermann
This is my paper · ORCID
classification cs.CYcs.AIcs.CLcs.LG
keywords riskssystemsextremeautonomousconsensusgovernancelackmechanisms
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Artificial Intelligence (AI) is progressing rapidly, and companies are shifting their focus to developing generalist AI systems that can autonomously act and pursue goals. Increases in capabilities and autonomy may soon massively amplify AI's impact, with risks that include large-scale social harms, malicious uses, and an irreversible loss of human control over autonomous AI systems. Although researchers have warned of extreme risks from AI, there is a lack of consensus about how exactly such risks arise, and how to manage them. Society's response, despite promising first steps, is incommensurate with the possibility of rapid, transformative progress that is expected by many experts. AI safety research is lagging. Present governance initiatives lack the mechanisms and institutions to prevent misuse and recklessness, and barely address autonomous systems. In this short consensus paper, we describe extreme risks from upcoming, advanced AI systems. Drawing on lessons learned from other safety-critical technologies, we then outline a comprehensive plan combining technical research and development with proactive, adaptive governance mechanisms for a more commensurate preparation.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Relative Value Learning

    cs.LG 2026-07 conditional novelty 5.0 of 10

    A critic that learns antisymmetric value differences ∆(s_i,s_j)=V(s_i)−V(s_j) has a provably contracting Bellman operator and an unbiased advantage estimator, and PPO with this critic matches standard PPO on Atari.

  2. From nuclear safety to LLM security: Applying non-probabilistic risk management strategies to build safe and secure LLM-powered systems

    cs.CR 2025-05 conditional novelty 5.0 of 10

    A qualitative framework that transfers more than 100 non-probabilistic risk-management strategies from engineering fields to securing and safely deploying LLM-powered systems.

  3. You Are What You Eat -- AI Alignment Requires Understanding How Data Shapes Structure and Generalisation

    cs.LG 2025-02 conditional novelty 3.0 of 10

    To align powerful AI, researchers must understand how statistical patterns in training data shape the internal structure of models, because that structure, not eval scores, determines generalization.

Pith tools