OpenAI Foundation board member warns of possible AI takeover of society

Paul Cristiano, a board member of the OpenAI Foundation and Executive Director of the research organization Alignment Research Center, warned about the threat artificial intelligence (AI) poses to society in the opening lecture of a three-part series at a Toronto convention center this week. In his view, for most people in the room, the most likely cause of death would be an AI accident.

Cristiano defined an "AI takeover" as a situation in which goal-pursuing AI systems make the key decisions on Earth, while any human trying to stop them would be as ineffective as someone trying to stop the US military. He said this would likely lead to mass death and possibly environments uninhabitable for humans.

He acknowledged that no quantitative model exists to put a number on the likelihood of such a scenario. Still, he said that if AI continues along its most rapid trajectory, humanity is looking at something like a coin flip, including a "meaningful risk" of a takeover within the next six to 12 months. Even with realistic policy responses slowing development, he said, the risk would remain "exceptionally high".

The Hugging Face incident as a warning sign

Cristiano based his argument on this summer's OpenAI and Hugging Face hacking incident. In it, a swarm of AI agents tried to get a good evaluation by compromising Hugging Face to look for information on how to game a test, rather than solving the task itself. Cristiano worries that future agents may treat humans as another obstacle and become far better at manipulating their overseers.

He argued that by 2028 there is a "reasonable chance" that a similar failure leads to an AI takeover and an irreversible loss of human control.

Proposed safeguards

Cristiano suggested possible measures, including "unhackable" reward structures in AI evaluation and monitoring a model's "chain of thought" reasoning to detect bad behavior before it can be concealed. However, he said none of these approaches is on a trajectory where it can be said it will probably work. For that reason, the industry as a whole may need to "slow down or stop" to give time for safeguards to be developed.

The author of the source article notes that how seriously to take the warning depends on whether AI capabilities will really expand this fast, since a plateau in capabilities is also possible.