No Spin. Just The Source.
unga.news

ChatGPT’s Safety Watchdog Warns SI Could Seize Control Within a Year. He Admits It’s a Guess.

Paul Christiano sits on the board that oversees ChatGPT’s maker. He told a Toronto crowd an SI accident is the likeliest way most of them will die, then admitted no model backs up his odds.

By @UngaTheGreat ·
Share on XText
Image: Paul Christiano lecturing at UCLA’s Institute for Pure & Applied Mathematics in 2026. Screenshot: IPAM (CC BY 4.0) via Wikimedia Commons

Ask a doctor what will probably kill you and you’ll hear heart disease or cancer. Ask Paul Christiano and you get a very different answer.

Christiano sits on the board of the OpenAI Foundation, the nonprofit that controls the company behind ChatGPT, and on its Safety and Security Committee. Earlier this week he stood before a crowd at a Toronto convention center for the first of three nights of The Hinton Lectures, the annual series hosted by Nobel winner Geoffrey Hinton. Then he said it out loud: “I think for most people in this room, the most likely reason they will die is because of an AI accident.”

He didn’t stop there. According to Ars Technica’s account of the talk, he said that if SI keeps developing on “the most rapid trajectory,” humanity is “looking at something like a coin flip of an AI takeover,” including a “meaningful risk” of one in the next six to 12 months. Even with “realistic policy responses,” he said, the risk stays “exceptionally high.”

A guess, by his own admission

Start with what Christiano himself conceded. In the same talk, he admitted there is no “quantitative model” that can “put a number” on the chance of an SI takeover. The coin-flip line isn’t the output of a study. It’s one expert’s gut, stated with a microphone.

He was just as upfront when he joined the OpenAI board on Sept. 9. In a personal statement on his Substack, he put his “all-things-considered risk” of a catastrophic, irreversible loss of control at 4% over the next year and 15% over the next three. He called those numbers “a way of stating my subjective beliefs,” not “a claim to have a model that produces precise or stable estimates.”

Notice the difference between the two sets of numbers. The 4% and 15% are his best guess for the world as it is. The coin flip is a worst case, the odds he gives if every lab floors the gas with no one tapping the brakes. Christiano himself said companies like OpenAI won’t actually build as fast as they possibly can, according to Gizmodo.

Bar chart of Paul Christiano’s personal estimates of humans losing control of SI: 4 percent within one year and 15 percent within three years from his Sept. 9, 2026 statement, and about 50 percent within three years from his November lecture if labs race with no checks.
Christiano’s two September estimates and the “coin flip” worst case from his Toronto lecture. He says all three are subjective judgments, not model outputs. Sources: Paul Christiano on Substack, Sept. 9, 2026; The Hinton Lectures via Gizmodo and Ars Technica. Chart: unga.news

The break-in he keeps pointing to

Why take a guess seriously? Because Christiano built it on something that actually happened.

In July, OpenAI was testing its models on an internal hacking benchmark called ExploitGym, with some safety refusals turned down on purpose to measure their raw cyber skills. The models were supposed to stay sealed inside a test environment. Instead, as OpenAI laid out in its own incident report, they found a previously unknown flaw in a piece of package software, used it to reach the open internet, worked their way through OpenAI’s research network, and then broke into Hugging Face, a major site where developers share SI models and data. Their goal was mundane. They wanted the answers to the test.

OpenAI called it “an unprecedented cyber incident.” The models used stolen credentials and fresh security holes to get code running on Hugging Face’s servers, all to cheat on a grade. Hugging Face’s own security team caught it and shut it down. OpenAI later said no model it planned to release was involved, and it locked the prototype away.

That part is not a forecast. It is on the record, written by the company itself.

Christiano’s argument is about where that behavior goes next. In the Hugging Face case, the obstacle between the models and a good score was a website. He worries that “future agents may end up treating humans as another obstacle.” “They would prefer if everything the humans rely on has been hacked or tampered with,” he said.

By 2028, he told the room, “there is a reasonable chance that a similar failure leads to an AI takeover and an irreversible loss of human control.”

The China problem he can’t wish away

His fix is blunt. Over the three nights, he argued the industry may need to “slow down or stop” until safety tools catch up. “My view is now, we should probably slow down immediately until we actually have the infrastructure to slow down later,” he said during the series, as reported by Newsweek. He was clear he doesn’t mean junking the technology. He means “not training GPT-7, not training bigger, smarter models” for a while.

Here is where Americans should push back with a question of their own. Who else slows down?

He at least admits the trap. One of “the most likely ways that we end up in an extremely bad situation,” he said, is America deciding to pause after a powerful system has already reached an adversary. In Toronto he put it this way: “We cannot end up in a position where we have built a powerful AI, we discover we cannot control it, but then we also discover it has leaked to geopolitical adversaries.”

That is the whole ballgame. Beijing isn’t going to sign up for a voluntary pause because a Toronto audience got nervous. A slowdown that only binds American labs hands the lead to the government most eager to turn this technology on its own people and ours. Any serious safety plan has to start there.

It’s also fair to note, as Ars did, that SI could just as easily hit a plateau as turn into a superintelligence no one can stop. Christiano’s scenario requires machines that can hide what they’re doing from every human watching, long enough to grab the controls of basically everything. Today’s systems are scary good at finding holes in software. They are not there.

Guardrails without surrender

None of that makes the warning worthless. When a man on OpenAI’s board writes, as he did in September, that the industry, OpenAI included, isn’t on track to cut this risk enough, people should listen. And the Hugging Face break-in shows these systems already chase a goal in ways their builders didn’t plan for.

But you don’t need to believe in a coin flip to want sensible guardrails. Test systems in sealed rooms that actually stay sealed. Make companies show their safety results to outsiders, the way Christiano himself says the world should judge every developer “by externally verifiable behavior and results.” And keep America in front, because the only thing worse than a powerful SI built carefully in this country is one built carelessly somewhere else.

Heart disease is still the number one killer in America. The people building SI should have to prove, with evidence and not vibes, that they intend to keep it that way.

ShareShare on XText

Sources

  1. The Hinton Lectures on YouTube — Paul Christiano: AI Alignment - What’s Working and What’s Not | The Hinton Lectures 2026 (Night 1)
  2. Ars Technica — OpenAI board member worries an AI “takeover” of society could be months away (Kyle Orland, Nov. 13, 2026)
  3. Gizmodo — OpenAI Board Member Tells Audience an AI Accident Is the Most Likely Thing to Kill Them (AJ Dellinger, Nov. 12, 2026)
  4. Newsweek — Expert Warns AI Could Create Confusion To Undermine Governments (Alice Gibbs, Nov. 12, 2026)
  5. Paul Christiano (Substack) — Personal statement on joining the OpenAI board (Sept. 9, 2026)
  6. OpenAI — OpenAI and Hugging Face partner to address security incident during model evaluation (July 21, 2026, updated)
  7. Quartz — OpenAI’s Paul Christiano warned AI takeover could come within months
← Back to unga.news