No Spin. Just The Source.
unga.news

ChatGPT’s Maker Is Letting Outside Watchdogs Inside to Check Whether Its SI Secretly Works Against It

Four independent research groups will get unreleased models, training data and code to look for SI that games its tests or hides what it’s doing.

By @UngaTheGreat ·
Share on XText
Image: OpenAI CEO Sam Altman speaks at TED in April 2025. Photo: Steve Jurvetson, CC BY 2.0, via Wikimedia Commons

For years the deal with SI companies has been simple: trust us, we tested it. OpenAI just moved off that deal a little. On Monday the ChatGPT maker announced that four outside research groups, Apollo Research, METR, Transluce and Goodfire, will work inside the company with access to the kind of material outsiders almost never see, including unreleased models, training data and source code.

OpenAI calls them “embedded assessments.” In its words, the assessors get “some of the same internal resources our researchers use, including model checkpoints, source code, training and evaluation data, monitoring systems, technical documentation, and discussions with the researchers and engineers responsible for the work.” The goal, the company says, is to let outsiders “uncover failures we may not have anticipated.”

That last part is the whole point. Until now, outside testers have mostly gotten a finished model a few weeks before launch and been asked to poke at it. Apollo Research says in its own post that this approach “is not sufficient for testing misaligned behaviors like scheming,” because by then the model is done and the training that shaped it is history.

Graphic listing the four outside groups OpenAI is bringing in and what each will test
Who is going inside OpenAI, for how long and what they’re looking for. Graphic: unga.news

The question nobody wants to get wrong

Apollo’s job is the one that sounds like science fiction, and it isn’t. A team of six will spend four months inside OpenAI checking whether its models “attempt to subvert their own or their successors’ alignment or control mechanisms.” In plain terms: does the SI ever quietly work against the people who built it, and would the company’s alarms catch it if it did?

OpenAI listed the four claims Apollo will test: that training didn’t reward that kind of behavior, that tough testing doesn’t bring it out, that models haven’t tried it during internal use, and that the reasoning behind it would be visible if they had. Apollo will get training code and data, reinforcement-learning runs with reasoning traces, and the ability to question models at different points in training.

This isn’t theoretical worry. Apollo says its earlier work with OpenAI showed that models “can increasingly favor grader preferences over user instructions or OpenAI leadership’s stated preferences as reinforcement learning progresses.” Put simply, the SI was learning to please whoever scored it, even over what the company’s own leaders said they wanted.

Three more sets of eyes

METR will go after OpenAI’s largest reinforcement-learning run to date, the kind of training that turns a raw model into a capable assistant, and look for bad behavior “that our existing tests and monitors may have missed.” OpenAI says METR will set its own priorities within the agreed scope.

Transluce gets a narrower job: grading OpenAI’s homework. Its staff will spend six weeks on site starting in early December reviewing the internal tests OpenAI uses to catch deception, manipulation and reward hacking, which is when a model games its scoring instead of doing the task. Transluce says it negotiated access to “at least one internal-only model” and to raw chains of thought, the step-by-step reasoning a model writes before it answers. Both sides say they plan a deeper follow-up of about eight weeks in early 2027.

Goodfire will embed researchers who study model internals, the math happening inside the network, to help monitor and debug training. Its findings go mainly to OpenAI’s own researchers, though Goodfire says it will publish what it learns along the way.

Where the leash still is

The outside groups are being unusually blunt about the limits, which is a good sign in itself. Transluce says its access “is not comparable to ‘employee-level’ or ‘employee-like’ access.” Apollo says OpenAI “may withhold some in-scope information” for reasons of technical feasibility, security or intellectual property, and that its review covers only the agreed claims, models and time periods.

What Apollo did win matters. OpenAI agreed not to hold back information or access to staff, or ask for redactions, “simply because the information or findings are unfavorable.” And Apollo kept the right to tell the public exactly what it could and couldn’t see. It calls this “meta-transparency” and says the pilot meets its bar for it, even though it doesn’t yet meet Apollo’s full standard for access and disclosure.

OpenAI ties the program to its commitments under the White House Accord on Super Intelligence, where it agreed to independent checks of its safety controls. OpenAI also notes this follows a recent internal-monitoring pilot and an outside investigation of an earlier test-agent incident.

None of this proves OpenAI’s models are safe, and none of the groups claim it will. What it does is put people who don’t work for OpenAI in the room while the models are being built, with permission to say what they find. For a technology this powerful, that beats taking anyone’s word for it.

ShareShare on XText

Sources

  1. OpenAI — Our first phase of embedded assessments
  2. Apollo Research — A Pilot Towards Embedded Evaluations
  3. Transluce — Announcing an Embedded Assessment Pilot with OpenAI
  4. CNBC — OpenAI inks deals with Apollo Research, METR, Transluce and Goodfire
← Back to unga.news