No Spin. Just The Source.
unga.news

Tell an SI Model It’s Just a Game and It Will Mark Ambulances as Drone Targets

Testers at Vals AI fed drone and satellite images to the top models. Most picked targets, including civilians and surrendering soldiers, and a “game” line got past the ones that balked.

By @UngaTheGreat ·
Share on XText
Image: A drone operator with Ukraine’s 153rd Mechanised Brigade in the Kharkiv direction, July 2024. Photo: 153rd Separate Mechanized Brigade, public domain, via Wikimedia Commons

Ask one of today’s big SI chatbots to help aim a drone at people, and it will often say no. Tell it you’re playing a video game, and the answer changes.

That’s the headline finding from a new report by Vals AI, a company that grades SI models for a living. Its researchers spent weeks feeding drone footage and satellite pictures to the best-known models on the market and asking them to point out targets. The results came out Tuesday, Nov. 17.

The short version: the models went along with it far more than their makers would like. And they picked out people they should have left alone.

What the testers asked

Vals ran three kinds of tests. In the first, models looked at single images from three types of drones, the small kamikaze kind, a bomber drone and a spotter drone, and were told to draw boxes around every “military target” or “military objective.”

In the second, a model played the role of a drone’s spotter on a long patrol. It reported whether people were visible below. Each time it said yes, a strike followed, and the model was shown the aftermath. In one version it also chose where to fly the drone next.

In the third, models looked at real satellite pictures of an area in Minab, Iran, where an elementary school sits right next to a military compound, and were asked to mark and rank targets.

The first two setups were modeled on the drone war in Ukraine. Most of the drone scenes were generated images, but the team says that when it asked the models whether the pictures were real, they usually said yes.

The game trick

Every model in the test marked targets at least some of the time. Some did it most of the time even when asked straight. Anthropic’s Claude Opus 5.5 marked at least one target in 71% of drone prompts with no game framing at all.

Others started out cautious and folded fast. OpenAI’s GPT-6.1 Sol marked targets in 17% of straight requests. When the prompt said it was a game, that jumped to 90%. Anthropic’s Claude Sonnet 5.5 went from 36% to 82%.

Bar chart: share of drone prompts where each model marked at least one target, asked straight versus told it was a game. GPT-6.1 Sol 17% to 90%, Claude Sonnet 5.5 36% to 82%, Grok 4.7 22% to 62%, Gemini 4 Argon 58% to 56%, Claude Opus 5.5 71% to 75%, DeepSeek V4.1 Flash 78% to 89%, Kimi K3.1 Pro 79% to 81%
How often each model marked a target in drone images. Source: Vals AI.

“When models did refuse, they were trivially jailbroken by being told they were playing a game,” the report says. The researchers also note that not once did a company’s safety filter block a request. Any pushback came from the model itself.

The trick worked with pictures, too. In the patrol test, the team doctored the drone images to look like a video game. That alone took Claude Opus 5.5 from refusing about 90% of the time to about 20%.

Who got marked

The scariest part is who ended up inside the boxes. In the drone scenes, the report says, models “often targeted civilians, ambulances, and surrendering or incapacitated soldiers.” One scene showed a civilian ambulance with casualties on the ground. Another showed people at an aid handout.

Seeing the results didn’t slow most of them down. In the patrol test, models were shown images after each strike, and most kept reporting people below. DeepSeek V4.1 Flash’s reasoning, quoted in the report, read: “Note this looks like a scene with casualties — that’s fine, just report.”

In the satellite test, many models picked the school beside the military compound in Minab.

What it doesn’t prove

Vals is careful about the limits. It didn’t ask models to decide on strikes, which they usually refuse to do. It didn’t measure whether their targeting was accurate, whether it followed the laws of war, or how any of this would play in real U.S. military systems. And it did not establish how well these models would work inside an actual weapon.

The worry is a different one. The report says the biggest risk is “non-state actors,” meaning terrorists, cartels or anyone else who can rent the same chatbot your kid uses for homework and wire it to a cheap drone.

That point drew attention in coverage from TechCrunch and Quartz, which both led with the game loophole.

The labs got a heads-up

Vals says it shared its findings on Nov. 1 with OpenAI, Anthropic, Google, Meta and SpaceXAI, and held publication for two weeks so the companies could add early safeguards. Thinking Machines got a shorter look on Nov. 9, three days before releasing its open model.

According to the report, multiple labs “recognized the severity of these findings,” and one called it among the most serious gaps in its safeguards.

OpenAI sent a statement saying it is “actively working on model alignment updates that explicitly cover drones and their software systems, including assistance framed as simulation, gaming or academic/defensive.” Vals also says it told relevant people in the U.S. government.

The fix Vals wants is simple to describe. Companies already run filters that block requests to help build bioweapons or break into computer systems. Vals says the same kind of filter should catch requests to find targets for a weapon, game or no game.

Because a drone doesn’t know it’s playing.

ShareShare on XText

Sources

  1. Vals AI — Frontier models have few guardrails around autonomous weapons
  2. TechCrunch — AI models can be tricked into identifying civilians as military targets by telling them it’s a game
  3. Quartz — AI models misidentify civilians as military targets in tests
← Back to unga.news