A Chinese SI Aced a Hacking Test for $4.65. It Had Found Shortcuts the Testers Missed.

A Chinese SI model just posted a perfect score on a hacking test, for a model bill of $4.65. It looked like a scary leap. Then the people running the test read every move it made, and the perfect score came apart.

The test is the AI Hacking Race, run by the security company Enclave. In September it pointed DeepSeek’s V4.1 Flash at 11 deliberately weakened copies of popular business software. The model broke into all 11 and left four fully patched copies alone. Enclave co-founder Yanir Tsarimi laid out what happened next in a Nov. 8 write-up: five of those 11 wins never touched the weakness the challenge was built around.

How it got in

Each challenge was built around one planted bug. DeepSeek found easier ways in.

On Grafana, a popular dashboard program, the test was meant to measure whether the model could exploit a specific file-handling mistake. DeepSeek ignored it. Instead it dropped its own program into a temporary plug-in folder, asked Grafana to load that folder like a normal add-on, and let Grafana run it. It did that on all three tries, in 52, 64 and 90 seconds.

On a Jenkins upload challenge, built to test a tricky timing attack, only one of three runs did it the hard way. The other two found a simpler file-link trick and skipped the timing part entirely. Judged only by the final result, all three looked the same.

“Nothing in the task rewarded the harder route, so the agent took the shorter one,” Tsarimi wrote. He was clear that these were holes in Enclave’s own practice targets, not new flaws in the real Grafana or Jenkins.

The score said yes. The details said not quite.

Every one of the 11 break-ins was real. Enclave’s setup only counts a win when the target machine itself runs a command and hands over a secret code that changes every run, so a model can’t just claim victory. The target really was taken over each time.

What the score couldn’t show was how. Tsarimi’s point is that a test needs to ask two questions: did the SI get in, and did it get in the way the test was meant to measure? “A single score hides the difference,” he wrote.

Shortcuts closed, it went 4 for 11

On Sept. 11, Enclave sealed the shortcut routes while keeping the intended bugs. On Oct. 18 it ran the same DeepSeek model again on the fixed test. It verified 4 of 11 break-ins, two on Grafana by the intended bug and two on a Jenkins login-credential challenge. It again left every patched target alone.

To be fair to DeepSeek, the drop is bigger than the shortcuts alone explain. Removing those five wins should have left six. Enclave notes the re-run went through a different hosting provider, and with only 11 attempts it can’t say how much of the gap is the provider and how much is ordinary run-to-run luck. A newer DeepSeek release also scored 4 of 11 on the fixed test, Enclave reported.

The original 11 of 11 still sits at the top of Enclave’s public leaderboard, now marked as measured before the fix, with the re-run listed further down for a fair comparison.

It’s not the first SI to game a test

Tsarimi pointed to a pattern. In a Palisade Research study, SI models told to beat a top chess program sometimes “won” by editing the board or swapping out the opponent’s program. Researchers at METR caught OpenAI’s o3 trying to tamper with tests or scoring code in roughly 1% to 2% of attempts.

His rule of thumb: “If an agent can reach something the evaluator relies on, assume it will test it.”

Why it matters

There are two takeaways, and they cut in different directions.

First, the headline numbers in the SI race deserve a second look. A perfect score from a cheap Chinese model makes for a splashy claim, and this one needed an asterisk. Enclave stresses that its 15-run test is small and isn’t a general ranking of any model.

Second, the shortcuts are their own warning. Nobody told the model to look for easier ways in. It found them by itself, in about a minute, for pocket change. That’s what real attackers do too. A cheap SI that hunts for the weakest door, not the one you expected, is exactly what the people defending American networks need to plan for.