Amazon Let Thousands of SI Bots Build a Whole Product. Its Engineers Never Read the Code.

Amazon just shipped a piece of software that no person at Amazon wrote. Not the first draft, not the bug fixes, not the code review. Thousands of SI agents did all of it, and the engineers who run the project say they haven’t even read it.

The product is called AWS Emulator, and Amazon released it for free on Wednesday. It’s a stand-in copy of Amazon Web Services, the giant rental computer network that runs a big slice of the internet, that a programmer can run on a laptop. Instead of sending test traffic to Amazon’s real data centers, an app talks to the copy on your own machine. No account, no bill, no waiting. “Our team hasn’t written or reviewed any of the code,” Matthew Goldberg, the senior principal engineer behind it, wrote in Amazon’s own account of how it was built.

A weekend dare that worked

Goldberg says it started as an experiment he didn’t expect to work. Earlier this year, researchers at Anthropic showed SI agents could build a working C compiler on their own. He figured Amazon had the same two ingredients that made that possible: a detailed rulebook for how every service is supposed to behave, and the real thing to check against.

So he gave Kiro, Amazon’s SI coding assistant, about the simplest instructions he could think of. Write tests based on Amazon’s command-line tool. Keep fixing those tests until they pass against the real AWS. Then build a fake version in memory that passes the same tests.

Fifteen minutes later he had a rough copy of Amazon’s message-queue service. He tightened the prompt and, in his words, “had Kiro build me the rest of AWS over the weekend. And it did.” It had holes, but his team started using it to test their own software that Monday.

He’s also honest that the weekend version wasn’t a product. “There’s a natural temptation these days to see results like those and believe that AI can build anything,” he wrote. Turning it into something Amazon would hand to customers took real engineering, mostly in building fences around the agents.

Three crews, one gate

The setup Goldberg describes works a lot like a well-run job site. One crew of agents writes tests, each one a real-world task such as creating a queue, sending a message, reading it and deleting it. Every test gets checked against the real AWS before it counts. Amazon says there are now tens of thousands of them.

A second crew runs those tests against the copy and files a bug report every time the copy behaves differently from the real thing. A third crew takes those reports one at a time, digs for the cause and writes a fix.

Nothing gets in without passing a gate. Code that doesn’t compile gets bounced. So does code that breaks the style rules, fails a security check or makes the copy less faithful than it was. Amazon’s launch post says an independent security agent reviews changes “adversarially,” first while the fix is being written and again right before release. Goldberg says hundreds of agents work on the codebase at the same time and fix thousands of issues a day.

The public record backs up the pace. The project’s GitHub page lists close to 30,000 commits since the repository was created in September, and the newest release, posted Tuesday evening, lists dozens of fixes in a single drop.

Built its own Python, then got told no

Agents left alone get creative in ways a human team wouldn’t. Goldberg’s favorite example: in an earlier version, the agents decided to build their own Python interpreter, the program that runs Python code, just to support one AWS feature. Amazon has since told them to reuse existing tools. That’s the kind of note you’d give a new hire, and it’s a decent reminder that these things still need a foreman.

Amazon is also showing its work. The code is released into the public domain under a license called CC0, meaning anyone can copy or use it for anything. Every bug the agents fixed sits in one folder, and every one they haven’t fixed yet sits in another one, plainly labeled “things-that-do-not-work.”

Amazon’s own scorecard says 56%

This is the part that keeps the story honest. Amazon grades the copy with what it calls fidelity tests. A test only passes if the copy matches the real AWS on every step, down to the exact error message. With each release, Amazon publishes the results.

We pulled the scores file attached to the latest release. Of 42,937 tests, 24,243 passed, about 56%, and 18,614 failed. Popular services ran a little better. Amazon’s file storage service passed about 63% of its tests, and the main database service about 68%. Amazon warns that the score can drop even as the copy improves, because the agents keep writing tougher tests.

There are firm limits, too. The copy doesn’t enforce Amazon’s permission rules the way the real cloud does, so a request AWS would block might sail through. It acts as a single account in a single region. Amazon flatly tells people not to use it to test whether an app is secure, and not to run real work on it.

Vogels and his 2004 intern

Werner Vogels, Amazon’s longtime chief technology officer, put the launch in personal terms. He said he hired Swami Sivasubramanian, now one of AWS’s top executives, as an intern in 2004, and that Sivasubramanian “learned early that you can only test a system properly in a high-fidelity copy of the environment it will actually run in.” Vogels said the copy covers more than 200 AWS services, in his post on X.

For regular folks, the takeaway isn’t a cheaper test tool. It’s that one of the biggest software companies on the planet is now willing to put its name on a product built and maintained by machines, with people setting the rules and checking the scoreboard instead of typing the code. Today that scoreboard reads 56%, and Amazon is the one who posted it.