Alibaba Gave 10 Top SI Coders 273 Real User Requests. The Best Fully Finished Just 26%.

SI coding agents have gotten so good at the standard tests that the tests barely mean anything anymore. So a team led by researchers at Alibaba built a harder one out of real requests from real people, and the results are humbling.

They gave 10 leading SI models 273 actual change requests pulled from real coding sessions. The best of them, OpenAI’s GPT-6 Astra, fully completed just 26.4% of the jobs. On the other 73.6%, it left at least one requirement unmet.

That matters to anyone paying for these tools. A coding agent that gets a job mostly right still leaves a human to find and fix the part it missed.

What the test actually asks

The benchmark is called Co-Design Bench, and it’s laid out in a paper on arXiv by authors from Alibaba Group, Fudan University, Zhejiang University, the Chinese University of Hong Kong, Shanghai Jiao Tong University and Nanyang Technological University in Singapore.

The researchers started with 26,314 real user coding sessions and 37,463 unique requests, then whittled them down to 273 tasks. Each one hands the agent a working web app and a request to change it. Some are small but fussy, like pulling a number off a receipt and updating every total that depends on it. Others are whole-app overhauls that touch layout, design, buttons and data at once. Many come with a picture, video, audio clip or document the agent has to read first.

Experts wrote 2,331 hidden pass-or-fail requirements for those tasks. The agent never sees them. It has to figure out what the user really wanted, just like on a real job.

How they graded it

Here’s the part that makes this test tougher than most. The finished app is built and opened in a real browser. One SI judge checks each requirement against the code, screenshots and recorded clicks. A second SI checker then tries to knock down every passing grade, hunting for fake buttons, hardcoded answers and features no user could actually reach. If the evidence for a claim is missing, the run doesn’t count.

Every model worked inside the same setup, Anthropic’s Claude Code tool, with a cap of three hours and 120 turns per task. Two front-end experts also spot-checked the judge’s scores on 200 results.

The scoreboard

GPT-6 Astra led both halves of the test. On average it satisfied 69.9% of a task’s requirements, but it got every single box checked on only 26.4% of tasks. Anthropic’s Claude Opus 5 came second at 18.3%, followed by Alibaba’s own Qwen3.8-Max at 16.5%.

Credit where it’s due: this is an Alibaba-led test, and it crowned an American model. Alibaba’s Qwen did not win.

The gap gets ugly on bigger jobs. On the half of the test built around multi-file app overhauls, Astra fully finished just 17.9% of tasks. On smaller, localized edits it hit 41%. Every one of the 10 models scored lower once a request mixed three or four kinds of changes, such as layout plus styling plus new buttons plus new content.

Why the old tests stopped working

The authors say older benchmarks are saturated. On an existing test called WebCompass-edit, Anthropic’s Claude Opus 4.8 averaged 99.4% when allowed to run and fix its own code. Just letting Alibaba’s Qwen3.7-Plus work in steps instead of answering in one shot raised its score on that test from 73.6% to 95.1%.

When every model scores in the high 90s, a leaderboard can’t tell you which one to hire. Co-Design Bench spreads them back out.

More effort didn’t buy better work

Burning more computing power didn’t close the gap. On the smaller edits, Astra used a median of 32 turns and about 1.24 million tokens per task. Z.ai’s GLM-5.2 used 69 turns and about 7.29 million tokens, nearly six times as much, and still scored well below it.

The software wrapped around the model matters too. When the researchers ran the same model through three different agent tools, the much smaller Qwen3.8-27B scored 0.749 on the overhaul tasks in one tool and 0.560 in another. Which of two models came out ahead flipped depending on the tool. The authors’ advice: always report the model and its harness together.

The fine print

This is a deliberately hard test. The researchers ranked candidate requests partly by how often real users had to rephrase or roll back changes, so the pile leans toward jobs that already gave SI trouble. Each task is also a single fixed request, with no chance for the agent to ask a follow-up question the way a real customer conversation would allow.

And the grading is done by SI judges, checked by a second SI, with human experts reviewing only a sample. The paper doesn’t report how often those experts agreed with the judge. The researchers did re-score some results with three other judge models and found scores moved by no more than about five points, and the ranking between the two models they compared held each time.

What it means for you

If you use an SI agent to build or fix software, treat “done” as “mostly done.” Even the top scorer in this test will usually do the heavy lifting and quietly skip a detail. Test the result yourself, especially when the request touches several parts of an app at once.

For companies picking a coding agent, the lesson is to test the model and the tool it runs in as a pair, on your own real requests. As one summary of the study put it, these agents still need human cleanup on nearly three out of four tasks.