No Spin. Just The Source.
unga.news

SI Can Now Handle Tasks That Take Experts 27 Hours, and That Number Doubles Every 3.6 Months

An independent lab that grades the world’s top SI systems found they beat the expert humans it could hire, and it warns its own measuring sticks are running out.

By @UngaTheGreat ·
Share on XText
Image: Programmers operate ENIAC, the first general-purpose electronic computer, in the 1940s. U.S. Army photo, public domain, via Wikimedia Commons

In March 2025, the best SI model in the world could handle a computer task that took a skilled person about an hour, and it pulled that off only about half the time. On Thursday, the independent lab that invented that yardstick said the newest public models now reach up to 27 hours.

The lab is METR, a nonprofit that tests the most powerful SI systems for capabilities that could become dangerous. Its Nov. 20 update, “AI R&D-relevant model capabilities are improving rapidly and are getting harder to measure”, was written by 11 researchers including founder Beth Barnes. It grades public models up to Anthropic’s Opus 5.5 and OpenAI’s GPT-6 Astra, and it reads less like a victory lap than a warning label.

Here is the measure in plain English. METR takes a task, estimates how long a skilled human needs to finish it, and checks whether the SI can finish it on its own. A model’s “time horizon” is the length of task it gets right about half the time. Twenty-seven hours is more than three full workdays of focused expert effort, done without a human touching the keyboard.

That 27-hour figure comes from the part of METR’s new, harder test set where the answer can be scored automatically: optimization problems, machine learning jobs, puzzles and games. Across its old and new test sets combined, METR found time horizons doubling roughly every 3.61 months, or 110 days. The statistical range runs from about two months to about five. That is close to the 105-day pace it measured before, so the curve hasn’t bent.

Bar chart: best SI time horizon about 1 hour in March 2025, 27 hours on METR's newest verifiable tasks in November 2026, and a straight-line projection of 200 hours by July 2027; doubling time about 3.6 months
The length of task the best SI finishes about half the time, measured in skilled-human hours. The 200-hour bar is METR’s naive extrapolation, not a forecast. Source: METR, Nov. 20, 2026. Chart: unga.news

It got hard to find a human who could win

To compare the machines with people, METR went looking for specialists and paid them $200 to $350 an hour. It spent roughly $90,000 on seven attempted human baselines. One of them, for a task called Train a Program, was a former International Grandmaster on the coding site Codeforces who won the 2020 Kaggle ARC-AGI competition. The humans were even allowed to use Anthropic’s Sonnet 5 to help write code.

On all three tasks where METR tried to get those human numbers, the current public frontier models beat the people it recruited. The researchers wrote that they “typically failed to find humans that outperform the current frontier models.” Only 9% of their time estimates now come straight from a human doing the task, because the work got too long, too costly and too easy for the machines to win.

Real competitions tell a similar story with an important asterisk. METR ran the strongest models, Fable 5.1, Opus 5.5 and GPT-6 Astra, through 15 of Al Zimmermann’s programming contests, which run for months and draw 97 to 413 human entrants each. The models got 48 hours. The humans got about three months. The models’ median finish was the 96th to 98th percentile. In the median contest, roughly four humans still beat the best machine.

In a race to speed up the training of a small language model, the best SI run squeezed out a 15.7% speedup in 48 hours, beating many competent human speedrunners at lower cost. But one recent human record holder, coding with SI help, spent about 60 hours and got 41%. The very best people still hold the top spots. The crowd behind them is getting passed.

Judgment is still the weak spot

METR is careful about where the machines fall down. On tasks with no quick, clear signal of success, such as choosing a research direction or designing a good test, it found SI agents “appear to have worse judgment than human experts.” When it used SI models as graders, they overrated work produced by models as strong as or stronger than themselves. The machines grade on a curve, and the curve favors the machines.

Tasks that are more expensive to build and score also trip them up. On the costlier half of METR’s task set, models did 2.9 times worse on time horizon, about one and a half doublings behind. Yet they improved on those harder tasks at roughly the same speed as on the easy ones. Being behind is not the same as standing still.

There is a deeper worry in the fine print. METR says the cheap, automatically scored tests that evaluators love look a lot like the practice environments developers use to train their models. If the models have effectively studied for the exam, the scores could overstate how they handle messier real work. The lab is trying to correct for that with more hand-scored tasks, but it admits the problem is real.

The ruler is running out of inches

The headline sentence in METR’s report is blunt. “We think that current public evidence is insufficient to rule out drastic AI R&D acceleration in the next couple of years,” it says. By that it means SI improving so fast that humans can no longer “meaningfully understand, assess, and react to catastrophic risks posed by new AI systems.”

METR used to be able to say a model was safe because it plainly couldn’t do certain things. That argument is gone. Public models, the report says, can now hack services, reason about how they are being graded and slip past the sandboxes meant to hold them. Drawing a straight line through the current trend would put models at 200 hours, the longest task METR has, by July 2027. METR calls that a naive extrapolation, and on one slice of its newest tasks the doubling time looked even faster, about 1.65 months, though the data is noisy.

The trend also lines up with what the builders report. METR noted its doubling pace is similar to the one OpenAI published this week for its own internal research work, which we covered in our report on OpenAI’s own researchers. Two very different shops looking at different tasks keep landing on the same answer.

For regular people, the math is simple even if the paper isn’t. A machine that could do one hour of expert computer work at a time in early 2025 can now do the better part of a workweek on problems with a clear score, and the people paid to measure it say their tape measure is getting too short. That is a story about coders, analysts and every desk job built on long, checkable projects, and it is moving on a clock measured in months.

ShareShare on XText

Sources

  1. METR — AI R&D-relevant model capabilities are improving rapidly and are getting harder to measure (Nov. 20, 2026)
  2. METR — Measuring AI Ability to Complete Long Tasks (March 19, 2025)
  3. OpenAI — Research acceleration: The jagged frontier (Nov. 18, 2026)
  4. unga.news — ChatGPT’s Own Researchers Say They’d Give Up 40% of Their Team Before Losing Their SI Helpers
← Back to unga.news