A New Free SI Lets Robots Watch a Video of What Happens Before They Try It

Picture a tube with a ball stuck inside. One cap twists off. The other is glued shut, but nothing in the picture tells you which is which.

A computer program was told to get the ball out. It didn’t touch a real tube. Every time it came up with a plan, a second program played that plan out as a short video, and the first program watched to see whether it worked. It pulled the red cap. Nothing. It twisted the red cap and, misreading the clip, decided the cap had come off. It spent three more tries shaking and pinching at that glued end. On the sixth try it went for the blue cap, and the ball rolled out.

That little test comes from the release notes for PAN 2, a new “world model” the Institute of Foundation Models put out on Nov. 10. The institute was set up by Mohamed bin Zayed University of Artificial Intelligence in Abu Dhabi, and PAN 2 was announced from its Silicon Valley lab in Sunnyvale, California, according to the press release. The point of it is simple to say and hard to do: let a machine see what will happen before it acts.

Predicting the next world instead of the next word

The chatbots most people use work by guessing the next word. PAN 2 tries to guess the next moment. You hand it a starting scene, either a photo or a description, and an action in plain words. It hands back a video, up to 20 seconds long, of what it thinks happens next.

The institute’s pitch on 𝕏 put it in one line.

Under the hood it is two programs working as a team, the technical write-up explains. A language model does the thinking first. It writes out, second by second, who does what and where the camera points. Then a 29-billion-parameter video model, which the team trained from scratch, fills in the stuff words leave out: how a hand grips a cup, how juice sloshes, how paper bends along a fold.

The team picked its training footage for real-world motion, not for looks. After the earliest round of training, they threw out video game clips, cartoons and computer renders so the model would learn from how things actually move. In a side test with two smaller versions, people comparing results blind preferred the one trained without the fake footage 58% of the time.

Why a robot would want this

A robot that can only learn by doing is slow and expensive to teach. Every dropped glass is a broken glass. Every bad move by a self-driving test car is a real risk. A model that can rehearse the move first, and show the robot how it plays out, lets the machine try ten plans and pick the best one without breaking anything.

PAN 2’s write-up shows another use: carrying a job forward one step at a time. Starting from a photo of a blank sheet on a table, the team gave it one folding instruction at a time and fed the last frame of each clip into the next. Fold by fold, the sheet became a paper airplane.

The ball-in-a-tube test showed something harder. The fact that one cap was glued lived only in the text given to PAN 2, never to the program trying to solve the puzzle. It had to learn it the way a person would, by trying things and watching what happened.

How it scored, warts included

The institute is upfront that this is far from finished. It built its own tests around everyday jobs with three to six steps each, like freeing a kite from a tree with a pole and handing it back to a child. PAN 2 finished 75.1% of the individual steps and 34% of the full plans. That 34% was the best of the six models the team tested, edging MiniMax-H3 at 32.5%, with closed models FLUX 3 and Seedance 2.5 behind both.

Put the other way, about two out of three multi-step plans still miss a step somewhere. MarkTechPost’s review flagged that, along with the fact that MiniMax-H3 still beat PAN 2 on single actions and on human preference. The scores are also the institute’s own.

The team found something useful in those numbers anyway. Across all six models, full-plan success tracked closely with single-step success on one steep curve. Their fitted curve suggests that pushing single-step reliability from 75% to 90% would roughly double full-plan success, from about a third to about two-thirds. In other words, small gains on each step should pay off big on long jobs.

On a public physics test called Physics-IQ, the institute says PAN 2 ranks first among open models and third overall.

Free to use, down to the training code

What sets PAN 2 apart is how much the institute gave away. The video model’s weights, the full training code and the code to run it are all posted under the Apache 2.0 license, which lets anyone use and change them, including companies, the project’s GitHub page shows. Few world models ship with their training code.

“We want researchers to be able to test PAN 2’s capabilities, understand its limits, and build on it,” said Hector Liu, director of the institute’s Silicon Valley lab. “That means releasing the model weights alongside the training and inferencing code.”

There is a catch for hobbyists. MarkTechPost notes it runs on data-center graphics chips, with the full 29-billion-parameter model held in memory. This is a tool for labs and robot builders, not for a laptop.

Eric Xing, the lead author on the research paper behind PAN 2’s design, framed the goal as getting machines to work in the messy real world rather than a tidy ideal one.

The institute says its next step is a simulated world that grows and stays consistent as a robot explores it, plus training robots on lots of rehearsed attempts so the lessons carry into the real world.

For now, the ball-in-a-tube test is the clearest picture of where this is going. The program fumbled, misread a clip and wasted three tries. But it figured out the glued cap without ever touching a tube. Do that a few thousand times in simulation, and the robot that finally reaches for the real thing has already made its mistakes somewhere they didn’t cost anything.