Nvidia’s New Robot Brain Learns Delicate Hand Skills by Watching Videos of People Working

Teaching a robot to do something delicate, like fold a shirt or focus a microscope, usually takes piles of practice runs on the robot itself. Nvidia says its new SI can learn a big chunk of that by watching people.

On Monday, at a robotics conference in Austin, Texas, Nvidia unveiled GR00T 2, a new “brain” for robots that it trained on robot practice runs and a huge library of first-person video of people doing hands-on work. Nvidia announced it on its official blog.

What it learned from

Nvidia’s research page lays out the recipe: 402,730 hours of video shot from the point of view of “people at work,” 10,184 hours of people using a handheld robot gripper across 2,897 tasks, and 23,500 hours of humans steering real robots by remote control. That is nearly 46 years of human video, played back nonstop.

Nvidia is giving away 100,000 hours of that human video for anyone to use, posted on the Hugging Face sharing site.

Does the human video actually help? Nvidia tested that directly. It trained four versions with none, 1%, 10% and all of the human video. On a real robot, average progress on 72 tasks rose from 50% to 70% as the human video grew.

What it can do

The new model also tries to predict what will happen next before it moves. Nvidia says it “predicts future video and robot actions together,” the way a person pictures reaching for a cup before doing it.

Nvidia’s headline numbers: with no lessons at all for a specific job, it completed 40 of 50 tested tasks at least once. Shown a task once, it succeeded 71% of the time on average. Shown five times, 95%.

It can also pass skills across different machines. Nvidia says it “learns from human demonstrations and transfers skills between different robot bodies without robot demonstrations,” so a skill learned from a person or one robot can carry over to a robot hand built differently.

Who’s using it

Companies are already testing it. Nexuni is trying it for robotic laundry, and MicroAGI is using it to swap network cables with a robot arm, according to The Robot Report. Foxconn and Siemens are testing it too, The Next Web reported.

The fine print

Every number above comes from Nvidia’s own testing, not an outside lab. And The Next Web noted that Nvidia’s own paper says the tasks were picked from ones the model could already learn well. In other words, those 95% results are a best case, not a promise for any job you hand it.

Still, the direction matters. If robots can learn from ordinary video of people working, instead of thousands of hours of robot-by-robot coaching, useful helpers in warehouses, labs and maybe someday homes get a lot cheaper to train.