Nvidia’s DreamDojo: AI System Teaches Robots from 44K Hours of Human Video

Nvidia this week unveiled DreamDojo, a new artificial intelligence system designed to accelerate the development of humanoid robots by leveraging a massive dataset of human video. The system, detailed in research published earlier this month, aims to reduce the substantial time and expense currently required to train robots to interact with the physical world.

The core of DreamDojo is a dataset dubbed DreamDojo-HV, comprising 44,000 hours of human egocentric videos – footage captured from a person’s point of view. This represents a significant leap in scale, exceeding the previously largest dataset for world model training by a factor of 15 in duration, 96 in skills demonstrated and 2,000 in the number of scenes, according to project documentation.

Researchers from Nvidia, UC Berkeley, Stanford, the University of Texas at Austin, and other institutions collaborated on the project. DreamDojo operates in two phases. First, it learns general physical principles by “pre-training with latent actions” from the large-scale human video dataset. Subsequently, it undergoes “post-training on the target embodiment with continuous robot actions,” refining its understanding for specific robot hardware.

This approach addresses a key challenge in robotics: the need for extensive, robot-specific data to teach robots how to manipulate objects in unstructured environments. By learning from human observation, DreamDojo allows robots to develop a foundational understanding of physics before physical interaction, potentially lowering development costs and accelerating deployment.

The system has demonstrated the ability to generate “realistic action-conditioned rollouts” across multiple robot platforms, including the GR-1, G1, AgiBot, and YAM humanoid robots. Researchers achieved real-time interaction speeds of 10 frames per second for over a minute through a distillation process, enabling applications like live teleoperation and on-the-fly planning.

The release of DreamDojo coincides with a period of significant investment in AI, and robotics. Nvidia CEO Jensen Huang recently stated that AI robotics represents a “once-in-a-generation” opportunity, particularly for manufacturing hubs. Huang also indicated that the next decade will observe “accelerated development for robotics technology.” Industry-wide capital expenditures for AI infrastructure are projected to reach $660 billion this year, according to Huang, with major tech companies like Meta, Amazon, Google, and Microsoft driving the increase.

The broader robotics industry has seen a surge in funding, with startups raising a record $26.5 billion in 2025. European industrial companies, including Siemens, Mercedes-Benz, and Volvo, have formed robotics partnerships, and Tesla CEO Elon Musk has predicted that 80 percent of his company’s future value will derive from its Optimus humanoid robots.

For companies evaluating humanoid robots, DreamDojo’s simulation capabilities offer potential benefits. Researchers highlight the possibility of “reliable policy evaluation without real-world deployment and model-based planning for test-time improvement,” allowing for extensive testing before physical trials. What we have is crucial, as robots often struggle to adapt to the unpredictable conditions of real-world environments.

The research team, led by Linxi “Jim” Fan, Joel Jang, and Yuke Zhu, with Shenyuan Gao and William Liang as co-first authors, intends to release the code publicly, though a specific timeline has not been announced.