AI creates 3D scenes from 2D photos in seconds

AI can create 3D scenes from 2D photos in milliseconds, reversing the process of the instant photo created 75 years ago. Using neural networks, it mimics the behavior of light to create raster-like images.

When the first instant photo was taken 75 years ago with a Polaroid camera, it was groundbreaking to instantly capture the 3D world in a realistic 2D image.

Now, Artificial Intelligence (AI) researchers have achieved the opposite: turning a collection of 2D still images into a 3D digital scene in a matter of seconds.

known as reverse representationthe process uses AI to mimic how light behaves in the real world, allowing a 3D scene to be reconstructed from a handful of 2D images taken from different angles.

The research team at NVIDIA, a multinational specialized in the development of graphics processing units, has developed an approach that accomplishes this task almost instantly, making it one of the first models of its kind to combine network training lightning-fast neural networks and fast rendering.

strong acceleration

This team applied their approach to a popular new technology called Neural Radiance Fields or NeRF. The result, called Instant NeRF, is the fastest NeRF technique to date, achieving more than 1,000x speedups in some cases, the company said in a statement.

The model requires only a few seconds to train on a few dozen still photos, plus data on the camera angles they were taken from, and can then render the resulting 3D scene, all this in tens of milliseconds.

What this technology does is capture the way light radiates from an object or within a scene to create bitmap-like images that can be viewed on a monitor, paper, or other device. .

According to its developers, this technology could be as important to 3D as digital cameras and JPEG compression have been to 2D photography: it increases the speed, ease and scope of 3D capture and sharing.

It was unveiled last week, and in something of a tribute to the early days of Polaroid imaging, this technology recreated an iconic photo of Andy Warhol taking an instant photo, then turning it into a 3D scene using Instant NeRF.

neural networks

The team explains that NeRFs use neural networks to represent and render realistic 3D scenes based on an input collection of 2D images.

The neural network requires a few dozen images taken from multiple positions around the scene, as well as the camera position of each of them, to reach 3D.

If there is too much movement during the 2D image capture process, the AI-generated 3D scene will look blurry, but AI manages to more accurately reconstruct the scene by predicting the color of light radiating in any direction , from any point in 3D space.

Speed ​​is another advantage of this technology, because if creating a 3D scene using traditional methods takes hours or more, depending on the complexity and resolution of the display, adding AI to the image reduces rendering time by several orders of magnitude. magnitude.


The technology could be used to create avatars or scenes for virtual worlds, capture video conference participants and their environments in 3D, or reconstruct scenes for 3D digital maps, its creators say.

Also to help autonomous robots and cars understand the size and shape of real-world objects, by capturing 2D images or video footage of those objects.

It could also be used in architecture and entertainment to rapidly generate digital representations of real environments that creators can modify and build.

Finally, the researchers are also exploring how this technique could be used to accelerate multiple AI challenges, including reinforcement learning, language translation, and general-purpose deep learning algorithms.

Related news

Leave a Reply

Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Never miss any important news. Subscribe to our newsletter.