Neuralangelo by Nvidia

NVIDIA Research has developed an AI model called Neuralangelo that utilizes neural networks for 3D reconstruction. This innovative tool has the capability to convert 2D video clips into intricate 3D structures, allowing for the creation of virtual replicas of real-world objects like buildings and sculptures. These generated 3D objects possess detailed textures and can be imported into design applications for further editing, making them suitable for a wide range of industries including video game development and industrial digital twins. Neuralangelo surpasses previous methods by effectively translating complex material textures, such as roof shingles, glass panes, and smooth marble, from 2D videos to 3D assets. Its high fidelity simplifies the process of rapidly creating virtual objects using footage captured by smartphones. By enabling the importation of highly detailed objects into virtual environments, Neuralangelo aids in the recreation of the real world in digital form, whether for video games or industrial digital twins. The model utilizes instant neural graphics primitives, the same technology found in NVIDIA Instant NeRF, to capture fine details like repetitive texture patterns, consistent colors, and vibrant color variations. It selects multiple frames from a 2D video, taken from different angles, to create an initial 3D representation of the scene, which is then optimized to enhance the details, similar to a sculptor refining the texture of a statue. The resulting 3D object or large-scale scene can be utilized in virtual reality applications, digital twins, or robotics development.