The definitive list of text-to 3D, image-to-3D and video-to-3D models and applications
.png?width=1280&auto=webp&quality=95&format=jpg&disable=upscale "A female 3D character generated by AI. AI has the potential to scale the creation of 3D assets.")
When it comes to technology hype, AI is overshadowing the metaverse to capture the attention of businesses and consumers everywhere.
But AI could further enhance the metaverse, at least in one sense: asset creation. AI has the potential to scale the creation of 3D assets for use in virtual environments.
AI Business explores the various AI 3D generation capabilities.
AI 3D generation uses artificial intelligence to generate 3D models or objects.
There are several techniques:
Text-to-3D models
Image-to-3D models
Video-to-3D models
Each technique can potentially help creators customize and generate virtual assets and environments at speed, especially helping those without prior 3D modeling expertise.
Potential applications of AI 3D generation are for creating assets in video games and metaverse environments, as well as design concepts for products in business. It could even be applied to creating virtual representations of factory floors, generating potential mock-ups of industrial plants.
Creators: Google, UC Berkeley
First published: September 2022
DreamFusion can generate 3D models from text inputs. It is essentially an expanded version of Dream Fields, a generative 3D system Google unveiled back in 2021.
DreamFusion requires little to no prior training – meaning it can generate 3D representations of objects without 3D data. It uses 2D images of an object generated by the Imagen text-to-image diffusion model to understand different perspectives of the model it is trying to generate.
Read the DreamFusion paper.
Explore 3D models generated by DreamFusion.
Read more about DreamFusion on AI Business.
Creator: OpenAI
First published: December 2022
Point-E works the same as OpenAI's image generation tool DALL-E – simply type in something like, 'a corgi wearing a Santa hat' and the system will generate a 3D model.
Point-E can generate 3D models in just one to two minutes on a single GPU. It also boasts image-to-3D capabilities, meaning images can be used in tandem for inputs alongside text.
Read the Point-E paper.
Access the Point-E code.
Read more about Point-E on AI Business.
Creator: Apple
First published: August 2022
Named after the famous Spanish architect, Gaudi was created by researchers at Apple to generate 3D scenes from text prompts. Users can redraw the scene from any angle using natural language – allowing for the creation of interior scans of buildings.
GAUDI also can generate new camera movements through 3D indoor scenes via text, such as a user input to 'go through the corridor.'
Read the GAUDI paper.
Explore GAUDI 3D generations.
Read more about GAUDI on AI Business.
Creators: TikTok parent ByteDance, University of California, San Diego
First published: 31 August 2023
MVDream is one of the newest models on this list. Researchers from ByteDance and the University of California, San Diego developed a diffusion model that can generate high-quality 3D renderings from text inputs.
MVDream can generate multiple views of the model that are geometrically consistent multi-view images.
The model can be fine-tuned for personalized 3D generation using tools like DreamBooth3D.
Read the MVDream paper.
Creator: Nvidia
First published: March 2023
Magic 3D is a pre-trained text-to-image diffusion model. It can create quality 3D mesh models from text prompts.
Magic3D utilizes a two-stage process:
It creates a coarse model using a low-resolution diffusion.
The system then uses a sparse 3D hash grid structure that is further optimized with an efficient differentiable renderer to create the final output.
Magic3D can create high-quality 3D mesh models in 40 minutes.
Read the Magic3D paper.
Creator: Johannes L. Schönberger, principal scientist at Microsoft Mixed Reality & AI Lab
First published: October 2021
COLMAP is an open source 3D reconstruction pipeline with a graphical and command-line interface. It can be used to generate detailed 3D environments from a handful of images.
COLMAP users can export a 3D mesh – although to refine it, programs like MeshLab would be required.
Access COLMAP via GitHub.
Read more about COLMAP on AI Business.
Creators: The Chinese University of Hong Kong, Shanghai AI Laboratory, Zhejiang University
First published: August 2023
PointLLM is designed to aid 3D generation. It's an approach that helps large language models like PaLM and Llama understand 3D data.
Using PointLLM, AI researchers can greatly utilize point clouds – data points that represent a 3D shape or object – in their datasets.
Read the PointLLM paper.
Access the PointLLM codes and dataset.
AI-powered video creation tool that can animate a brand's characters through text inputs.
Simply provide Gliastar with a 2D or rough 3D representation of your character, integrate the model into the GliaStar AI-Animation system, which will then automatically generate a 3D animation of the desired mascot in minutes.
Try Gliastar for yourself - https://www.gliastar.com/
3DFY.ai can create high-quality 3D models from a text prompt or a single image.
3DFY.ai 3D generation pipeline can automatically generate 3D outputs at speed – with users able to test out the text-to-3D web service to try out prompts of their own.
Try 3DFY.ai for yourself - https://3dfy.ai/
Fotor lets you generate 3D models from text descriptions. Users can create detailed 3D models for various applications.
Fotor also has an AI 3D character generator where users can create unique and detailed characters in minutes.
Check out Fotor - https://www.fotor.com/features/ai-3d-model-generator/
With Spline, users can generate 3D models and representations in real time using natural language inputs.
Spline can be used not only to make 3D models – but also animations and textures.
Explore Spline AI - https://spline.design/ai
Meshcapade builds virtual avatars – using generative AI to create accurate digital humans for business applications, like e-commerce settings.
Meshcapade's SMPL system uses machine learning to capture the statistics of an individual's 3D body shape.
Try Meshcapade for yourself - https://meshcapade.com/
With Masterpiece Studio, you can generate 3D assets using AI.
Masterpiece Studio builds 3D creation software that users with non-technical backgrounds can use to create 3D content from concept to animation.
The Masterpiece team partnered with Meta to build Masterpiece X, which allows creators to remix existing 3D assets. It's also accessible on the Meta Quest store.
Explore Masterpiece Studio - https://masterpiecestudio.com/
With Luma, users can generate photorealistic 3D assets and environments in minutes.
Luma is available via mobile – so iOS users can use its 3D generation tools from their handsets.
Check out Luma AI - https://lumalabs.ai/
Read more about:
ChatGPT / Generative AIYou May Also Like
