The definitive list of text-to 3D, image-to-3D and video-to-3D models and applications

Ben Wodecki, Jr. Editor

September 12, 2023

5 Min Read
A female 3D character generated by AI. AI has the potential to scale the creation of 3D assets.
Getty Images

When it comes to technology hype, AI is overshadowing the metaverse to capture the attention of businesses and consumers everywhere.

But AI could further enhance the metaverse, at least in one sense: asset creation. AI has the potential to scale the creation of 3D assets for use in virtual environments.

AI Business explores the various AI 3D generation capabilities.

What is AI 3D generation?

AI 3D generation uses artificial intelligence to generate 3D models or objects.

There are several techniques:

  • Text-to-3D models

  • Image-to-3D models

  • Video-to-3D models

Each technique can potentially help creators customize and generate virtual assets and environments at speed, especially helping those without prior 3D modeling expertise.

Potential applications of AI 3D generation are for creating assets in video games and metaverse environments, as well as design concepts for products in business. It could even be applied to creating virtual representations of factory floors, generating potential mock-ups of industrial plants.

Text-to-3D AI models

DreamFusion

Creators: Google, UC Berkeley

First published: September 2022

DreamFusion can generate 3D models from text inputs. It is essentially an expanded version of Dream Fields, a generative 3D system Google unveiled back in 2021.

Related:12 Language Models You Need to Know

DreamFusion requires little to no prior training – meaning it can generate 3D representations of objects without 3D data. It uses 2D images of an object generated by the Imagen text-to-image diffusion model to understand different perspectives of the model it is trying to generate.

Read the DreamFusion paper.

Explore 3D models generated by DreamFusion.

Read more about DreamFusion on AI Business.

Point-E

Creator: OpenAI

First published: December 2022

Point-E works the same as OpenAI's image generation tool DALL-E – simply type in something like, 'a corgi wearing a Santa hat' and the system will generate a 3D model.

Point-E can generate 3D models in just one to two minutes on a single GPU. It also boasts image-to-3D capabilities, meaning images can be used in tandem for inputs alongside text.

Read the Point-E paper.

Access the Point-E code.

Read more about Point-E on AI Business.

GAUDI

Creator: Apple

First published: August 2022

Named after the famous Spanish architect, Gaudi was created by researchers at Apple to generate 3D scenes from text prompts. Users can redraw the scene from any angle using natural language – allowing for the creation of interior scans of buildings.

GAUDI also can generate new camera movements through 3D indoor scenes via text, such as a user input to 'go through the corridor.'

Related:AI Image-Generation Models and Tools: The Ultimate List

Read the GAUDI paper.

Explore GAUDI 3D generations.

Read more about GAUDI on AI Business.

MVDream

Creators: TikTok parent ByteDance, University of California, San Diego

First published: 31 August 2023

MVDream is one of the newest models on this list. Researchers from ByteDance and the University of California, San Diego developed a diffusion model that can generate high-quality 3D renderings from text inputs.

MVDream can generate multiple views of the model that are geometrically consistent multi-view images.

The model can be fine-tuned for personalized 3D generation using tools like DreamBooth3D.

Read the MVDream paper.

Magic3D

Creator: Nvidia

First published: March 2023

Magic 3D is a pre-trained text-to-image diffusion model. It can create quality 3D mesh models from text prompts.

Magic3D utilizes a two-stage process:

  • It creates a coarse model using a low-resolution diffusion.

  • The system then uses a sparse 3D hash grid structure that is further optimized with an efficient differentiable renderer to create the final output.

Magic3D can create high-quality 3D mesh models in 40 minutes.

Read the Magic3D paper.

Image-to-3D AI models

COLMAP

Creator: Johannes L. Schönberger, principal scientist at Microsoft Mixed Reality & AI Lab

First published: October 2021

COLMAP is an open source 3D reconstruction pipeline with a graphical and command-line interface. It can be used to generate detailed 3D environments from a handful of images.

COLMAP users can export a 3D mesh – although to refine it, programs like MeshLab would be required.

Access COLMAP via GitHub.

Read more about COLMAP on AI Business.

PointLLM

Creators: The Chinese University of Hong Kong, Shanghai AI Laboratory, Zhejiang University

First published: August 2023

PointLLM is designed to aid 3D generation. It's an approach that helps large language models like PaLM and Llama understand 3D data.

Using PointLLM, AI researchers can greatly utilize point clouds – data points that represent a 3D shape or object – in their datasets.

Read the PointLLM paper.

Access the PointLLM codes and dataset.

AI 3D generation applications to try

Gliastar

AI-powered video creation tool that can animate a brand's characters through text inputs.

Simply provide Gliastar with a 2D or rough 3D representation of your character, integrate the model into the GliaStar AI-Animation system, which will then automatically generate a 3D animation of the desired mascot in minutes.

Try Gliastar for yourself - https://www.gliastar.com/

3DFY.ai

3DFY.ai can create high-quality 3D models from a text prompt or a single image.

3DFY.ai 3D generation pipeline can automatically generate 3D outputs at speed – with users able to test out the text-to-3D web service to try out prompts of their own.

Try 3DFY.ai for yourself - https://3dfy.ai/

Fotor

Fotor lets you generate 3D models from text descriptions. Users can create detailed 3D models for various applications.

Fotor also has an AI 3D character generator where users can create unique and detailed characters in minutes.

Check out Fotor - https://www.fotor.com/features/ai-3d-model-generator/

Spline AI

With Spline, users can generate 3D models and representations in real time using natural language inputs.

Spline can be used not only to make 3D models – but also animations and textures.

Explore Spline AI - https://spline.design/ai

Meshcapade

Meshcapade builds virtual avatars – using generative AI to create accurate digital humans for business applications, like e-commerce settings.

Meshcapade's SMPL system uses machine learning to capture the statistics of an individual's 3D body shape.

Try Meshcapade for yourself - https://meshcapade.com/

Masterpiece Studio

With Masterpiece Studio, you can generate 3D assets using AI.

Masterpiece Studio builds 3D creation software that users with non-technical backgrounds can use to create 3D content from concept to animation.

The Masterpiece team partnered with Meta to build Masterpiece X, which allows creators to remix existing 3D assets. It's also accessible on the Meta Quest store.

Explore Masterpiece Studio - https://masterpiecestudio.com/

Luma AI

With Luma, users can generate photorealistic 3D assets and environments in minutes.

Luma is available via mobile – so iOS users can use its 3D generation tools from their handsets.

Check out Luma AI - https://lumalabs.ai/

Read more about:

ChatGPT / Generative AI

About the Author(s)

Ben Wodecki

Jr. Editor

Ben Wodecki is the Jr. Editor of AI Business, covering a wide range of AI content. Ben joined the team in March 2021 as assistant editor and was promoted to Jr. Editor. He has written for The New Statesman, Intellectual Property Magazine, and The Telegraph India, among others. He holds an MSc in Digital Journalism from Middlesex University.

Keep up with the ever-evolving AI landscape
Unlock exclusive AI content by subscribing to our newsletter!!

You May Also Like