LucidPix is using AI to add 3D effects to mobile photography – we find out how
While artificial intelligence has already embedded itself in business applications, it has plenty of room to grow in the consumer sector.
LucidPix, a photography app for iOS and Android, is a good example of how AI needn’t be pigeonholed as something dry and analytical: it can also be used to enable creativity.
Or perhaps, more accurately, enhance creativity. LucidPix can take a picture snapped on any smartphone and make it appear three-dimensional.
To do this, the AI-based system examines the image pixel by pixel, predicting what it would look like from a slightly different angle, letting users tilt and move the image as if they were really there and slightly adjusting their viewpoint.
“To mimic creativity with AI is incredibly hard because AI is, from the get-go, not very creative,” explained Han Jin, CEO and founder of Lucid VR, who was cited in Forbes’ 30 Under 30 in consumer technology back in 2018.
“It’s trained to perform what it should do, by us,” he pointed out – which is almost the opposite of being creative.
“For us, the best part is we draw out the creativity of that image by making it 3D, and let the AI predict the content. I would say in 90% of the cases, we are pretty good, but there’s still some room for improvement, and clearly, it will take some time until AI becomes creative and precise.”
The app has done an impressive job since launching in beta at the start of 2020.
In fact, the global coronavirus lockdown has caused a spike in interest, as people seek new ways to connect remotely, and the app has passed the two million download mark, attracting investment from the founders of Oculus and Guitar Hero along the way.
From hardware to AI
The path to developing LucidPix was a lot less straightforward than you might think.
Initially, the company had a different vision for its 3D photography dream: hardware.
Prior to Lucid VR, Jin and co-founder Adam Rowell worked with the military to design large-scalesignal processing systems for the battlefield, helping computers predict the locations of mines and bombs.
“You can imagine something like that, where you have a huge truck with tons and tons of GPUs, which is almost the size of a tank or two. Today, that’s doable on a mobile device and that’s impressive, right?”
In 2017, the company created a $499 stereoscopic camera which could take 3D photographs; it eventually shelved the concept, in part because, in Jin’s own words, “hardware is really hard” and in part because AI technology has developed at such an incredible speed that alternative, software-based solutions are now possible.
The need to design your own hardware has been largely eliminated by advances in mobile System-on-Chips (SoCs), manufactured by the likes of Apple, Samsung and Huawei, but particularly Qualcomm, which still remains the dominant force in mobile hardware.
“Three to four years ago, they became powerful enough to make very rudimentary, very basic functionality possible,” Jin explained.
“Lately, like since 2018 and 2019, they’ve been able to perform very, very compute-heavy processing.”
It’s that processing that Jin and his team have taken advantage of, initially doing everything at the ‘edge’ – meaning the smartphone handles the AI models.
“The biggest benefit on the edge is real-time processing. The speed of it... I think it takes less than one second,” he explained.
Of course, not all smartphones are equipped with hardware-based AI accelerators. In fact, it doesn’t take a genius to realize that the vast majority of handsets in the world simply can’t run AI, given it has been just a decade since the original iPhone launched.
So how does the company cope?
Simple: it offers an alternative solution for phones without the processing resources – it outsources the work to the cloud.
“Our app is also deployed in South Africa or India and Bangladesh, and those kinds of areas have devices which just don’t have the power to do it, and that’s why we have been also supplying and complementing it with server processing,” Jin continued.
“That takes a little bit longer, I would say like one to two seconds, but I think it’s acceptable for a consumer, and that’s what we are aiming for.”
The system, powered by Microsoft Azure cloud servers, is another feature that has only become possible recently.
“A couple of years ago, I think that would have taken you ten to 20 minutes,” he said.
Clearly that would be unreasonable for a product where people need to see immediate results – including all modern photography apps.
“We have done a lot of consumer testing over the last couple of years and the one thing which surprises us every single year is that people are becoming more demanding in terms of processing speed, in terms of smoothness of the process, and yeah, user experience is everything nowadays.”
On the subject of that experience, how can the app be improved?
Jin highlighted the 10 percent of use cases where the app struggles: “When you have an image and it’s very crowded. You have a lot of objects in there stacked behind each other, and sometimes our AI thinks one object is supposed to be in the front, versus in the back. It’s just very, very computationally-demanding.”
Transparency, privacy and the consumer potential
Generally though, the user response seems positive, with scores of 4.5/5 and 3.5/5 on Google Play and the iOS App Store respectively.
With over two million cumulative downloads, it seems that people are happy with artificial intelligence in their photography app – and that’s pretty significant, given the hot water that viral sensation FaceApp got into last year with regards to data privacy.
“When someone downloads the app, the first thing we ask them is whether they understand that we are using AI to generate those images for them,” Jin explained.
“The second thing is that some of their images need to be sent to our servers for processing.”
Anonymity isn’t an issue: with LucidPix, you don’t even need to create an account.
“Being able to upload pictures with us never knowing who you are, where you live, and however you use your pictures is very important for us,” Jin said.
“We didn’t want consumers to start loving this kind of format and then end up getting concerned by data privacy issues. So that’s what we try to avoid.”
Anonymous pictures submitted by users are used to train the system and make it better. Anybody unnerved by this may be humanizing a product that sees the world in ones and zeroes.
“The good thing about AI training is AI doesn’t see your image,” Jin explained.
“All it does is break it down: it looks at the color, it looks at every pixel, and it assigns a number to that pixel.
“Is it a naked person, or is it a dog? The AI system will never understand the content of the image, and we don’t see the images either. There’s no privacy being leaked or any data issues because the AI will not be able to use it.”
That’s in stark contrast to apps like FaceApp, where abuse could have more sinister implications. “If you do combine it, like FaceApp, with your face and with your age and name, that’s something else, because you’re assigning those kinds of personal data points to the image itself.”
Whether consumers are happy with this or just skipping the fine print, the potential for AI-infused digital imaging is clearly there, as is the hardware, and the market.
How about the competition?
“The one thing I noticed is there are not too many consumer apps out there which are using AI, and I think in the next couple of years, that will change,” Jin said.
“I’ve seen new things coming up – videos where people want to put their faces on Shakira and look like they’re dancing in the music video – and I think consumers will appreciate those kinds of things more and more.”
Jin thinks consumers will even pay for it – something that, historically, has been far from guaranteed for mobile apps.
Assuming you have an idea that can capture the public’s imagination, the sky could be the limit.
To find out more about the exciting potentials that AI could bring to smartphone applications, download our eBook: 'AI in your pocket: Better mobile computing with smarter apps'