German carmaker wants to create hologram ‘head-up’ displays in cars.

When Volkswagen was testing augmented reality in a head-up display on a vehicle – those holographic directional arrows, driving speed and other information that appear on a windshield to guide drivers – it thought about using Microsoft’s HoloLens AR headset.

But the problem was that when the HoloLens was put in a moving vehicle, its sensors lost their tracking ability and its holograms disappeared. That’s because the headset’s two types of sensors − visible light cameras and an inertial measurement unit (IMU) – disagreed about whether there was motion. The IMU recognized motion and the cameras did not.

This is similar to why humans get motion sickness: when the eyes see one thing but other senses feel something else.

Volkswagen approached Microsoft to find a solution to this problem.

The solution

To get around this impasse, VW connected an in-car GPS to the headset to give the wearable a better understanding of its position and surroundings. The German automaker came upon the discovery as it was tinkering with HoloLens in a bid to create head-up displays for cars, providing drivers with information on things such as speed and direction.

It addresses a long-standing HoloLens issue of moving surroundings confusing the headset’s sensors. The enhancement was developed in collaboration with Volkswagen, which has been experimenting with using the headset as a head-up display in its vehicles.

“We think mixed reality information is the most intuitive information we could provide to enhance our customers’ user experience,” said Andro Kleen, head of the data science team at Volkswagen Group Innovation. “What you see there, and what you need to process, is very close to what humans normally see and process. It’s not so abstract.”

VW initially began using HoloLens as a means to teach inexperienced drivers to become more confident on the roads. However, the motion understanding issue caused this use case to flounder.

Since the fix a few months ago, Microsoft said it has begun rolling out the moving platform feature, which has already attracted interest from several firms in the maritime space as it is currently only available for use on large ships. However, the company said it plans to refine and expand the service for use in elevators, trains, cars and other moving environments.

“We think of this as moving toward a mobility system where different products and mobility solutions will be connected,” Kleen added. “The basic assumption is that this technology will become lighter and smaller ... (and) as that happens, more people will get their hands on it and integrate it into their daily lives — and thus into their way of moving from A to B.”

One market that has expressed interest in Microsoft’s hologram headset is defense. Last April, the U.S. Army struck a $21.9 billion deal with the tech provider to purchase 120,000 HoloLens systems. The units will be custom-built and will be used to project digital images in the wearer's field of view of battlefield objects and objectives.