Apple Holds the Keys to the Future of Mixed Reality

Since their acquisition of Augmented Reality innovators Metaio in May 2015, the expectation has been that Apple has something up it’s sleeve. Something that will disrupt the AR / VR sector in the same way they changed mobile technology forever with the iPhone.

While there have been a few patents which have been filed and frequent rumours about a possible headset, nothing concrete has emerged. My sense is that they have been close to announcing something revolutionary but have held back due to the sudden burst of activity from equally secretive competitors Magic Leap and the remarkable work from companies like Meta.

With the breakneck pace of innovation, whatever comes next from Apple will have to be a revelation. Particularly given their continued reluctance to embrace VR and AR in any meaningful way in iOS.

What’s clear is that they have many of the parts of the puzzle to create something new and exciting in the Mixed Reality space.

Apple What?

The Apple Watch has struggled somewhat to justify its price tag. As a piece of hardware in its own right it falls a little short. However, as a control surface for a Mixed Reality wearable device it would come into its own.

Using the touchscreen of the watch as a control for a cursor and the Force Touch  to trigger selections would provide a user friendly input method for a wearable AR device.

Mono Cameras Are So 2016

I was a little early in my prediction that the iPhone would adopt a 3D depth-sensing camera. After Apple acquired 3d-scanning specialists PrimeSense in 2013 I was convinced that the iPhone 7 would be the first Apple device to allow us to take 3D selfies but the expectation is that the next major release will finally feature this technology.

For a Mixed Reality obsessive like myself this is hugely exciting. It creates the possibility of real-time hand tracking. This would allow users to place their iPhone into a VR headset and be able to use gestures such as pinching, grabbing and swiping in mid air to manipulate virtual objects. This is the way that Hololens, the Meta Glasses and the Leap Motion device work and would make mixed reality on iPhone truly intuitive.

Virtually Real

At their InsideAR conference in 2014, Metaio unveiled two research projects that promised to add an amazing level of realism to mixed reality.

The first was their SLAM (Simultaneous localisation and mapping) tracking. This allowed a device to build up a continuous 3d model of a location as it moved through the space. This opened up the possibility of having truly immersive augmented and mixed reality without the need for any printed trigger images.

The second was a truly surprising and slightly mesmerizing demonstration of realistic lighting simulation on a 3d model. By studying how light was falling on the user’s face using the rear-facing camera, they were able to update the virtual lighting on a 3d model in augmented reality. This was displayed over the top of the input from the front-facing camera.  This created a truly believable sense that the object was part of the real-world. It even cast a shadow onto the ground from the correct angle and with realistic intensity.

If Apple can combine these technologies with a more powerful, higher resolution iPhone they’ll have a compelling mixed reality offering. That said, anything short of the lightweight, wireless connected wearable device that many are expecting could still be too little too late.




2017-05-26T13:10:38+00:00May 25th, 2017|