Designing for virtual and augmented reality: Our trip to Microsoft's Hololens Academy

We were invited to visit Microsoft’s Hololens Academy in Redmond, Washington, where we spent the day learning and hacking the Hololens.

The Hololens is a virtual reality glasses/headset combo developed by Microsoft. In a way, it "projects" crystal clear holograms into your 3D environment and lets you interact with them.

From a designer's standpoint, there was a lot to take in from Microsoft's presentation. It was validating to see we shared the same ideas and made the same conclusions about various design problems.

I won't dive too much into the the actual experience of the “Academy” portion of the day - I want to focus on problems we had in designing an AR experience, how we solved them, and how Microsoft solved them.

Our most pressing problems were figuring out how to show the user where they were “selecting” with their eyes, how to show an “on hover” state, and how to actually select an object with just your gaze.

What we tested

While we were building a demo, we had a problem getting users to realize that where their head was pointing was where they were “selecting” in space.

We tested several different solutions. In the end, we settled on a reticle, which is a series of fine lines or fibers in the eyepiece of an optical device - such as a telescope or microscope, or on the screen of an oscilloscope - used as a measuring scale or an aid in locating objects.

Companies like Google are also using reticles and have great resources explaining why these are a good idea. We just wanted to make sure before we went whole hog on using a reticle.

Google Cardboard image from Google

Our solutions

The reticle ended up being a ball of light in 3D space that followed your head position. It was as unobtrusive and visually pleasing as we could make it. We tried several variations: a ball that pulsed, breathed, had shadow or glow, had no glow, had an outline, and others.

reticle idea 1 reticle idea 2 reticle idea 3

Our favorite: the simple circle of light -- no glow and no breathing. The only concession was to change the size of the circle with depth; if your gaze was resting on an object in the distance that was recognizable to our demo, the circle would shrink in relation to the distance, which gave the impression of the circle moving back in space. We found that this helped with the “illusion” of 3D and was generally received as positive by viewers.

Microsoft’s interpretations of the reticle

It was interesting to see Microsoft's various versions of the reticle and how it changed in different applications. I won't assume to have figured out all of Microsoft's engineers' learning and expertise in just one day, but I still gleaned valuable information.

In one application, Microsoft tackled the reticle in a remarkably similar way to us: they used a still, floating ball of light that changed in size according to its place in space. The difference between our version and theirs was their different paradigm of interaction - the inclusion of gestures for selection, i.e. Airtap (Microsoft’s version of selection includes tapping your index finger and thumb together in front of you).

Another reticle they used was donut-shaped and looked to be made out of a solid material, like wood or metal or stone. What was particularly interesting about this version of the reticle is that it not only changed size according to its place is space, but it also changed orientation based on the “edge” it was being placed on. Unlike the floating ball of light, this reticle was treated with physics of a real object and felt solid.

Microsoft Reticles

What this shows

Fundamentally, we know some things about sight: the human eyes need someplace to “focus” and our brain needs a visual representation. What we don’t know is how to best accomplish this. I’m convinced that neither we nor Microsoft have figured out how to best give the mind what it wants, but our explorations in reticle development have so far proved to be fruitful.