It took a lot of emails, whining talking to the right people (thank you!), some luck and the dedication from our MVP Lead, Andrew DeBerry and others – but finally, me and the rest of the Kinect for Windows MVP group (or Emerging Experiences MVP group as we call ourselves now) got a chance to experience HoloLens in person during the MVP Summit in November.

This was a pretty big deal for me. I’ve gotten into the small Kinect Emerging Experiences group because of my interest and passion towards new and almost green-field ways of human-computer interaction. And HoloLens is the first mixed reality device that has a chance of being widely available. It has never stopped tickling my fantasy since it first has been introduced. I’ve read every piece of review and report on personal experience from those who were lucky enough to actually try it. I had a pretty good idea on what to expect – but wanted desperately to experience it for myself.

This post is the summary of my experiences. It will be long, but worth it.

Holographic Academy

The Holographic Academy is Microsoft’s dedicated showcase area for HoloLens. It is located in building 37 on the Microsoft Campus. Once we left all our bags and phones at the nearby lockers, we (about 20 of us) gained entry to a rather large room. The room was scarcely lit, but it had more than enough light for us to see well. In the center of the room was a round stage, on which stood a tall, muscular guy – looking just like a drill sergeant from the movies. Around him were about 5 or 6 stations, each with computers, a few HoloLens devices, a table, TVs hanging on the wall, a couch, and a Microsoft employee to help. We were directed to split into groups of 5 each, and each group went to a different station. My group ended up being only four people, and we had three HoloLens devices, so chances were good 🙂

The air-click gesture

The “sergeant” started to speak, loudly, with authority, but fortunately without any of the scary uptones. He turned out to be a pretty nice guy, welcoming us to the Holographic Academy, and walking us through the basic gestures of “air tapping” (clicking), and “bloom”, which opens up the Start menu. Once it became clear that we weren’t about to go through a rigorous bootcamp training, and got our congrats for mastering the air gestures, my attention started to wonder. I looked at the device right next to me on the table, connected to the PC nearby for charging. It looked exactly like the one in the pictures, so no surprises there. However, I could see the display area – it was about 3×2 centimeters, but I had no way of actually measuring it. Still, it looked much larger than the Epson AR glass I tried a couple of weeks earlier.

Putting it on

Soon, we had the chance of putting on the device. It has an internal band, which can be expanded and contracted based on the size of your head. The purpose of this band is to bear the weight of the device, and distribute it evenly on your head – it’d be way too heavy to just rest on your nose. The actual device can then be tilted independently of this holder unit – in fact, one of the first surprises to me as a lifetime wearer of glasses was that the nose bridge wasn’t even supposed to rest on your nose. All in all, HoloLens sat on my head comfortably, and soon I’d forget about the weight of it. Ah, and we didn’t have to measure my pupil distance – that seems to be something the device no longer needed or did automatically.

The head pant can be expanded and contracted using the wheel in the middle. It is a very premium and comfortable experience.

On the left side, there are two buttons to control the volume, and on the right side, two buttons control the brightness.

This is what the volume and brightness buttons look like

When I first looked through it, I saw a pale blue border, indicating the actual screen. Well, it is not a screen, but more on that later. Soon, HoloLens started booting, and I saw the familiar, light blue Windows logo as it did so. Yapp, it’s Windows 10 all right!

At this point I was pretty disappointed with the field of view… it was not just small, but looked like a 16:4 aspect ratio. I soon realized that the top half of the screen faded away, but the bottom side had a very sharp edges. I started moving the headset around, and yes, it seemed like either my glasses or my eyebrows actually obstructed the view. As I moved the movable part of the headset down a bit, I managed to get a full 16:9 aspect ratio display out of it. I know this because I could move myself so that the entire display area covered one of the TV sets on the wall.

Another, very interesting observation: you can not only tilt, but also move the glass closer to or further from your eyes. The travel distance is about 5-8 centimeters, which is quite a lot. And while I did this, the actual field of view has not changed! If HoloLens had a display in front of my eyes, I’d have expected it to shrink as I move it away. But the perceived size of the display area remained the same – this suggests that HoloLens actually uses some kind of projection, and not simply a small display in front of you.

In less than a minute, the logo disappeared, and a small 3D graphic took its place – a line drawing, resembling three mountains, made out of just a couple of lines. A text saying that HoloLens is scanning the environment was displayed. And soon, magic started to happen.

Scanning the Room

The spatial mapping process

The room scanning (Spatial Mapping) looks somewhat like what you see on the videos – triangles made of light start to cover the real objects in front of you. There are a couple of differences though:

• The triangles were not filled blue triangles, but outlines, made of white light
• It took about 20-30 seconds for the scan to finish
• The triangles had sides of about 5-10 real life centimeters. This is more than enough to discover the walls, furniture and other objects in the room, but the resulting mesh is pretty low resolution for anything more than that. I have no idea whether it is just for the coordination of the device in the 3D space, whether the HoloLens is capable of a higher resolution environment mapping, or even if apps can actually access this information.

Start!

During the “boot camp”, we were asked to first use the “Origami” app, and then we were told that we could experiment freely, although a lot of the apps on the device may not work or not work well. So, when the scan finished, the start menu was presented, hanging in the air in a fixed space in front of me. The way HoloLens interaction works is somewhat like a mouse – you have a pointer which you can direct with your head, and the air click gesture to activate whatever the pointer is over. All the usual effects – “mouseover” and click animations, and even sounds are in place.

The way you move your pointer is by moving your head – not your gaze. The pointer is in the middle of the screen, and looks like a small circle. The air click gesture can be performed pretty much anywhere in front of your body. However, simply bending your finger down and up is not enough – it is not a coincidence that we’ve been trained “hard” to move our entire finger and touch our thumbs. If you do this gesture right, it works well, and detection is pretty reliable.

So, as a newbie Holographic Academy graduate, I obediently moved my head so that the pointer was over the Origami app, and clicked – I mean, air-tapped.

Origami

An early version of the Origami app. We were shown a much more refined and complex version.

The Origami app starts out as a holographic cube. It moves along with your gaze, and as you look around, it stays mostly in center – but sticks to the floor, the walls and the tables. I moved my head so that it was on the table, and put it there, using the air-tap gesture.

This was the first time I actually examined the holograms themselves. And I think, Microsoft choose a very good name when they decided to call these things “holograms”. They look exactly as what you’d expect after watching too many Star Wars movies. Perfect illusion of 3D objects hanging in space. Still, you’d never mistake a HoloLens hologram for a real life object. There is one fundamental difference: holograms are actually made of light. Real objects reflect light, and thus they are not bright in a dark room (except for lamps and some tricky lighting, but I digress). Holograms are made of light themselves, but their light is not reflected on the furniture around them.

But that’s where the similarities between R2-D2’s projection in Kenobi’s cave end with HoloLens. Because the holograms in HoloLens’s field of view are absolutely amazing. The holograms stick in place. You can move your head around, and they remain exactly where you put them. You can move yourself, and examine the hologram from every direction. You can jump up and down (believe me, I tried, looking more like an idiot than usual), and they are still there. No jumps, no glitches, no nothing. The Holograms are where you put them, and they stay there. They are also extremely solid – there’s barely any transparency to be seen. Of course, this also depends on the brightness level set on the device.

The 3D illusion is also perfect. If you go to an IMAX movie, the 3d can be breathtaking – but you’ll never confuse it with reality. You always know that it’s an illusion. Not so with HoloLens. The holographic objects are “just there”. You don’t have to convince your brain that you’re looking at a 3D thing, because you ARE. There is none of that over-emphasized “look, I am 3D” feeling that you get with 3D movies. Things are just naturally there, and naturally 3D. And this is extremely important to keep the illusion alive – because holograms need to work together, and next to the real, three dimensional reality. This is the real mind-blowing part of the HoloLens tech – the 3D illusion is so perfect, you don’t have to suspend your disbelief, because there is none of that to begin with. Except for the field of view…

OK, back to the Origami experience, which I placed on a small table earlier. There are two slopes in the hologram, seemingly made of folded paper, both of which have an origami ball suspended above them. If you air-tap, the balls fall down on the ramps, roll down, and there is an explosion at the bottom of the ramp as the balls hit the table. Then, the table “opens up”, and through the hole you can peek into a new world. The world has origami birds, clouds, mountains and a blue sky – and you’re looking at it from above. The illusion is perfect, it’s as if you opened a Portal in Valve’s game, and are now looking down from the sky. You can walk around the table, and peek into this portal from every direction – the illusion stays impeccable.

Sounds

One of  the speakers

The Origami experience (along with some others) emit a 3D spatial sound, which gives you an important pointer to what’s happening around you and even behind you. Unfortunately, there was something wrong with my device, and I couldn’t get the spatial sound illusion working, even though my spatial hearing is fine in the real world. This was probably some sound driver issue, limited to the device I tried. Others in my group had no sound problems, but my HoloLens actually rebooted itself when I tried to launch the Cortana app. (remember, HoloLens is Windows 10). Also, voice commands didn’t work for me – there was supposed to be a reset world command for the Origami experience, which it didn’t get even when the helper in our area said it leaning close to my headset. BTW, try to do that with a VR headset – it would really freak out the wearer.

The “Holograms” app

After I had enough of the Origami app, I performed the “bloom” gesture again to bring back the Start menu.

(This is the Bloom gesture, but the resulting Start menu doesn’t look like this.)

The next app I tried was a simple one – you could select holograms from a list, and place it anywhere in the space. If I recall correctly, the app was called “Holograms”. Once you’ve selected a Hologram, it followed your gaze (meaning where your head was pointing at, not your eyes), and stuck to any surface you were looking at. I could then fix the Hologram in place with an air-tap gesture. Some of these holograms were just static 3D objects, some would animate. If I wanted to move a Hologram, I had to air-tap on it, and a surrounding box would appear. There were text options below the box to delete the Hologram, move it or resize it. Deleting worked much like you’d expect. However, I had some cognitive issues with the resizing. The problem here is that the cursor is usually moved with your gaze (head). However, when resizing, I had to perform a pinch gesture with my finger, and move it in the 3D space. Basically, when performing the “drag and drop” operation, you have to move your hands – but in other cases, the pointer moves with your gaze. Multiple times I moved my head to move the resize handlers, when I should have used my hands at that point. The reason for this is understandable – gaze is a 2D pointer, but when you want to move stuff around, you want to move them in 3D. The HoloLens perfectly followed my hand movement in all three dimensions, but this experience was still somehow confusing to my brain trained for mouse usage.

The first thing I did was to place a rainbow Hologram on the floor, about 2-3 meters from me. As I said before, the 3D illusion was perfect, but you couldn’t confuse the Hologram with anything real, because it was made of light (and the objects around didn’t get lit by the light the Hologram radiated).

The next thing I tried was to put a holographic space suit helmet on the head of a fellow MVP, patiently sitting on the couch and waiting for her turn. I could easily move, rotate and resize the helmet to fit it on her neck. The hologram completely blocked out her head, I couldn’t see her face – until she moved a bit and was only half covered with the helmet 🙂 Still, the illusion was great, and I could move around and look at her helmet from all directions. Now that I think of it, I must have looked like a freaky stalker, staring at her head in my futuristic glasses, with mouth half open in wonder… Sorry 🙂

One thing I haven’t tried though is to place large Holograms. I guess I am so used to small screens that it didn’t even occur to me that computer generated objects can be as big as a human, or even larger. Just imagine using this technology to design clothing, or any machinery, and seeing the result in 3D in real time, in the real environment!

I also placed a small ballerina Hologram next to the small orange table, on the floor. As I moved around the table, it started to cover the Hologram, as a real object would! Well, mostly… because of the low resolution environment mesh, the surface area of the round table was pretty scarcely modelled – and therefore, it couldn’t hide the hologram perfectly. This was probably the weakest holographic experience I had – when a real life object was supposed to hide the hologram, but it couldn’t because the real life object’s mesh was too rude.

I even played with covering the depth sensors on the glass (there are two on each side, looking somewhat outwards). When I covered one, the tracking remained stable. However, when I covered the other one as well, the tracking was lost, and I was back to the initial Environment Scanning, when the environment 3D mesh with the triangles were built up.

Holographic Browsing

The next thing I tried was launching the Edge browser. It worked as you’d expect – a 2D window, floating in front of you in space. Selecting links and navigating was simple enough. However, the text clarity was not perfect, I could see a disturbing ghost image on the smaller text. It may have been because I have pretty strong glasses and there’s more than 1 dioptry difference between my eyes. I should have closed one of my eyes to see if the text would’ve became clearer – but unfortunately, I didn’t think of this while there. Scrolling the browser was simple, a similar drag and drop gesture as what you used for resizing objects.

The Last Mind Blowing Experience

At this point I was running out of time, but so, the last thing I tried was to launch the Photos app. You know, the Windows 10 app that has all you photos? I was expecting another “2D app floating in space”. But boy, was I wrong!

It started out much like the Edge browser – you select the app from the Start menu, place it in space, and start interacting with it. But the thing about HoloLens is that you can move close to the Holograms. And that’s what I did.. I moved closer to the app, looking for the point where the field of view limitations would hinder the experience. But instead, I found something else. I found that the actual photos were in front of the app! Field of view concerns totally forgotten, I moved even closer. And yes – the Photos app was not the same app running on the “standard” Windows 10. It looked similar, but the UI elements were actually in 3D! The photos were in front of the background of the app, and casted small shadows. I moved to the side, and I could actually see the gap between the photos and the rest of the app. And the rest of the app actually had some thickness to it! Not just a thin paper floating in the air – the app’s toolbar had thickness, the app’s background was also a solid 3D object.

To me, this app was the most mind blowing stuff I saw. I was expecting most of the other things, based on what I read and heard of HoloLens before. I just wanted to see those with my own eyes. But… seeing an app I used every day in 3D, with real substance, real width – wow. Just wow. It sounds cliché, but these final few minutes with a solid, 3D version of an everyday 2D app made me realize that this is the device that really transforms computing to a new dimension.

• HoloLens
• NUI