HoloLens Mixed Reality Streaming Done Right

Most people will have a huge smiley on their face when you show the HoloLens to them. With way over a hundred demos behind me, only 3 or 4 didn’t agree right away, that mixed reality is the future.

joy
My friend is experiencing the HoloLens for the first time, with a huge smile on his face

So, demonstrating HoloLens is a very grateful job. But it can be pretty frustrating, too, because:

  • You can’t see what the user sees, and can’t offer help or explanations;
  • If there are more people in the room, they’ll be bored as they have no idea what the current lucky person is experiencing.

Luckily, Microsoft has added a way to wirelessly project the so-called Mixed Reality view from the HoloLens. This displays the GPU-generated holograms that the user sees over the video coming from the RGB camera, and streams it real-time to a computer.

hololens tracking
Example of a mixed reality capture – the furniture is real, while the game character and the coins are not.

The problem is, that to get this streaming right, you need to have a lot of things working together. Most of all, you need a fast and reliable Wi-Fi connection. But even then, there is usually a delay measurable in seconds, often as much as 5-6 seconds. This makes it extremely difficult to explain what you’re seeing on stage (because you have to explain what you saw 5 seconds ago, which what the audience sees right now). And when you’re trying to help somebody, it can be downright frustrating, because you say stuff like “yes, there is is… oh wait, there it was 6 seconds ago, move back… not there… the other one… now air-tap… let’s wait a little until I can see that you successfully airtapped…”. In a business scenario this even comes along as unprofessional, and can make the HoloLens look like it’s not ready yet.

To illustrate the delay, I put on the HoloLens, and launched the Mixed Reality Capture Live Preview in the HoloLens Device Portal. Then I opened up and closed the Start Menu. Conditions were fairly good, so I “only” experienced a 4 second delay:

4 sec delay.gif
The ~4 second streaming delay. This is far from ideal for presentations or helping others.

After months of experimenting, and dozens of demonstrations, we at 360world found a way to reduce the latency significantly. This has worked for us a dozen times already, under varying circumstances, including our own office, a client’s office and even in very busy conference locations. And the even better news is that it is easy to implement.

Step 1 – Use Windows 10 Anniversary Edition

You should use Windows 10 Anniversary Edition on the computer you want to stream the Mixed Reality Capture to. (you can just use this computer or a projector to share the results with a larger audience). The reason is that for Step 3, you need a new AE feature.

Step 2 – Enable Mobile Hotspot on the Computer

This is a new feature in the Windows 10 Anniversary Edition. You can access it from Settings / Network & Internet, and Mobile Hotspot:

SharedImage.pngAs you can see, the computer has to be connected to the Internet (and has to have a Wi-Fi adapter) for the Mobile hotspot to be enabled. If you can’t see the above warning, all is OK – turn on the “Share my Internet connection with other devices” checkbox.

Step 3 – Connect the HoloLens to the Mobile Hotspot on your Computer

And here comes the trick: once the hotspot is set up, you need to connect your HoloLens directly to the hotspot:

SharedImage 3.png
The HoloLens is connected to the mobile hotspot on the computer

Now air-tap on the Advanced options link, and take a note of the IP address of the HoloLens. You will need it soon.

Step 4 – the HoloLens Windows App

For the best streaming results, forget the device portal. What you need is the Microsoft HoloLens app from the Windows Store. This app has most (but not all) of the features of the Device Portal, and seems to perform much better when it comes to live streaming the Mixed Reality Capture.

Once you have the app installed, click on the + button, and add your HoloLens to the form. This is where you need the IP address of the HoloLens you – hopefully – recorded earlier:

 

SharedImage 4.png
The Add your HoloLens screen of the HoloLens UWP app

 

You should now see the HoloLens you just added, and hopefully it is online. If not, make sure that the HoloLens is still turned on. You may need to wait  few seconds before the HoloLens app can detect the device.

 

SharedImage 5.png
Select the HoloLens you want to connect to.

Click on the connected HoloLens, and

Step 5 – Enjoy!

Click on the first icon called “Live Stream”, and Live Stream should start. To further reduce latency (and not to cause audio issues), you may want to turn off the audio in the … menu.

Here is the result:

 

much better.gif
Much better!

As you can see, the latency is well below 1s, in fact, about 0.5s! This is more than satisfactory for any kind of live demo, or helping a first-time HoloLens user.

In fact, depending on how noisy the airwaves around you are, you can even switch it to High Quality mode from the default “Balanced” setting.

This solution is still not foolproof. The connection is wireless, and packet losses occur that can add more and more delays to the stream. If this happens, just navigate away from the live stream and quickly come back, and you’re good as new.

nexus2cee_50190-have-you-tried-turning-it-off-hfqe_thumb

To sum up:

  1. Use Windows 10 Anniversary Edition
  2. Turn on Mobile Hotspot
  3. Connect your HoloLens to the Mobile Hotspot on your computer and make sure you know the IP address
  4. Install the Microsoft HoloLens app on your computer, and connect it to your HoloLens
  5. Use “Live Stream” and tune it as you like.

 

Enjoy demoing HoloLens!

 

 

Advertisements

9 thoughts on “HoloLens Mixed Reality Streaming Done Right

  1. As of late, virtual and augmented reality devices have become the new tools of the future. For years classic movies like The Matrix, Tron, Gamer, and even Total Recall have left viewers with the yearning to escape or just experience another world. In fact Nintendo created one of the fist attempts for this reality with the virtual boy which was a little before its time and was not appreciated. As of today in the gaming world we have quite a few devices for virtual reality Oculus Rift, HTC Vive, Playstation VR and a few other. These devices aren’t as immersive as the movies aforementioned but users still have the chance of loosing themselves in these realms to an extent. Even with this small success i fear even today’s version of virtual reality video games will not have the longevity true fans hope for.

    Microsoft has taken a different approach to virtual reality with this augmented reality bringing the false world to you instead of taking you to a false world. I suspect this new reality will have the longevity VR dose not because people can use during there everyday lives. Anyone with a single creative bone in there body will get the full experience out of the Hololense device. The biggest reason this device will surpass the others is because it doesn’t immerse the user in virtual reality taking them away from everyday responsibilities in truth it enhances the physical world making everyday activities, duties, and pass time more achievable and tolerable. My only worry is the $3,000 and $5,000 price tag if the device stays at such a ridiculous price regular consumers will be less likely to purchase. For Hololense to succeed the price must be deducted tremendously.

    Liked by 1 person

  2. You saved my day! Had to do a presentation of our brand new HoloLens and the stream kept stopping all the time, until I found this article and everything went perfect! Thanks a ton!

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s