Category Archives: Immersive MR

HoloLens and MR Device Dev Advisory Mar-2018

I’m currently doing HoloLens development with VS 2017 v15.6.4, Unity 2017.3.1p3, MRTK 2017.1.2, and W10 17128 (Insider Build).

Unity 2017.3.1p3, a patch release, includes the 2017.3.1p1 fix for hologram instability:

(993880) – XR: Fixed stabilization plane not getting set correctly via the SetFocusPointForFrame() API, resulting in poor hologram stabilization and color separation on HoloLens.

There continues to be uncertainty about whether this fixes all the stabilization problems or not – though it’s definitely better than it has been over the past several months.

UnityWebRequest continues to always return true for the isNetworkError property. Use isError or isHttpError instead. The older WWW class probably shouldn’t be used anymore. There are reports that media downloads aren’t working with UnityWebRequest while other file types are.

So if you have something working now, and have work-arounds in place, you probably shouldn’t upgrade. I know of HoloLens developers who are still very happy working in the older Unity 5.6.3.

April is shaping up to be very interesting. According to Unity, they will be releasing Unity 2018.1.0 then. For UWP/HoloLens developers, this means the addition of the .NET Standard 2.0 API compatibility level. .NET Standard 2.0 can be thought of as the set of APIs commonly supported by both .NET Core 2.0 (what UWP uses) and .NET Framework 4.6.1 (used for Windows apps and in the IDE). By supporting this, Unity 2018.1.0 should provide us with the ability to write much more common MonoBehaviour script code that works in both the IDE and on the HoloLens without using precompiler directives.

Of course, this is only useful if the HoloLens actually supports .NET Core 2.0, which is why the announcement of the RS4 Technical Preview is such a big deal. This is the first major firmware update for the HoloLens since release, and brings with it all the changes to Windows platform since the Anniversary Update (build 10.0.14393) which was also known as RS1 and which supports .NET Core 1.0.

Redstone 4 (build 10.0.17133), also known as the Spring Creators Update, is supposed to drop for PCs in mid-April. Which coincidentally is also when Unity 2018.1.0 is supposed to drop. So it would not be out of the question to expect a version of RS4 for HoloLens to drop at around the same time.

What sets RS4 for HoloLens apart from RS4 for Windows? For one thing, on the HoloLens we will have access to a new feature called Research Mode, providing access to low level sensor data such as the ToF depth camera and potentially the 4 mono cameras and the microphones. This in turn can be used to try out new algorithms beyond what the HoloLens already currently uses for data analysis.

On the UI front, the MR Design Labs interface tools have finally been integrated into the dev branch of the Mixed Reality Toolkit. Fingers crossed that this will make its way into the main branch in April also.

Finally, Magic Leap’s mixed reality headset, dubbed the Magic Leap One, had its debut at GDC this month. They also opened their creator portal to all developers, with links to documentation, the Lumin SDK, a special version of Unity 2018 to develop ML apps and a simulator to test gesture and controller interactions.

In the interest of full disclosure, I’ve been developing for the Magic Leap for a while under NDAs and inside a locked room ensorcelled by eldritch spells. It’s a great device and finally creates some good competition for the HoloLens team at Microsoft.

The first reaction among people working with the HoloLens and occluded MR devices may be to be defensive and tribal. Please resist this instinct.

A second, well-funded device like the Magic Leap One means all that much more marketing dollars from both Microsoft and Magic Leap spent on raising the profile of Mixed Reality (or Spatial Computing, as ML is calling it). It means healthy competition between the two device makers that will encourage both companies to improve their tech in efforts to grow and hold large swaths of the AR market. It also means a new device to which most of your spatial development skills will easily transfer. In other words, this is a good thing my MR homies. Embrace it.

And from the development side, there are lots of things to like about Magic Leap. Lumin is Linux/Mono based, which means a higher level of compatibility between the platform and pre-existing Unity assets from the Asset store. It also supports development in Unreal. Lastly, it also supports development on a Mac, potentially offering a way for crossover between the design, gaming and enterprise dev worlds. This in turn raises interest in high-end AR and will make people take a second look at HoloLens and the occluded MR devices.

It doesn’t take a weatherman to know it’s going to be a great summer for Mixed Reality / Spatial Computing developers.

HoloLens and MR Device Dev Advisory Jan-2018

I’ve come to accept that doing HoloLens and MR Device development means working with constant issues. As long as I can stay on top of what these issues are, I feel less like pulling out my hair. And I’m not just a member of the hair club for men – I also want to help you avoid hair loss with monthly updates.

I’m currently doing HoloLens development with VS 2017 v15.3.3, Unity 2017.2.0f3, MRTK 2017.1.2, and W10 17074 (Insider Build).

shared_buffer

This month saw the release of Unity 2017.3.0f3, which introduced a fix but also some new bugs. The fix is to the HoloLens stabilization issues introduced in December’s Unity 2017.2.1 release which caused holograms to be jittery. In Unity 2017.3.0f3, a new player setting in the editor called shared depth buffer fixes this. Just expand the Windows Mixed Reality node under XR Settings and check off Enable Depth Buffer Sharing. On the other hand, this seems to conflict with the stabilization logic in the MRTK, so you may see some jumpiness (but no jitteriness!) so you’ll want to remove that older logic from your code.

2017.3.0f3 also introduced problems with WWW (Unity’s older internet communication class). Basically, it doesn’t work when running in UWP anymore (though it does on other platforms and in the editor), so if your code or any assets you may be depending on for internet communication depend on WWW, you’ll have issues.

If you are using older stable builds of the MR Toolkit (up to 2017.1.2), you’ll start getting warnings and errors in 2017.2.0f03 and above about outdated APIs. Unity introduces API changes with each monthly release. The UnityEngine.VR.WSA namespace, for instance, is now UnityEngine.XR.WSA (sometimes the Unity editor will automatically fix this  for you when you migrate a project but often it doesn’t). In a couple of cases (like in the HandGuidance class) you’ll notice that the InteractionManager APIs have changed.

UnityEngine.XR.WSA.Input.InteractionManager.SourceLost += InteractionManager_SourceLost;
            UnityEngine.XR.WSA.Input.InteractionManager.SourceUpdated += InteractionManager_SourceUpdated;
            UnityEngine.XR.WSA.Input.InteractionManager.SourceReleased += InteractionManager_SourceReleased;

becomes:

UnityEngine.XR.WSA.Input.InteractionManager.InteractionSourceLost += InteractionManager_SourceLost;
            UnityEngine.XR.WSA.Input.InteractionManager.InteractionSourceUpdated += InteractionManager_SourceUpdated;
            UnityEngine.XR.WSA.Input.InteractionManager.InteractionSourceReleased += InteractionManager_SourceReleased;

The signature of the event handlers also change. Just follow Visual Studio Intellisense’s notes on how to fix the signatures.

HoloLens and MR Device Dev Advisory

indie_trap

 

[Update 10/17/17: I’m currently doing HoloLens development with VS 2017 v5.3.3, Unity 2017.1.2f1, MRTK 2017.1.2, and W10 17010. (This is one version of Windows 10 past what is recommended.) Unity 2017.2 for WMR development. There’s a new release of Visual Studio 2017 5.4 that I haven’t installed, yet. I am using Unity 2017.3.0 beta 5 for WebGL work.]

[Update 9/8/17: I’m currently having success in HoloLens development with Visual Studio 2017 (15.3.3), Unity 2017.1.0p5 (x64), SDK 14393, and HTK v1.2017.1.1 . This allows me to deploy Release builds, Unity C# Projects, and lets me debug remotely (my breakpoints are getting hit) from the regular File | Build Settings window. From the HoloToolkit Build and Deploy window, checking Unity C# Project throws an error. (To work around this, I am using two different deployment folders, one for debugging and one for non-debugging deployments.) I don’t think it matters, but I’m on Windows 10 insider build 16281. Will update if anything changes.]

[Update 9/7/17: Unity 2017.2.0b10 is definitely no go for HoloLens development. It just tries to run from the tile window and then goes black screen. Probably necessary for Immersive MR development, though.]

Sometimes it’s hard to know when to move forward and when to go backwards. If you are developing with Microsoft Mixed Reality products today (by which I mean on or around September 5th, 2017) you may need to do both.

Because common tools consisting of Visual Studio, Unity 3D and the open source Mixed Reality Toolkit (formerly the HoloToolkit) are used for both, and because we are at a point where multiple tools are being released at a rapid rate, conflicts are bound to happen, unfortunately.

For HoloLens development, it is best to roll back to the last working version of the tools you had. If you aren’t sure what those were, then I’d recommend using Visual Studio 2015 update 3, Unity 3D 5.6.3 patch 2 and HoloToolkit 1.5.8.0.

For immersive mixed reality, you want the latest of everything. As of today, that means Unity 2017.2.0b10 (and then b11 when that comes out). You will need Visual Studio 2017 update 3 (15.3) and finally you will need to be on the latest Windows Insider Build of Windows 10 from the fast ring (for instance build 16281, which has some very nice updates to the MR Portal).

These are the versions I could get working consistently. Other combinations, for HoloLens development, could potentially raise the following issues (though your mileage may vary – if so, please let me know in the comments):

  • unable to create Master builds, dev or release okay
  • apps deploy successfully but crash before the splash screen
  • unable to use remote debugging on your deployed app – get a dll missing error or “Please ensure that target device has developer mode enabled” error
  • IL weaving issues (this occurs for Visual Studio update 3 (15.3.3) and is fixed in patch 2017.1.0p5 as well as 5.6.3.0p2 (if you are using Visual Studio 2015 update 3)
  • Unable to create Unity C# Project builds (for debugging)

MR development with the wrong tool combination has raised some of these issues, such as the C# Project builds and IL weaving. In addition, there may be clipping issues if you don’t have the latest tools.

buildsetting

Note: You don’t really need Visual Studio 2017 for HoloLens development. Use VS 2015, instead. You only require VS 2017 if you are developing for Windows SDK 10.0.15063 or above. The HoloLens device, however, still runs on windows build 14393. When you export your solution from Unity, just be sure to change the UWP SDK property in Build Settings from its default value “Latest installed” to “10.0.14393.0”.

Note 2: You cannot currently publish your immersive MR app to the Windows Store. Just hold tight for now. It’s being worked out.

Quick recap: for now, you need different tool versions for HoloLens and Immersive MR Headset development. Use older tools for HoloLens but the latest and greatest for MR Headsets.

I will try to keep this updated as facts on the ground change.