The New XR SDK Pipeline with HoloLens 2: A Walkthrough

The HoloLens 2 toolchain is under continuous development. In addition, the development of the Mixed Reality Toolkit (MRTK), the developer kit for HoloLens, must be synced with the continuous development of the Unity Game Engine. For these reasons, it is necessary for developers to be careful about using the correct versions of each tool in order to avoid complications.

HoloLens2

As of February, 2021, I’m  doing HoloLens development with Visual Studio 2019 v16.5.5, Unity Hub 2.4.2, Unity 2019.4.20f, Mixed Reality Feature Tool 1.0.2102 beta,  MRTK 2.5.4 + MRTK 2.6.0-preview, and Windows 10 Pro 10.0.18363.

MRTK, MR Feature Tool beta,  and the new Unity XR SDK pipeline

The most accurate documentation on getting started with HoloLens 2 development, at this time, is found in the MRTK github repository and on Microsoft’s MR Feature Tool landing page. There are nevertheless some gaps between these two docs that it would be helpful to fill in.

The new Unity XR SDK pipeline is a universal plugin framework that replaces something now known as the legacy Unity XR pipeline. Because of all the moving parts, many people have trouble getting the new pipeline working correctly, especially if they have not kept up with the changes for a while or if they are new to the HoloLens 2.

You will want to download Unity Hub if you don’t have it already.  Unity Hub will help you manage multiple versions of Unity on your development machine. It is fairly common to switch between different versions of Unity if you are working in VR and Mixed Reality as you go back and forth between older versions for maintenance and pursue newer versions for the latest features. As a rule, never upgrade the Unity version of an existing project if things are working well enough.

Create a new unity project

Use the Unity Hub Installs tab to get the latest version of Unity 2019.4, which you will need in order to successfully work with the latest MRTK. Later versions of Unity will not currently work for developing HoloLens 2 applications with the MRTK 2.5.4. Versions earlier than Unity 2018.4 also will not work well.

Some of the documentation mentions using Unity 2020.2 with OpenXR. This is something totally different. Just ignore it.

new_file

Start by creating a new Unity 2019.4 project.

new_2019

When you do this from Unity Hub, you can use the pull down menu to select from any of the versions installed on your development computer.

package_mgr

When your Unity app has been created, open the Unity Package Manager from the Windows menu.

xr_plugin

In Package Manager, select the Windows XR Plugin in the left panel. Then click the Install button in the lower left corner of the main panel.

xr_legacy_etc

This will also install the following components automatically: XR Interaction Subsystems, XR Legacy Input Helpers, and XR Plugin Management.

*** Notice that the component called Windows Mixed Reality is not installed. Don’t touch it. This is a left over from the legacy Unity XR pipeline and will eventually be removed from Unity. You will use a different method to get the MRTK into your project. ***

Configure project settings for HoloLens 2

You should now edit your project settings. Open the project settings panel by selecting Edit | Project Settings in the menu bar.

project_settings1

1. Select XR Plug-in Management in the left-hand pane and then, in the main window, check off Windows Mixed Reality to select it as your plug-in provider. This lets the new XR SDK pipeline work with your MRTK libraries (which you will import in the next section).

buffer16

2. Under XR Plug-in Management, select the Windows Mixed Reality entry in the left-hand pane. Make sure Depth Buffer 16 Bit is selected as your Depth Buffer Format and that Shared Depth Buffer is checked off. This will improve the performance of your spatial computing application.

project_settings_qual

3. Select Quality in the left-hand pane. To improve performance, click the down arrow under the Windows logo to set your default quality setting to Low for your spatial computing application.

tmpro

4. While you are configuring your project settings, you might as well also import TextMesh Pro. Select TextMesh Pro in the left-hand pane and click on the Import TMP Essentials button in the main window. TMP will be useful for drawing text objects in your spatial computing application.

player_packagename

5. Select Player  in the right-hand pane to edit your player settings. Change the Package name entry to something relevant for your project. (The project name is how the HoloLens identifies your application. If you are quickly prototyping and deploying projects and forget to change the package name, you will get an obscure message about saying your Template3D package is already installed. This is just the default package name on all new Unity projects.)

You are now ready to import the Mixed Reality Toolkit.

Retrieve MRTK components with MR Feature Tool

The HoloLens team has created a new tool called the Mixed Reality Feature Tool for Unity to help you acquire and deploy the correct version of the MRTK to your project.

feature_tool

1. After downloading the feature tool, you can go into settings and check off the Include preview releases box in order to get the 2.6.0-preview.20210220.4 build of MRTK. Alternatively, you can use MRTK version 2.5.4 if you are uncomfortable with using a preview build.

features

2. Follow the wizard steps to select and download the MRTK features you are interested in. At a minimum, I’d recommend selecting the Toolkit Foundation, Toolkit Extensions, Toolkit Tools, Toolkit Standard Assets, and Toolkit Examples.

fature_path

3. On the Import features screen, select the path to your Unity project. Validate that everything is correct before importing the selected features into your Unity HoloLens 2 project.

4. Click on the Import button.

Configure build settings for HoloLens 2

As the MRTK libraries are added to your project, your Unity project may hang while it is being refreshed if you still have it open (which is very okay).

add_scene

1. When the refresh is done, the Mixed Reality Toolkit item will appear your Unity menu bar.

apply

2. At the same time, an MRTK configuration window will also pop up in the Unity IDE. Accept the suggested project setting changes by selecting Apply.

switch_settings

3. Click on File | Build Settings… to open the Build Settings window. Select Universal Windows Platform in the left pane.

switch

4. Upon selecting Universal Windows Platform as your build target, you will be prompted to switch platforms. Click on the Switch Platform button to confirm that you will be building an application for the UWP platform. This will initiate a series of updates to the project that may freeze the IDE as the project is refreshed.

switch_uwp

5. After your switch to the Universal Windows Platform, MRTK may prompt you to make additional changes to your project. Apply the recommended changes.

switch_settings

6. For a release deployment, you will want the following build settings:

    • Target Device should be HoloLens.
    • Architecture is ARM64. This is the processor used in the HoloLens 2.
    • Build Type is D3D Project. Any other build type will project a standard 2D UWP window into the HoloLens.
    • Build configuration is Master for a release build, rather than a Release, as odd as that seems. This is the most lightweight build you can deploy to your device, and consequently the best performing.
    • Compression Method should be LZ4HC for release builds. This takes a lot longer to compile, but is the most efficient in a deployed application.

You can close the Build Settings window.

Summary

This walkthrough is intended to get you through the initial steps for integrating the new Unity XR SDK pipeline with the Mixed Reality Toolkit. For completeness, I will walk you through setting up and deploying a HoloLens scene in the next post.

When GameStop Killed XBox One Kinect

producers

If you look up the Xbox One Kinect (informally known as the Kinect 2) on the GameStop website, you’ll read in the product description that “[t]he best Xbox One experience is with Kinect.”

Over the course of the Xbox One’s life, there were approximately 38 games that supported Kinect body tracking. None of them were triple-A games. This is out of 2682 games for the Xbox One. While Microsoft initially planned to require that the Kinect be always on, by the time of the Xbox One’s release on November 2013, this requirement was removed.  By the summer of 2014, Microsoft unbundled the Kinect from their game console, allowing people to purchase the Xbox One at a lower price point that was more competitive with PlayStation 4. The final blow came in late 2015, when Microsoft removed their Kinect support for navigating the Xbox dashboard.

Before going into some theories on what happened to the Kinect, I wanted to give my “they’re all dirty” metaphor for the recent rise and fall of the GameStop stock price. The weak GameStop business was being shorted by hedge funds. Small investors gathered on Reddit decided to fight this by pumping money into GameStop stocks in order to inflate the price artificially. They typically used the app Robinhood, which doesn’t charge trading fees, to do this. In the end, the hedge funds appear to have hedged their best, because even as they lost money on their shorts, they made money by fulfilling the trades coming through Robinhood from these reddit investors.

Isn’t this the plot of Mel Brooks’ The Producers?  While the purpose of the stock market is supposed to be efficiently moving investor money into the hands of companies in order to create value, short-selling is a speculative financial instrument to allow people to bet that certain companies will fail.  Like Leo Bloom, hedge funds like Melvin Capital and Citadel recognized that sometimes you can make more money with a failed venture than with a successful one.

In order to improve the odds of failure, Leo Bloom and Max Bialystock stack the deck by finding the worst script, the worst director and the worst cast for their Broadway show. Similarly, in order to improve the odds of driving down the price of GameStop stock, Citadel let people know that they were shorting the stock. Who would invest in a company that Wall Street big guns were trying to destroy?

The problem for The Producers is that the worst play, Springtime for Hitler, the worst director (who turned it into a Busby Berkeley style musical), and the worst cast (drugged addled hippies), come together to create something that people can enjoy ironically. The play is so bad, it is good.

producers_aud

The worst director and worst cast in the GameStop saga are the Robinhood app and the reddit community /wallstreetbets. Robinhood allows (and encourages) inexperienced investors to bet against Wall Street professionals, which is about as successful as betting against the house in Las Vegas. /wallstreetbets, in turn, allows users to try out betting systems. The latest one depends on treating the stock market ironically, assuming that investment is primarily about manipulating markets rather than finding good companies to invest in. The only difference between /wallstreetbets and the hedge funds, is that one is made up of market outsiders and the other by insiders. Late capitalism. Post-truth investment.

There was a time when GameStop wasn’t just a carcass being fought over by carrion feeders looking for a quick meal. In 2013, GameSpot was a quickly growing company that made its money reselling second-hand console game disks.

In the lead up to the release of the XBox One, it turns out that Microsoft was attempting to kill this aftermarket. Even into the middle of 2013, Microsoft was considering dropping the optical drive from its hardware altogether and making the purchase of games completely cloud-based, like Steam.

ps4vid

It is clear from the confusion around the May, 2013 Xbox One reveal that this idea had lingering ramifications for the strategy around connectivity. Two requirements for a digital only game distribution system are a need for all consoles to be online, at least part of the time, and complex digital licensing verification systems. It turned out that the aftermarket in video games, brokered through third-parties like GameStop, was a much bigger deal than Microsoft realized and their inability to explain how people would be able to exchange and sell used games inspired one of the great marketing trolls of all time, when Sony created a commercial demonstrating how to exchange PlayStation games.

Today any teenager can explain to you the market forces that are destroying GameStop’s business model. There is no need for a company to provide an aftermarket for video games when no one uses disks anymore. Everything is digital in 2021 and everything is online. Almost like an act of revenge for 2013, Microsoft is even strong arming its Microsoft Gold subscribers to upgrade to the Xbox Game Pass by raising prices for the former. Xbox Game Pass allows users to have access to a broad range of games without having to buy those games individually, including the top games from the past two to three years.

Microsoft was ahead of its time in 2013. But what made it want to get rid of disks? One theory is that without a disk drive, Microsoft would have been able to drop the launch price of its console by $50. As it turned out, the Xbox with a disk drive and bundled with an Xbox Kinect, brought the initial price of an Xbox One to $499. The Sony PlayStation 4 launched at a $399 price point.

kinect-for-windows

This one hundred dollar difference turned out to be nearly fatal for the Xbox, which was forced to unbundle the Kinect 2 from its Xbox One by the middle of 2014, finally making their console competitive on price with the PlayStation. It was even able to undercut the price of the PlayStation by selling an unbundled Xbox One for $349 shortly after. This suggests that without an optical drive, the Xbox might have sold for only $50 more than the PlayStation 4, or even for the same price, at launch, while including a key differentiator with the Kinect.

Why did Microsoft insist on bundling the Kinect with the Xbox One in the first place? The problem for Microsoft was that in order to make the Kinect successful, it needed triple-A game companies to create games that used it. But this entails extra design and development costs for game companies. There is no way they would take on this additional cost without a guarantee of a user base that owned Kinect devices. There was a virtuous circle – or perhaps a vicious one – in which game makers need players with Kinects before they will create games for the Kinect, while console buyers need to be shown games that highlight the Kinect before they will buy a console that requires them to buy a Kinect.  In the end, neither of these things happened.

There was an underlying reason that Microsoft wanted to get Kinects into consumer living rooms. While the Kinect’s primary feature is its body tracking, which could be used as a controller for playing games and navigating screens, it’s secondary feature is a directional microphone plugged into Microsoft’s cutting edge speech recognition. It could have become an essential interface between consumers and the commercial internet, with Microsoft as the essential broker for these transactions and interactions.

Echo

As usual Microsoft was ahead of its time, and even as it quickly killed the Kinect in 2014, Amazon was releasing its own natural language devices built around Alexa, which soon expanded into a tool for not only accessing data on the internet, but also for integrating with services and controlling home devices.

But alas, GameStop created an aftermarket for game disks, that prevented Microsoft from getting rid of its Xbox One optical drive, that caused the Xbox One to lose on price to the PlayStation 4, that caused the XBox to drop the Kinect, that caused Microsoft to cede the living room device market to Amazon.