The HoloLens Toolchain and XAML Grids

occlusion2occlusion

What tech stack will be used to develop applications for HoloLens, Microsoft’s innovative new augmented reality platform?

Keeping in mind that Alex Kipman is the visionary behind both HoloLens and the Kinect sensor, some clues can be gleaned from the current Kinect SDK. The Kinect SDK initially supported development in WPF and C++ with DirectX. Over time, however, and in line with Microsoft’s internal shift to be more open and embrace the tech stacks of non-Microsoft communities, the Kinect SDK has grown to include support for Unity3D, Cinder, OpenFrameworks, OpenCV and even MATLAB.

Legos vs Play-Doh

lego_vs_playdoh

This is a two-pronged approach to tooling that Microsoft luminary Rick Barraza has famously characterized as the distinction between building with legos and sculpting with play-doh.  Legos represent what Microsoft has traditionally been extremely good at: creating reusable components. Reusable components abstract the underlying technology layers so even beginning developers can accomplish difficult tasks without needing to understand the intricacies of networking, graphics cards, or memory management. As long as all you want to do is the 95% of tasks that Microsoft components support, legos may be all that you ever need. Microsoft has almost single-handedly created the modern enterprise software developer community based on component building, drag-and-drop IDEs, and copy /paste.

But what if you are not interested in building applications that look like everyone else’s? What if you want to do the 5% of things that reusable components do not allow you to do. This has typically been difficult. Not only does it require having a fine grain understanding of the underlying operating system but also a desire to hack around Microsoft’s safeguards. Because Microsoft had for so long embraced the component model of software development, it internally saw its role as one of safeguarding applications from attempts to follow any other model of software development. API classes are typically sealed to prevent extension and if one were to ask to have them unsealed, the inevitable reply was always “what would you use that for”?

Now lets consider a more playful software world in which it makes sense for classes to be unsealed by default so we can just start squishing them around in our hands to see what comes of it. This is software as exploration and in fact there is a whole community, frequently styled “creative coders”, who work in this way. The processing software programming platform created by Casey Reas and Ben Fry is the epitome of this movement. It was originally created as a better way to teach computer programming since it is built around drawing shapes rather than displaying text. From this simple and divergent starting point, all “hello world” applications are dramatically different.  Even more profound, however, it became clear that through simple loops and seed values, vastly different effects could be generated using only a few lines of code. Playing with processing feels like sculpting with play-doh. Other homologous tool chains were eventually created that shared processing’s emphasis on the visual rather than the textual: OpenFrameworks, Arduino, Cinder and Unity3D.

Which is a long-about way of saying that for premium experiences, these creative coding tool chains will likely be the tools of choice. If you want to get a jump start on HoloLens development, go learn these platforms:

 

HologramFramwork APIs

 hologramAPIs

I mentioned above that Microsoft has a two-pronged approach to tooling software developers. So far I have only mentioned the play-doh side. As we come closer to the Windows 10 release, however, there has been increased activity in the long dormant WPF platform team – enough to suggest that some sort of Holographic support might be released with a new version of WPF. WPF, after all, is Microsoft’s premier platform for component-based development. If the marketing ideal for HoloLens is to make as many HoloLens supported applications as possible straight out of the gate (as it should be), then an easy to use platform for building new HoloLens applications as well as porting old ones to the new paradigm is an obvious pre-requisite.

The image above is from enterprising colleagues who inspected the Windows 10 symbol packages to find out what kind of holographic support would be natively built into the new OS.  The initial impression is that low-hanging integration will be possible by using some sort of texture mapping model. For example, Silverlight provided a component called the VideoBrush that allowed any control that supported brushes to use a video rather than a solid or gradient texture as a background image. This included even complex 3D shapes or skewed geometries.

HoloLens Grids

occlusion4

To me, this suggests a grid-based programming model for quickly painting applications as textures onto valid surfaces identified by the HoloLens’s built-in depth sensors. The depth sensors will use computer vision algorithms to identify and tag surfaces in a room that can be used to project digital content. The user’s movements and any desirable digital-realistic skewing will be taken care of by the underlying holographic framework. For now, let’s assume that interactions will also be taken care of automatically and will follow a mouse-like hover/press idiom for convenience.

 simplegrid

XAML-based languages like WPF have a unique layout component called a Grid. Unlike tables in html, XAML grids define the layout and the content separately. The layout is specified in ColumnDefinitions and RowDefinitions – for instance one may specify a grid that is 3 X 3, or 2 X 1 (as above), and so on. Content is written out (or dragged) below the column and row definitions. Their placement in the layout is then defined by attaching positional directives on the content as shown below.

 grid2

In this code snippet, I haven’t defined a row so there is just one row by default. I have defined two columns which are a zero based array. Finally, I’ve specified on the green panel that I want it to be positioned in the first column by writing Grid.Column=”0” . The red panel, in turn, is placed in column 1, the second column in the series. The resulting WPF application is shown below:

grid3

In this case, the application is not particularly impressive. We can imagine the same code being written against the Holographic Framework with the following results, however:

occlusion3

And here is what the code for your first “Hello, World” application might look like:

holoworld

occlusion5

It is, I hope, not difficult to see how we can then go from a standard XAML Grid like the one above to a HoloLens-enabled Grid like the one below:

occlusion

At this point, given the dearth of information currently available (I’m dumpster diving through Windows 10 symbol packages, after all) this is obviously just a wild guess. I believe it is a plausible programming model, however, and would provide a royal road to quickly generate applications for Microsoft’s newest and brightest technology innovation.

The Coming Holo Wars and How to Survive Them

 

this is the way the RL world ends

cloud atlas: new seoul

We are the holo men,

We are the stuffed men.

Leaning together

Headpiece filled with straw. Alas!

Our dried voices, when

We whisper together,

Are quiet and meaningless

As wind in dry grass

Or rat’s feet over broken glass

In our dry cellar.

— T. S. Eliot

 

“Disruptive technology” is one of the most over-used phrases in contemporary marketing hyper-speech. Borrowing liberally from previous generations’ research into the nature of political and scientific revolutions (Leon Trotsky, Georges Sorel, Thomas Kuhn), self-promoting second raters have pillaged the libraries of these scholars of disruption and have co-opted their intellects in the service of filling the world with useless gadgets and vaporware. When everything is a disruptive technology, nothing is.

Just as Sorel drew on historical examples of general strikes to form his narrative of idealized proletarian revolution and Kuhn identified three examples of scientific revolution: the transition from the Ptolemaic to the Copernican model of the solar system, the abandoning of phlogiston theory, and the shift from Newtonian to relativistic physics – to distill his theory of the “paradigm shift”, we can similarly take one step back in order to find the treasure hidden in the morass of marketing opportunism.

There have been three* major shakeups in the tech sector over the past several decades; each one was marked by the invocation of the “war” metaphor, the leveraging of large sums of money and massive shifts in the fortunes of well known companies.

The PC Wars – the commoditization of the personal computer in the 80s led to the diminishing of IBM and a surprising victor, Microsoft, which realized that the key to winning the PC Wars lay not with the hardware but with the operating system that made the hardware accessible. Following that model, the mid- to late-90s saw the rise of the Internet, various attempts to create portal solutions, and a pitched battle between Netscape and Microsoft to produce the dominant browser. 

The Browser Wars – the Browser Wars saw the rise and fall of companies like Yahoo! and AOL and the eventual victor turned out not be the best browser but the best search engine: Google. More recently we’ve been going through the Mobile Wars in which Apple has been the clear winner – but also Amazon, Twitter and Facebook.

The Mobile Wars – covering both the rise of smart phones as well as tablet devices, the Mobile Wars have born fruit in the way we view consumer experiences, have shifted software development from desktop to web development, have made JavaScript a first class language, have made responsive design the de facto standard, have made the freelance creative designer the Renaissance person of the 21st century, and perhaps most important have accelerated geolocation technology. Geolocation, as will be shown below, is a key player in the next technology war.

 

between the idea and the reality

 

jupiter ascending

Shape without form, shade without color,

Paralyzed force, gesture without motion;

 

As a devotee of Adam Sandler movies, I was pleased to see him teamed with Judd Apatow and Seth Rogan in 2009’s Funny Men. Adam Sandler movies are up there with “Pretty Woman” and “Dumb and Dumber” in the cable industry as movies that can be shown at any time of day and still be guaranteed to draw viewers. There is a false moment in the middle of the movie, however, in which Adam Sandler and Seth Rogan are flown out to perform at a private party for MySpace. What’s MySpace you ask? It was a social network that was crushed in the dust by Facebook, of which you have probably heard, along with other even more obscure networks like Friendster and Bebo. MySpace are portrayed in the movie as an up-and-rising social network through a last-gasp cross-marketing placement with Universal Studios.

A major characteristic of today’s tech wars is that we do not remember the losers. It does not even matter how big these corporations were during their period of being winners. Once they are gone, it is as if they are completely erased from the timeline, their reputations liquidated in the same fashion as their Aeron chairs and stock options.

To be a winner in the tech wars is to be a survivor of the tech wars. This applies not just to corporations but also to the marketing, business and technical people who are carried in the wake of rising and falling technology trends. IT groups across the US now face the problem of trends they have ignored finally reaching the C-levels as they are being asked about their mobile strategies and why their applications are not designed to be responsive – and perhaps even whey they continue to be written in vb6 or delphi.

These casualties of the Mobile Wars must be wondering what choices they could have made differently over the past several years and what choices they should be making over the next. How does one survive the conflict that comes after the Mobile Wars?

 

between the motion and the act

 

2001 a space odyssey

Those who have crossed

With direct eyes, to death’s other kingdom

Remember us — if at all — not as lost

Violent souls, but only

As the holo men,

The stuffed men.

 

Surviving and even thriving in the coming Holo Wars is possible if you keep an eye out for the contours of future history – if you know what is coming. The first key is knowing who the major players are: Microsoft, Facebook, Google – though there is no guarantee any of them will still be standing when the Holo Wars are over.

Microsoft has catapulted to the front of the Holo Wars with its announcement of the HoloLens on January 21st. HoloLens is the brainchild of Alex Kipman, who also spearheaded the product development of the Kinect. It is expected to be built on some of the technology developed for the Kinect v2 sensor combined with new holographic display technology – possibly involving eye movement tracking – that has yet to be revealed.

Facebook became a participant in the Holo Wars when it bought Palmer Luckey’s company Oculus VR in mid-2014. The Oculus Rift, a virtual reality headset, is basically two mobile display screens placed in front of a user’s eyeballs in order to show stereoscopic digital visualizations. The key to this technology is John Cormack’s ingenious use of sensors to track and anticipate head movements to rotate and skew images in a realistic way in the virtual world revealed by the Rift.

Google participates in several ways. Even though the explorer program is now closed, Google Glass arrived with great fanfare and created excitement around the fashion and consumer uses of this heads-up display technology. Following Google’s major investment in Rony Abovitz’s Magic Leap in October 2014, a maker of mysterious augmented reality technology, it now appears that this is the more likely future direction of Google Glass or whatever it is eventually called. Magic Leap, in turn, has added some amazing names to its payroll including Gary Bradski of OpenCV fame and Neal Stephenson, the author of Snow Crash. The third leg of Google’s investment in a holographic future is the expertise in geolocation it has acquired over the past decade.

The next key to surviving the Holo Wars is to understand what skills will be needed when the fighting starts. The first skill is a deeper knowledge of computer graphics. Since the rise of the graphical user interface, software development platforms have increasingly abstracted away the details of generating pixels and managing human-computer interactions. Future demands for spatially aware pixels will force developers to relearn basic mathematical concepts, linear algebra, trigonometry and matrix math.

In addition to mathematics, machine learning will be important as a way of making overwhelming amounts of data manageable. Modern computer interactions are relatively simple. Users sit in one place, in a fixed position respective to the machine, and rarely deviate from this position. Input is passed through transducers that reduce desire and intent into simple signals. Digital reality experiences, on the other hand, not only receive gestural information which must be interpreted but also physical orientation, world coordinates, facial expressions and speech commands. A basic knowledge of Bayesian probability and stochastic calculus will be part of the tool chest of anyone who wants to successfully navigate the Holo joblists of the future.

To reforge ourselves with skills for surviving the next seven years, designers must also become better programmers and software programmers must become more creative. The freelance creative, a job role that expanded dramatically during the Mobile Wars, will have an even brighter future in a world pervaded by augmented reality experiences. In order to make the shift, however, creatives will need to move beyond their comfort zone of creating PSDs in Photoshop and learn motion graphics as well as basic computer programming. Programmers likewise will need to move beyond the conceit that coding is an inherently creative activity; moving data around from point A to point B is no more creative than moving books around a sprawling Amazon warehouse and then packing them up for shipping is a poetic.

Real creative coding involves learning how to construct digital-to-physical experiences with Arduino, how to program self-generating visual algorithms with Processing, how to create 3D worlds in Unity and how to create complex visual interactions with openFrameworks and Cinder. These activities will become the common vocabulary of the future programmers of augmented experiences. Hiring managers and recruiters will expect to find them on resumes and without them, otherwise experienced tech workers be unhireable or worse, relegated to maintaining legacy web applications.

 

not with a bang but a whimper

 

enders game

The eyes are not here

There are no eyes here

In this valley of dying stars

In this holo valley

This broken jaw of our lost kingdoms

In this last of meeting places

We grope together

 

How can one tell if these prescriptions for the future Holo Wars are real and actionable or simply more marketing hype attempting to take advantage of people’s natural gullibility regarding technical gadgets? Aren’t we always being burned by overly optimistic portrayals of the future that never come to pass? Where are our flying cars? Where are our remote work locations?

In order for the Holo Wars to play out, certain milestones need to be achieved. Consequently, if you start seeing these milestones realized, you will know that you are in fact living through a fight over the next disruptive technology that will destroy some major tech corporations while affirming others at the apex of the tech world, one that will also reward those that have positioned themselves with useful skills for this future economy and punish those who do not. These milestones are: technology, monetization, persistent holographic objects, belief circles, overlapping dissociative realities.

Technology: the first phase is occurring now with the three major players discussed above and several additional players such as Metaio, Qualcomm and Samsung engaged in building up consumer augmented reality hardware and supporting technologies such as geolocation and gestural interfaces.

Monetization: innovation costs money. The initial hardware and infrastructure effort will likely be subsidized by the major players. Over time, the monetization model will likely follow what we see on the internet with “free” consumer experiences being subsidized by ads. There will be a struggle between premium subscription based experiences offering to remove the ads while providing better, higher resolution experiences with better content. These portal solutions will also contend against free and low-cost plug-in content provided by hackers and freelance creatives. How this plays out will depend largely on whether the premium content providers will be able to block out independents through standards and compatibility issues as well as whether hackers will find ways to overcome these roadblocks. There is also the possibility that some of the players might be looking at a much longer game and will foster an open AR content generation community rather than attempt to crush it. If the AR economy opens up in this way, a new service sector will grow made up of one set of people generating digital worlds for another set to live in.

Persistent Holographic Objects: virtual worlds are typically subjective experiences. They can be made inter-subjective, as they are in MMOs, by creating virtual topology in which people co-exist and co-operate. In augmented worlds, on the other hand, shared topology is an inherent feature. AR shared topology is called reality. In order to make AR worlds truly inter-subjective, rather than simply objective or subjective, shared holo objects must be part of the experience. Pesistent holo objects such as a digital fountain, a digital garden, or a digital work of art will have a set location and orientation in the world. AR players will need to travel to these locations physically in order to experience them. Unlike private AR or VR experiences in which each player views copies of the same digital object, with a shared experience each player can be said to be looking at the same persistent holo object from different points of view. In order to achieve persistent holographic objects, we will require finer grained geolocation than we currently have. AR gear must also be improved to become more usable in direct sunlight.

Belief Circles: a healthy indie creative fringe-economy and persistent holographic objects will make it possible to customize intersubjective experiences. People have a natural tendency to form cliques, parties and communities. Belief circles, a term coined by Vernor Vinge, will provide coherent community experiences for different guilds based on shared interests and shared aspirations. Users will opt in and out of various belief circles as they see fit. The same persistent holographic objects may appear differently to members of different circles and yet be recognized as sharing a common space and perhaps a common purpose. For instance, the holosign in front of the local Starbucks will have a permanent location and consistent semantic purpose, in AR space, but a polymorphic appearance. To paraphrase a truism, beauty will be in the eye of one’s belief circle.

Overlapping Dissociative Realities: divergent intersubjectivities will produce both a greater awareness of synchronicity – and a sense of deja vu as AR content is copied freely into multiple locations — as well as an increased sense of cognitive dissonance. Consider the example of going into Starbucks for coffee. The people waiting in line will likely each be members of varying belief circles and consequently will be having different experiences of the wait. This is not a large departure since we typically do not care about what other people in line are doing and even avoid paying attention unless they take too long making a selection. In this case, divergent belief circles make it easier to follow our natural instinct to avoid each other. Everyone in the holo valley is anonymous if they want to be. When one arrives at the head of the line, however, something more interesting happens. Even though the customer and the barista likely belong to different belief circles, they must interact, communicate, and perform an economic exchange; these two creatures from different worlds. What will that be like? Will one then lift a corner of the holo lenses in order to rub a sore eye only to discover that this isn’t a Starbucks at all but really a Dunkin’ Donuts which had silently bought out the other chain in a hostile takeover the previous week? Will your coffee taste any different if it looks exactly the same?

* 1996 was witness to a small skirmish between OpenGL and Direct3D that has subsequently come to be known as the API Wars. While the API Wars have had long lasting ripples, I don’t see them as having the tectonic effect of the other historical phenomena I am describing – plus anyways Thomas Kuhn only provides three major examples of his thesis and I wanted to stick to that particular design pattern.

[Much gratitude to Joel and Nate for collaborating on these scenarios over a highly entertaining lunch.]