Tech means never having to say you’re sorry

spatial

There have been a series of amazing turns of event in the mixed reality world lately. The big headliners for me are:

1) Magic Leap laid off about a thousand employees due to diminishing funds but then was able to get a lifeline of $350 million, which will save the jobs of the remaining 300-400 engineers. The creative teams and sales teams appear to have been gutted in the first round of layoffs, unfortunately.

2) Microsoft HoloLens announced general availability of the HoloLens 2 on the Microsoft Store starting in July. Also availability in more countries starting in the fall.

3) The Unity XR SDK is getting closer to shipping, or has already shipped but is only working well with some platforms for now? Obviously some things need to be ironed out, but this appears to be the future of cross-platform AR development.

4) Spatial.io, the cross-platform XR collaboration platform, has made its product free.

Along with these there have been a series of refreshingly honest video interviews with some of the central people in the current evolution of mixed reality that help to frame our understanding of what has been going on at Microsoft and at Magic Leap over the past five years.

The XR Talk podcast is always great (thanks Roland for introducing me to it). This meandering interview with Graeme Devine, post-Leap, is particularly fascinating. There’s a great story of how he delivered the blade Orcrist (or was it Glamdring?) to Neal Stephenson in order to tempt him to come work with Magic Leap.

This week also saw the hosting of MR Dev Days conference on altspaceVR, which was a fascinating and wonderfully international experience.  Big thanks to Jesse McCulloch and everyone else responsible for throwing it together. The highlight of the show was a very frank conversation between Rene Schulte and Alex Kipman which I can’t recommend enough.

kipman

During the fireside chat (and the keynote the previous day) Kipman acknowledged the drawn out distribution of the HoloLens 2 and thanked developers for their patience. He also discussed the bucket problem (I think that’s what it’s called?) in which losing a a bucket of credibility requires a lot of buckets to regain that same level of credibility (pretty sure I messed up that metaphor).

We’ve seen a lot of that in the MR world this month. The financial problems at Magic Leap will have put a lot of people off. The fact that all the laid off employees have refrained from criticizing the company in the aftermath has been  surprising and probably speaks well for the company culture.

Meanwhile in the HoloLens world, public and private message boards indicate a lot of frustration with the product team. Original messaging suggested the devices would be out in early 2019, but over a year later, individual devs still have problems getting devices.

Looked at objectively, it’s pretty clear that if the HoloLens team could have gotten more devices to indie devs they would have and that the delays were not intentional. But knowing that also doesn’t necessarily make the bad feelings go away, given the difference between knowing and feeling, and this  in turn may have a depressing effect on any excitement around the wider release in July and then in the fall.

For what it’s worth, I think an apology goes a long way, and a heartfelt, personalized acknowledgment of the people who might feel slighted will go the greatest way. The difficulty here is that in a corporate culture like Microsoft’s, acknowledgement of mistakes is as alien to the normal way of doing things as – well, to be honest – as it is in the Trump administration. The culture of the Trump administration, after all, comes out of common practice in the modern corporation.

This isn’t always the case, though, and I have two pieces of evidence. Microsoft is very good at giving out chachkas, and even sent out an impressive gift pack for their online Build conference. The two best things I ever got from Microsoft, though, were personalized notes.

The first is a note from the Ben Lower / Heather Mitchell days of the Kinect program. Somebody wrote this out by hand, providing both an acknowledgment of who I am and what I had done (and to be honest, I was surprised they even knew who I was at the time):

WIN_20200523_18_10_50_Pro

The next is a card from the early days of the HoloLens program (Venessa Arnauld / Aileen Mcgraw).

WIN_20200523_18_10_27_Pro

These are my two most prized possessions from Microsoft over the past decade. Friends and associates have similar mementos they memorialize at home. The lesson from these two examples, for me, is that in tech you don’t always have to say you are sorry. It is often good enough, and probably more meaningful,  to acknowledge the legitimate concerns, understandable feelings,  and obvious humanity of the people who make you successful.

That’s a lot of personalized messages, but also the sort of thing that can easily repair broken or damaged relationships with your developer community.

The Fork in Mixed Reality

Futuristic Spaceman - Programmer Reading Projected Information

Yogi Berra gnomically said, “when you come to a fork in the road, take it.”  On the evening of Friday, February 1st, 2019 at approximately 9 PM EST, that’s exactly what happened to Mixed Reality.

The Mixed Reality Toolkit open source project, which grew out of the earlier HoloLens Toolkit on github, was forked into the Microsoft MRTK and a cross-platform XRTK (read the announcement). While the MRTK will continue to target primarily Microsoft headsets like the HoloLens and WMR, XRTK will feature a common framework for HoloLens, Magic Leap, VR Headsets, Mobile AR – as well as HoloLens 2 and any other MR devices that eventually come on the market.

So why did this happen? The short of it is that open source projects can sometimes serve multiple divergent interests and sometimes they cannot. Microsoft was visionary in engineering and releasing the original HoloLens MR Headset. They made an equally profound and positive step back in 2016 by choosing to open source the developer SDK/Framework/Toolkit (your choice) that allows developers to build Unity apps for the HoloLens. This was the original HoloLens Toolkit (HLTK).

While the HLTK started as a primarily Microsoft engineering effort, members of the community quickly jumped in and began contributing more and more code to the point that the Microsoft contributions became a minority of overall contributions. This, it should be noted, goes against the common trend of a single company paying their own engineers to keep an open source project going. The HLTK was an open source success story.

In this regard, it is worth calling out two developers in particular, Stephen Hodgson and Simon Jackson, for the massive amounts of code and thought leadership they have contributed to the MR community. Unsung heroes barely captures what they have done.

In 2017 Microsoft started helping to build occluded WinMR (virtually the same as VR) devices with several hardware vendors and it made sense to create something that supported more than just the HoloLens. This is how the MRTK came to be. It served the same purpose as the HLTK, to accelerate development with Unity scripts and components, but now with a larger perspective about who should be served.

In turn, this gave birth to something that is generally known as MRTK vNext, an ambitious project to support not just Microsoft devices but also platforms from other vendors. And what’s even more amazing, this was again driven by the community rather than by Microsoft itself. Microsoft was truly embracing the open source mindset and not just paying lip service to it as many naysayers were claiming.

But as Magic Leap, the other major MR Headset vendor, finally released their product in fall, 2018, things began to change. Unlike Microsoft, Magic Leap developed their SDK in-house and threw massive resources at it. Meanwhile, Microsoft finally started throwing their engineers at the MRTK again after taking a long hiatus. This may have been in response to the Magic Leap announcement or equally could have been because the team was setting the stage for a HoloLens 2 announcement in early 2019.

And this was the genesis of the MR fork in the road: For Microsoft, it did not make sense to devote engineering dollars toward creating a platform that supported their competitors’ devices. In turn, it probably didn’t make sense for engineers from Google, Magic Leap, Apple, Amazon, Facebook, etc. to devote their time toward a project that was widely seen as  a vehicle for Microsoft HMDs.

And so a philosophical split needed to occur. It was necessary to fork MRTK vNext. The new XRTK (which is also pronounced “Mixed Reality Toolkit”) is a cross-platform framework for HoloLens as well as Magic Leap (Lumin SDK support is in fact already working in XRTK and is getting even more love over the weekend even as I write).

But XRTK will also be a platform that supports developing for Oculus Rift, Oculus Go, HTC Vive, Apple ARKit, Google ARCore, the new HoloLens 2 which may or may not be announced at MWC 2019, and whatever comes next in the  Mixed Reality Continuum.

So does this mean it is time to stick a fork in the Microsoft MRTK? Absolutely not. Microsoft’s MRTK will continue to do what people have long expected of it, supporting both HoloLens and Occluded WinMR devices (that is such a wicked mouthful — I hope someone will eventually give it a decent name like “Windows Surface Kinect for Azure Core DotNet Silverlight Services” or something similarly delightful).

In the meantime, while Microsoft is paying its engineers to work on the MRTK, XRTK needs fresh developers to help contribute. If you work for a player in the MR/VR/AR/XR space, please consider contributing to the project.

Or to word it in even stronger terms, if you give half a fork about the future of mixed reality, go check out  XRTK  and start making a difference today.

Magic Leap Store publishes first pay app

seedling

The Magic Leap store (aka Magic Leap “World”) has published its first app for $9.99. This is an app created by Insomniac but published by Magic Leap itself, so in some sense is a trial run for its store. “Seedling” already made its first appearance at the Leap conference in Los Angeles in October.

$9.99 is also an interesting price, perhaps signaling a target for apps on Magic Leap devices. Back in the day, $.99 was the target price for Apple Store apps. When Microsoft came out with Windows Phone, they marketed the idea that apps should sell for more than that on their platform (more towards $1.49 or $1.99). On Steam, the magic price point for games seems to be $20 to $60.

sketchup

For the HoloLens, which uses the online Windows Store as its distribution channel, the most frequent price point seems to be free. This makes sense since even with a purported 50K HoloLens devices currently in the world, the total market is still too small to support a reasonably priced game. Trimble initially went the other way with their SketchUp Viewer,  which lists for about $1.5K, apparently trying to recoup their investment with a high price tag. Their subsequent HoloLens offering, part of a collaboration service, is free.

In order to buy Seedling, I had to go into my online magic leap creator’s account and add a payment method. This is an interesting aspect of all current VR and AR devices: entering data is rarely – and entering financial data is never – done through the actual device. We still live in a world where one must switch to either a phone or a computer in order to establish the credentials that will be used through the device.

This is ultimately a pre-NUI UX problem involving the difficulty of doing data entry without a keyboard and mouse (though we are finally getting comfortable with doing this on our smart phones, thanks to the rising comfort level with using web apps on tiny screens). This will be an ongoing problem for developing apps for the enterprisy market, where the exchange of data is pretty key.

Who knows, maybe solving this UX dilemma for the enterprise will end up being the killer app we’ve all been waiting for. I wonder how much someone would charge for it?

Why Augmented Reality is harder than Virtual Reality

hololensx519_0

At first blush, it seems like augmented reality should be easier than virtual reality. Whereas virtual reality involves the generation of full stereoscopic digital environments as well as interactive objects to place in those environments, augmented reality is simply adding digital content to our view of the real world. Virtual reality would seem to be doing more heavy lifting.

perspective

In actual fact, both technologies are creating illusions to fool the human eye and the human brain. In this effort, virtual reality has an easier task because it can shut out points of reference that would otherwise belie the illusion. Augmented reality experiences, by contrast, must contend with real world visual cues that draw attention to the false nature of the mixed reality content being added to a user’s field of view.

In this post, I will cover some of the additional challenges that make augmented reality much more difficult to get right. In the process, I hope to also provide clues as to why augmented reality HMDs like HoloLens and Magic Leap are taking much longer to bring to market than AR devices like the Oculus Rift, HTC Vive and Sony Project Morpheus.

terminator vision

But first, it is necessary to distinguish between two different kinds of augmented reality experience. One is informatics based and is supported by most smart phones with cameras. The ideal example of this type of AR is the Terminator-vision from James Cameron’s 1984 film “The Terminator.” It is relatively easy to to do and is the most common kind of AR people encounter today.

star wars chess

The second, and more interesting, kind of AR requires inserting illusory 3D digital objects (rather than informatics) into the world. The battle chess game from 1977’s “Star Wars” epitomizes this second category of augmented reality experience. This is extremely difficult to do.

The Microsoft HoloLens and Magic Leap (as well as any possible HMDs Apple and others might be working on) are attempts to bring both the easy type and the hard type of AR experience to consumers.

Here are a few things that make this difficult to get right. We’ll put aside stereoscopy which has already been solved effectively in all the VR devices we will see coming out in early 2016.

cloaked predator

1. Occlusion The human brain is constantly picking up clues from the world in order to determine the relative positions of objects such as shading, relative size and perspective. Occlusion is one that is somewhat tricky to solve. Occlusion is an effect that is so obvious that it’s hard to realize it is a visual cue. When one body is in our line of sight and is positioned in front of another body, that other body is partially hidden from our view.

In the case where a real world object is in front of a digital object, we can clip the digital object with an outline of the object in front to prevent bleed through. When we try to create the illusion that a digital object is positioned in front of a real world object, however, we encounter a problem inherent to AR.

In a typical AR HMD we see the real world through a transparent screen upon which digital content is either projected or, alternatively, illuminated as with LED displays. An obvious characteristic of this is that digital objects on a transparent display are themselves semi-transparent. Getting around this issue would seem to require being able to make certain portions of the transparent display more opaque than others as needed in order to make sure our AR objects look substantial and not ghostly.

 citizen kane

2. Accommodation It turns out that stereoscopy is not the only way our eyes recognize distance. The image above is from a scene in Orson Welles’s “Citizen Kane” in which a technique called “deep focus” is used extensively. Deep focus maintains clarity in the frame whether the actors and props are in the foreground, background or middle ground. Nothing is out of focus. The technique is startling both because it is counter to the way movies are generally shot but also because it is counter to how our eyes work.

If you cover one eye and use the other to look at one of your fingers, then move the finger toward and away from you, you should notice yourself refocusing on the finger as it moves while other objects around the finger become blurry. The shape of the cornea actually becomes more rounded when objects are close in order to cause light to refract more in order to reach the retina. For further away objects, the cornea flattens out because less refraction is needed. As we become older, the ability to bow the cornea lessens and we lose some of our ability to focus on near objects – for instance when we read. In AR, we are attempting to make a digital object that is really only centimeters from our eyes appear to be much further away.

Depending on how the light from the display passes through the eye, we may end up with the digital object appearing clear while the real world objects supposedly next to it and at the same distance appear blurred.

vergence_accomodation

3. Vergence-Accommodation Mismatch The accommodation problem is one aspect of yet another VR/AR difficulty. The term vergence describes the convergence and divergence of the two eyes from one another as objects move closer or further away. An interesting aspect of stereoscopy – which is used both for virtual reality as well as augmented reality to create the illusion of depth – is that the distance at which the two eyes coordinate to see an object is generally different from the focal distance from the eyes to the display screen(s). This consequently sends two mismatched signals to the brain concerning how far away the digital object is supposed to be. Is it the focal length or the vergence length? Among other causes, vergence-accommodation mismatch is believed to be a contributing factor to VR sickness. Should the accommodation problem above be resolved for a given AR device, it is safe to assume that the vergence-accommodation mismatch will also be solved.

 4. Tetherless Battery Life Smart phones have changed our lives among other reasons because they are tetherless devices. While the current slate of VR devices all leverage powerful computers to which they are attached, since VR experiences are all currently somewhat stationary (the HTC Vive being the odd bird), AR needs to be portable. This naturally puts a strain on the battery, which needs to be relatively light since it will be attached to the head-mounted-display, but also long-lived as it will be powering occasionally intensive graphics, especially for games.

5. Tetherless GPU Another strain on the system is the capability of the GPU. Virtual reality devices can be fairly intense since they require the user to purchase a reasonably powerful and somewhat expensive graphics card. AR devices can be expected to have similar graphics requirements as VR with much less to work with since the GPU needs to be onboard. We can probably expect a streamlined graphics pipeline dedicated to and optimized for AR experiences will help offset lower GPU capabilities.

6. Applications Not even talking about killer apps, here. Just apps. Microsoft has released videos of several impressive demos including Minecraft for HoloLens. Magic Leap up to this point has only shown post-prod, heavily produced illustrative videos. The truth is that everyone is still trying to get their heads around designing for AR. There aren’t really any guidelines for how to do it or even what interactions will work. Other than the most trivial experiences (e.g. weather and clock widgets projected on a wall) this will take a while as we develop best practices while also learning from our mistakes.

Conclusion

With the exception of V-AM, these are all problems that VR does not have to deal with. Is it any wonder, then, that while we are being led to believe that consumer models of the Oculus Rift, HTC Vive and Sony Project Morpheus will come to market in the first quarter of 2016, news about HoloLens and Magic Leap has been much more muted. There is simply much more to get right before a general rollout. One can hope, however, that dev units will start going out soon from the major AR players in order to mitigate challenge #6 while further tuning continues, if needed, on challenges #1-#5.

The HoloCoder’s Bookshelf

WP_20150430_06_43_49_Pro

Professions are held together by touchstones such as as a common jargon that both excludes outsiders and reinforces the sense of inclusion among insiders based on mastery of the jargon. On this level, software development has managed to surpass more traditional practices such as medicine, law or business in its ability to generate new vocabulary and maintain a sense that those who lack competence in using the jargon simply lack competence. Perhaps it is part and parcel with new fields such as software development that even practitioners of the common jargon do not always understand each other or agree on what the terms of their profession mean. Stack Overflow, in many cases, serves merely as a giant professional dictionary in progress as developers argue over what they mean by de-coupling, separation of concerns, pragmatism, architecture, elegance, and code smell.

Cultures, unlike professions, are held together not only by jargon but also by shared ideas and philosophies that delineate what is important to the tribe and what is not. Between a profession and a culture, the members of a professional culture, in turn, share a common imaginative world that allows them to discuss shared concepts in the same way that other people might discuss their favorite TV shows.

This post is an experiment to see what the shared library of augmented reality and virtual reality developers might one day look like. Digital reality development is a profession that currently does not really exist but which is already being predicted to be a multi-billion dollar industry by 2020.

HoloCoding, in other words, is a profession that exists only virtually for now. As a profession, it will envelop concerns much greater than those considered by today’s software developers. Whereas contemporary software development is mostly about collecting data, reporting on data and moving data from point A to points B and C, spatial software development will be more concerned with environments and will have to draw on complex mathematics as well as design and experiential psychology. The bookshelf of a holocoder will look remarkably different from that of a modern data coder. Here are a few ideas regarding what I would expect to find on a future developer’s bookshelf in five to ten years.

 

1. Understanding Media by Marshall McLuhan – written in the 60’s and responsible for concepts such as ‘the global village’ and hot versus cool media, McLuhan pioneered the field of media theory.  Because AR and VR are essentially new media, this book is required reading for understanding how these technologies stand side-by-side with or perhaps will supplant older media.

2. Illuminations by Walter Benjamin – while the whole work is great, the essay ‘The Work of Art in the Age of Mechanical Reproduction’ is a must read for discussing how traditional notions about creativity fit into the modern world of print and now digital reproduction (which Benjamin did not even know about). It also deals at an advanced level with how human interactions work on stage versus film and the strange effect this creates.

3. Sketching User Experiences by Bill Buxton – this classic was quickly adopted by web designers when it came out. What is sometimes forgotten is that the book largely covers the design of products and not websites or print media – products like those that can be built with HoloLens, Magic Leap and Oculus Rift. Full of insights, Buxton helps his readers to see the importance of lived experience when we design and build technology.

4. Bergsonism by Gilles Deleuze – though Deleuze is probably most famous for his collaborations with Felix Guattari, this work on the philosophical meaning of the term ‘’virtual reality’, not as a technology but rather as a way of approaching the world, is a gem.

5. Passwords by Jean Baudrillard – what Deleuze does for virtual reality, Baudrillard does for other artifacts of technological language in order to show their place in our mental cosmology. He also discusses virtual reality along the way, though not as thoroughly.

6. Mathematics for 3D Game Programming and Computer Graphics by Eric Lengeyl – this is hardcore math. You will need this. You can buy it used online for about $6. Go do that now.

7. Linear Algebra and Matrix Theory by Robert Stoll – this is a really hard book. Read the Lengeyl before trying this. This book will hurt you, by the way. After struggling with a page of this book, some people end up buying the Manga Guide to Matrix Theory thinking that there is a fun way to learn matrix math. Unfortunately, there isn’t and they always come back to this one.

8. Phenomenology of Perception by Maurice Merleau-Ponty – when it first came out, this work was often seen as an imitation of Heiddeger’s Being and Time. It may be the case that it can only be truly appreciated today when it has become much clearer, thanks to years of psychological research, that the mind reconstructs not only the visual world for us but even the physical world and our perception of 3D spaces. Merleau-Ponty pointed this out decades ago and moreover provides a phenomenology of our physical relationship to the world around us that will become vitally important to anyone trying to understand what happens when more and more of our external world becomes digitized through virtual and augmented reality technologies.

9. Philosophers Explore the Matrix – just as The Matrix is essential viewing for anyone in this field, this collection of essays is essential reading. This is the best treatment available of a pop theme being explored by real philosophers – actually most of the top American philosophers working on theories of consciousness in the 90s. Did you ever think to yourself that The Matrix raised important questions about reality, identity and consciousness? These professional philosophers agree with you.

10. Snow Crash by Neal Stephenson – sometimes to understand a technology, we must extrapolate and imagine how that technology would affect society if it were culturally pervasive and physically ubiquitous. Fortunately Neal Stephenson did that for virtual reality in this amazing book that combines cultural history, computer theory and a fast paced adventure.

The Coming Holo Wars and How to Survive Them

 

this is the way the RL world ends

cloud atlas: new seoul

We are the holo men,

We are the stuffed men.

Leaning together

Headpiece filled with straw. Alas!

Our dried voices, when

We whisper together,

Are quiet and meaningless

As wind in dry grass

Or rat’s feet over broken glass

In our dry cellar.

— T. S. Eliot

 

“Disruptive technology” is one of the most over-used phrases in contemporary marketing hyper-speech. Borrowing liberally from previous generations’ research into the nature of political and scientific revolutions (Leon Trotsky, Georges Sorel, Thomas Kuhn), self-promoting second raters have pillaged the libraries of these scholars of disruption and have co-opted their intellects in the service of filling the world with useless gadgets and vaporware. When everything is a disruptive technology, nothing is.

Just as Sorel drew on historical examples of general strikes to form his narrative of idealized proletarian revolution and Kuhn identified three examples of scientific revolution: the transition from the Ptolemaic to the Copernican model of the solar system, the abandoning of phlogiston theory, and the shift from Newtonian to relativistic physics – to distill his theory of the “paradigm shift”, we can similarly take one step back in order to find the treasure hidden in the morass of marketing opportunism.

There have been three* major shakeups in the tech sector over the past several decades; each one was marked by the invocation of the “war” metaphor, the leveraging of large sums of money and massive shifts in the fortunes of well known companies.

The PC Wars – the commoditization of the personal computer in the 80s led to the diminishing of IBM and a surprising victor, Microsoft, which realized that the key to winning the PC Wars lay not with the hardware but with the operating system that made the hardware accessible. Following that model, the mid- to late-90s saw the rise of the Internet, various attempts to create portal solutions, and a pitched battle between Netscape and Microsoft to produce the dominant browser. 

The Browser Wars – the Browser Wars saw the rise and fall of companies like Yahoo! and AOL and the eventual victor turned out not be the best browser but the best search engine: Google. More recently we’ve been going through the Mobile Wars in which Apple has been the clear winner – but also Amazon, Twitter and Facebook.

The Mobile Wars – covering both the rise of smart phones as well as tablet devices, the Mobile Wars have born fruit in the way we view consumer experiences, have shifted software development from desktop to web development, have made JavaScript a first class language, have made responsive design the de facto standard, have made the freelance creative designer the Renaissance person of the 21st century, and perhaps most important have accelerated geolocation technology. Geolocation, as will be shown below, is a key player in the next technology war.

 

between the idea and the reality

 

jupiter ascending

Shape without form, shade without color,

Paralyzed force, gesture without motion;

 

As a devotee of Adam Sandler movies, I was pleased to see him teamed with Judd Apatow and Seth Rogan in 2009’s Funny Men. Adam Sandler movies are up there with “Pretty Woman” and “Dumb and Dumber” in the cable industry as movies that can be shown at any time of day and still be guaranteed to draw viewers. There is a false moment in the middle of the movie, however, in which Adam Sandler and Seth Rogan are flown out to perform at a private party for MySpace. What’s MySpace you ask? It was a social network that was crushed in the dust by Facebook, of which you have probably heard, along with other even more obscure networks like Friendster and Bebo. MySpace are portrayed in the movie as an up-and-rising social network through a last-gasp cross-marketing placement with Universal Studios.

A major characteristic of today’s tech wars is that we do not remember the losers. It does not even matter how big these corporations were during their period of being winners. Once they are gone, it is as if they are completely erased from the timeline, their reputations liquidated in the same fashion as their Aeron chairs and stock options.

To be a winner in the tech wars is to be a survivor of the tech wars. This applies not just to corporations but also to the marketing, business and technical people who are carried in the wake of rising and falling technology trends. IT groups across the US now face the problem of trends they have ignored finally reaching the C-levels as they are being asked about their mobile strategies and why their applications are not designed to be responsive – and perhaps even whey they continue to be written in vb6 or delphi.

These casualties of the Mobile Wars must be wondering what choices they could have made differently over the past several years and what choices they should be making over the next. How does one survive the conflict that comes after the Mobile Wars?

 

between the motion and the act

 

2001 a space odyssey

Those who have crossed

With direct eyes, to death’s other kingdom

Remember us — if at all — not as lost

Violent souls, but only

As the holo men,

The stuffed men.

 

Surviving and even thriving in the coming Holo Wars is possible if you keep an eye out for the contours of future history – if you know what is coming. The first key is knowing who the major players are: Microsoft, Facebook, Google – though there is no guarantee any of them will still be standing when the Holo Wars are over.

Microsoft has catapulted to the front of the Holo Wars with its announcement of the HoloLens on January 21st. HoloLens is the brainchild of Alex Kipman, who also spearheaded the product development of the Kinect. It is expected to be built on some of the technology developed for the Kinect v2 sensor combined with new holographic display technology – possibly involving eye movement tracking – that has yet to be revealed.

Facebook became a participant in the Holo Wars when it bought Palmer Luckey’s company Oculus VR in mid-2014. The Oculus Rift, a virtual reality headset, is basically two mobile display screens placed in front of a user’s eyeballs in order to show stereoscopic digital visualizations. The key to this technology is John Cormack’s ingenious use of sensors to track and anticipate head movements to rotate and skew images in a realistic way in the virtual world revealed by the Rift.

Google participates in several ways. Even though the explorer program is now closed, Google Glass arrived with great fanfare and created excitement around the fashion and consumer uses of this heads-up display technology. Following Google’s major investment in Rony Abovitz’s Magic Leap in October 2014, a maker of mysterious augmented reality technology, it now appears that this is the more likely future direction of Google Glass or whatever it is eventually called. Magic Leap, in turn, has added some amazing names to its payroll including Gary Bradski of OpenCV fame and Neal Stephenson, the author of Snow Crash. The third leg of Google’s investment in a holographic future is the expertise in geolocation it has acquired over the past decade.

The next key to surviving the Holo Wars is to understand what skills will be needed when the fighting starts. The first skill is a deeper knowledge of computer graphics. Since the rise of the graphical user interface, software development platforms have increasingly abstracted away the details of generating pixels and managing human-computer interactions. Future demands for spatially aware pixels will force developers to relearn basic mathematical concepts, linear algebra, trigonometry and matrix math.

In addition to mathematics, machine learning will be important as a way of making overwhelming amounts of data manageable. Modern computer interactions are relatively simple. Users sit in one place, in a fixed position respective to the machine, and rarely deviate from this position. Input is passed through transducers that reduce desire and intent into simple signals. Digital reality experiences, on the other hand, not only receive gestural information which must be interpreted but also physical orientation, world coordinates, facial expressions and speech commands. A basic knowledge of Bayesian probability and stochastic calculus will be part of the tool chest of anyone who wants to successfully navigate the Holo joblists of the future.

To reforge ourselves with skills for surviving the next seven years, designers must also become better programmers and software programmers must become more creative. The freelance creative, a job role that expanded dramatically during the Mobile Wars, will have an even brighter future in a world pervaded by augmented reality experiences. In order to make the shift, however, creatives will need to move beyond their comfort zone of creating PSDs in Photoshop and learn motion graphics as well as basic computer programming. Programmers likewise will need to move beyond the conceit that coding is an inherently creative activity; moving data around from point A to point B is no more creative than moving books around a sprawling Amazon warehouse and then packing them up for shipping is a poetic.

Real creative coding involves learning how to construct digital-to-physical experiences with Arduino, how to program self-generating visual algorithms with Processing, how to create 3D worlds in Unity and how to create complex visual interactions with openFrameworks and Cinder. These activities will become the common vocabulary of the future programmers of augmented experiences. Hiring managers and recruiters will expect to find them on resumes and without them, otherwise experienced tech workers be unhireable or worse, relegated to maintaining legacy web applications.

 

not with a bang but a whimper

 

enders game

The eyes are not here

There are no eyes here

In this valley of dying stars

In this holo valley

This broken jaw of our lost kingdoms

In this last of meeting places

We grope together

 

How can one tell if these prescriptions for the future Holo Wars are real and actionable or simply more marketing hype attempting to take advantage of people’s natural gullibility regarding technical gadgets? Aren’t we always being burned by overly optimistic portrayals of the future that never come to pass? Where are our flying cars? Where are our remote work locations?

In order for the Holo Wars to play out, certain milestones need to be achieved. Consequently, if you start seeing these milestones realized, you will know that you are in fact living through a fight over the next disruptive technology that will destroy some major tech corporations while affirming others at the apex of the tech world, one that will also reward those that have positioned themselves with useful skills for this future economy and punish those who do not. These milestones are: technology, monetization, persistent holographic objects, belief circles, overlapping dissociative realities.

Technology: the first phase is occurring now with the three major players discussed above and several additional players such as Metaio, Qualcomm and Samsung engaged in building up consumer augmented reality hardware and supporting technologies such as geolocation and gestural interfaces.

Monetization: innovation costs money. The initial hardware and infrastructure effort will likely be subsidized by the major players. Over time, the monetization model will likely follow what we see on the internet with “free” consumer experiences being subsidized by ads. There will be a struggle between premium subscription based experiences offering to remove the ads while providing better, higher resolution experiences with better content. These portal solutions will also contend against free and low-cost plug-in content provided by hackers and freelance creatives. How this plays out will depend largely on whether the premium content providers will be able to block out independents through standards and compatibility issues as well as whether hackers will find ways to overcome these roadblocks. There is also the possibility that some of the players might be looking at a much longer game and will foster an open AR content generation community rather than attempt to crush it. If the AR economy opens up in this way, a new service sector will grow made up of one set of people generating digital worlds for another set to live in.

Persistent Holographic Objects: virtual worlds are typically subjective experiences. They can be made inter-subjective, as they are in MMOs, by creating virtual topology in which people co-exist and co-operate. In augmented worlds, on the other hand, shared topology is an inherent feature. AR shared topology is called reality. In order to make AR worlds truly inter-subjective, rather than simply objective or subjective, shared holo objects must be part of the experience. Pesistent holo objects such as a digital fountain, a digital garden, or a digital work of art will have a set location and orientation in the world. AR players will need to travel to these locations physically in order to experience them. Unlike private AR or VR experiences in which each player views copies of the same digital object, with a shared experience each player can be said to be looking at the same persistent holo object from different points of view. In order to achieve persistent holographic objects, we will require finer grained geolocation than we currently have. AR gear must also be improved to become more usable in direct sunlight.

Belief Circles: a healthy indie creative fringe-economy and persistent holographic objects will make it possible to customize intersubjective experiences. People have a natural tendency to form cliques, parties and communities. Belief circles, a term coined by Vernor Vinge, will provide coherent community experiences for different guilds based on shared interests and shared aspirations. Users will opt in and out of various belief circles as they see fit. The same persistent holographic objects may appear differently to members of different circles and yet be recognized as sharing a common space and perhaps a common purpose. For instance, the holosign in front of the local Starbucks will have a permanent location and consistent semantic purpose, in AR space, but a polymorphic appearance. To paraphrase a truism, beauty will be in the eye of one’s belief circle.

Overlapping Dissociative Realities: divergent intersubjectivities will produce both a greater awareness of synchronicity – and a sense of deja vu as AR content is copied freely into multiple locations — as well as an increased sense of cognitive dissonance. Consider the example of going into Starbucks for coffee. The people waiting in line will likely each be members of varying belief circles and consequently will be having different experiences of the wait. This is not a large departure since we typically do not care about what other people in line are doing and even avoid paying attention unless they take too long making a selection. In this case, divergent belief circles make it easier to follow our natural instinct to avoid each other. Everyone in the holo valley is anonymous if they want to be. When one arrives at the head of the line, however, something more interesting happens. Even though the customer and the barista likely belong to different belief circles, they must interact, communicate, and perform an economic exchange; these two creatures from different worlds. What will that be like? Will one then lift a corner of the holo lenses in order to rub a sore eye only to discover that this isn’t a Starbucks at all but really a Dunkin’ Donuts which had silently bought out the other chain in a hostile takeover the previous week? Will your coffee taste any different if it looks exactly the same?

* 1996 was witness to a small skirmish between OpenGL and Direct3D that has subsequently come to be known as the API Wars. While the API Wars have had long lasting ripples, I don’t see them as having the tectonic effect of the other historical phenomena I am describing – plus anyways Thomas Kuhn only provides three major examples of his thesis and I wanted to stick to that particular design pattern.

[Much gratitude to Joel and Nate for collaborating on these scenarios over a highly entertaining lunch.]

Why Magic Leap is Important

 magic-leap-shark-640x426

This past weekend a neighbor invited our entire subdivision to celebrate an Indian holiday called Diwali with them – The Festival of Lights. Like many traditions that immigrant families carry to the New World in their luggage, it had become an amalgamation of old and new. The hosts and other Indians from the neighborhood wore traditional South-East Asian formalwear. I was painfully underdressed in an old oxford, chinos and flip-flops. Others came in the formalwear of their native countries. Some just put on jackets and ties. We organized this Diwali as a pot-luck and had an interesting mix of biryanis, spaghetti, enchiladas, pancakes with syrup, borscht, tomato korma, Vietnamese spring rolls and puri.

The most important part of the celebration was the lighting of fireworks. For about two solid hour, children ran through a smoky cul-de-sac waving sparklers while firecrackers went off around them. Towards the end of this celebration, one of our hosts pulled out her iPhone in order to Facetime with her father in India and show him the children playing in the background just as they would have back home, forming a line of continuity between continents using a 1500 year old ritual and an international cellular system. Diwali is called the Festival of Lights, according to Wikipedia, because it celebrates the spiritual victory of light over darkness and ignorance.

When I got home I did some quick calculations. In order to get to that Apple moment our host had with her father – we no longer have Hallmark moments but only Apple moments today – took approximately seven years. This is the amount of time it takes for a technology to seem fantastic and impractical – because we don’t believe it can be done and can’t imagine how we would use it in everyday life if it was – to having it be unexceptional.

2001-telepresence (1)

Video conferencing has been a staple of science fiction for ages, from 2001: A Space Odyssey to Star Trek. It was only in 2010, however, that Apple announced the FaceTime app making it generally available to anyone who could afford an iPhone. I’m basing the seven years from fantasy to facticity, though, on length of time since the initial release of the iPhone in 2007.

Magic Leap, the digital reality technology that has just received half a billion dollars of funding from companies like Google, is important because it points the way to what can happen in the next seven years. I will paint a picture for you of what a world with this kind of digital reality technology will look like and it’s perfectly okay if you feel it is too out there. In fact, if you end up thinking what I’m describing is plausible, then I haven’t done a good enough job of portraying that future.

Magic Leap is creating a wearable product which may or may not be called Dragonstone glasses and which may or may not be a combination of light field technology – like that used in the Lytro camera – and depth detection – like the Kinect sensor. They are very secretive about what they are doing exactly. When Leap Magic CEO Rony Abovitz talks about his product, however, he uses code to indicate what it is and what it isn’t.

In an interview with David Lidsky, Abovitz let slip that Dragonstone is “not holography, it’s not stereoscopic 3-D. You don’t need a giant robot to hold it over your head, you don’t need to be at home to use it. It’s not made from off-the-shelf parts. It’s not a cellphone in a View-Master.” At first reading, this seems like a quick swipe at Oculus Rift, the non-mobile, stereoscopic virtual reality solution built from consumer parts by Oculus VR and, secondarily, Samsung Gear VR, the mobile add-on to Samsung’s Galaxy Note 4 that turns it into a virtual reality device with stereoscopic audio. Dig a little deeper, however, and it’s apparent that his grand sweep of dismissal takes in a long list of digital reality plays over the years.

Let’s start with holography. Actually, let’s start with a very specific hologram.

let the wookie win

The 1977 holographic chess game from Star Wars is the precursor to both virtual and augmented reality as we think of them – for convenience, I am including them all under the “digital reality” rubric. No child saw this and didn’t want it. From George Lucas imaginative leap, we already see an essential aspect of the digital experience we crave that differentiates it from the actual technology we have. Actual holography involves a frame that we view the virtual image through. In Lucas’s vision, however, the holograms take up space and have a location.

harryhausen

What’s intriguing about the Star Wars scene is that as a piece of film magic, the technology behind the chess game wasn’t particularly innovative. It’s pretty much just the same claymation techniques Ray Harryhausen and others had been using since the 50’s and involves superimposing a animated scene over a live scene. The difference comes in how George Lucas incorporates it into the story. Whereas all the earlier films that mixed live and animated sequences sought to create the illusion that the monsters were real, in the battle chess scene, it is clear that they are not – for instance because they are semi-transparent. Because the elements of the chess game are explicitly not real within the movie narrative – unlike Wookies, Hutts, and Ton-tons – they are suddenly much more interesting. They are something we can potentially recreate.

AR

The difference between virtual reality and augmented reality is similarly one of context. Which is which depends on how we, as the observer, are related to the digital experience. In the case of augmented reality, the context is the real world into which digital objects are inserted. An example of this occurs in Empire Strikes Back [1980], where the binoculars on Hoth provide additional information presented as an overlay on the real world.

The popular conception of virtual reality, as opposed to the technical accomplishment, probably dates to the publication of William Gibson’s Neuromancer in 1984. Gibson’s “cyberspace” is a fully digital immersive world. Unlike augmented reality where the context is our reality, in cyberspace the context is a digital space into which we, as observers and participants, are superimposed.

 titan

To schematize the difference, in augmented reality, reality is the background and digital content is in the foreground; in virtual reality, the background that we perceive is digital while the foreground is a combination of digital and actual objects. I find this to be a clean way of distinguishing the two and preferable to the tendency to distinguish them based on different degrees of immersion. To the extent that contemporary VR is built around improving the video game experience, we see that POV games have, as a goal, to create increasingly realistic world – but what is more realistic than the real world. On the other side, augmented reality, when done right, have the potential to be incredibly immersive.

magic quadrant

We can subdivide augmented reality even further. We’ll actually need to in order to elucidate why AR in Magic Leap is different from AR in Google Glass. Overlaying digital content on top of reality can take several forms and tends to fall along two axes. An AR experience is either POV or non-POV. It can also be either informational or interactive.

terminator_view

Augmented Reality in the POV-Informatics quadrant is often called Terminator Vision after the 1984 sci-fi Austrian body-builder augmented film. I’m not sure why a computer, the Terminator, would need a display to present data to itself, but in terms of the narrative it does wonders for the audience. It gives a completely false sense of what it must be like to think like a computer.

google glass

Experiences in the non-POV-Informatics quadrant are typically called Heads-Up-Displays or HUD. They have their source in military applications but are probably best known from first-person-shooters where the view-point is tied to objects like windshields or gun-sights rather than to the point-of-view of the player. They also don’t take up the entire view and consequently we can look away from them – unlike Terminator Vision. Google Glass is actually an example of a HUD – though it is sometimes mistaken for TV — since the display only fills up the right corner of the visual field.

fiducial

Non-POV interactive can be either magic mirror experiences or hand-held games and advertisements involving fiducials. This is a common way of creating augmented reality experiences for the iPad and smartphones. The device camera is pointed toward a fiducial, such as a picture in a catalog, and a 3-D model is layered over the video returned by the camera. Interestingly Qualcomm, one of the backers in Magic Leaps recent round of funding, is also a leader in developing tools for this type of AR experience.

hope

POV interactive, the final quadrant, is where Magic Leap falls. I don’t need to describe it because its exemplar is the sort of experience that Rony Abovitz says Dragonstone is not – the hologram from Star Wars. The difference is that where Abovitz is referring to the sort of holography we can do in actual reality, Magic Leap’s technology is the kind of holography that, so far, we have only been able to do in the movies.

If you examine the two images I’ve included from Star Wars IV, you’ll notice that the holograms are seen not from a single point of view but from multiple points of view. This is a feature of persistent augmented reality. The digital AR objects virtually exist in a real-world location and exist that way for multiple people. Even though Luke and Ben have different photons shooting at their eyes displaying the image of Leia from different perspectives, they are nevertheless looking at the same virtual Princess.

This kind of persistence, and the sort of additional technology required to make it work, helps to explain part of the reason Google is interested in it. Google, as we know, already has its own augmented reality play. Where Google brings something new to a POV interactive AR experience is in its expertise in geolocation, without which persistent AR entities would be much harder to create.

This sort of AR experience does not necessarily imply the use of glasses. We don’t know what sort of pseudo-technology is used the the Star Wars universe, but there are indications that it is some sort of projection. In Vernor Vinge’s sci-fi novel Rainbow’s End [2006], persistent augmented reality is projected on microscopic filaments that people experience without wearables.

Because Magic Leap is creating the experience inside a wearable close-range display, i.e. glasses, additional tricks are required. In addition to geolocation – which is only a guess at this point – it will also require some sort of depth sensor to determine if real-world objects are located between the viewer and the object’s location. If there is, then the occlusion of the virtual entity has to be simulated in the visualization – basically, a chunk has to be cut out of the image.

magic-leap-whale

If I have described the Magic Leap technology correctly – and there’s a good chance I have not given the secretiveness around it – then what we are looking at seven years out is a world in which everything we see is constantly being photoshopped in real-time. At a basic level, this fulfills the Magic Leap promise to re-enchant the world with digital entities and also makes sense of their promotional materials.

There are also some interesting side-effects. For one, an augmented world would effectively turn everything and everyone into a potential billboard. Given Google’s participation, this seems even likely. As with the web, advertisements will pay for the content that populates an augmented reality world. Like the web and mobile devices, the same geolocation that makes targeted content possible may also be used to track our behavior.

magic

There are additional social consequences. Many strange aspects of online behavior may make its way into our world. Pseudo-anonymity, which can encourage bad behavior in good people, can become a larger aspect of our world. Instead of appearing as themselves, people may prefer enhanced versions of themselves or even avatars.

jedi_council

In seven years, it may become normal to sit across a conference table from a giant rabbit and Master Chief discussing business strategies. Constant self-reinvention, which is a hallmark of the online experience, may become even more prevalent. In turn, reputation systems may also become more common as a way to curb the problems associated with anonymity. Liking someone I pass in the street may become much more literal.

Jedi

There is also, however, the cool stuff. Technology, despite all the frequent articles to the contrary, has the power to bring people together. Imagine one day being able to share an indigenous festival with loved ones who live thousands of miles away. My eleven year-old daughter has grown up with friends from around the world whom she has met online. Technology allows her not only to chat with them with texts, but also to speak with them while she is performing chores or walking around the house. Yet she has never met any of them. In seven years, we may live in a world where physical distance no longer implies emotional distance and where sitting around chatting face-to-face with someone you have never actually met face-to-face does not seem at all strange.

For me, Magic Leap points to a future where physical limitations are no longer limitations in reality.