Tech means never having to say you’re sorry

spatial

There have been a series of amazing turns of event in the mixed reality world lately. The big headliners for me are:

1) Magic Leap laid off about a thousand employees due to diminishing funds but then was able to get a lifeline of $350 million, which will save the jobs of the remaining 300-400 engineers. The creative teams and sales teams appear to have been gutted in the first round of layoffs, unfortunately.

2) Microsoft HoloLens announced general availability of the HoloLens 2 on the Microsoft Store starting in July. Also availability in more countries starting in the fall.

3) The Unity XR SDK is getting closer to shipping, or has already shipped but is only working well with some platforms for now? Obviously some things need to be ironed out, but this appears to be the future of cross-platform AR development.

4) Spatial.io, the cross-platform XR collaboration platform, has made its product free.

Along with these there have been a series of refreshingly honest video interviews with some of the central people in the current evolution of mixed reality that help to frame our understanding of what has been going on at Microsoft and at Magic Leap over the past five years.

The XR Talk podcast is always great (thanks Roland for introducing me to it). This meandering interview with Graeme Devine, post-Leap, is particularly fascinating. There’s a great story of how he delivered the blade Orcrist (or was it Glamdring?) to Neal Stephenson in order to tempt him to come work with Magic Leap.

This week also saw the hosting of MR Dev Days conference on altspaceVR, which was a fascinating and wonderfully international experience.  Big thanks to Jesse McCulloch and everyone else responsible for throwing it together. The highlight of the show was a very frank conversation between Rene Schulte and Alex Kipman which I can’t recommend enough.

kipman

During the fireside chat (and the keynote the previous day) Kipman acknowledged the drawn out distribution of the HoloLens 2 and thanked developers for their patience. He also discussed the bucket problem (I think that’s what it’s called?) in which losing a a bucket of credibility requires a lot of buckets to regain that same level of credibility (pretty sure I messed up that metaphor).

We’ve seen a lot of that in the MR world this month. The financial problems at Magic Leap will have put a lot of people off. The fact that all the laid off employees have refrained from criticizing the company in the aftermath has been  surprising and probably speaks well for the company culture.

Meanwhile in the HoloLens world, public and private message boards indicate a lot of frustration with the product team. Original messaging suggested the devices would be out in early 2019, but over a year later, individual devs still have problems getting devices.

Looked at objectively, it’s pretty clear that if the HoloLens team could have gotten more devices to indie devs they would have and that the delays were not intentional. But knowing that also doesn’t necessarily make the bad feelings go away, given the difference between knowing and feeling, and this  in turn may have a depressing effect on any excitement around the wider release in July and then in the fall.

For what it’s worth, I think an apology goes a long way, and a heartfelt, personalized acknowledgment of the people who might feel slighted will go the greatest way. The difficulty here is that in a corporate culture like Microsoft’s, acknowledgement of mistakes is as alien to the normal way of doing things as – well, to be honest – as it is in the Trump administration. The culture of the Trump administration, after all, comes out of common practice in the modern corporation.

This isn’t always the case, though, and I have two pieces of evidence. Microsoft is very good at giving out chachkas, and even sent out an impressive gift pack for their online Build conference. The two best things I ever got from Microsoft, though, were personalized notes.

The first is a note from the Ben Lower / Heather Mitchell days of the Kinect program. Somebody wrote this out by hand, providing both an acknowledgment of who I am and what I had done (and to be honest, I was surprised they even knew who I was at the time):

WIN_20200523_18_10_50_Pro

The next is a card from the early days of the HoloLens program (Venessa Arnauld / Aileen Mcgraw).

WIN_20200523_18_10_27_Pro

These are my two most prized possessions from Microsoft over the past decade. Friends and associates have similar mementos they memorialize at home. The lesson from these two examples, for me, is that in tech you don’t always have to say you are sorry. It is often good enough, and probably more meaningful,  to acknowledge the legitimate concerns, understandable feelings,  and obvious humanity of the people who make you successful.

That’s a lot of personalized messages, but also the sort of thing that can easily repair broken or damaged relationships with your developer community.

10 Questions with Noah A S

noah2

Every group of friends has one person who holds the others together. In the world of Magic Leap, this person is Noah Aubrey Schiffman.

When the HoloLens first came out, the HoloLens team tried to create their own community website and forums. But people felt more comfortable hanging out in the HoloDevelopers Slack group that Jesse McCulloch created (and now Jesse works at Microsoft). When the Magic Leap came out at the end of 2018, a friend and I started a Slack group for it while others created a Discord channel to gather the community.

However, it was the twitter thread and #leapnation tag that Noah created which eventually became the gathering spot for MR developers, hobbyists and fans.

Why? you might ask. I think communities develop around people whose sincere enthusiasm reflects and reveals the common purpose inside the rest of us. In the world of magic leap, this hearth keeper is Noah, unofficial community ambassador to the magicverse, first of his name. Long may he reign.


What movie has left the most lasting impression on you?

Terminator II: Judgement Day (with the fear of Skynet) Or The Matrix (the idea of living in a simulation)

What is the earliest video game you remember playing?

It might have been something on an old-style Mac. Probably the game Sockworks which is for young toddlers.

Who is the person who has most influenced the way you think?

Probably my mother or a few of my friends.

When was the last time you changed your mind about something?

I do it a lot.. so I guess it was this week.

What’s a skill people assume you have but that you are terrible at?

ah a skill I don’t have that people assume I have.. Development, in something. It could be javascript I’ve not made much anything yet.

What inspires you to learn?

More learning, I guess, Isn’t it a cycle?

What do you need to believe in order to get through the day?

I don’t really need to believe very much I’m good when it comes to coping? Is this the question?

What’s a view that you hold but can’t defend?

It’s when I know something is coming or on the way but I signed an NDA so I cannot talk about it.

What will the future killer Mixed Reality app do?

– Something social! *Or* It will give you news! (doesn’t twitter do both?)

What book have you recommended the most?

Snow Crash.

10 Questions With Charles Poole

charles_poole

Charles Poole, the owner of IS Studios,  is currently one of the most experienced mixed reality developers in the business. Like many of the other people well known for their development chops on the HoloLens and Magic Leap One, he fell into it accidentally. Through a combination of determination and blind luck, as well as the ability to pick up a new UX paradigm that requires technical acumen with both .NET and Unity, he is currently one of those rare people with 3+ years of hands-on MR design, development and project management experience. You’ll have to ask him yourself for the full story, but it basically comes down – as with so many others – to getting his hands on a very expensive device and learning to make it hum (ideally  using spatial audio).

Charles is soft spoken and kind. One of the very interesting things about his background is that he is a mathematician – and so in that small subclass of software developers who actually knows math! There’s nothing nicer in the world of programming than having a friend you can hit up when you are having problems with an algorithm or with your matrix math.


What movie has left the most lasting impression on you?

Hackers, I think watching Hackers in 95/96 shaped my childhood and later choices when it came to education and what I spent my time on.

What is the earliest video game you remember playing?

Super Mario Brothers on the NES, or Sky Kid, also on NES. I remember playing it for hours just to get to the 3rd or 4th level, then watching my father get a lot further.

Who is the person who has most influenced the way you think?

Maybe Buckminster Fuller, Neal Stephenson maybe Michael Crichton. I read a lot as a child, I feel as though all the views I was exposed to through fiction and non-fiction had a big influence in how I see the world and approach problems. In general the problems seem really big, cause a lot of drama, are entertaining to read and experience, then the solution just happens to come together from a character that has the experience to pull a solution out of their ass.

When was the last time you changed your mind about something?

A big one recently, and kind of mild, was using Photon for the multiplayer aspects of my work. I was against Photon for a long time, I wanted to be in control of every aspect of what I was building. So I’d do things like make a custom socket server, write the server in Dark rift, use WebRTC. One of the most important things about freelancing is using every tool you have to accelerate development, while keeping it altogether. We had to make a decision recently about a multiplayer backend that could scale to thousands of users, but still be self hosted, and the time-frame was extremely compressed, so I revisited Photon, specifically PUN2 which had been released since the last time I had used PUN, and it felt like it had come a long way in the time since I had used it last.

Simpler and more personal – My daughter’s kindergarten teacher had been pushing for her to repeat kindergarten. I was staunchly against it, she was getting top marks, won the science fair over 5th graders, and with something she had actually done and came up with on her own, we only bought the materials. But she just wasn’t emotionally ready for the pace to get quicker in first grade, and her teacher made her excited about helping out for another year. So, we agreed to have her repeat kindergarten, because she loves to learn, and we didn’t want to make school into something she hated.

What’s a skill people assume you have but that you are terrible at?

Managing my time, I’m terrible at managing my time, I tend to get sucked into a project and neglect everything else. I would work every day from 9am – 9pm or later. I had to step back and put a rigid stop time on my day so I would spend time with my kids and not just work through their whole childhood.

What inspires you to learn?

I want to do everything myself, and push myself outside my developer comfort zone everyday. I’ll say ‘yes’ to things just for the challenge of figuring it out.

What do you need to believe in order to get through the day?

That things can only get better. I started off this dev journey making a thousand bucks a month, living in a tiny apartment with my wife and two kids. Every day, week, month feels like things have gotten better for us, at some point I want to turn around and help make other people’s lives better too.

What’s a view that you hold but can’t defend?

That anything is possible with enough hard work. I have an applied math background, and have seen sparks of insight and intuition I know I’d never have, but I still feel like I’d get there eventually if I put enough hours into it.

What will the future killer Mixed Reality app do?

Something agent based, an intelligent agent that acts as your exocortex. AI/ML is the future of Human Computer Interaction, the killer app won’t feel like an app, it will just be part of your life.

What book have you recommended the most?

Rainbow’s End by Vernor Vinge, it’s shaping up to be the most prescient book I’ve read. It was written in 2006 but the trends he wrote about are what we’re starting to see today, the nascent AR technology.

10 Questions with Suzanne Borders

suzanneborders_mm

Suzanne is the CEO of BadVR – which IMO wins the prize for best company name and probably could easily make a top 10 list for band names, also. Suzanne’s company works with the fascinating world of data visualizations in VR and MR. She is also the recipient of one of the coveted 2019 Magic Leap grants and is a member of Magic Leap’s Independent Creator Program. I met her briefly at an MR event in Mountain View, CA in early 2019 and besides being an amazing advocate for the importance of true 3D data visualizations in spatial experiences, has successfully shown everyone how to be a leader and promoter of mixed reality in the XR world.

What movie has left the most lasting impression on you?

This one is tough! I’m a huge film buff and there have been so many movies that have deeply impacted me and altered my understanding of the universe.

That being said, I think the most impactful film I’ve ever watched is “The Holy Mountain” by Alejandro Jodorowsky. It’s such an explosion of creativity, a surrealistic fever dream that functions on so many levels as a commentary on the human desire to seek truth and enlightenment. Jodorowsky is unlike any other filmmaker out there, a true magician that makes film into high art without losing the ability to make impactful statements about the universal human condition. Any of his films could really be considered my favorite but “The Holy Mountain’ in particular speaks to me the most because it best captures the hero’s journey; our collective desire to seek something greater from life than what we’re given. A lot of surrealistic film is just weird for the sake of being weird and therefore loses impact because it doesn’t use the symbolism of surrealism to make any sort of deeper statement. Jodorowsky is a surrealist in the best sense of the term – all his bizarre unexpected images convey meaning and activate archetypical feelings, drives, and desires in his audience. He’s a master of the subconscious and knows how to access and wield communicative power in this area. Because of this, he’s my creative hero and I look to his work often for inspiration, especially when attempting to craft products that have the ability to touch user’s subconscious. I think this is key when unlocking broad market appeal for products or film or art in general. To really touch and impact a wide audience the experience, the artist or creator must touch on, and involve, a universal archetype. Jodorowsky’s films taught me this lesson and showed me how to execute on it. I want to give a big shout-out and thank you to my filmmaker friend Ryan, who introduced me to them. He, in many ways, has fundamentally changed how I approach any creative challenge by showing me Jodo’s work. 

Beyond “The Holy Mountain,” I’m a big fan of “Belladonna of Sadness” (you will not find a more beautifully animated film ever), “Apocalypse Now” (Brando as Kurtz and his monologue at the end talking about the clarity of evil is a perennial favorite; combining Conrad’s “The Heart of Darkness” with the Vietnam war was a stroke of pure genius), “Funeral Parade of Roses” (a Japanese film that sets the ancient story of Oedipus into the transgender alternative subculture in 1960s Japan; I love it for its ability to utilize archetypical images and stories in an unexpected and creative way), “Hiroshima Mon Amour” (any media by Marguerite Duras is an automatic favorite), and “Last Tango in Paris” (I adore Brando, he’s an absolute legend, and this film touches on so many truths of the human existence, our longing for connection, the power of anonymity, my own personal life makes this film more powerful to me than it will to many, but none the less I adore it). And of course, the visual style and occult symbolism of Dario Argento’s films is a forever favorite (“Suspiria” being the pinnacle of Argento’s work IMO).

Lastly, Fellini’s “8 1/2” was the first film I watched as a child that really unlocked for me the power of cinema and storytelling. Prior to watching it, I had dismissed film as some inferior commercial medium. I saw it as cheap mindless entertainment for the masses without substance or meaning. For me at that time, my understanding of film was limited to boring and poorly made summer blockbusters. I remember clearly popping in the 8 1/2 VHS tape at age 17 without any expectation, just another mindless story to pass the long summer hours of adolescence. But the story that jumped out from the screen – starting with Fellini’s infamous opening dream sequence – absolutely captivated me. I found myself profoundly touched at the end of the film, crying even, and realized that I had been changed forever for having watched it. The message of the film – our flawed desire for human connection and all the broken and dysfunctional ways we pursue it – resonated with me at such a level that I have, decades later, never forgotten that moment. From that point on, I considered film and storytelling a high art that held the potential to change the world. Of course, not all film or stories rise to this potential and I’ve continued to be disappointed by mainstream commercial film in such a major way that I don’t even engage with it anymore. But 8 1/2 made me realize the potential of film as a medium for spiritual transformation. It showed me the power of storytelling had to bring humanity together and demonstrated the medium’s ability to hold up to the audience a mirror of themselves, helping them pursue a deeper understanding of both themselves the world around them.

Obviously, I adore film. It is one of my biggest sources of creative inspiration for all my technical work. I love immersive tech because one builds experiences, not screens. MR holds the same potential to affect deep spiritual change and transformation in users and that interests me immensely.

What is the earliest video game you remember playing?

LOOM! I remember playing it on the first computer my father bought for our family, when I was 6 or 7 years old. I remember spending hours and hours sitting in front of the computer playing, captivated by the beautiful game art. LucasFilm games are the best, but in particular Loom really did it for me. I loved (and still love) that the primary way Bobbin Threadbare (main character) interacted with his world was through music and sound. Such an original and creative idea!

Plus, you could cast spells to literally rip apart the fabric of existence, calling forth the lord of the dead, ripping open cemeteries to speak to the souls of the deceased. You could exist beyond space and time and your character could visit this beautiful lake floating in the void, populated by swans who spoke to you in parables of truth. As a goth kid and a lover of poetry, this was beyond transformative for me. I wanted to live in Loom! Additionally, the game came with this amazing backstory about a world full of guilds and weavers of destiny. I used to listen to the backstory tape, complete with a dramatic reenactment, and pretend I was Bobbin Threadbare. Loom will forever be my favorite game of all time.

Myst is a very close second!

Who is the person who has most influenced the way you think?

This is a difficult one – there have been so many amazing mentors in my life and each one of them has taught me something important, about myself, about my experience of the world.

As mentioned, Jodorowsky has been a major influence on me and all that I create. I’ve followed him around the world and I’ve actually met him in real life. I was fortunate enough to have him read my tarot in Paris and that reading truly changed my life. I won’t go into details because it was a deeply personal reading, but it transformed me without doubt. I also was lucky enough to meet him again at the Egyptian Theater in Los Angeles and at this event he dropped many nuggets of wisdom as well. 

I’ve also learned a lot from the coterie or filmmaker friends that I’ve developed here in Los Angeles. The one in particular who introduce me to Jodorowsky has taught me a lot about the creative journey. He’s taught me how to dive into my creative subconscious to identify those valuable universal, broadly resonate true ideas. I’ve always been fascinated with the ability to broadly affect so many different types of people with one single idea and I wanted to translate that to my products. When you talk to someone who wrote or directed a hugely successful film, you find they have this ability to take a concept and distill it down into its most basic form. However, instead of that process being reductive or simplistic, you find that this distillation strengthens the idea and makes it more crystalline and clear and most importantly, universally accessible. The ability to take complex, nuanced, ideas and make them resonate with the broadest audience possible is one that I value highly. I’m very glad to have had a group of people who’ve helped teach me this skill. Regardless of the difference in our industries.

When was the last time you changed your mind about something?

Whew boy, I change my mind all the time, constantly, on a second to second basis! I’m always ingesting data about my world, through experiences, books, travel, websites, music, films, poems, and products. Even subconsciously, my mind is always picking up on new data about my world, which then changes my understanding of the universe. Plus, I believe everything constantly changes, so I have to keep pace with this change and adjust my thoughts and theories to mesh with the latest information.

A system that runs off absolutes and stasis is brittle and bound for failure. Only by being nimble and changeable can any system truly be strong and resilient. As such, I agree very much with Nassim Nicholas Taleb and his concept of anti-fragility. Anti-fragility involves growth through stress and I’d like to think all of my internal world models fall into this category by being responsive in real time to new data that stresses their limits, structures, and boundaries. 

What’s a skill people assume you have but that you are terrible at?

Everyone assumes I’m skilled at math because my company works with data. But I’m actually numerically dyslexic (yes, it’s a real thing) and numbers have always been a real struggle for me. That’s one of the many inspirations for BadVR – my desire to work with data but my lack of technical skill with which to do so. I am in many ways the non-technical person for whom my product is built; I am my user. This gives me the power and the passion to build and also gives me the empathy needed to deliver an effective product that makes data accessible to everyone.

Of course, being acutely aware of this shortcoming I’ve assembled a team of very highly talented mathematical geniuses that augment my own weaknesses. So, to allay any question about my company’s ability to deliver a highly technical product, I want to underscore the idea that my company is not solely comprised of me. The heart and soul of BadVR is our team, and they are deeply capable in all the ways that I am not. That variety of skills and talent is what makes us powerful. We all balance each other’s weaknesses and strengths and, in doing so, create something better than any of us could ever achieve independently. 

What inspires you to learn?

I don’t need inspiration for this! I’m endlessly curious about everything, all the time. I never learned to stopped asking “why?” Learning is my default state of being. Anytime I see anything, or experience anything, it inspires me to ask more questions, to dig deeper, to understand further. My google search history is full of things like “how did dinosaurs procreate? What is dirt? Why is dark meat dark?” I just wonder and google and learn all the time. Every experience is an impetus for learning; a reason to dive into the whys, hows, and whats of yet another line of inquiry.

What do you need to believe in order to get through the day?

I have to believe that life doesn’t end with death. That I will again see the people I love who I’ve lost. If there isn’t an afterlife or there isn’t an alternate timeline we’re we meet again, I can’t continue. I’ve lost too many loved ones to be able to function without the belief that I will see them again. It goes without saying then that I believe in reincarnation, in the broadest sense. I strongly believe that the people we love never leave us and that in some way we end up back together. It’s not an evidence-based belief – besides anecdotal evidence anyways – but I must believe it. I do believe it. I will always believe it. Other, the loneliness is crushing, overwhelming; the feeling akin to being forever a planetary stranger at the very end of the world.

What’s a view that you hold but can’t defend?

I have plenty of beliefs that don’t have scientific, evidence-based support. I can always defend every belief I have if you allow anecdotal evidence or emotional appeals. Some examples include my belief in the tarot, in astrology, in dream work, psychic powers, aliens, the collective subconscious, Bigfoot, the Missouri Skunk Ape, and ghosts. I’d be more than happy to argue their existence on an emotional and anecdotal level with anyone. But science of course doesn’t support or embrace such parapsychology and cryptozoology. This doesn’t stop me from believing, though. Many of the most important questions in life cannot be answered by science. I think the scientific method is important for lesser questions but for the big questions of life like “Why are we here? What is our purpose? What is the meaning of life?” — science fails. I’m more interested in the answers offered by faith and spirituality than I am in the answers offered by science, for these sorts of questions. In the face of the eternal, science can seem so small and pedantic. But of course, for the mundane it is very important.

What will the future killer Mixed Reality app do?

Visualize data and allow for immersive analysis! Data is the killer app for mixed reality. I firmly believe that, and I fully believe my company, BadVR, will be the industry standard tool for working with data immersively. I may be biased as BadVR is my company, but hey that’s what I believe! Our unique approach, mixing art with logic, the abstract with the concrete, is exactly the way this product needs to be approached. In the future, everyone will be able to easily see and interact with incredibly large, abstract and geospatial datasets with ease. We will think of data as an oracle; a source of truth. It’s important that everyone be able to access such a powerful product, which is a major focus of BadVR – universal accessibility.

What book have you recommended the most?

The Panic Fables” by Alejandro Jodorowsky. A book of spiritual comics that delivers small truths via 1-page comics. It’s an easy entry point into the Jodo-sphere!

Narcopolis” by Jeet Thayil. One of my all-time favorite passages can be found in this novel. It’s about a large cast of characters who frequent an opium den in Bombay (before it became Mumbai). Thayil is one of the few writers who can write prose that reads like poetry. I am a forever a huge fan!

The Hour of the Star” by Clarice Lispector. She deconstructs language and storytelling to deliver a narrative about a poor Brazilian girl and her search for meaning and transcendence in a world that doesn’t want or even see her. It is a visceral gut-punch of truth. Anything by Lispector is wonderful, but this story in particular is my favorite.

I will leave you with a quote from Lispector:

“I do not know much. But there are certain advantages in not knowing. Like virgin territory, the mind is free of preconceptions. Everything I do not know forms the greater part of me. And with this I understand everything. The things I do not know constitute my truth.”

The Fork in Mixed Reality

Futuristic Spaceman - Programmer Reading Projected Information

Yogi Berra gnomically said, “when you come to a fork in the road, take it.”  On the evening of Friday, February 1st, 2019 at approximately 9 PM EST, that’s exactly what happened to Mixed Reality.

The Mixed Reality Toolkit open source project, which grew out of the earlier HoloLens Toolkit on github, was forked into the Microsoft MRTK and a cross-platform XRTK (read the announcement). While the MRTK will continue to target primarily Microsoft headsets like the HoloLens and WMR, XRTK will feature a common framework for HoloLens, Magic Leap, VR Headsets, Mobile AR – as well as HoloLens 2 and any other MR devices that eventually come on the market.

So why did this happen? The short of it is that open source projects can sometimes serve multiple divergent interests and sometimes they cannot. Microsoft was visionary in engineering and releasing the original HoloLens MR Headset. They made an equally profound and positive step back in 2016 by choosing to open source the developer SDK/Framework/Toolkit (your choice) that allows developers to build Unity apps for the HoloLens. This was the original HoloLens Toolkit (HLTK).

While the HLTK started as a primarily Microsoft engineering effort, members of the community quickly jumped in and began contributing more and more code to the point that the Microsoft contributions became a minority of overall contributions. This, it should be noted, goes against the common trend of a single company paying their own engineers to keep an open source project going. The HLTK was an open source success story.

In this regard, it is worth calling out two developers in particular, Stephen Hodgson and Simon Jackson, for the massive amounts of code and thought leadership they have contributed to the MR community. Unsung heroes barely captures what they have done.

In 2017 Microsoft started helping to build occluded WinMR (virtually the same as VR) devices with several hardware vendors and it made sense to create something that supported more than just the HoloLens. This is how the MRTK came to be. It served the same purpose as the HLTK, to accelerate development with Unity scripts and components, but now with a larger perspective about who should be served.

In turn, this gave birth to something that is generally known as MRTK vNext, an ambitious project to support not just Microsoft devices but also platforms from other vendors. And what’s even more amazing, this was again driven by the community rather than by Microsoft itself. Microsoft was truly embracing the open source mindset and not just paying lip service to it as many naysayers were claiming.

But as Magic Leap, the other major MR Headset vendor, finally released their product in fall, 2018, things began to change. Unlike Microsoft, Magic Leap developed their SDK in-house and threw massive resources at it. Meanwhile, Microsoft finally started throwing their engineers at the MRTK again after taking a long hiatus. This may have been in response to the Magic Leap announcement or equally could have been because the team was setting the stage for a HoloLens 2 announcement in early 2019.

And this was the genesis of the MR fork in the road: For Microsoft, it did not make sense to devote engineering dollars toward creating a platform that supported their competitors’ devices. In turn, it probably didn’t make sense for engineers from Google, Magic Leap, Apple, Amazon, Facebook, etc. to devote their time toward a project that was widely seen as  a vehicle for Microsoft HMDs.

And so a philosophical split needed to occur. It was necessary to fork MRTK vNext. The new XRTK (which is also pronounced “Mixed Reality Toolkit”) is a cross-platform framework for HoloLens as well as Magic Leap (Lumin SDK support is in fact already working in XRTK and is getting even more love over the weekend even as I write).

But XRTK will also be a platform that supports developing for Oculus Rift, Oculus Go, HTC Vive, Apple ARKit, Google ARCore, the new HoloLens 2 which may or may not be announced at MWC 2019, and whatever comes next in the  Mixed Reality Continuum.

So does this mean it is time to stick a fork in the Microsoft MRTK? Absolutely not. Microsoft’s MRTK will continue to do what people have long expected of it, supporting both HoloLens and Occluded WinMR devices (that is such a wicked mouthful — I hope someone will eventually give it a decent name like “Windows Surface Kinect for Azure Core DotNet Silverlight Services” or something similarly delightful).

In the meantime, while Microsoft is paying its engineers to work on the MRTK, XRTK needs fresh developers to help contribute. If you work for a player in the MR/VR/AR/XR space, please consider contributing to the project.

Or to word it in even stronger terms, if you give half a fork about the future of mixed reality, go check out  XRTK  and start making a difference today.

Magic Leap One vs HoloLens v1 Comparison

side-by-side

I’m currently sitting in my room at the L.A. Grand Hotel waiting for the L.E.A.P. conference to start. I’ve been holding off on this comparison post because I had promised Dennis Vroegop I would give it first as a talk at the Techorama Netherlands conference – which I did last week. I will do a feature comparison based on publicly available information, then highlight features unique to the Magic Leap, and then distinguish subtle but important differences that only become apparent from spending months with these devices at the developer level. Finally I want to point out design improvements in the Magic Leap that are so good for Mixed Reality that I predict they will be incorporated into the next version of HoloLens.

Keep in mind that this is a comparison of two different generations of devices. The Magic Leap One is coming out two years after the HoloLens and would be expected to be better. At the same time, the HoloLens v2 is being released some time in 2019 and can be expected to be better still.

1. Field of View

In raw numbers, the field of view of the Magic Leap One is approximately 25% better than the HoloLens. The HoloLens field of view is estimated to be about 29-30 degrees wide and 17 degrees high. The Magic Leap One is 40 degrees wide by 30 degrees high. There is a corresponding difference in resolution, with the HoloLens offering 1268 by 720 per eye and the Magic Leap One providing 1280 by 960 per eye.

The Magic Leap One uses the same wave guide display technology that the HoloLens does, however, so how did they pump up the FOV? First, the ML1 has a more powerful battery than the HoloLens does, and it’s often been claimed by Microsoft that FOV is largely dependent on the power of the projection. This is probably offset, though, by the fact that the ML1 is using more power to project in two planes instead of only one like the HoloLens does (with 6 Waveguide layers compared to 4 in the HoloLens).

Another trick is that the waveguides in the Magic Leap are closer to the wearer’s eyes than they are in the HoloLens. As a consequence, you can wear glasses underneath the HoloLens while you cannot do so comfortably under the Magic Leap device.

In addition to this, Jasper Brekelmans and Dennis Vroegop suggested over coffees along the Amstel River (in a conversation about David Copperfield) that because one’s peripheral vision is closed off in the ML1, the perceived FOV may be even larger than the actual. The theory behind this is that, due to the widespread use of glasses, we have become used to not paying attention to our peripheral vision so much and consequently are comfortable with this tunneling of our vision.

Blocking off the peripheral field of view might cause issues in certain industrial settings, but the general effect is that what you can see as a proportion of your overall FOV is much larger in the ML1 than it is in the HoloLens. Or another way of putting this is that the empty areas of your FOV, as a proportion of your available FOV, is much smaller than it is in the HoloLens.

On top of this, the aspect ratio of the FOV in the ML1 is much taller than in the HoloLens, which may end up doing a better job at accommodating vertical seccaddic movements of the eyes.

inset

Because of the narrower gap between the device and the wearer’s eyes, the Magic Leap can’t accommodate glasses as the Hololens can. To compensate, Magic Leap is developing relationships with online eyeglass manufacturers to provide prescription inserts that can be placed in front of the waveguides and magnetically lock into place. There’s some controversy over whether this is a good or a bad thing. Some developers have expressed concern that this will make demoing Magic Leap at events more difficult than demoing HoloLens, since those with poor vision will either not be able to participate or, alternatively, we will be forced to carry around a large suitcase of prescription inserts to every event.

On the other hand, when I think of what MR will be like in the future, I tend to think of them resembling real glasses (and not electronic contacts, which simply scare me). When they reach the size and ubiquity of modern glasses, it will make sense for each person to have their own personalized device with their appropriate prescription. Magic Leap is on the right track in this case. It’s just in the intervening period that we have to figure out how to share our limited, expensive devices with others.

HoloLens v1 Magic Leap One
Price $3000 – $5000 $2300
OS Windows Android variant
Field of View ~30 deg x 17 deg 40 deg x 30 deg
Resolution 1268 x 720 per eye 1280 x 960 per eye
Depth Sensor Time of Flight Time of Flight
Display Type Wave Guide Wave Guide
Hand Gestures Recognized 2 9
Underlying comic book technology Light Engines Light Fields
Controller Click 6 DOF
Hand tracking limited fingers (3 joints each)
Processing unit Above nose Light pack
Audio Spatial Sound Spatial Sound

2. Hardware Specs (It’s all about the battery)

HoloLens v1 Magic Leap One
Intel Atom x5-Z8100
1.04 GHz
Intel Airmont (14nm)
4 Logical Processors
64-bit capable
NVIDIA® Tegra X2 SOC
2 Denver 2.0 64-bit cores + 4 ARM Cortex A57 64-bit cores
(2 A57’s and 1 Denver accessible to applications)
8086h (Intel) GPU. NVIDIA Pascal™, 256 CUDA cores; Graphic APIs: OpenGL 4.5, Vulkan, OpenGL ES 3.3+
2GB RAM 8GB RAM
64GB Storage 128GB Storage

The Magic Leap One is overall a much beefier machine than the current HoloLens. While both the HoloLens and the Magic Leap One advertise a 3 hour battery life, these can mean vastly different things. In order to drive all of its extra hardware, the Magic Leap One needs a much beefier battery. The ML1 is powered by a twin-cell battery with 36.77 Wh, running at 3.83 V.  The HoloLens has a 1.65 Wh battery.

For overall performance, the larger battery means the world meshes (i.e. surface reconstruction, world mapping) are much denser and more frequently updated on the Magic Leap than on the HoloLens. The Time-of-Flight depth camera can fire off more frequently and for longer periods.

The larger battery and beefier specs also translate to much better 3D performance. The HoloLens is able to run 30,000 polygons at 60 fps. Beyond that, the fps begins to drop. The Magic Leap runs upwards of 1 million polygons at 60 fps.

On the downside, that more powerful battery rig needs a fan to cool it whereas the HoloLens is passively cooled. In laboratory and medical scenarios where a sterile environment must be maintained, active cooling with a fan could be a problem.

3. The HoloLens and Tracking

The HoloLens uses 4 monochrome cameras (“environment aware sensors”), an accelerometer, magnetometer and gyroscope in a sensor fusion configuration, and a custom HPU to perform head tracking. The Magic Leap one has a similar setup minus the HPU.

The HoloLens tracking is still somewhat better than the ML1’s. It loses tracking less frequently and digital content is less jittery when seen up close or while the wearer is in motion.

Overall, though, tracking performance is fairly close between the two devices.

4. Magic Leap Extras

The ML1 has a couple of features that are simply outside of the box. One is the eye tracking. There are inward facing cameras that track the wearer’s eye movements as invisible IR flashes.

The tracking is not continuous and is captured at a much lower resolution level than the displays. While they shouldn’t be used for direct user interactions, they are great for providing context for other interactions. It would be great if someone would write a keyboard that uses eye tracking to select keys. In the meantime, I wrote this heat vision demo that uses eye tracking to burn the walls of my house — I think of it as “Superman with a Migraine”. Note the eye-blink tracking.

The other cool extra in the Magic Leap is two planes of focus. Most VR devices have a single plane of focus at infinity. The HoloLens has a single plane of focus set at two meters.

In the magic leap one, when you look at near objects, objects further away (on the outer plane) seem to go out of focus. When you look at objects close up, the objects further away go out of focus. I would guess that the close plane is around a meter and the out one about 3 meters but I’m not really sure. In the Lumin OS .91, there is also a sporadic green shift in the near plane (which I expect will be fixed soon).

5. The Tether

tether

The Magic Leap One is made up of two parts: the Light Pack and the Light Wear. They are connected by a cable. The Light Wear contains all the sensors, projectors and displays while the Light Wear, worn at the hip, contains all the computer bits and the battery.

This is an engineering choice that allows for a much larger power source. Without the tether solution, a large battery would not be possible. Without the large battery, the ML1’s enhanced depth sensing, improved graphics processing and larger field of view would not be possible.

In addition, this design makes the Magic Leap a much more comfortable fit on the head. The weight distribution is better than on the HoloLens, it is lighter, and it doesn’t require extra straps.

The tether solution is actually so effective that I would be surprised if the HoloLens v2 does not follow a similar design. The original one-piece “tetherless” solution Microsoft came up with for the HoloLens was visionary, but severely limiting.

6. Developing

If you have ever developed in Unity for the Android (or really any other device) then you know how to develop for Magic Leap in Unity. You press a button and your app compiles to an .mpk image (Android uses “.apk” file extensions). If your device is attached, you can deploy directly by clicking on “build and run”.

Magic Leap apps can also be built with the Unreal Engine.

HoloLens apps run on a Unity player sandboxed in a UWP app. The development cycle consequently involves exporting your HoloLens app as a Visual Studio project targeting UWP and then building and deploying in UWP. In general (and it may just be me) this has been tedious.

It became even worse when the immersive WinMR devices (or occluded WinMR – basically Microsoft VR) devices came out last year and the basic tools used for HoloLens development, known as the HoloLens Toolkit and then the Mixed Reality Toolkit, was expanded to supported both kinds of device. Because of some issues with Unity, building for WinMR required certain versions of Unity and above while developing for HoloLens required certain versions of Unity and below. And this state went on for several months to the point that finding the correct Windows SDK paired with the right MRTK version paired with the correct Unity version became a closely kept alchemical formula passed from developer to developer.

This experience may not be the same for everyone but it left me a bit traumatized. By contrast, Magic Leap development is simply a pleasure. I can build and see the results very quickly in my device. I can wear the device for hours at a time. I typically only stop development when the ML battery runs down and I have to let it recharge. I don’t have a Magic Leap Hub, which would allow me to charge while I dev, but I intend to get one.

The Magic Leap toolkit is still not quite as capable as the open source Mixed Reality Toolkit managed by Stephen Hodgson and others.

The Magic Leap also has a simulator rather than an emulator for developing without a device. This actually makes sense since the Hololens emulator runs the HoloLens OS in a virtual machine, which might be tricky given the much larger specs of the Magic Leap.

7. Interactions

6dof

The Magic Leap supports robust hand and gesture tracking as well as a 6DOF controller. The DOF in 6DOF stands for degrees of freedom. We know not only the direction the controller is pointing in (3DOF) but also its position.

tap

I love the controller. I love it so much it made me finally admit to myself that I hate the HoloLens tap gesture. No one ever gets it right. It’s awkward. It’s uncomfortable and makes me feel like I’m performing a kung fu move.

mantis

By contrast, a controller just makes sense. The UX for MR, I believe, should always support three layers of interactions. Mixed reality UX should support hand gestures for ease of use. It should fall back to the controller for precision movements. It should finally fall back on the delta pad on the controller for accessibility.

For all of my antipathy toward the HoloLens tap, however, I have to say I miss the HoloLens bloom gesture (escape), which I keep trying to use in Magic Leap to no avail. Instead, in Magic Leap holding the controller’s Home button for three seconds is the escape gesture, which I don’t really like. It also bothers me that hand gestures aren’t supported in the core desktop (the Icon grid) – but this is still the Creator’s Edition (translation: dev edition) after all.

[Late edit thanks to SH: it should also be pointed out that the Lumin OS (the desktop layer) currently doesn’t support hand gestures, which I find baffling. For now, you can’t get past the login and other initial screens without a paired phone or a controller.]

Summing Up

So is the Magic Leap One better than the HoloLens v1? Oh yes. By leaps and bounds.

1. The development workflow is much more straight forward and pleasant.

2. The increased battery size and beefier hardware makes it possible to do things, performance wise, that the HoloLens tended to stop us from doing. Phone and tablet level experiences are doable now.

3. The Magic Leap One has a much better interaction model than the HoloLens does. How did anyone ever do MR without a controller? (Actually, everyone used an XBox controller in the end in order to get any sort of real work done, but we don’t talk about that much.)

Is it time to jump back into Mixed Reality development?

If you spent $3.2K to $5K for a HoloLens, then you owe it to yourself to spend $2,300 for a Magic Leap. It’s the device you originally wanted. The HoloLens was a brilliant device back in 2016 and really the first of its kind, but it had limitations. Many of the projects you were never able to realize in HoloLens (in the small dev community that developed around HoloLens, we all know what these are) are now doable with the improved Magic Leap specs. Additionally, your enterprise stories are much easier to sell with the controller. Instead of spending 5 minutes of your precious pitch time explaining how tap works, you can now just let your potential investors and clients go straight into the demo with a controller they basically already know how to use.

Is there a future in spatial computing?

Now there is. There was a brief pause between 2016 and the middle of 2018, but we currently have two great devices available with another shoe dropping soon. Microsoft will be coming out with a HoloLens v2 sometime in the first half of 2019 which I would predict will implement the tethered design Magic Leap is using. This will be an improvement over the current Magic Leap which in turn will be driven to improve its own tech.

Microsoft has an advantage because it started this journey back in the Kinect days and has the resources of Microsoft Research to draw on. Magic Leap has an advantage because, well, they aren’t Microsoft and don’t face the internal political problems a large tech giant does (though no doubt they have their own). More importantly, they have their own U.S.-based production lines (as well as production lines in Mexico) and are less reliant on China, which hopefully means they are capable of much quicker turn-arounds and initial SKU production.

When do we get smaller devices that wear like glasses?

I have no idea, but try to think in terms of 3, 5, 10 years. We always overestimate what can be done in 3 years but always underestimate how much things will change in 10. Somewhere in the middle, we will intersect with our MR futures.

Your comments, corrections and criticisms are welcome in the comments below. I’ll try to keep up with them and incorporate what you say into the main article as appropriate.