Virtual Reality – The Imaginative Universal

Immersion and the Star Wars Galactic Star Cruiser

May 4, 2022May 5, 2022 James Ashley

In the second week of March, I took my family to the Galactic Starcruiser at Disneyworld in Orlando, Florida, informally known as the Star Wars Hotel. The Starcruiser is a two-night immersive Star Wars experience with integrated storylines, themed meals, costumes, rides and games. For those familiar with the Disneyworld vacation experience, it should be pointed out that even though the Star Wars themed Galaxy’s Edge area in Hollywood Studios, it isn’t a resort hotel. Instead, it can best be thought of as a ride in itself.

The design is that of a cruise ship, with a dining hall and helm in the “front” and an engine room in the “back”, and a space bar off of the main muster area. The NPCs and the staff never break character, but work hard to maintain the illusion that we are all on real space cruise. Besides humans, the “crew” is also staffed with aliens and robots – two essential aspects of Star Wars theming.

In line with the cruise experience, you even do a one-day excursion to a nearby alien planet. I’ve had trouble writing about this experience because it felt highly personal, lighting off areas of my child brain that were set aside for space travel fantasies. At the same time, it is also very nerdy, and the intersection of the highly nerdy and the highly personal is dangerous territory. Nevertheless, it being May 4th today, I felt I could not longer put it off.

How you do Immersion?

“Immersion” is the touchstone for what people and tech companies are calling the Metaverse. Part of this is a carry over from VR pitching, and was key to explaining why being inside a virtual reality experience was different and better than simply playing a 3D video game with a flat screen and a controller.

But the term “immersion” hides as much as it reveals. How can “immersion” be a distinguishing feature of virtual reality when it is already a built-in aspect of real reality? What makes for effective immersion? What are the benefits of immersion? Why would anyone pay to be immersed in someone else’s reality? Is immersion a way to telling a story or is storyline a component of an immersive experience?

A Russian doll aspect of the starcruiser is the “Simulation Room” which, in the storyline of the ship, is an augmented area in that recreates the climate of the planet the ship is headed toward. The room is equipped with an open roof which happens to perfectly simulate the weather in central Florida. The room also happens to be where the Saja scholars provide instruction on Jedi history and philosophy.

Space Shrimp (finding the familiar in the unfamiliar)

I’m the sort of person who finds it hard to every be present in the moment. I’m either anticipating and planning for the next day, the next week, the next few years, or I am reliving events from the past which I wish had gone better (or wish had never happened at all).

For two and a half days on this trip, I was fully captivated by the imaginary world I was living through. There wasn’t a moment after about the first hour when I was thinking about anything but the mission I was on and the details of the world I was in. I didn’t feel tempted to check my phone or know what was happening in the outside world.

An immersive experience, it seems to me, is one that can make you forget about the world in this way, by replacing it with a more captivating world and not letting go of you. I’ve been going over in my head the details of the star wars experience that make this work and I think the blue shrimp we had for dinner one night is the perfect metaphor for how Disney accomplishes immersion.

To create immersion, there can be nothing that references the outside world. The immersive experience must be self-contained and everyone in the immersive experience, from cabin boy to captain, must only reference things inside the world of the starcruiser. Fortunately Star Wars is a pre-designed universe. This helps in providing the various details that are self-referential and remind us of the world of the movies rather than the world of the world.

A great example of this is the industrial overhead shower spout and the frosted glass sliding shower door in our cabin. They are small details but harken back to the design aesthetic of the star wars movies, which contain, surprisingly, a lot of blue tinted frosted glass.

This extends to the food. All the food is themed, in a deconstructionist tour de force, to appear twisted and alien. We drank blue milk and ate bantha steaks. We feasted on green milk and salads made from the vegetation found on the planet Falucia.

And here there is a difficulty. Humans have a built-in sense of disgust of strange foods that at some point protected our ancestors from accidentally poisoning themselves. And so each item of food had to indicate, through appearance or the name given on the menu, what it was an analog of in the real world. I often found myself unable to enjoy a dish until I could identify what it was meant to be (the lobster bisque was especially difficult to identify).

What I took from this was that for immersion to work, things have to be self-referential but cannot be totally unfamiliar. As strange as each dish looked, it had to be, like the blue shrimp, analogous with something people knew from the real world outside the ship. Without these analogical connections, the food will tend to create aversion and anxiety instead of the sense of immersion intended.

One take way is that as odd as the food sometimes looked, the food analogs were always meals familiar to Americans. Things common to other parts of the world, like chicken feet or durian fruit or balut, would not go over well even though they taste good (to many people).

A second take away is that the galactic food has to be really, really good. In modern American cuisine, it is typical to provide the story behind the food explaining each ingredient’s purpose, where it comes from and how to use it in the dish (is it a salad or a garnish?). The galactic food can’t provide these value-add story points and only has fictitious ones.

In the case of the food served on the starcruiser, then, each dish has to stand on its own merits, without the usual restaurant storytelling elements that contribute to the overall sense that you are eating something expensive and worthy of that expense. Instead, each dish requires us to taste, smell, and feel the food in our mouths and decide if we liked it or not. I don’t think I’ve ever had to do that before.

World building – (decrepit futurism)

The world of Star Wars is one of decrepit futurism. It is a world of wonders in decline.

There are other kinds of futurism like the streamlined retro-futurism of the 30s and 50s or contemporary Afro-futurism. The decrepit futurism of Star Wars takes a utopic society and dirties it up, both aesthetically and morally. The original Star Wars starts off at the dissolution of the Senate marking a political decline. George Lucas doubles down on this in the prequels making this also a spiritual decline in which the Jedi are corrupted by a malignant influence and end up bringing about the fall of their own order. The story of the sequels (which is the period in which the galactic space voyage takes place) is about the difficultly and maybe impossibility of restoring the universe to its once great heights.

As beautiful and polished as all the surfaces are on the star cruiser, the ship is over 200 years old and has undergone massive renovations. Despite this, the engines continue to provide trouble (which you get to help fix). Meanwhile, the political situation in the galaxy in general and on the destination planet in particular is fraught, demanding that voyagers choose which set of storylines they will pursue. Will they help the resistance or be complicit with the First Order? Or will they opt out of this choice and instead be a Han Solo-like rogue pursuing profit amid the disorder?

The metaphysics of the Star Wars universe is essentially fallibilist and flawed – which in turn opens the way for moral growth and discovery.

The decrepit futurism of Star Wars has always seemed to me to be one of the things that makes it work best because it artfully dodges the question of why things aren’t better in a technologically advanced society. Decrepit futurism says that things once were (our preconceptions of what the future and progress entails is preserved) but have fallen from the state of grace through a Sith corruption. In falling short, the future comes down to the level where the rest of us live.

It’s also probably why Luke, in the last trilogy, never gets to be the sort of teacher we hoped he would be to Rey. The only notion we have of the greatness and wisdom of a true Jedi master comes from glimpses we get through Yoda in The Empire Strikes Back, but he is only able to achieve this level of wisdom by losing everything. Greatness in Star Wars is always something implied but never seen.

Storytelling (narrative as an organizing principle)

Much is made of storytelling and narrative in the world of immersive experiences. Some people talk as if immersion is simply a medium for storytelling – but I think it is the other way around. Immersion is created out of world building and design that distract us from our real lives. The third piece of immersion is storytelling.

But one thing I discovered on the Galactic Starcruiser is that the stories in an immersive experience don’t have to be all that great – they don’t have to have the depth of a Dostoevsky novel. Instead they can be at the level of a typical MMORPG. They can be as simple as go into the basement and kill rats to get more information. Hack a wall terminal to get a new mission. Follow the McGuffin to advance the storyline.

Narrative in an immersive experience is not magic. It’s just a way of organizing time and actions for people, much the way mathematical formulas organize the relationship between numbers or physics theorems organize the interactions of physical bodies. Narratives help us keep the thread while lots of other things are going on around us.

The main difficulty of a live theater narrative, like the one on the starcruiser, is that the multiple story lines have to work well together and work even if people are not always paying attention or even following multiple plots at the same time. Additionally, at some point, all of the storylines must converge. In this case, keeping things simple is probably the only way to go.

Crafting a narrative for immersive experiences, it seems to me, is a craft rather than an art. It doesn’t have to provide any revelations or tell us truths about ourselves. It just has to get people from one place in time to another.

The real art, of course, is that exercised by the actors who must tell these stories over and over and improvise when guests throw them a curve ball while keeping within the general outline of the overarching narrative. And being able to do this for 3 days at a time is a special gift.

Westworld vs the Metaverse (what is immersion)

Using the Galactic Starcruiser as the exemplar of an immersive experience, I wanted to go back to the question of how immersion in VR is different from immersion in reality. To put it another way, what is the difference between Westworld and the Metaverse?

There seems to be something people are after when they get excited about the Metaverse and I think it’s at bottom the ability to simulate a fantasy. Back when robots were all the rage (about the time Star Wars was originally made in the 70s) Michael Crichton captured this desire for fantasy in his film Westworld. The circle of reference is complete when one realizes that Chrichton based his robots on the animatronics at Disneyland and Disneyworld.

So what’s the difference between Westworld and the Metaverse? One of the complaints about the Metaverse (and more specifically VR) is that the lack of haptic feedback diminishes the experience. The real world, of course, is full of haptic feedback. More than this, it is also full of flavors and smells, which you cannot currently get from the Metaverse. It can also be full of people that can improvise around your personal choices so that the experience never glitches. This provides a more open world type of experience, whereas the Metaverse as it currently stands will have a lot of experiences on rails.

From all this, it seems as if the Metaverse aspires to be Westworld (or even the Galactic Starcruiser) but inevitably falls short sensuously and dynamically.

The outstanding thing about the Metaverse, though, is that it can be mass produced – precisely because it is digital and not real. The Starcruiser is prohibitively expensive dinner theater which I was able to pull off through some dumb luck with crypto currencies. It’s wonderful and if you can afford it I highly encourage you to go on that voyage into your childhood.

The Metaverse, on the other hand, is Westworld-style immersion for the masses. The bar to entry for VR is relatively low compared to a real immersive experience. Now all we have to do is get the world building, design, and storylines right.

XR FRAMEWORK WARS: MRTK VS XRTK VS UNITY XR SDK

May 29, 2020June 28, 2020 James Ashley

Thanks to the organizers of the AWE 2020 conference for letting me speak on the history and future of mixed reality and virtual reality frameworks on May 27. This was my first experience presenting at a completely online conference. It was a learning experience for me and extremely well managed by the organizers.

My VRLA Shopping List

May 10, 2018May 10, 2018 James Ashley1 Comment

I meant to finish this earlier in the week. I spent the past weekend in Los Angeles at the VRLA conference in order to hear Jasper Brekelmans speak about the state of the art in depth sensors and visual effects. One of the great things about VRLA is all the vendor booths you can visit that are directly related to VR and AR technology. Nary a data analytics platform pitch or dev ops consulting services shill in sight.

Walking around with Jasper, we started compiling a list of how we would spend our Bitcoin and Ethereum fortunes once they recover some of their value. What follows is my must-have shopping list if I had so much money I didn’t need anything:

1. Red Frog HoloLens mod

First off is this modified HoloLens by Red Frog Digital. The fabrication allows the HoloLens to balance much better on a user’s head. It also applies no pressure to the bridge of the nose, but instead distributes it across the user’s head. The nicest thing about it is that it always provides a perfect fit, and can be properly aligned with the user’s eyes in about 5 seconds. They designed this for their Zombie Maze location-based experience and are targeting it for large, permanent exhibits / rides.

2. Cleanbox … the future of wholesome fun

If you’ve ever spent a lot of time doing AR and VR demos at an event, you know there are three practical problems you have to work around:

seating devices properly on users’ heads
cleaning devices between use
recharging devices

Cleanbox Technology provides a solution for venue-based AR/VR device cleaning. Place your head-mounted display in the box, close the lid, and it instantly gets blasted with UV rays and air. I’d personally be happy just to have nice wooden boxes for all of my gear – I have a tendency to leave them lying on the floor or scattered across my computer desk – even without the UV lights.

3. VR Hamster Ball

The guy demoing this never seemed to let anyone try it, so I’m not sure if he was actually playing a hamster sim or not. I just know I want one as a 360 running-in-place controller … and as a private nap space, obviously.

4. Haptic Vest and Gauntlets

Bhaptics was demoing their TactSuit, which provides haptic feedback along the chest, back, arms and face. I’m going to need it to go with my giant hampster ball. They are currently selling dev units.

5. Birdly

A tilt table with an attached fan and a user control in the form of flapping wings is what you need for a really immersive VR experience. Fortunately, this is exactly what Birdly provides.

6. 5K Head-mounted Display

I got to try out the Vive Pro, which has an astounding 2K resolution. But I would rather put my unearned money down for a VRHero 5K VR headset with 170 degree FOV. They seem to be targeting industrial use cases rather than games, though, since their demo was of a truck simulation (you stood in the road as trucks zoomed by).

7. A globe display

Do I need a giant spherical display? No, I do not need it. But it would look really cool in my office as a conversation piece. It could also make a really great companion app for a VR or AR experience.

8. 360 Camera Rig with Red Epic Cameras

Five 6K Red Dragon Epic Cameras in a 360 video rig may seem like overkill, but with a starting price of around $250K, before tripod, lenses and a computer powerful enough to process your videos – this could make the killer raffle item at any hi-tech conference.

9. XSens Mocap Suit

According to Jasper, the XSens motion capture fullbody, lycra suit with realtime kinematics is one of the best available. I think I was quoted a price something like $7K(?) to $13K(?) Combined with my hamster ball, it would make me unstoppable in PvP competitive Minecraft.

10. AntVR AR Head-mounted display

AntVR will be launching a kickstarter campaign for their $500 augmented reality HMD in the next few weeks. I’d been reading about it for a while and was very excited to get a chance to try it out. It uses a Pepper’s ghost strategy for displaying AR, has decent tracking, works with Steam, and at $500 is very good for its price point.

11. Qualcomm Snapdragon 845

qualcomm

The new Qualcomm Snapdragon 845 Chip has built-in SLAM – meaning 6DOF inside-out tracking is now a trivial chip-based solution – unlike just two years ago when nobody outside of robotics had even heard of SLAM algorithms. This is a really big deal.

Lenovo is using this chip in its new (untethered) Mirage Solo VR device – which looks surprisingly like the Windows Occluded MR headset they built with Microsoft tracking tech. At the keynote, the Lenovo rep stumbled and said that they will support “at least” 6 degrees of freedom, which has now become an inside joke among VR and AR developers. It’s also spoiled me, because I am no longer satisfied with only 6DOF. I need 7DOF at least but what I really want is to take my DOF up to 11.

12. Kinect 4

This wasn’t actually at VRLA, and I’m not ultimately sure what it is (maybe a competitor for the Google computer vision kit?) but Kinect for Azure was announced at the /build conference in Seattle and should be coming out sometime in 2019. As a former Kinect MVP and a Kinect book author, this announcement mellows me out like a glass of Hennessy in a suddenly quiet dance club.

While I’m waiting for bitcoin to rebound, I’ll just leave this list up on Amazon for, like, in case anyone wants to fulfill it for me or something. In the off chance that that actually comes through, I can guarantee you a really awesome unboxing video.

Streaming and Recording VR Tracking Data

August 11, 2017August 11, 2017 James Ashley

Jasper Brekelmans is already pretty famous for creating Brekel Pro Tools, which turns the sensor streams from various 3D cameras, including the Kinect, into useful 3D data that can be used for 3D animations, visual effects and general research. OpenVR Recorder expands on this capability by also ingesting high quality data used for tracking virtual reality devices like the Oculus Rift and the HTC Vive.

The original Kinect, introduced in 2010, created a small revolution in the 3D sensor industry. It helped to drive down the price of 3D tracking and inspired new research into time-of-flight, structured light pattern and marker-based techniques that dramatically improved the accuracy with which we capture and understand 3D space. In turn, this has driven innovation in the area of interpreting 3D spaces using 2D cameras – which goes into solutions like Apple’s ARToolkit.

Though it may not be immediately evident, the current developments in VR and AR are built on top of these leaps and bounds in 3D sensor technology. The HoloLens uses it for position tracking and spatial mapping. The HoloLens solution is going into the SLAM-based position tracking tech for the Mixed Reality Headsets from HP and Acer. At the same time, VR tech from Oculus and HTC are constantly streaming 3D data in order to provide accurate tracking of both the user and handheld controllers.

Since this data is already flowing through the air around you when you are in VR space, there must be a way to capture it, right? That’s what Jasper Brekelmans has done with his new tool, OpenVR Recorder.

If you’re already working with the Kinect for visual effects, motion capture, or research, then this new software should be included as an essential part of your tool chest. There’s even a free trial available.

Black Mirror Explains the Difference between Virtual Reality and Augmented REality

November 28, 2016March 10, 2017 James Ashley

Why Augmented Reality is harder than Virtual Reality

September 9, 2015 James Ashley

hololensx519_0

At first blush, it seems like augmented reality should be easier than virtual reality. Whereas virtual reality involves the generation of full stereoscopic digital environments as well as interactive objects to place in those environments, augmented reality is simply adding digital content to our view of the real world. Virtual reality would seem to be doing more heavy lifting.

perspective

In actual fact, both technologies are creating illusions to fool the human eye and the human brain. In this effort, virtual reality has an easier task because it can shut out points of reference that would otherwise belie the illusion. Augmented reality experiences, by contrast, must contend with real world visual cues that draw attention to the false nature of the mixed reality content being added to a user’s field of view.

In this post, I will cover some of the additional challenges that make augmented reality much more difficult to get right. In the process, I hope to also provide clues as to why augmented reality HMDs like HoloLens and Magic Leap are taking much longer to bring to market than AR devices like the Oculus Rift, HTC Vive and Sony Project Morpheus.

terminator vision

But first, it is necessary to distinguish between two different kinds of augmented reality experience. One is informatics based and is supported by most smart phones with cameras. The ideal example of this type of AR is the Terminator-vision from James Cameron’s 1984 film “The Terminator.” It is relatively easy to to do and is the most common kind of AR people encounter today.

star wars chess

The second, and more interesting, kind of AR requires inserting illusory 3D digital objects (rather than informatics) into the world. The battle chess game from 1977’s “Star Wars” epitomizes this second category of augmented reality experience. This is extremely difficult to do.

The Microsoft HoloLens and Magic Leap (as well as any possible HMDs Apple and others might be working on) are attempts to bring both the easy type and the hard type of AR experience to consumers.

Here are a few things that make this difficult to get right. We’ll put aside stereoscopy which has already been solved effectively in all the VR devices we will see coming out in early 2016.

cloaked predator

1. Occlusion The human brain is constantly picking up clues from the world in order to determine the relative positions of objects such as shading, relative size and perspective. Occlusion is one that is somewhat tricky to solve. Occlusion is an effect that is so obvious that it’s hard to realize it is a visual cue. When one body is in our line of sight and is positioned in front of another body, that other body is partially hidden from our view.

In the case where a real world object is in front of a digital object, we can clip the digital object with an outline of the object in front to prevent bleed through. When we try to create the illusion that a digital object is positioned in front of a real world object, however, we encounter a problem inherent to AR.

In a typical AR HMD we see the real world through a transparent screen upon which digital content is either projected or, alternatively, illuminated as with LED displays. An obvious characteristic of this is that digital objects on a transparent display are themselves semi-transparent. Getting around this issue would seem to require being able to make certain portions of the transparent display more opaque than others as needed in order to make sure our AR objects look substantial and not ghostly.

citizen kane

2. Accommodation It turns out that stereoscopy is not the only way our eyes recognize distance. The image above is from a scene in Orson Welles’s “Citizen Kane” in which a technique called “deep focus” is used extensively. Deep focus maintains clarity in the frame whether the actors and props are in the foreground, background or middle ground. Nothing is out of focus. The technique is startling both because it is counter to the way movies are generally shot but also because it is counter to how our eyes work.

If you cover one eye and use the other to look at one of your fingers, then move the finger toward and away from you, you should notice yourself refocusing on the finger as it moves while other objects around the finger become blurry. The shape of the cornea actually becomes more rounded when objects are close in order to cause light to refract more in order to reach the retina. For further away objects, the cornea flattens out because less refraction is needed. As we become older, the ability to bow the cornea lessens and we lose some of our ability to focus on near objects – for instance when we read. In AR, we are attempting to make a digital object that is really only centimeters from our eyes appear to be much further away.

Depending on how the light from the display passes through the eye, we may end up with the digital object appearing clear while the real world objects supposedly next to it and at the same distance appear blurred.

vergence_accomodation

3. Vergence-Accommodation Mismatch The accommodation problem is one aspect of yet another VR/AR difficulty. The term vergence describes the convergence and divergence of the two eyes from one another as objects move closer or further away. An interesting aspect of stereoscopy – which is used both for virtual reality as well as augmented reality to create the illusion of depth – is that the distance at which the two eyes coordinate to see an object is generally different from the focal distance from the eyes to the display screen(s). This consequently sends two mismatched signals to the brain concerning how far away the digital object is supposed to be. Is it the focal length or the vergence length? Among other causes, vergence-accommodation mismatch is believed to be a contributing factor to VR sickness. Should the accommodation problem above be resolved for a given AR device, it is safe to assume that the vergence-accommodation mismatch will also be solved.

4. Tetherless Battery Life Smart phones have changed our lives among other reasons because they are tetherless devices. While the current slate of VR devices all leverage powerful computers to which they are attached, since VR experiences are all currently somewhat stationary (the HTC Vive being the odd bird), AR needs to be portable. This naturally puts a strain on the battery, which needs to be relatively light since it will be attached to the head-mounted-display, but also long-lived as it will be powering occasionally intensive graphics, especially for games.

5. Tetherless GPU Another strain on the system is the capability of the GPU. Virtual reality devices can be fairly intense since they require the user to purchase a reasonably powerful and somewhat expensive graphics card. AR devices can be expected to have similar graphics requirements as VR with much less to work with since the GPU needs to be onboard. We can probably expect a streamlined graphics pipeline dedicated to and optimized for AR experiences will help offset lower GPU capabilities.

6. Applications Not even talking about killer apps, here. Just apps. Microsoft has released videos of several impressive demos including Minecraft for HoloLens. Magic Leap up to this point has only shown post-prod, heavily produced illustrative videos. The truth is that everyone is still trying to get their heads around designing for AR. There aren’t really any guidelines for how to do it or even what interactions will work. Other than the most trivial experiences (e.g. weather and clock widgets projected on a wall) this will take a while as we develop best practices while also learning from our mistakes.

Conclusion

With the exception of V-AM, these are all problems that VR does not have to deal with. Is it any wonder, then, that while we are being led to believe that consumer models of the Oculus Rift, HTC Vive and Sony Project Morpheus will come to market in the first quarter of 2016, news about HoloLens and Magic Leap has been much more muted. There is simply much more to get right before a general rollout. One can hope, however, that dev units will start going out soon from the major AR players in order to mitigate challenge #6 while further tuning continues, if needed, on challenges #1-#5.

The HoloCoder’s Resume

July 12, 2015June 24, 2017 James Ashley

In an ideal world, the resume is an advertisement for our capabilities and the interview process is an audit of those claims. Many factors have contributed to complicating what should be a simple process.

The first is the rise of professional IT recruiters and the automation of the resume process. Recruiters bring a lot to the game, offering a wider selection of IT job candidates to hiring companies, on the one hand, and providing a wider selection of jobs to job hunters, on the other. Automation requires standardization, however, and this has led to an overuse of key search terms when matching candidates to positions. The process begins with job specs from the hiring company — which parenthetically often have little to do with the actual job itself and highlights the frequent disconnect between IT departments and HR departments. A naive job hunter would try to describe their actual experience, which typically will not match the job spec as written by HR. At this point the recruiter helps the job hunter modify the details of her resume to match the template provided by the hiring company by injecting and prioritizing key buzzwords into the resume. “I’m sorry but Lolita, Inc will never hire you unless you have synesthesia listed in your job history. You do have experience with synesthesia, don’t you?”

All of this gerrymandering is required in order to get to the next step, the job interview. Unfortunately, the people doing the job interview have little confidence in the resume as a vehicle for accurately describing a candidate’s actually abilities. First of all, they know that recruiters have already gone over it to eliminate useful information and replace it with keywords instead. Next, the interviewers typically haven’t actually seen the HR job specs and do not understand what kind of role they are hiring for. Finally, none of the interviewers have any particular training in doing job interviews or any particular skill in ascertaining what a candidate knows. In short, the interviewer doesn’t know what he’s looking for and wouldn’t know how to get it if he did.

A savvy interviewer will probably realize that he is looking for the sort of generalist that Joel Spolsky describes as “smart and gets things done,” but how do you interview for that? The tools the interviewer is provided with are not generic but instead highly specific technology skills. At some point, this impedance mismatch between technology specific interview questions on the one had and a desire to hire generalists on the other (technology, after all, simply changes too quickly to look for only one skillset) lead to an increased reliance on behavioral questions and eventually Google-style language games. Neither of these, it turns, out, particularly help in hiring good candidates.

Once we historically severed any attempt to match interview questions to actual skills, the IT interview process was allowed to become a free floating hermeneutic exercise. Abstruse but non-specific questions involving principles and design patterns have taken over the process. This has led to two strange outcomes. On the one hand, job applicants are now required to be fluent in technical information they will never actually use in their jobs. Literary awareness of ten year old blog posts by Martin Fowler are more important than actually knowing how to get things done. And if the job interviewer exhibits any self-awareness when he turns down a candidate for not being clear on the justified uses of the CQRS pattern (there are none), it will not be because the candidate didn’t know something important for the job but rather because the candidate was unwilling to play the software architecture language game, and anyone unwilling to play the game is likely going to be a poor cultural fit.

The other consequence of an increased reliance on abstruse and non-essential IT knowledge has been the rise of the Architect across the industry. The IT industry has created a class of software developers who cannot actually develop software but instead specialize in telling other people what is wrong with their code. The architect is a specialization that probably indicates a deviant phase in the software industry – but at the same time it is a natural outcome of our IT job spec – resume – interview process. The skills of a modern software architect – knowledge of abstruse information and jargon often combined with an inability to get things done – is what we currently look for through our IT cargo cult hiring rituals.

This distinction between the ritual of IT hiring and the actual goals of IT hiring become most apparent when we look for specific as opposed to generalist skills. We hire generalists to be on staff over a long period. We hire specialists to perform difficult but real tasks that can eventually be handed over to our generalists – when we need to get something specific done.

Which gets us to the point of this post. What are the skills we should look for when hiring for a HoloLens developer? And what are the skills a HoloLens developer should be highlighting on her resume?

At this point in time, when there is still no SDK generally available for the HoloLens and all HoloLens coders are working for Microsoft and under various NDAs, it is hard to say. Fortunately, important clues have been provided by the recent announcement of the first consulting agency dedicated to the HoloLens and co-founded by someone who has been working on HoloLens applications for Microsoft over the past year. The company Object Theory was just started by Michael Hoffman and Raven Zachary and they threw up a website to advertise this new venture.

Among the tasks involved in creating this sort of extremely specialized website is explaining what capabilities you offer. First, they offer experience since Hoffman has worked on several of the demos that Microsoft has been exhibiting at conferences and in promotional videos. But is this enough of a differentiator? What skills do they have to offer to a company looking to build a HoloLens application?

This is part of the fascination of their “Work” page. It cannot describe any actual work since the company just started and hasn’t technically done any technical work. Instead, it provides a list of capabilities that look amazingly like resume keywords – but different from any keywords you may have come across:

- - - - Entirely new Natural User Interfaces (NUI)
      - Surface reconstruction and object persistence
      - 3D Spatial HRTF audio
      - Mesh reduction, culling and optimization
      - Baked shadows and ambient occlusion
      - UV mapping
      - Optimized render shaders
      - Efficient WiFi connectivity to back-end services
      - Unity and DirectX
      - Windows 10 APIs

These, in fact, are probably the sorts of skills you should be putting on your resume – or learning about in order to put on your resume – if getting a job programming HoloLens is your goal.

The verso side of this coin is that the list can also be turned into a great set of interview questions for someone thinking of hiring for HoloLens development, for instance:

Explain the concept of NUI to me.

Tell me about your experience with surface reconstruction and object persistence.

What is 3D spatial HRTF audio and why is it important for engineering HoloLens apps?

What are mesh reduction, mesh culling and mesh optimization?

Do you know anything about baked shadows and ambient occlusion?

Describe how you would go about performing UV mapping.

What are optimized render shaders and when would you need them?

How does the HoloLens communicate with external services such as a database?

What are the advantages and disadvantages of developing in Unity vs DirectX?

Describe the Windows 10 APIs that are used in HoloLens application development.

Then again, maybe these questions are a bit too abstruse?

HoloLens App Development with Unity

June 25, 2015 James Ashley

A few months ago I wrote a speculative piece about how HoloLens might work with XAML frameworks based on the sample applications Microsoft had been showing.

Even though Microsoft has still released scant information about integration with 3D platforms, I believe I can provide a fairly accurate walkthrough of how HoloLens development will occur for Unity3D. In fact, assuming I am correct, you can begin developing games and applications today and be in a position to release a HoloLens experience shortly after the hardware becomes available.

To be clear, though, this is just speculative and I have no insider information about the final product that I can talk about. This is just what makes sense based on publicly available information regarding HoloLens.

Unity3D integration with third party tools such as Kinect and Oculus Rift occurs through plugins. The Kinect 2 plugin can be somewhat complex as it introduces components that are unique to the Kinect’s capabilities.

The eventual HoloLens plugin, on the other hand, will likely be relatively simple since it will almost certainly be based on a pre-existing component called the FPSController (in Unity 5.1 which is currently the latest).

To prepare for HoloLens, you should start by building your experience with Unity 5.1 and the FPSController component. Here’s a quick rundown of how to do this.

Start by installing the totally free Unity 5.1 tools: http://unity3d.com/get-unity/download?ref=personal

Next, create a new project and select 3D for the project type.

Click the button for adding asset packages and select Characters. This will give you access to the FPSController. Click done and continue. The IDE will now open with an practically empty project.

At this point, a good Unity3D tutorial will typically show you how to create an environment. We’re going to take a shortcut, however, and just get a free one from the Asset Store. Hit Ctrl+9 to open the Asset Store from inside your IDE. You may need to sign in with your Unity account. Select the 3D Models | Environments menu option on the right and pick a pre-built environment to download. There are plenty of great free ones to choose from. For this walkthrough, I’m going to use the Japanese Otaku City by Zenrin Co, Ltd.

After downloading is complete, you will be presented with an import dialog box. By default, all assets are selected. Click on Import.

Now that the environment you selected has been imported, go the the scenes folder in your project window and select a sample scene from the downloaded environment. This will open up the city or dungeon or forest or whatever environment you chose. It will also make all the different assets and components associated with the scene show up in your Hierarchy window. At this point, we want to add the first-person shooter controller into the scene. You do this by selecting the FPSController from the project window under Assets/Standard Assets/Characters/FirstPersonCharacter/Prefabs and dragging the FPSController into your Hierarchy pane.

This puts a visual representation of the FPS controller into your scene. Select the controller with your mouse and hit “F” to zoom in on it. You can see from the visual representation that the FPS controller is basically a collision field that can be moved with a keyboard or gamepad that additionally has a directional camera component and a sound component attached. The direction the camera faces ultimately become the view that players see when you start the game.

Here is another scene that uses the Decrepit Dungeon environment package by Prodigious Creations and the FPS controller. The top pane shows a design view while the bottom pane shows the gamer’s first-person view.

You can even start walking through the scene inside the IDE by simply selecting the blue play button at the top center of the IDE.

The way I imagine the HoloLens integration to work is that another version of FPS controller will be provided that replaces mouse controller input with gyroscope/magnetometer input as the player rotates her head. Additionally, the single camera view will be replaced with a two camera rig that sends two different, side-by-side feeds back to the HoloLens device. Finally, you should be able to see how all of this works directly in the IDE like so:

There is very good evidence that the HoloLens plugin will work something like I have outlined and will be approximately this easy. The training sessions at the Holographic Academy during /Build pretty much demonstrated this sort of toolchain. Moreover, this is how Unity3D currently integrates with virtual reality devices like Gear VR and Oculus Rift. In fact, the screen cap of the Unity IDE above is from an Oculus game I’ve been working on.

So what are you waiting for? You pretty much have everything you already need to start building complex HoloLens experiences. The integration itself, when it is ready, should be fairly trivial and much of the difficult programming will be taken care of for you.

I’m looking forward to seeing all the amazing experiences people are building for the HoloLens launch day. Together, we’ll change the future of personal computing!

Marshall McLuhan and Understanding Digital Reality

June 23, 2015 James Ashley

Understanding McLuhan

While slumming on the internet looking for new content about digital media I came across this promising article entitled Virtual Reality, Augmented Reality and Application Development. I was feeling hopeful about it until I came across this peculiar statement:

“Of the two technologies, augmented reality has so far been seen as the more viable choice…”

What a strange thing to write. Would we ever ask whether the keyboard or the mouse is the more viable choice? The knife or the fork? Paper or plastic? It should be clear by now that this is a false choice and not a case of having your cake or eating it, too. We all know that the cake is a lie.

But this corporate blog post was admittedly not unique in creating a false choice between virtual reality and augmented reality. I’ve come across this before and it occurred to me that this might be an instance of a category mistake. A category mistake is itself a category of philosophical error identified by the philosopher Gilbert Ryle to tackle the sticky problem of Cartesian dualism. He pointed out that even though it is generally accepted in the modern world that mind is not truly a separate substance from mind but is in fact a formation that emerges in some fashion out of the structure of our brains, we nevertheless continue to divide the things of the world, almost as if by accident, into two categories: mental stuff and material stuff.

sony betamax

There are certainly cases of competing technologies where one eventually dies off. The most commonly cited example is the Betamax and VHS. Of course, they both ultimately died off and it is meaningless today to claim that either one really succeeded. There are many many more examples of apparently technological duels in which neither party ultimately falls or concedes defeat. PC versus Mac. IE vs Chrome. NHibernate vs EF. etc.

The rare case is when one technology completely dominates a product category. The few cases where this has happened, however, have so captured our imaginations that we forget it is the exception and not the rule. This is the case with category busters like the iPhone and the iPad – brands that are so powerful it has taken years for competitors to even come up with viable alternatives.

What this highlights is that, typically, technology is not a zero sum game. The norm in technology is that competition is good and leads to improvements across the board. Competition can grow an entire product category. The underlying lie, however, is perhaps that each competitor tells itself that they are in a fight to the death and that they are the next iPhone. This is rarely the case. The lie beneath that lie is that each competitor is hoping to be bought out by another larger company for billions of dollars and has to look competitive up until that happens. A case of having your cake and eating it, too.

stop_staring

There is, however, a category in which one set of products regularly displace another set of products. This happens in the fashion world.

Each season, each year, we change out our cuts, our colors and accessories. We put away last year’s fashions and wouldn’t be caught dead in them. We don’t understand how these fashion changes occur or what rules they obey but the fashion houses all seem to conform to these unwritten rules of the season and bring us similar new things at the proper time.

This is the category mistake that people make when they ask things such as which is more viable: augmented reality or virtual reality? Such questions belong to the category of fashion (which is in season: earth tones or pastels?) and not to technology. In the few unusual cases where this does happen, then the category mistake is clearly in the opposite direction. The iPhone and iPad are not technologies: they are fashion statements.

Virtual reality and augmented reality are not fashion statements. They aren’t even technologies in the way we commonly talk about technology today – they are not software platforms (though they require SDKs), they are not hardware (though they are useless without hardware), they are not development tools (you need 3D modeling tools and game engines for this). In fact, they have more in common with books, radio, movies and television than they do to software. They are new media.

A medium, etymologically speaking, is the thing in the middle. It is a conduit from a source to a receiver – from one world to another. A medium lets us see or hear things we would otherwise not have access to. Books allow us to hear the words of people long dead. Radio transmits words over vast distances. Movies and television let us see things that other people want us to see and we pay for the right to see those things. Augmented reality and virtual reality, similarly, are conduits for new content. They allow us to see and hear things in ways we haven’t experienced content before.

The moment we cross over from talking about technology and realize we are talking about media, we automatically invoke the spirit of Marshall McLuhan, the author of Understanding Media: The Extensions of Man. McLuhan thought deeply about the function of media in culture and many of his ideas and aphorisms, such as “the medium is the message,” have become mainstays of contemporary discourse. Other concepts that were central to McLuhan’s thought still elude us and continue to be debated. Among these are his two media categories: hot and cold.

understanding media

McLuhan claimed that any media is either hot or cold, warm or cool. Cool mostly means what we think it means metaphorically; for instance, James Dean is cool in exactly the way McLuhan meant. Hot media, in turn, is in most ways what you would think it is: kinetic with a tendency to overwhelm the senses. To illustrate what he meant by hot and cold, McLuhan often provides contrasting examples. Movies are a hot medium. Television is a cold medium. Jazz is a hot medium. The twist is a cool medium. Cool media leave gaps that the observer must fill in. It is highly participatory. Hot media is a wall of sensation that does not require any filling in: McLuhan characterizes it as “high definition.”

I think it is pretty clear, between virtual reality and augmented reality, which falls into the category of a cool medium and which a hot one.

To help you come to your own conclusions about how to categorize augmented reality glasses and the virtual reality goggles, though, I’ll provide a few clues from Understanding Media:

“In terms of the theme of media hot and cold, backward countries are cool, and we are hot. The ‘city slicker’ is hot, and the rustic is cool. But in terms of the reversal of procedures and values in the electric age, the past mechanical time was hot, and we of the the TV age are cool. The waltz was hot, fast mechanical dance suited to the industrial time in its moods of pomp and circumstance.”

“Any hot medium allows of less participation than a cool one, as a lecture makes for less participation than a seminar, and a book for less than dialogue. With print many earlier forms were excluded from life and art, and many were given strange new intensity. But our own time is crowded with examples of the principle that the hot form excludes, and the cool one includes.”

“The principle that distinguishes hot and cold media is perfectly embodied in the folk wisdom: ‘Men seldom make passes at girls who wear glasses.’ Glasses intensify the outward-going vision, and fill in the feminine image exceedingly, Marion the Librarian notwithstanding. Dark glasses, on the other hand, create the inscrutable and inaccessible image that invites a great deal of participation and completion.”

audrey hepburn glasses

Minecraft in Virtual Reality and Augmented Reality

June 17, 2015 James Ashley

Microsoft recently created possibly the best demo they have ever done on stage for E3. Microsoft employees played Minecraft in a way no one has ever seen it before, on a table top as if it was a set of legos. Many people speculated on social media that this may be the killer app that HoloLens has been looking for.

What is particularly exciting about the way the demo captured people’s imaginations is that they can start envisioning what AR might actually be used for. People are even getting a firm grip on the differences between Virtual Reality, which creates an immersive experience, and augmented reality which creates a mixed experience overlapping digital objects with real world objects.

Nevertheless, there is still a tendency to see virtual reality exemplified by the Oculus Rift and augmented reality exemplified by HoloLens and Magic Leap as competing solutions. In fact they are complementary solutions. They don’t compete with one another any more than your mouse and your keyboard do.

Bill Buxton has famously said that everything is best for something and worst for something else. By contrasting the Minecraft experience for Oculus and HoloLens, we can better see what each technology is best at.

The Virtual Reality experience for Oculus is made possible by a free hacking effort called Minecrift. It highlights the core UX flavor of almost all VR experiences – they are first person, with the player fully present in a 3D virtual world. VR is great for playing Minecraft in adventure or survival mode.

Adventure mode with HoloLens is roughly equivalent to the adventure mode we get today on a PC or XBOX console with the added benefit that the display can be projected on any wall. It isn’t actually 3D, though, as far as we can tell from the demo, despite the capability of displaying stereoscopic scenes with HoloLens.

What does work well, however, is Minecraft in creation mode. This is basically the god view we have become familiar with from various strategy and resource games over the years.

God View vs First Person View

In a fairly straightforward way, it makes sense to say that AR is best for a god-centric view while VR is best for a first-person view. For instance, if we wanted to create a simulation that allows users to fly a drone or manipulate an undersea robot, virtual reality seems like the best tool for the job. When we need to create a synoptic view of a building or even a city, on the other hand, then augmented reality may be the best UX. Would it be fair to say that all new UX experiences fall into one of these two categories?

Most of our metaphors for building software, building businesses and building every other kind of buildable thing, after all, are based on the lego building block and it’s precursors the Lincoln log and erector sets. We play games as children in order, in part, to prepare ourselves for thinking as adults. Minecraft was built similarly on the idea of creating a simulation of a lego block world that we could not only build but also virtually play in on the computer.

lego_astronaut

The playful world of Lego blocks is built on two things: the blocks themselves formed into buildings and scenes and the characters that we identify with who live inside the world of blocks. In other words the god-view and the first-person view.

coffee prince

It should come as no surprise, then, that these two core modes of our imaginative lives should stay with us through our childhoods and into our adult approaches to the world. We have both an interpersonal side and an abstract, calculating side. The best leaders have a bit of both.

You apparently didn’t put one of the new coversheets on your TPS report

The god-view in business tends to be the synoptic view demanded by corporate executives and provided in the form of dashboards or crystal reports. It would be a shame if AR ended up falling into that use-case when it can provide so much more and in more interesting ways. As both VR and AR mature over the next five years, we all have a responsibility to keep them anchored in the games of our childhood and avoid letting them become the faults and misdemeanors of the corporate adult world.

Update 6/20

A recent arstechnica article indicates that the wall-projected HoloLens version of Minecraft in adventure mode can be played in true 3D:

One other impressive feature of the HoloLens-powered virtual screen was the ability to activate a three-dimensional image, so that the scene seemed to recede into the wall like a window box. Unlike a standard 3D monitor, this 3D image actually changed perspective based on the viewing angle. If I went up near the wall and looked at the screen from the left, I could see parts of the world that would usually be behind the right side of the wall, as if the screen was simply a window into another world.