On The Gaze as an input device for Hololens

May 8, 2015 James Ashley

microsoft-hololens-build-anatomy

The Kinect sensor and other NUI devices have introduced an array of newish interaction patterns between humans and computers: tap, touch, speech, finger tracking, body gestures. The hololens provides us with a new method of interaction that hasn’t been covered extensively from a UX perspective before: The Gaze. Hololens performs eye tracking in order to aid users of the AR device to activate menus.

Questions immediately arise as to the role this will play in surveillance culture, and even more in the surveillance of surveillance culture. While sensors track our gaze, will they similarly inform us about the gaze of others? Will we one day receive notifications that someone is checking us out? Quis custodiet ipsos custodes? To the eternal question who watches the watchers, we finally have an answer. HoloLens does.

lacan gaze

Even though The Gaze has not been analyzed deeply from a UX perspective, it has been the object of profound study from a phenomenological and a structuralist point of view. In this post I want to introduce you to five philosophical treatments of The Gaze covering the psychological, the social, the cinematic, the ethical and the romantic. To start, the diagram above is not from an HCI book as one might reasonably assume but rather from a monograph by the psychoanalyst Jacques Lacan.

the_ambassadors

A distinction is often drawn between Lacan’s early studies of The Gaze and his later conclusions about it. The early work relates it to a “mirror stage” of self-awareness and concerns the gaze when directed to ourselves rather than to others:

“This event can take place … from the age of six months on; its repetition has often given me pause to reflect upon the striking spectacle of a nursling in front of a mirror who has not yet mastered walking or even standing, but who … overcomes, in a flutter of jubilant activity, the constraints of his prop in order to adopt a slightly leaning-forward position and take in an instantaneous view of the image in order to fix it in his mind.”

This notion flowered in the later work The Split Between the Eye and the Gaze into a theory of narcissism in which the subject sees himself/herself as an objet petit a (a technical term for an object of desire) through the distancing effect of the gaze. Through this distancing, the subject also become alienated from itself. What is probably essential for us in this work – as students of emerging technologies – is the notion that the human gaze is emotionally distancing. This observation was later taken up in post-colonial theory as the “Imperial Gaze” and in feminist theory as “objectification.”

Michel Foucault achieved fame as a champion of the constructivist interpretation of truth but it is often forgotten that he was also an historian of science. A major theme in his work is the way in which the gaze of the other affects and shapes us – in particular the “scientific gaze.” Being watched causes us discomfort and we change our behavior – sometimes even our perception of who we are – in response to it. The grand image Foucault raises to encapsulate this notion is Jeremy Bentham’s Panopticon, a circular prison in which everyone is watched by everyone else.

THE HOUSEHOLD OF PHILIP IV or LAS MENINAS by Juan Bautista Martinez del Mazo (c1612-15-1667) after Diego Velazquez (1599 - 1660), at Kingston Lacy, Dorset

Where Lacan concentrates on the self-gaze and Foucault on the way the gaze makes us feel, Slovoj Zizek is concerned with the appearance of The Gaze when we gaze back at it. He writes in an essay called “Why Does the Phaullus Appear” from the collection Enjoy Your Symptom:

Shane Walsh (Jon Bernthal) - The Walking Dead - Season 2, Episode 12 - Photo Credit: Gene Page/AMC

“Let us take the ‘phantom of the opera,’ undoubtedly mass culture’s most renowned specter, which has kept the popular imagination occupied from Gaston Leroux’s novel at the turn of the century through a series of movie and television versions up to the recent triumphant musical: in what consists, on a closer look, the repulsive horror of his face? The features which define it are four:

“1) the eyes: ‘his eyes are so deep that you can hardly see the fixed pupils. All you can see is two big black holes, as in a dead man’s skull.’ To a connoisseur of Alfred Hitchcock, this image instantly recalls The Birds, namely the corpse with the pecked-out eyes upon which Mitch’s mother (Jessica Tandy) stumbles in a lonely farmhouse, its sight causing her to emit a silent scream. When, occasionally, we do catch the sparkle of these eyes, they seem like two candles lit deep within the head, perceivable only in the dark: these two lights somehow at odds with the head’s surface, like lanterns burning at night in a lonely, abandoned house, are responsible for the uncanny effect of the ‘living dead.’”

Obviously whatever Zizek says about the phantom of the opera applies equally well to The Walking Dead. What ultimately distinguishes vampires, zombies, demons and ghosts lies in the way they gaze at us.

While Zizek finds in the eyes a locus for inhumanity, the ethicist Emmanual Levinas believes this is where our humanity resides. These two notions actually complement each other, since what Zizek indentifies as disturbing is the inability to project a human mind behind a vacant stare. As Levinas says in a difficult and metaphysical way in his masterpiece Totality and Infinity:

“The presentation of the face, expression, does not disclose an inward world previously closed, adding thus a new region to comprehend or to take over. On the contrary, it calls to me above and beyond the given that speech already puts in common among us…. The third party looks at me in the eyes of the Other – language is justice. It is not that there first would be the face, and then the being it manifests or expresses would concern himself with justice; the epiphany of the face qua face opens humanity…. Like a shunt every social relation leads back to the presentation of the other to the same without the intermediary of any image or sign, solely by the expression of the face.”

The face and the gaze of the other implies a demand upon us. For Levinas, unlike Foucault, this demand isn’t simply a demand to behave according to norms but more broadly posits an existential command. The face of the other asks us implicitly to do the right thing: it demands justice.

new_optics_06

The final aspect of the gaze to be discussed – though probably the first aspect to occur to the reader – is the gaze of love, i.e. love at first sight. This was a particular interest of the scholar Ioan P. Couliano. In his book Eros and Magic in the Renaissance Couliano examines old medical theories about falling in love and cures for infatuation and obsession. He relates this to Renaissance theories about pneuma [spiritus, phantasma], which was believed to be a pervasive fluid that allowed objects to be sensed through apparently empty air and become transmitted to the brain and the heart. In this regard, Couliano raises a question that would only make sense to a true Renaissance man: “How does a woman, who is so big, penetrate the eyes, which are so small?” He quotes the 13th century Bernard of Gordon:

“The illness called ‘hereos’ is melancholy anguish caused by love for a woman. The ‘cause’ of this affliction lies in the corruption of the faculty to evaluate, due to a figure and a face that have made a very strong impression. When a man is in love with a woman, he thinks exaggeratedly of her figure, her face, her behavior, believing her to be the most beautiful, the most worthy of respect, the most extraordinary with the best build in body and soul, that there can be. This is why he desires her passionately, forgetting all sense of proportion and common sense, and thinks that, if he could satisfy his desire, he would be happy. To so great an extent is his judgment distorted that he constantly thinks of the woman’s figure and abandons all his activities so that, if someone speaks to him, he hardly hears him.”

And here is Couliano’s gloss of Bernard’s text:

RokebyVenus

“If we closely examine Bernard of Gordon’s long description of ‘amor hereos,’ we observe that it deals with a phantasmic infection finding expression in the subject’s melancholic wasting away, except for the eyes. Why are the eyes excepted? Because the very image of the woman has entered the spirit through the eyes and, through the optic nerve, has been transmitted to the sensory spirit that forms common sense…. If the eyes do not partake of the organism’s general decay, it is because the spirit uses those corporeal apertures to try to reestablish contact with the object that was converted into the obsessing phantasm: the woman.”

[As an apology and a warning, I want to draw your attention to the use of ocular vocabulary such as “perspective,” “point of view,” “in this regard,” etc. Ocular phrases are so pervasive in the English language that it is nearly impossible to avoid them and it would be more awkward to try to do so than it is to use them without comment. If you intend to speak about visual imagery, take my advice and pun proudly and without apology – for you will see that you have no real choice in the matter.]

HoloCoding resources from Build 2015

May 1, 2015 James Ashley

darren!

The Hololens team has stayed extremely quiet over the past 100 days in order to have a greater impact at Build. Alex Kipman was the cleanup batter on the first day keynote at Build with an amazing overview of realistic Hololens scenarios. This was followed by Hololens demos as well as private tutorials on using Hololens with a version of Unity 3D. Finally there were sessions on Hololens and a pre-recorded session on using the Kinect with the brand new RoomAlive Toolkit.

darren?

Here are some useful links:

Day One Keynote (fast forward to 2:28:20)
Developing for Hololens with Alex Kipman
Case Studies of Hololens App Development
Hacking Augmented Reality with Kinect with Ben Lower (Kinect) and Andy Wilson (MSR IllumiRoom)

There were two things I found particularly interesting in Alex Kipman’s first day keynote presentation.

The first was the ability of the onstage video to actually capture what was being shown through the hololens but from a different perspective. The third person view of what the person wearing the hololens even worked when the camera moved around the room. Was this just brilliant After Effects work perfectly synced to the action onstage? Or were we seeing a hololens-enabled camera at work? If the latter – this might be even more impressive than the hololens itself.

mission_impossible_five

Second, when demonstrating the ability to pin movies to the wall using Hololens gestures, why was the new Mission Impossible trailer used for the example? Wouldn’t something from, say, The Matrix been much more appropriate.

Perhaps it was just a licensing issue, but I like to think there was a subtle nod to the inexplicable and indirect role Tom Cruise has played in the advancement of Microsoft’s holo-technologies. Minority Report and the image of Cruise wearing biking gloves with his arms raised in the air, conductor-like, was the single most powerful image invoked with Microsoft first introduced the Kinect sensor. As most people know by now, Alex Kipman was the man responsible not only for carrying the Kinect nee Natal Project to success, but now for guiding the development of the Hololens. Perhaps showing Tom Cruise onstage at Build was a subtle nod to this implicit relationship.

The HoloCoder’s Bookshelf

April 30, 2015 James Ashley

Professions are held together by touchstones such as as a common jargon that both excludes outsiders and reinforces the sense of inclusion among insiders based on mastery of the jargon. On this level, software development has managed to surpass more traditional practices such as medicine, law or business in its ability to generate new vocabulary and maintain a sense that those who lack competence in using the jargon simply lack competence. Perhaps it is part and parcel with new fields such as software development that even practitioners of the common jargon do not always understand each other or agree on what the terms of their profession mean. Stack Overflow, in many cases, serves merely as a giant professional dictionary in progress as developers argue over what they mean by de-coupling, separation of concerns, pragmatism, architecture, elegance, and code smell.

Cultures, unlike professions, are held together not only by jargon but also by shared ideas and philosophies that delineate what is important to the tribe and what is not. Between a profession and a culture, the members of a professional culture, in turn, share a common imaginative world that allows them to discuss shared concepts in the same way that other people might discuss their favorite TV shows.

This post is an experiment to see what the shared library of augmented reality and virtual reality developers might one day look like. Digital reality development is a profession that currently does not really exist but which is already being predicted to be a multi-billion dollar industry by 2020.

HoloCoding, in other words, is a profession that exists only virtually for now. As a profession, it will envelop concerns much greater than those considered by today’s software developers. Whereas contemporary software development is mostly about collecting data, reporting on data and moving data from point A to points B and C, spatial software development will be more concerned with environments and will have to draw on complex mathematics as well as design and experiential psychology. The bookshelf of a holocoder will look remarkably different from that of a modern data coder. Here are a few ideas regarding what I would expect to find on a future developer’s bookshelf in five to ten years.

1. Understanding Media by Marshall McLuhan – written in the 60’s and responsible for concepts such as ‘the global village’ and hot versus cool media, McLuhan pioneered the field of media theory. Because AR and VR are essentially new media, this book is required reading for understanding how these technologies stand side-by-side with or perhaps will supplant older media.

2. Illuminations by Walter Benjamin – while the whole work is great, the essay ‘The Work of Art in the Age of Mechanical Reproduction’ is a must read for discussing how traditional notions about creativity fit into the modern world of print and now digital reproduction (which Benjamin did not even know about). It also deals at an advanced level with how human interactions work on stage versus film and the strange effect this creates.

3. Sketching User Experiences by Bill Buxton – this classic was quickly adopted by web designers when it came out. What is sometimes forgotten is that the book largely covers the design of products and not websites or print media – products like those that can be built with HoloLens, Magic Leap and Oculus Rift. Full of insights, Buxton helps his readers to see the importance of lived experience when we design and build technology.

4. Bergsonism by Gilles Deleuze – though Deleuze is probably most famous for his collaborations with Felix Guattari, this work on the philosophical meaning of the term ‘’virtual reality’, not as a technology but rather as a way of approaching the world, is a gem.

5. Passwords by Jean Baudrillard – what Deleuze does for virtual reality, Baudrillard does for other artifacts of technological language in order to show their place in our mental cosmology. He also discusses virtual reality along the way, though not as thoroughly.

6. Mathematics for 3D Game Programming and Computer Graphics by Eric Lengeyl – this is hardcore math. You will need this. You can buy it used online for about $6. Go do that now.

7. Linear Algebra and Matrix Theory by Robert Stoll – this is a really hard book. Read the Lengeyl before trying this. This book will hurt you, by the way. After struggling with a page of this book, some people end up buying the Manga Guide to Matrix Theory thinking that there is a fun way to learn matrix math. Unfortunately, there isn’t and they always come back to this one.

8. Phenomenology of Perception by Maurice Merleau-Ponty – when it first came out, this work was often seen as an imitation of Heiddeger’s Being and Time. It may be the case that it can only be truly appreciated today when it has become much clearer, thanks to years of psychological research, that the mind reconstructs not only the visual world for us but even the physical world and our perception of 3D spaces. Merleau-Ponty pointed this out decades ago and moreover provides a phenomenology of our physical relationship to the world around us that will become vitally important to anyone trying to understand what happens when more and more of our external world becomes digitized through virtual and augmented reality technologies.

9. Philosophers Explore the Matrix – just as The Matrix is essential viewing for anyone in this field, this collection of essays is essential reading. This is the best treatment available of a pop theme being explored by real philosophers – actually most of the top American philosophers working on theories of consciousness in the 90s. Did you ever think to yourself that The Matrix raised important questions about reality, identity and consciousness? These professional philosophers agree with you.

10. Snow Crash by Neal Stephenson – sometimes to understand a technology, we must extrapolate and imagine how that technology would affect society if it were culturally pervasive and physically ubiquitous. Fortunately Neal Stephenson did that for virtual reality in this amazing book that combines cultural history, computer theory and a fast paced adventure.

What is a HoloCoder?

April 26, 2015 James Ashley

holodeck

Over the past few years we’ve seen the rapid release of innovative consumer technologies that are all loosely related by their ability to scan 3D spaces, interact with 3D spaces or synthesize 3D spaces. These include the Kinect sensor, Leap Motion, Intel Perceptual Computing, Oculus Rift, Google Glass, Magic Leap and HoloLens. Additional related general technologies include projection mapping and 3D printing. Additional related tools include Unity 3D and the Unreal Engine.

Despite a clear family resemblance between all of these technologies, it has been difficult to clearly define what that relationship is. There has been a tendency to categorize all of them as simply being “bleeding edge”, “emerging” or “future”. The problem with these descriptors is that they are ultimately relative to the time at which a technology is released and are not particularly helpful in defining what holds these technologies together in a common gravitational pull.

definitions

I basically want to address this problem by engaging in a bit of word magic. Word magic is a sub-category of magical thinking and is based on a form of psychological manipulation. If you have ever gone out to Martin Fowler’s Bliki then you’ve seen the practice at work. One of the great difficulties of software development is anticipating the unknown: the unknown involved in requirements, the unknown related to timelines, and the unknown concerned with the correct tactics to accomplish tasks. In a field with a limited history and a tendency not to learn from other related fields, the fear of the unknown can utterly cripple projects.

Martin Fowler’s endless enumeration of “patterns” on his bliki takes this on directly by giving names to the unknown. If one reads his blog carefully, however, it quickly becomes clear that most, though not all, of these patterns are illusory: they are written at such an abstract level that they fail to provide any prescriptive advice on how to solve the problems they are intended to address. What they do provide, however, is a sense of relief that there is a “name” that can be used to plug up the hole opened up in time by the fear of the unknown. Solutions architects can return to their teams (or their managers) and pronounce proudly that they have found a pattern to solve the outstanding problem that is hanging over everyone – all that remains is to determine what each “name” actually means.

In this sense, the whole world of software architecture – which Glassdoor ranked as the 11th best job of 2015 — is a modern priesthood devoted to prophetic interpretations of “design patterns”.

I similarly want to use word magic to define the sort of person that works with the sorts of technology I listed at the top of this article. I think I can even do it quite simply with familar imagery.

A holocoder is someone who works with technologies that are inspired by and/or anticipate the Star Trek Holodeck.

interpretations

The part of the definition that states “inspired by and/or anticipate” may seem strange but it is actually quite essential. It is based on a specific temporal-cybernetic theory concerning the dissemination of ideas which I will attempt to describe but which is purely optional with respect to the definition.

But first: how can a theory be both essential and optional? This is an issue that Niels Bohr, one of the fathers of quantum mechanics, tackled frequently. In the early 30’s Bohr was travelling through eastern Europe on a lecture tour. During part of the tour, a former student met him at his inn and noticed him nailing a horse shoe over the door of his room. “Professor Bohr”, he asked, “what are you doing?” Niels Bohr replied, “The Inn Keeper informed me that a horse shoe over the door will bring me luck.” The student was scandalized by this. “But Herr Professor,” the student objected, “surely as a physicist and intellectual such as yourself does not believe in these silly superstitions.” “Of course not,” Bohr answered. “But the Inn Keeper reassured me that the horse shoe will bring me luck whether I believe in it or not.”

Here is the optional theory of the Holodeck. Certain technologies, it seems to me, can have such an influence that they shape the way we think about the world. We have seen many examples of this in our past such as the printing press, the automobile, the personal computer and the cell phone. Furthermore we anticipate the advent of similar major technologies in our future. These technologies have what is called a “psychic resonance” and change the very metaphors we use to describe our world. To give a simple example, whereas we originally used mental metaphors to explain computers in terms of “memory”, “processing” and even “computing”, today we use computer metaphors to help explain how the brain works. The arrival of the personal computer caused a shift and a reversal in what semioticians call the relationship between the explanans and the explanandum.

wesley in the holodeck

Psychic impact is transmitted over carriers called “memes”. Memes are basically theoretical constructs that are phenomenally identical to what we call “ideas” but behave like viruses. Memes travel through air as speech and along light waves as images in order to spread themselves from host to host. Traditionally the psychic impact of a meme is measured by the meme’s density over a given space. Besides density, the psychic impact can also be measured based on the total volume of space it is able to infect. Finally, the effectiveness of a meme can also be measured based on its ability to spread into the future. For instance, works of literature and cultural artifacts such as religions and even famous sayings are examples of memes that have effectively infected the future despite a distance of thousands of years between the point of origin of the infection and the temporal location of the target.

While the natural habitat of bacteria like e coli is in the gastrointestinal tract, the natural habitat of memes is in the brain and this leads to a fascinating third form of mimetic transmission. At the level of microtubules in the brain where memes happen to live, we enter the Planck scale in which classical physics do not apply in the way that they do at the macro level. At this scale, effects like quantum entanglement create spooky behaviors such as quantum communication. While theoretically people still cannot communicate with each other in time since that level of semiotics is still governed by classical physics, there is an opening for mimetic viruses to actually be transmitted backwards in time as if they were entering a transporter in one brain and rematerialized in another brain in the past. This allows for a third manner of mimetic spread: in space, forward in time, and finally backwards in time.

Riker in the Holodeck

As an aside, and as I said above, this is an _optional_ theory of psychic impact through time. A common and totally valid criticism is that it appeals to quantum mystery which tends to be misused to justify anything from ghosts to religious cults. The problem with appeals to “quantum mystery” is that this simply provides a name for a problem rather than prescribing actual ways to make predictions or anticipate behavior. In other words, like Martin Fowler’s bliki, it is word magic that provides interpretations of things but not actual solutions. Against such criticisms, however, it should be pointed out that I am explicitly engaged in an exercise in word magic, in which case using certain techniques of word magic – such as quantum mystery – is perfectly legitimate and even natural.

Through quantum entanglement acting on memes at the microtubule level, a technology from our possible future which resembles the Star Trek holodeck has such a large psychic impact that it resonates backwards in time until it reaches and inhabits the brains of the writers of a futuristic science fiction show in the late 80’s and is introduced into the show as the Holodeck. Through television transmissions, the holodeck meme is then broadcast to millions of teenagers who eventually enter the tech industry, become leaders in the tech industry, and eventually decide to implement various aspects of the holodeck by creating better and better 3D sensors, 3D simulation tools and 3D visualization technologies – both augmented and virtual. In other words, the Holodeck reaches backwards in time to inspire others in order to effectively give birth to itself, ex nihilo. Those that have been touched by the transmission are what I am calling holocoders.

and/or

Alternatively, this theory of where holocoders come from can be taken as a metaphor only. In this case, holocoders are not people being pulled toward a common future but instead people being pushed forward from a common past. Holocoders are people inspired directly or indirectly by a television show from the late 80’s that involved a large room filled with holograms that could be used for entertainment as well as research. Holocoders work on any or all of the wide variety of technologies that could potentially be combined to recreate that imagined experience.

the dreamatorium

Anyways, that’s my theory and I’m sticking to it. More importantly, these technologies are deeply entangled and deserve a good name, whether you want to go with holocoding or something else (though the holodeck people from the future highly encourage you to use the terms “holocoder”, “holocoding” and “holodeck”).

appendix

There are two other important instances of environment simulators which for whatever reason do not have the same impact as the Star Trek holodeck but are nevertheless worth mentioning.

The first is the X-Men Danger Room which is an elaborate obstacle course involving holograms as well as robots used to train the X-Men. While the Danger Room goes back to the 60’s, the inclusion of holograms actually didn’t happen until the early 90’s, and so actually comes after the Star Trek environment simulator.

Clifford D. Simak published Way Station in 1963 (and won a Hugo award for it). It actually anticipates two Star Trek technologies – transporters as well as an environment simulator. Enoch Wallace, the hero of the story, works the earth relay station for intergalactic aliens who transport travelers over vast distances by sending them over shorter hops between the way stations of the title. Because he is so isolated in his job, the aliens support him by allowing him to pursue a hobby. Because Wallace enjoys hunting, the aliens build for him an environment simulator that lets him do big game hunting for dinosaurs.

How to Read Star Wars Comics

April 19, 2015 James Ashley

Are you trying to find a guide to reading Star Wars comic books online? You’ve come to the right place. And obviously — Do. Or do not. There is no try.

Lucas Film just released the second teaser for Star Wars VII. My wife and I found ourselves tearing up as we watched it on her ipad, demonstrating that nostalgia is the only thing stronger than the Force. We are of the generation that first saw Star Wars in a theater in the 70’s. We remember a time before Star Wars existed and yet it has always been the background myth of our lives as we grew up. What Gilgamesh was to Mesopotamians or Siegfried and Brunhilde to Germans, Luke, Han and Leia are to us.

Timed with the release of the teaser, Marvel Comics has added a ton of Star Wars related comic books to their digital comics service Marvel Unlimited. These comic books span a period from the 90’s to the present in which Dark Horse Comics started spinning up stories from the Star Wars expanded mythology (many based on the books) that fill in gaps left by the movies as well as extending the storyline beyond Star Wars VI. In 2015, Disney, which owns both the Star Wars franchise as well as Marvel Comics, moved the Star Wars publication rights from Dark Horse Comics to Marvel Comics, which is apparently how these classic Dark Horse comics are now appearing online.

accessing the digital comics

If you want to know what happens in Star Wars after the battle of Endor (but before J. J. Abrams retcons over it) then this is your opportunity. You can even do it for free if you want. Go to the Marvel Unlimited website and enter the promotion code starwars to get one free month – though I’d recommend skipping this and getting an annual subscription for $69. Marvel Unlimited has a decent web interface, but the best way to use the service is with the iPad app.

(Scott Hanselman has a good but critical review of the service written in 2011 that deserves to be read. It’s worth mentioning, though, that the service as well as the UX have greatly improved over the past four years.)

Once you have your subscription your main problem is going to be seeing the trees for the forest. There are thousands of comics in the Marvel catalog and they tend to be listed in alphabetical order. This is sensible, but not particularly helpful if you want to read the continuing Star Wars saga in mythologically chronological order. Additionally, while Marvel is offering large chunks of the Star Wars graphic novel canon, there are pieces missing. This is an additional difficulty in trying to get the full story straight.

reading in the correct order

While there are lots of comics available through the subscription that are contemporaneous with the events in the movies, in this post I’m just going to try to help you to read the Star Wars comics being offered through Marvel Unlimited in the correct order starting just after the battle of Endor.

The first comic of the New Republic Era available is Star Wars: Boba Fett – Twin Engines of Destruction (1997) — I’m using the titles as they are listed in the “browse” tab of the Marvel Unlimited app and including the year to highlight the difference between the chronology and the publication order. I’m also heavily indebted to Wookieepedia (you read it right) for all the correct timeline information.

The story picks up with what is known as the Thrawn Trilogy. In Marvel Unlimited, these are cataloged under three different series of about 5 comics each:

Star Wars: Heir to the Empire (1995 – 1996)

Star Wars: Dark Force Rising (1997)

Star Wars: The Last Command (1997 – 1998)

This leads us into the Dark Empire Trilogy in which the emperor turns out not to be as dead as he could be. Dark Horse’s first Dark Empire series is also in many ways what first made the Star Wars comics attractive as a vector for transmitting expanded universe stories. A few comics slip in between Dark Empire II and Empire’s End of which only the Boba Fett story is currently available on MU.

Star Wars: Dark Empire (1991 – 1992)

Star Wars: Dark Empire II (1994 – 1995)

Star Wars: Boba Fett – Agent of Doom (2000)

Star Wars: Empire’s End (1995)

Boba Fett grew as a character mainly because he had an awesome costume and fans just wanted to see more of it. The same could be said of the main character in the next two series. Crimson Empire follows the exploits of one of Emperor Palpatine’s elite bodyguards.

Star Wars: Crimson Empire (1997)

Star Wars: Crimson Empire II – Council of Blood (1998-1999)

The Chewbacca series (2000) is a commemorative four issue run with stories told by Chewbacca’s friends because he is dead at this point in the Star Wars chronology (::sniff::) which will be overwritten by J. J. Abrams faster than you can unsay “Kaaahhhhhn” as J. J. retcons the expanded Star Wars universe.

At this point we leap a century forward and get into the Star Wars: Legacy comics where we follow the adventures of Cade Skywalker, Ania Solo and lots of other people with familiar-but-not-quite-right sounding names. MU lists three collections in the browse tab.

Star Wars Legacy (2006-2010) – 50 issues!

Star Wars: Legacy – War (2010 – 2011) – six issues

Star Wars: Legacy (2013 – 2014) [aka Star Wars: Legacy II] – 18 issues

then what … ?

And that’s as far as it goes for now. If you need more to read, you can go back in time and start pounding the 55 issues of Knights of the Old Republic digital comics which will provide the Jedi back story from thousands of years before the movies. On the other hand, you might also want to branch out and see what else MU has to offer. Here’s some other books on Marvel Unlimited that I would highly recommend.

Nick Fury, Agent of SHIELD #1 (1968)

This single issue written and illustrated by legend Jim Steranko changed the game in comic books. Even as pop art was bringing high art low, Steranko lifted the comic book genre and opened the possibility to start considering comics an art form – or as we prefer to say today, start considering “graphic novels” an art form.

Marvel 1602 (2002-2003)

While there have been many takes on alternate Marvel timelines, Neil Gaiman’s turn with these eight issues is one of the most interesting. He imagines the classic Marvel heroes finding their place in 17th century Europe.

Eternals (2006)

In the 70’s, Marvel experimented with making Erich von Daniken’s Chariots of the Gods the basis for a comic book and had mixed success. Decades later, Neil Gaiman came along and wrote a seven issue series based on the earlier work to create an amazing story of aliens turning ancient humans into super heroes for their own mysterious purpose. The aliens in question, by the way, happen to be the Celestials who are part of the back story for James Gunn’s Guardians of the Galaxy movie. See – everything ties together in the Marvel universe.

Guardians of the Galaxy (2008)

The comics are as good as the movie. There are a few more characters and the ones you know are slightly different. Rocket and Groot are the same, though. This run of the comics basically revives a bunch of Silver Age characters, modernizes them and throws them together to amazing effect.

Annihilation: Conquest – Starlord (2007)

But if you want to do it right and find out how Peter Quill aka Starlord first meets Rocket, Groot and Bug (who’s Bug you ask?) then you might want to also read the four issues of Annihilation: Conquest – Starlord which is just a part of the much bigger Marvel space event called Annihilation: Conquest. Really, all of Annihilation: Conquest is worth reading because then you’ll get to know more about Quasar, Ronan the Accuser, the Heralds of Galactus, and the Nova Corps.

Annihilation (2006 – 2007)

But if you really really want to do it right, then you’ll read the Annihilation comic event before you read either Annihilation: Conquest or Guardians of the Galaxy. This is where Peter Quill first gets retconned into the contemporary world. In the case that you are fully committing to the effort, the correct order for reading most of the back story for the Guardians movie would be:

Annihilation Prologue (2006), Annihilation (2006 – 2007), Annihilation: Quasar / Annihilation: Nova / Annihilation: Ronan / Annihilation: Silver Surfer / Annihilation: Super Skrull [these are all overlapping series], Annihilation: Conquest Prologue (2007), Annihilation: Conquest (2007), Annihilation: Conquest – Quasar / Annihilation: Conquest – Starlord / Annihilation: Conquest – Wraith / Annihilation: Conquest – Heralds of Galactus, Guardians of the Galaxy (2008), The Thanos Imperative: Ignition (2010), The Thanos Imperative (2010), The Thanos Imperative: Devastation (2010).

It’s totally worth it.

Agents of Atlas (2006 — 2007)

Agents of Atlas, like Guardians of the Galaxy, is an instance of Marvel retconning characters that were abandoned in the 50’s and brought together for a series in the 00’s. Interestingly, this is the second time they have been retconned. The first time was in a What If? one-off from the 70’s. FBI agent Jimmy Woo leads a rag-tag team of super-powered beings against the nefarious criminal organization known as the Atlas Foundation. His team includes Namora of Atlantis, the goddess Venus, Marvel Boy the Uranian, Gorilla Man and M-11 the robot. Chronologically in the Marvel universe, Jimmy Woo’s team is actually considered the original Avengers formed to rescue President Dwight Eisenhower from the clutches of Atlas and then later mysteriously disbanded. This series of six issues from 2006 uncovers what really happened to the team. Another series of 11 issues of Agents of Atlas was released in 2009, which was followed up in 2010 by a five issue series simply titled Atlas.

Nextwave: Agents of H.A.T.E.

A twelve issue S.H.I.E.L.D. parody written by comic legend Warren Ellis and beautifully drawn by Stuart Immonen. Super powered heroes discover that they aren’t working for the good guys after all, but that their organization are actually the baddies. They decide to do something about it. Ellis said of the series, “It’s an absolute distillation of the superhero genre. No plot lines, characters, emotions, nothing whatsoever. It’s people posing in the street for no good reason.”

Runaways (2003 — 2004)

Misfit, middle-class teenagers discover that their parents really are evil after all when they accidentally witness them performing a human sacrifice to Elder Gods. They also come to discover that, like their super-villain parents, they possess super powers.

Journey Into Mystery (2011)

The god of mischief Loki is dead but a young boy appears claiming to be Loki reborn. He struggles however because, being Loki, everybody hates him and nobody trusts him. Written by Kieron Gillen, this is the story of how Loki attempts to redeem himself. It is by turns hilarious and heart breaking. Start with issue #622 if you can and try to at least get to issue #645 which wraps up the Loki story.

Secret Avengers #20 (2010)

The entire Secret Avengers series is great. I would especially recommend that you read issue #20 which follows a single storyline in the life of Natalia Romanov, aka Black Widow.

Marvel Zombies (2005-2006)

Marvel Unlimited gives you every variation on Marvel zombies you could possibly want, from the original series to the five follow ups to Marvel Zombies Christmas Carol to Marvel Zombies vs. Marvel Apes. I recommend at least having a taste of the first five issue run.

Secret Warriors (2008 — 2011)

In the first comic of this 28 issue run, Nick Fury of S.H.I.E.L.D. discovers that for his entire career, his arch enemy Hydra (“Hail Hydra!”) has been secretly controlling S.H.I.E.L.D. itself. The last 60 years of secret wars have been a farce scripted by the Nazi Baron Strucker. (This is actually the basis for the storyline in the film Winter Soldier as well as the Agents of S.H.I.E.L.D. TV series.) But Nick doesn’t give up. Instead, he pulls together a team to take Hydra down once and for all.

One Kinect to rule them all: Kinect 2 for XBox One

April 5, 2015 James Ashley

Yes. That’s a bit of a confusing title, but it seems best to lay out the complexity upfront. So far there have been two generations of the Kinect sensor which combine a color camera, a depth sensing camera, an infrared emitter (basically used for the depth sensing camera) and a microphone array which works as a virtual directional shotgun microphone. Additional software called the Kinect SDK then allows you to write programs that read these data feeds as well as interpolating them into 3D animated bodies that are representations of people’s movements.

Microsoft has just announced that they will stop producing separate versions of the Kinect v2, one for windows and one for the XBox One, but will instead encourage developers to purchase the Kinect for Windows Adapter instead to plug their Kinects for XBox One into a PC. In fact, the adapter has been available since last year, but this just makes it official. All in all this is a good thing. With the promise that Universal Windows Apps will be portable to XBox, it makes much more sense if the sensors – and more importantly the firmware installed on them – are exactly the same whether you are on a PC running Windows 8/10 or an XBox running XBox OS.

This announcement also vastly simplifies the overall Kinect hardware story. Up to this point, there weren’t just two generations of Kinect hardware but also two versions of the current Kinect v2 hardware, one for the Xbox and one for Windows (for a total of four different devices). The Kinect hardware, both in 2010 and in 2013, has always been built first as a gaming device. In each case, it was then adapted to be used on Windows machines, in 2012 and 2014 respectively.

The now discontinued Kinect for Windows v2 differed from the Kinect for the Xbox One in both hardware and software. To work with Windows machines, the Kinect for Windows v2 device uses the specialized power adapter to pump additional power to the hardware (there is a splitter in the adapter that attaches the hardware to both a USB port as well as a wall plug). The Xbox One, being proprietary hardware, is able to pump enough juice to its Kinect sensor without needing special adapter. Additionally, the firmware for the original Kinect for Windows v1 sensor diverged over time from the Kinect for Xbox’s firmware – which led to differences in how the two versions of the hardware performed. It is now clear that this will not happen with Kinect v2.

Besides the four hardware devices and their respective firmware, the loose term “Kinect” can also refer to the software APIs used to incorporate Kinect functionality into a software program. Prior to this, there was a Kinect for Windows SDK 1.0 through 1.8 that was used to program against the original Kinect for Windows sensor. For the Kinect for XBox One with the Kinect for Windows Adapter, you will want to use the Kinect for Windows SDK 2.0 (“for Windows” is still part of the title for now, even though you will be using it with a Kinect for XBox One, though of course you can still use it with the Kinect for Windows v2 sensor if you happen to have bought one of those prior to their discontinuation). There are also other SDKs floating around such as OpenNI and Libfreenect.

[Much gratitude to Kinect MVP Bronwen Zande for helping me get the details correct.]

50 Shaders of Grey

April 1, 2015 James Ashley

This isn’t actually a blog post yet. Just a title and a promise. Someday I will backfill this post with amazingly deviant technical information about pixel shaders, vertex shaders, algorithmic drawing, matrix math, ray tracing, kernel convolutions and nipple clamps. Stay tuned.

Unity 5 and Kinect 2 Integration

March 27, 2015 James Ashley7 Comments

pointcloud

Until just this month one of the best Kinect 2 integration tools was hidden, like Rappuccini’s daughter, inside a walled garden. Microsoft released a Unity3D plugin for the Kinect 2 in 2014. Unfortunately, Unity 4 only supported plugins (bridges to non-Unity technology) if you owned a Unity Pro license which typically cost over a thousand dollars per year.

On March 3rd, Unity released Unity 5 which includes plugin support in their free Personal edition – making it suddenly very easy to start building otherwise complex experiences like point cloud simulations that would otherwise require a decent knowledge of C++. In this post, I’ll show you how to get started with the plugin and start running a Kinect 2 application in about 15 minutes.

(As an aside, I always have trouble keeping this straight: Unity has plugins, openFrameworks as add-ins, while Cinder has bricks. Visual Studio has extensions and add-ins as well as NuGet packages after a confusing few years of rebranding efforts. There may be a difference between them but I can’t tell.)

1. First you are going to need a Kinect 2 and the Unity 5 software. If you already have a Kinect 2 attached to your XBox One, then this part is easy. You’ll just need to buy a Kinect Adapter Kit from the Microsoft store. This will allow you to plug your XBox One Kinect into your PC. The Kinect for Windows 2 SDK is available from the K4W2 website, though everything you need should automatically install when you first plug your Kinect into your computer. You don’t even need Visual Studio for this. Finally, you can download Unity 5 from the Unity website.

2. The Kinect 2 plugin for Unity is a bit hard to find. You can go to this Kinect documentation page and scroll half-way down to find the link called Unity Pro Packages. Aternatively, here is a direct link to the most current version of the plugin as of this writing.

3. After you finish downloading the zip file (currently called KinectForWindows_UnityPro_2.0.1410.zip), extract it to a known location. I like to use $\Documents\Unity. Inside you will find three plugins as well as two sample scenes. The three Kinect plugins are the basic one, a face recognition plugin, and a gesture builder plugin, each wrapping functionality from the Kinect 2 SDK.

4. Fire up Unity 5 and create a new project in your known folder. In my case, I’m creating a project called “KinectUnityProject” in the $\Documents\Unity folder where I extracted the Kinect plugins and related assets.

5. Now we will add the Kinect plugin into our new project. When the Unity IDE opens, select Assets from the top menu and then select Import Package | Custom Package …

6. Navigate to the folder where you extracted the KinectforWindows_Unity components and select the Kinect2.0.xxxxx.unitypackage file. That’s our plugin along with all the scripts needed to build a Kinect-enabled Unity 5 application. After clicking on “Open”, an additional dialog window will open up in the Unity IDE called “Importing Package” with lots of files checked off. Just click on the “Import” button at the lower right corner of the dialog to finish the import process. Two new folders will now be added to your Project window under the Assets folder called Plugins and Standard Assets. This is the baseline configuration for any Kinect project in Unity.

7. Now we’ll get a Kinect with Unity project quickly going by simply copying one of the sample projects provided by the Microsoft Kinect team. Go into file explorer and copy the folder called “KinectView” out of the KinectforWindows_Unity folder where you extracted the plugins and paste it into the Assets directory in your project folder. Then return to the Unity 5 IDE. A warning message will pop up letting you know that there are compatibility issues between the plugin and the newest version of Unity and that files will automatically be updated. Go ahead and lie to the Unity IDE. Click on “I Made a Backup.”

8. A new folder has been added to your Project window under Assets called KinectView. Select KinectView and then double click on the MainScene scene contained inside it. This should open up your Kinect-enabled scene inside the game window. Click on the single arrow near the top center of the IDE to see your application in action. The Kinect will automatically turn on and you should see a color image, an infrared image, a rendering of any bodies in the scene and finally a point cloud simulation.

9. To build the app, select File | Build & Run from the top menu. Select Windows as your target platform in the next dialog and click the Build & Run button at the lower right corner. Another dialog appears asking you to select a location for your executable and a name. After selecting an executable name, click on Save in order to reach the final dialog window. Just accept the default configuration options for now and click on “Play!”. Congratulations. You’ve just built your first Kinect-enabled Unity 5 application!

les fruits dangereux

March 21, 2015 James Ashley

potato_clock

What ever happened to the potato clock? In a revived period of do-it-yourselfers, arduino artists and 3d printing presses – a bright new age of artisanal software and hardware development – what has happened to the epitome of nerdish home engineering? At one point in time you couldn’t escape your teenage years without an awareness of the potato clock – stashed somewhere between the x-ray glasses and the sea monkeys – and then poof, suddenly they vanish from the collective consciousness.

Over lunch with my good friends Joel and Nate, I raised this question and Nate was quick to identify the exact date on which the potato clock disappeared. It happened in 2001. On September 11th, to be precise. On that day, many seemingly innocent things were recognized for the danger they are. The test of this perceptual shift occurred on January 31, 2007 when a marketing campaign for the cartoon series Aqua Teen Hunger Force turned into a bomb scare. Whereas levity typically brings with it perspective and reestablishes old norms, in this case – five years after 9/11 – the guerrilla campaign involving led lights was seen as offensive, in poor taste and led to the resignation of Jim Samples, the man who built up Cartoon Network from nothing. Which led us to ask, over our fruit salads, what other common objects – objects like the lowly pototo — might be revealed to have a sinister aspect.

Children have always known the threatening qualities of fruits and vegetables. Adults, on the other hand, have always been somewhat oblivious to this implicit threat that children are aware of in their bones, and have been known to eat kale and Brussels sprouts with reckless abandon. We considered what it would take to capture the danger implicit in fruits and vegetables and realized all it required was a bit of tape and a phone.

Consider the banana. At first appearance, it is a pleasant and unassuming fruit with a waxy peel and a soft, sweet interior. Attach a phone to it with some electrical tape and leave it in the lobby at the airport and its true nature reveals itself. Drop it in the mailbox at the post office and see just how innocent it really is.

Leave this at the entrance to a police station and see who laughs.

Context obviously matters. A banana in a fruit basket suggests a certain functional role. A banana with electronics taped to it sets up a different sort of context. Through experimentation, we found that even the type of tape used can shift the context in subtle but meaningful ways. Silvery duct tape, for instance, is much more threatening than glossy black electrical tape when attached to fruit.

Which begs the question, what is the most threatening fruit? Here is an apple attached to an AT&T Pantech cell phone – one of the last free-with-contract AT&T phones that did not require a data plan and the automatic $25 data fee charged to your account whether you are using data or not – why can’t I just use it for phone calls and wifi, AT&T? In that scenario, the AT&T cellular contract is obviously the most frightening thing.

But what happens when we exchange the Pantech for an HTC Windows Phone?

Or an iPhone for that matter? Which is more intimidating: an iPhone 5 or an iPhone 6?

Here is a Microsoft Band wrapped around an apple. There’s a metaphor in there somewhere.

iPhone 5 and a kiwi.

Samsung Galaxy 4 with onion and duct tape.

I normally associate electrical tape with bombs and duct tape with kidnapping – which makes duct tape more viscerally terrifying for me. Some people claim they associate duct tape with ducks but I think that’s a canard. Onions make me want to cry.

One of life’s riddles: is a coconut a fruit or a vegetable?

No matter what I do, I can’t make strawberries look threatening.

Carrots, on the other hand, are Nature’s terrorists.

These carrots are organic, by the way. The extra cost is worth it for the additional fear factor.

Rubber bands can be intimidating in the right context. Especially when that context is celery. The safety pin is overkill, maybe?

This is a Nokia Lumia Windows Phone 7 developer unit with both front-facing and rear-facing cameras. It is attached to a red plum.

Kindle White meets cantaloupe.

And of course, potatoes: fear incarnate.

The Next Book

March 3, 2015 James Ashley

min_lib

The development community deserves a great book on the Kinect 2 sensor. Sadly, I no longer feel I am the person to write that book. Instead, I am abandoning the Kinect book project I’ve been working on and off over the past year in order to devote myself to a book on the Microsoft holographic computing platform and HoloLens SDK. I will be reworking the material I’ve so far collected for the Kinect book as blog posts over the next couple of months.

As anyone who follows this blog will know, my imagination has of late been captivated and ensorcelled by augmented reality scenarios. The book I intend to write is not just a how-to guide, however. While I recognize the folly of this, my intention is to write something that is part technical manual and part design guide, part math tutorial, part travel guide and part cookbook. While working on the Kinect book I came to realize that it is impossible to talk about gestural computing without entering into a dialog with Maurice Merleau-Ponty’s Phenomenology of Perception and Umberto Eco’s A Theory of Semiotics. At the same time, a good book on future technologies should also cover the renaissance in theories of consciousness that occurred in the mid-90’s and which culminated with David Chalmers’ masterwork The Conscious Mind. Descartes, Bergson, Deleuze, Guattari and Baudrillard obviously cannot be overlooked either in a book dealing with the topic of the virtual, though I can perhaps elide a bit.

A contemporary book on technology can no longer stay within the narrow limits of a single technology as was common 10 or so years ago. Things move at too fast a pace and there are too many different ways to accomplish a given task that choosing between them depends not only on that old saw ‘the right tool for the job’ but also on taste, extended community and prior knowledge. To write a book on augmented reality technology, even when sticking to one device like the HoloLens, will require covering and uncovering to the uninitiated such wonderful platforms as openFrameworks, Cinder, Arduino, Unity, the Unreal Engine and WPF. It will have to cover C#, since that is by and large the preferred language in the Microsoft world, but also help C# developers to overcome their fear of modern C++ and provide a roadmap from one to the other. It will also need to expose the underlying mathematics that developers need to grasp in order to work in a 3D world – and astonishingly, software developers know very little math.

Finally, as holographic computing is a wide new world and the developers who take to it will be taking up a completely new role in the workforce, the book will have to find its way to the right sort of people who will have the aptitude and desire to take up this mantle. This requires a discussion of non-obvious skills such as a taste for cooking and travel, an eye for the visual, a grounding in architecture and an understanding of how empty spaces are constructed, a general knowledge of literary and social theory. The people who create the next world, the augmented world, cannot be mere engineers. They will also need to be poets and madmen.

I want to write a book for them.