Jasper Brekelmans is already pretty famous for creating Brekel Pro Tools, which turns the sensor streams from various 3D cameras, including the Kinect, into useful 3D data that can be used for 3D animations, visual effects and general research. OpenVR Recorder expands on this capability by also ingesting high quality data used for tracking virtual reality devices like the Oculus Rift and the HTC Vive.
The original Kinect, introduced in 2010, created a small revolution in the 3D sensor industry. It helped to drive down the price of 3D tracking and inspired new research into time-of-flight, structured light pattern and marker-based techniques that dramatically improved the accuracy with which we capture and understand 3D space. In turn, this has driven innovation in the area of interpreting 3D spaces using 2D cameras – which goes into solutions like Apple’s ARToolkit.
Though it may not be immediately evident, the current developments in VR and AR are built on top of these leaps and bounds in 3D sensor technology. The HoloLens uses it for position tracking and spatial mapping. The HoloLens solution is going into the SLAM-based position tracking tech for the Mixed Reality Headsets from HP and Acer. At the same time, VR tech from Oculus and HTC are constantly streaming 3D data in order to provide accurate tracking of both the user and handheld controllers.
Since this data is already flowing through the air around you when you are in VR space, there must be a way to capture it, right? That’s what Jasper Brekelmans has done with his new tool, OpenVR Recorder.
If you’re already working with the Kinect for visual effects, motion capture, or research, then this new software should be included as an essential part of your tool chest. There’s even a free trial available.
This is the first in a series of interviews intended to help people get to know the movers & shakers as well as the drones & technicians (sometimes the same person is all four) who are making mixed reality … um … a reality. I’ve borrowed the format from Vox but added some new questions.
Though not widely known outside of certain circles, when you ask experienced HoloLens developers who they most admire, Jasper’s name usually comes up. Jasper is the creator of the Brekel Toolset, an affordable tool for doing motion capture with the Kinect sensor. He also works with HoloLens, Oculus, and the Vive and his innovative projects have been featured on RoadToVR and other venues. His work on collaboration between multiple Vive headsets was mind-blowing—but then again, so was his HoloLens motion capture demo with a live dancer, his HoloLens integration with Autodesk MotionBuilder, and his recent release of the OpenVR Recorder.
Without further ado, here are Jasper’s answers to 10 questions:
What is the earliest video game you remember playing? Pac-Man (arcade) and Donkey Kong (handheld).
Who is the person who has most influenced the way you think? A work mentor and some close personal friends.
When was the last time you changed your mind about something? Probably on a weekly basis on something or other.
What’s a programming skill people assume you have but that you are terrible at? Heavily math based algorithms and/or coding for mobile platforms.
What inspires you to learn? The goal of having new possibilities with freshly learned skills.
What do you need to believe in order to get through the day? That what I do matters to others.
What’s a view that you hold but can’t defend? That humanity will be better off once next generations have grown up with true AR glasses/lenses technology, have played with virtual galaxies and value virtual objects similarly to physical objects for certain purposes.
What will the future killer Mixed Reality app do? Empower users in their daily live without them realizing it while at the same time letting new users realize what they miss instantly.
Virtual Reality had its own section at CES this year in the Las Vegas Convention Center South Hall. Oculus had a booth downstairs near my company’s booth while the OSVR (Open Source Virtual Reality) device was being demonstrated upstairs in the Razer booth. The Project Morpheus (now Playstation VR) was being demoed in the large Sony section of North Hall. The HTC Vive Pre didn’t have a booth but instead opted for an outdoor tent up the street from North Hall as well as a private ballroom in the Wynn Hotel to show off their device.
It would be convenient to be able to tell you which VR head mounted display is best, but the truth is that they all have their strengths. I’ll try to summarize these pros and cons first and then go into details about the demo experiences further down.
HTC Vive Pre and Oculus Rift have nearly identical specs
Pro: Vive currently has the best peripherals (Steam controllers + Lighthouse position tracking), though this can always change
Pro: Oculus is first out of the gate with price and availability of the three major players
Con: Oculus and Vive require expensive latest gen gaming computers to run in addition to the headsets ($900 US +)
Pro: PlayStation VR works with a reasonably priced PlayStation
Pro: PlayStation Move controllers work really well
Pro: PlayStation has excellent relationships with major gaming companies
Con: PlayStation VR has lower specs than Oculus Rift or HTC Vive Pre
Con: PlayStation VR has an Indeterminate release date (maybe summer?)
Pro: OSVR costs only $299 US, making it the least expensive VR device
Con: OSVR has the lowest specs and is a bit DIY
Pro: OSVR is a bit DIY
You’ll also probably want to look at the numbers:
HTC Vive Pre
2160 x 1200
2160 x 1200
1920 x 1080
1920 x 1080
1920 x 1080
Res per eye
1080 x 1200
1080 x 1200
960 x 1080
960 x 1080
960 x 1080
60 / 75 Hz
The London Heist
Titans of Space
Let’s talk about Oculus first because they started the current VR movement and really deserve to be first. Everything follows from that amazing initial Kickstarter campaign. The Oculus installation was an imposing black fortress in the middle of the hall with lines winding around it full of people anxious to get a seven minute demo of the final Oculus Rift. This was the demo everyone at CES was trying to get into. I managed to get into line half an hour early one morning because I was working another booth. Like at most shows, all the Oculus helpers were exhausted and frazzled but very nice. After some hectic moments of being handed off from person to person, I was finally led into a comfortable room on the second floor of Fortress Oculus and got a chance to see the latest device. I’ve had the DK2 for months and was pleased to see all the improvements that have been made to the gear. It was comfortable on my head and easy to configure, especially compared to the developer kit model that I need a coin in order to adjust. I was placed into a fixed-back chair and an Xbox controller was put into my hand (which I think means Oculus Rift is exclusively a PC device until the Oculus Touch is released in the future) and I was given the choice of eight or so games including a hockey game in which I could block the puck and some pretty strange looking games. I was told to choose carefully as the game I chose would be the only game I would be allowed to play. I chose the space game, Eve Valkyrie, and until my ship exploded I flew 360 degrees through the void fighting off an alien armada while trying to protect the carriers in my space fleet.
What can one say? It was amazing. I felt fully immersed in the game and completely forgot about the rest of the world, the marketing people around me, the black fortress, the need to get back to my own booth, etc. If you are willing to pay $700 – $800 for your phone, then paying $600 for the Oculus Rift shouldn’t be such a big deal. And then you need to spend another $900 or more for a PC that will run the rift for you, but then at least you’ll have an awesome gaming machine.
Or you could also just wait for the HTC Vive Pre which has identical specs and feels just as nice and even has its own space game at launch called Elite: Dangerous. While the Oculus booth was targeted at fans, in large part, the Vive was shown in two different places to different audiences. A traveling HTC Vive bus pulled out tents and set up on the corner opposite Convention Hall North. This was for fans to try out the system and involved an hour wait for outdoor demos while demos inside the bus required signing up. I went down the street the the Wynn Hotel where press demos run by the marketing team were being organized in one of the hotel ballrooms. No engineers to talk to, sadly.
Whereas Oculus’s major announcement was about pricing and availability as well as opening up pre-orders, HTC’s announcement was about a technology breakthrough that didn’t really seem like much of a breakthrough. A color camera was placed on the front of HMD that outlines real-world objects around the player in order, among other things, to help the player avoid bumping into things when using the Vive Pre with the Lighthouse peripherals in order to walk around a VR experience.
The Lighthouse experience is cool but the experience I most enjoyed was playing Elite: Dangerous with two mounted joysticks. This is a game I’ve played on the DK2 until it stopped working with the DK2 following my upgrade to Windows 10 (which as a Microsoft MVP I’m pretty much required to do) so I was pretty surprised to see the game in the HTC press room and even more surprised when I spent an hour chatting away happily to one of ED’s marketing people.
So this is a big tangent but here’s what I think happened and why the ED Oculus support became rocky a few months ago. Oculus appears to have started courting Eve: Valkyrie a while back, even though Elite: Dangerous was the more mature game. Someone must have decided that you don’t need two space games for one device launch, and so ED drifted over to the HTC Vive camp. And suddenly, support for the DK2 went on the backburner at ED while Oculus made breaking changes in their SDK release and many people who had gotten ED to play with the Rift or gotten the Rift to play with ED were sorely disappointed. At this point, you can make Elite: Horizons (the upgrade from ED) work in VR with Oculus but it is tricky and not documented. You have to download SteamVR, even if you didn’t buy Elite: Horizons from Stream, and jury rig your monitor settings to get everything running well in the Oculus direct mode. Needless to say, it’s clear that Elite’s games are going to run much more nicely if you buy Steam’s Vive and run it through Steam.
As for comparing Oculus Rift and HTC Vive Pre, it’s hard to say. They have the same specs. They both will need powerful computers to play on, so the cost of ownership goes beyond simply buying the HMD. Oculus has the touch controllers, but we don’t really know when they will be ready. HTC Vive has the Lighthouse peripherals that allow you to walk around and the specialized Steam controllers, but we don’t know how much they will cost.
For the moment, then, the best way to choose between the two VR devices comes down to which space flying game you think you would like more. Elite: Dangerous is mainly a community exploration game with combat elements. Eve: Valkyrie is a space combat game with exploration elements. Beyond that, Palmer Luckey did get the ball rolling on this whole VR thing, so all other things being equal, mutatis mutandis, you should probably reward him with your gold. Personally, though, I really love Elite: Horizons and being able to walk around in VR.
But then again, one could always wait for PlayStation VR (the head-mounted display formerly known as Project Morpheus). The PlayStation VR demo was hidden in the back of the PlayStation demos, which in turn was at the back of the Sony booth which was at the far corner of the Las Vegas Convention Center North Hall. In other words, it was hard to find and a hike to get to. Once you go to it, though, it became clear that this was, in the scheme of things, a small play for the extremely diversified Sony. There wasn’t really enough room for the four demos Sony was showing and the lines were extremely compressed.
Which is odd because, for me at least, the PlayStation VR was the only thing I wanted to see. It’s by far the prettiest of the four big VR systems. While the resolution is slightly lower than that of the Oculus Rift or HTC Vive Pre, the frame rate is higher. Additionally, you don’t need to purchase a $900 computer to play it. You just need a PlayStation 4. The PlayStation Move controllers, as a bonus, finally make sense as VR controllers.
Best of all, there’s a good chance that PlayStation will end up having the best VR games (including Eve: Valkyrie) because those relationships already exist. Oculus and HTC Vive will likely clean up on the indie-game market since their dev and deployment story is likely going to be much simpler than Sony’s.
I waited forty minutes to play the newest The London Heist demo. In it, I rode shotgun in a truck next to a London thug as motorcycles and vans with machine gun wielding riders passed by and shot at me. I shot back, but strangely the most fascinating part for me was opening the glove compartment with the Move controllers and fiddling with the radio controls.
Prepare for another digression or just skip ahead if you like. While I was using Playstion Move controllers (those two lit up things in the picture above that look like neon ice-cream cones) in the Sony booth to change the radio station in my virtual van, BMW had a tent outside the convention center where they demoed a radio tuner in one of their cars that responded to hand gestures. One spun ones finger clockwise to scan through the radio channels. Two fingers pressed forward would pause a track. Wave would dismiss. Having worked with Kinect gestures for the past five years, I was extremely impressed with how good and intuitive these gestures were. They can even be re-programmed, by the way, to perform other functions. One night, I watched my boss close his eyes and perform these gestures from memory in order to lock them into his motor memory. They were that good, so if you have a lot of money, go buy all four VR sets as well as a BMW Series 7 so you can try out the radio.
But I digress. The London Heist is a fantastic game and the Playstation VR is pretty great. I only wish I had a better idea of when it is being released an how much it will cost.
Another great thing about the Sony PlayStation VR area was that it was out in the open unlike the VR demos from other companies. You could watch (for about 40 minutes, actually) as other people went through their moves. Eventually, we’ll start seeing a lot of these shots contrasting what people think they are doing in VR with what they are really doing. It starts off comically, but over time becomes very interesting as you realize the extent to which we are all constantly living out experiences in our imaginations and having imaginary conversations that no one around us is aware of – the rich interior life that a VR system is particularly suited to reveal to us.
I found the OSVR demo almost by accident while walking around the outside of the Razer booth. There was a single small room with a glass window in the side where I could spy a demo going on. I had to wait for Tom’s Hardware to go though first, and also someone from Gizmodo, but after a while they finally invited me in and I got to talk to honest to goodness engineers instead of marketing people! OSVR demoed a 3D cut scene rather than an actual game and there was a little choppiness which may have been due to IR contamination from the overhead lights. I don’t really know. But for $299 it was pretty good and, if you aren’t already the proud owner of an Oculus DK2, which has the same specs, it may be the way to go. It also has upgradeable parts which is pretty interesting. If you are a hobbyist who wants to get a better understanding of how VR devices work – or if you simply want a relatively inexpensive way to get into VR – then this might be a great solution.
You could also go even cheaper, down to $99, and get a Samsung Gear VR (or one of a dozen or so similar devices) if you already have a $700 phone to fit into it. Definitely demo a full VR head-mounted display first, though, to make sure the more limited Gear VR-style experience is what you really want.
I also wanted to make quick mention of AntVR, which is an indie VR solution and Kickstarter that uses fiducial markers instead of IR emitters/receivers for position tracking. It’s a full walking VR system that looked pretty cool.
If walking around with VR goggles seems a bit risky to you, you could also try a harness rig like Omni’s. Ignoring the fact that it looks like a baby’s jumporee, the Omni now comes with custom shoes so running inside it is easier. With practice, it looks like you can go pretty fast in one of these things and maybe even burn some serious calories. There were lots of discussions about where you would put something like this. It should work with any sort of VR setup: the demo systems were using Oculus DK2. While watching the demo I kept wanting to eat baby carrots for some reason.
According to various forecasters, virtual reality is going to be as important a cultural touchstone for children growing up today as the Atari 2600 was for my generation.
To quickly summarize (or at least generalize) the benefits of each of the four main VR systems coming to market this year:
1. Oculus Rift – first developed and first to release a full package
2. HTC Vive Pre – best controllers and position tracking
Valve’s recent announcement about their new Vive headset for virtual reality as well as Epic’s announcement that the Unreal Engine is now free made me realize that it is time to once again catalog the current set of future technologies vying for our attention. Just as pre-NUI computer users need the keyboard and mouse, the post-NUI user needs sensors and just as the pre-NUI user required a monitor to see what she was doing, the post-NUI user needs a headset. Here is the list for 2015 from which, you will notice, Google Glass is now absent: