Almost exactly twenty years ago, my family and I were living near Burlington, Vermont, and I was working remotely for a small graphics software company in California. They were great people to work for, but I had the sense that their business wasn’t doing well, and living as I was far from potential employers, I had to be proactive in anticipating problems. So I cast about for other work; two different lines of inquiry led to Microsoft, and I ended up as a contractor working on the VGA driver for the first version of Windows NT, which was a little more than a year from being released.
After a couple of months of that, I was offered the opportunity to interview for a full-time position. That was a harder decision than you might think, because if I got the job, we would have to pick up and move across the country. We liked Vermont; it was a good place to raise children, it was beautiful, my wife was working on a master’s degree and putting down roots, and it was a low-pressure lifestyle. I’d have more job security at Microsoft, and my work would be interesting, but we’d be giving up a lot.
In the end, I decided to interview, and when I was offered a job as the NT graphics driver lead, I took it. A few months later, the dev lead for GDI retired, and I ended up in that position for the first two versions of NT. Working on NT was a great experience on the whole. For one thing, Dave Cutler gave me a whole new perspective on what it meant to write good software, and what it took to do that. For another, I helped bring an operating system into existence that directly benefited a lot of people. In his book Flow, Mihaly Csikszentmihalyi describes work satisfaction as a function of having challenging but achievable goals combined with a sense that the work is worthwhile but not overwhelmingly so, and my work on NT fell right in that sweet spot.
How did it all work out in the end? Well, I’m not at Microsoft anymore, but we’re still content living in the Seattle area 20 years later, so the move was a good one. Equally important, prior to NT I had worked mostly on low-impact projects for small companies; starting with NT, I’ve gotten to work on interesting stuff that really matters. So things worked out well as far as I’m concerned.
I’d say it worked out well for Microsoft as well, since I played a significant role in getting GDI and the graphics drivers done and shipped, and NT became a huge success. (To be clear, I was just one programmer among many excellent ones on the NT team, and I’d love to work with most of them again. In fact, I’m working with two of them now at Valve; if you were part of that NT team and you’re ready for a new challenge in an amazing environment, drop me a line.)
But here’s the funny part: I completely bombed the first of the five interviews in my interview loop. The interviewer asked me Windows debugger questions; alas, I didn’t know much about Windows, let alone Windows debuggers, back then. I could not possibly have done worse. If they had continued with the usual interview process, there’s no way I would have gotten the job, because I just didn’t have the kind of experience Microsoft looked for in their standard interview loop. However, the next interviewer, Darryl Havens, said, “Okay, that was a waste of time. What do we have to do to get you here?”, and that set the tone for the rest of the day. (Thank you, Darryl!) Darryl knew that the graphics drivers had been badly neglected until I started working on them, and NT couldn’t ship until they were solid; hiring me was the fastest way to fix that.
If Microsoft had stuck to its interview process, I wouldn’t have been hired, and that would not have been good for either Microsoft or me. One implication of this that has stuck with me through the years is that it’s a bad idea to get too attached to a particular way of thinking about or doing anything. The world is complicated and constantly evolving, so it’s essential to constantly reexamine your plans, decisions, processes, assumptions, and mental models to see if they’re still tracking reality.
My first post on this blog talked about why augmented reality (AR) could well be the next great platform shift, and I still think that’s likely to be true. However, as I’ve worked on AR, I’ve been checking my assumptions, and as part of that process I’ve been thinking about whether a drive straight for AR or a path that includes VR as well – especially in the near term – makes more sense. There are good arguments for both sides, and it’s been an interesting exercise in visualizing the future. In this post, I’ll follow one line of thought that argues for an increased emphasis on VR; in the next post, I’ll follow another that concludes that AR should remain the dominant area for R&D, even in the immediate future. I don’t yet know what the correct choice is, so don’t expect any profound conclusions at the end, but the thought processes are interesting in their own right, and provide some insight into how the future of wearable computing could evolve.
As you read, please keep in mind that I’m not saying this is how it will be, but rather here’s a way it could be. The point is not to wrap things up with a neat bit of prophecy, since I don’t know what the future will hold, but rather to get you thinking, and to start a discussion that I’m looking forward to continuing in the comments.
Before I begin, I’d like to make it clear that this post and the next reflect my thinking, not Valve’s, and don’t represent a product or strategy announcement in any way. They’re just thought experiments on my part, trying to catch a glimpse of what promises to be a really interesting future.
A few definitions
If you’re not familiar with VR and AR, VR is the one where you sit down, put on a headset, and find yourself completely immersed in a virtual world like Snow Crash’s Metaverse or Ready Player One’s OASIS (and if you haven’t read Ready Player One, run don’t walk; it’s a great read, especially if you grew up in the 80’s, but even if not – I didn’t, and I still loved it). AR is the one where you put on glasses and walk around, and find that the real world is still there, but modified to a lesser or greater extent, as in Rainbow’s End’s belief circles or the Rivet Couture virtual society of “To Hie from Far Cilenia.”
So with VR, you might take a seat at your computer, put on your VR headset, and find yourself in Middle Earth or a starship or a Team Fortress 2 level. With AR, as you walk down the (real) street wearing your AR glasses you might find that there are (virtual) aliens shooting at you, or that when you encounter (real) members of your Belief Circle they’re wearing (virtual) medieval costumes and glowing faintly, or, to continue the TF2 analogy, that everyone you see is wearing virtual hats.
The sort of AR I just described, which is what I’m going to talk about in this post, is unconstrained AR – what I call walk-around AR, the kind that works wherever you go. That’s certainly the long-term goal, because it’s a platform shift, but for the next few years it’s something of a strawman, because there are a lot of challenging technical issues to be ironed out before it’s ready for prime time. In contrast, highly constrained AR, for example tabletop or room-scale AR, is considerably more feasible than walk-around AR right now, and certainly has some potentially interesting uses. However, it’s obviously not as generally useful as walk-around AR, is less immersive than VR, and is currently farther from a consumer-ready product than VR, with less capable, more expensive hardware. Nonetheless, constrained AR is a strong counter-argument to VR’s near-term advantage, and will feature prominently in the next post.
There’s also a third sort of wearable display technology, which I’ll call HUDSpace, based on the display of 2D information on see-through glasses, much like having a phone or tablet in view at all times; this is the direction Google appears to be going in with Project Glass. I include in this category very lightweight AR such as having people’s names floating over their heads, arrows to guide you turn by turn to your destination, and information popping up when you’re near points of interest. There’s a great deal of value to this, and it’s clearly going to happen, but it’s considerably less technologically demanding than AR or VR, has little opportunity for deep entertainment experiences, seems largely like an extension of the smartphones we have today rather than a genuinely new platform, and is just way less cool to me, so I’m going to focus on AR and VR.
So if AR is where we’re all headed, why is VR worth bothering with? Two reasons: in the long run, VR-like experiences may be how we use our spiffy AR glasses much of the time, and in the short run, VR is poised to take off well before AR.
Why VR is interesting now
Right now, VR is much closer to becoming a consumer product than AR. Perhaps the biggest reason for this is that VR hardware is more capable and easier to make right now. The Oculus Rift, which is intended to ship at a consumer price, has a 90-degree horizontal field of view; in contrast, I’ve never heard of see-through AR glasses with anything like that field of view at any price, and while they may exist, it’s hard to see how they could be made at consumer prices with anything like current technology. (Video-passthrough AR glasses could of course have the same field of view as the Rift, since all that would be required would be to add a camera, but I don’t think video-passthrough AR will be good enough for a number of years, for reasons discussed here.) Also, because VR is used in a fixed location, it can be tethered, sweeping away a host of hard power problems that walk-around AR has to deal with, and enabling the use of far more powerful CPUs and GPUs. Alternatively, VR headsets can be designed to run for just an hour or two between recharges; in contrast, AR has to have the same order of battery life as a phone or tablet. Furthermore, because VR is restricted to one location, it’s much easier to develop tracking technology for. And since you’re not going to wear a VR head mounted display in public, or walk around with it, it doesn’t have to be as stylish, and while it still has to be light and comfortable, it is considerably less constrained than AR glasses that have to look like fat sunglasses. Finally, VR can use existing controllers initially; you’ll be able to play VR games with standard game pads, for example, although I think new VR input will have to evolve quickly in order for VR to really reach its potential. In contrast, the input scheme for AR is an open question.
In terms of hardware problems to be solved, VR is closely related to AR, and in many cases figuring something out in VR’s more tractable space will help in AR as well. In this respect, resources devoted to VR R&D aren’t subtracted from AR efforts; in fact, this may be the most effective way to make progress on technology related to AR, because VR hardware can be made fully functional and iterated on much more rapidly than AR at this point.
This is particularly true because a VR marketplace appears to be emerging as I write this, in the form of the Oculus Rift and support for it in Doom 3: BFG Edition, Hawken, and other games, while AR is still some distance from viable products. It’s far easier to push technology forward when there are real customers to provide feedback, real products to provide incentive for better, cheaper components, and real revenue to spur competition, and VR will likely have all those long before AR does.
VR is more approachable on the software side as well. New experiences often evolve from existing experiences; it’s hard to make a complete break with the past in every respect, if only because your audience will be confused, and also because it’s hard for developers to solve multiple problems in a new space simultaneously. There’s a direct path to at least some interesting VR experiences; PC and console games like first-person shooters and flight, space, and car sims are designed for immersion, and should seem like they’re on steroids in VR. It’s even more obvious what interesting HUDspace experiences are; a few are listed above. However, it’s not at all clear what will constitute compelling walk-around AR experiences. I have no doubt that they exist, but they’re unknown right now. (It’s a lot clearer what might be interesting for constrained AR, and we’ll look at the implications of that in the next post.)
VR for the long run
So VR looks pretty good in the short run; how about after that? Even though I think it’s likely that in the long run (defined as five to ten years) AR will have a more radical effect on our lives, it’s possible that VR-like experiences will be where we will spend more of our time once we have really good AR glasses. The key is that AR glasses will be able to get darker or lighter on demand, because that’s necessary in order to work well in both dimly lit rooms and bright sunlight. That means they’ll be able to become almost completely dark at any time – and when they do, they’ll effectively be VR glasses. So your AR glasses will be able to provide both AR and VR experiences.
That’s interesting because VR experiences are richer in important ways. VR is more immersive, and that’s a big plus for many types of games. VR also has better contrast, since it doesn’t have to compete with photons from the outside world, so virtual images will look better. Because VR doesn’t have to interact with the real world, it doesn’t suffer from any of the inconsistencies that inevitably arise in AR; for example, lighting and shadowing in VR can be completely consistent. VR also avoids all the work that’s required in AR to figure out what real-world objects are in the field of view at any time, and to calculate how virtual and real images interact. Another point in VR’s favor is that it has no equivalent to the per-pixel opaquing limitation of AR, so VR software has complete control over the image that reaches the eye. Furthermore, small amounts of latency and tracking error may be more acceptable in VR, because the virtual images don’t have to match the real scene; since we’re not going to get to zero latency or perfect tracking anytime soon, that’s potentially a significant plus. (However, it’s also entirely possible that small amounts of latency and tracking error could cause simulator sickness under conditions of full immersion; this is one of many areas that we’re all going to learn a lot more about in the next few years.)
So AR is the only way to go when you want the virtual and real worlds to interact, but VR and VR-like experiences seem best for purely virtual experiences. (Here, “VR-like” means AR when it dynamically becomes opaque enough so that the virtual world is visually dominant.)
And it’s arguable that you spend most of your time in experiences that are more virtual than real (or at least that I do).
Our lives are more virtual than you might think
You’re probably thinking that you don’t spend any significant amount of time in virtual experiences, but consider: as you read this, you’re looking at a screen. Imagine you’re doing it on a head-mounted display, and you’ll see that it maps better to VR than to AR. Sure, you could have the text floating in your field of view while still seeing the real world, but why? It seems far more useful to just look at a virtual screen in VR, since all that’s of interest is the text. You could have lots of virtual screens up in 3-space around you, and you could have information presented in all sorts of other ways as well.
Similarly, the real world often doesn’t play an important role in watching TV or movies, or playing video games; certainly it does when you’re with friends, but when you’re alone, the real world doesn’t particularly enhance the experience. And if you ask yourself what percentage of your waking time you spend looking at a screen by yourself, you’ll find it’s a majority if you’re anything like me. So that’s why I say that VR-like experiences may be where we’ll spend a lot of our time once we have good AR glasses; until that time, this argues that VR by itself is interesting.
This is not to say being able to see the real world at the same time as the virtual world doesn’t have benefits; I’ll discuss that aspect in the next post. One thing that absolutely has to be figured out for VR is how to become not-blind instantly, for example by touching a control on the glasses that switches to a camera view; being unable to see without taking the HMD off just isn’t going to be acceptable in a consumer device.
Finally, there’s a wild card that could change the long-term balance between AR and VR dramatically. My thinking to date has assumed that AR will be a major platform shift that fundamentally changes the way we interact with computers, while VR won’t, except to the extent that VR-like experiences are part of the AR future. However, it’s possible that that VR will be a major platform shift all on its own; we could all end up spending our time sitting in our living rooms wearing VR headsets and haptics, while the traditional movie, TV, and videogame businesses wither. (In fact, I’d be surprised if that wasn’t the case someday, but I think it’ll be a long while before that happens.) We all know what that would imply, since we’ve all watched Star Trek – that way lies the Holodeck. If that happens, VR is more than interesting; it’s a big part of the future.
All of which implies that VR and VR-like experiences seem likely to be important in the long run.
Summing up the case for VR
None of the foregoing says that standalone VR is going to be more important or successful than AR in the next five to ten years, although that could happen. AR is most likely going to change the way we interact with the world, much as PCs and smartphones did, long before VR makes it to the Holodeck. However, it seems likely that VR is much closer to being deliverable in a truly workable form than walk-around AR, and it also seems likely that VR-like experiences will be an important part of the ultimate AR future. Given which, there’s a strong case to be made that while the long-term goal is to produce superb, do-everything AR glasses, VR and VR-like experiences are worth pursuing as well, both in the near term and down the road.