Ramblings in Valve Time

When I put up my first post, back in April, I was immediately buried in email. Over the next few months, I gradually dug out from under the pile, responding to most of the people who had written. However, there were a few hundred that I put in a folder for later, and there they sat until it seemed too late to respond. A lot of those emails were from people asking how they could get into the game industry, or what they should do to get hired at Valve, and for them, the best answers I can give are in this post and the Valve Handbook. If you mailed me back in April, and if after reading the above links and the blog to this point your question or topic remains unaddressed, please send your mail again – my turnaround time is much better these days.

The first video game I ever wrote ran in monochrome, at 160×72 resolution. My next four games moved up to four colors at 320×200 resolution. The game after that (Quake) normally ran with 256 colors at 320×200 resolution, but could go all the way up to 640×480 on the brand-new Pentium Pro, the first out-of-order Intel processor.

Those sound like Stone Age display modes, now that games routinely run in 24-bit color at 1600×1200 or even 2560×1600, but you know what? All those games looked great at the time. Quake at 640×480 would look pathetically low-resolution now, but when it shipped, even 320×200 looked great; it’s all a matter of what you’re used to.

That’s relevant just now because the first generation of consumer-priced VR head-mounted displays is likely to top out at 960×1080 resolution, for the simple reason that that’s what you get when you split a 1080p screen across two eyes, and 1080p is probably going to be the highest-resolution panel available in the near future that’s small enough to fit in a head-mounted display. At first glance, that doesn’t seem so bad; it falls short of 2560×1600, or even 1600×1200, but it’s half of the latter, so it’s in the same resolution ballpark as monitors. And besides, it’s way higher-resolution than any of my earlier games, and in fact it’s higher-resolution than anything that was available for more than 15 years after the PC was introduced, and, as I noted, those lower-resolution graphics looked great then. By analogy, VR should be in good shape at 960×1080, right?

Alas, it’s not that simple, because when it comes to resolution, it’s all relative. What do I mean by that? There are two very different interpretations, both applicable to the present discussion. We’ve seen the first one already: how good a given resolution looks depends on what you’re used to looking at. 160×72 looks great when the alternative is a text-based game, but less so next to a state-of-the-art game at 2560×1600. This first interpretation applies to VR in two senses. The first is that VR will inevitably be compared to current PC graphics – clearly not a favorable comparison. However, the second is that, like my early games, VR will also be judged against previous VR graphics in the PC space, and that’s a favorable comparison indeed, since there are none. For the latter reason, if VR is a unique enough experience, people will surely be very forgiving about low resolution; the brain is very good at filling in details, given an otherwise compelling experience, as happened, for example, with Quake at 320×200.

Another way to think about resolution, however, is relative to the field of view the pixels are spread across. The total number of pixels matters, of course, but the density of the pixels matters as well, and it’s here that VR faces some unique issues. Let’s run some numbers on that.

My very first game ran on a monitor that I’d estimate to have a horizontal field of view of maybe 15 degrees at a normal viewing distance. At 160×72, that’s about 11 pixels per horizontal degree.

A 30” monitor at 2560×1600 has about a 50-degree field of view at a normal viewing distance. That’s roughly 50 pixels per horizontal degree, and approximately the same is true of a 20” monitor at 1600×1200.

The first consumer VR head-mounted displays should have fields of view that are no less than a 90 degrees, and I’d hope for more, because field of view is key to a truly immersive experience. At 960×1080 resolution, that yields slightly less than 11 pixels per horizontal degree – the same horizontal pixel density as the CP/M machine I wrote my first game for in 1980, and barely one-fifth of the horizontal pixel density we routinely use now.

And that’s only the horizontal pixel density. The vertical pixel density is the same, and in combination they mean that a first-generation consumer head-mounted display will have about one-twentieth of the two-dimensional pixel density of a desktop monitor. As another way to understand just how low a wide field of view drives pixel density, consider that the iPhone 5 is 640×1136 – two-thirds as many pixels as the upcoming head-mounted displays, packed into a vastly smaller field of view; at a normal viewing distance, I’d estimate the iPhone has roughly 100 pixels per degree, so overall pixel density could be close to one-hundred times that of upcoming VR head-mounted displays.

It is certainly true that the brain can fill in details, especially when viewing scenes filled with moving objects. However, it would be highly optimistic to believe that a reduction in pixel density of more than an order of magnitude wouldn’t be obvious, and indeed it is. It’s certainly hard to miss the difference between these two images, which reflect the same base image at two different pixel densities:

And that’s only a 4X difference – imagine what 20X would be like.

If there were no monitors to compare to, low pixel density might not be as noticeable, but there are, not to mention omnipresent mobile devices with even higher pixel densities. Also, games that depend on very precise aiming may not work well on a head-mounted display where pixel location is accurate to only five or six arc-minutes. For that reason, antialiasing, which effectively provides subpixel positioning, will be very important for at least the first few generations of VR.

That’s not to say that the upcoming VR head-mounted displays won’t be successful; a huge field of view, together with high-quality tracking and low latency, can produce a degree of immersion that’s unlike anything that’s come before, with the potential to revolutionize the whole gaming experience. But I can tell you from personal experience that the visual difference between a 960×1080 40-degree horizontal field of view head-mounted display and a 640×800 90-degree HFOV HMD (both of which I happen to have worked with recently) is enormous – what looks like a blurry clump of pixels on one looks like a little spaceship you could reach out and touch on the other – and that’s only a ten-times difference.

So I’m pretty confident that we’ll be begging for more resolution from our head-mounted displays for a long time. Obviously, that was also the case for decades with monitors; the difference here is that every day we’ll encounter much higher pixel densities on our monitors, our laptops, our tablets, and even our phones than on our head-mounted displays, and that comparison is going to be a challenge for the consumer VR industry for some time to come.

Given which, the obvious question is: how high does VR resolution need to go before it’s good enough? I don’t know what would be ideal, but getting to parity with monitors in terms of pixel density seems like a reasonable target. Given a 90-degree field of view in both directions, 4K-by-4K resolution would be close to achieving that, and 8K-by-8K would exceed it. That doesn’t sound all that far from where monitors are now, but actually it’s four to sixteen times as many pixels; there’s no existing video link that can pump that many pixels – in stereo – at 60 Hz (which is the floor for VR), not to mention the lack of panels or tiny projectors that can come close to those resolutions (and the lack of business reasons to develop them right now), so pixel density parity is not just around the corner. However, if VR can become established as a viable market, competitive pressures of the same sort that operated (and continue to operate) in the 3D graphics chip business will drive VR resolutions, and hence pixel densities, rapidly upward. VR could well become the primary force driving GPU performance as well, because it will take a lot of rendering power to draw 16 megapixels, antialiased, in stereo, at 60 Hz – to say nothing of 64 megapixels.

Believe me, I can’t wait to have a 120-by-120-degree field of view at 8K-by-8K resolution – it will (literally) be a sight to behold. But I’m not expecting to behold it for a while.

The year is 2015. Wearable glasses have taken off, and they’re game-changers every bit as much as smartphones were, because these descendants of Google Glass give you access to information everywhere, all the time. You’re wearing these glasses, I’m wearing these glasses, all the early adopters are wearing them. We use them to make and receive phone calls, send and receive texts, do instant messaging, do email, get directions and route guidance, browse the Web, and do pretty much everything we do with smartphones today.

However, these glasses don’t support true AR (augmented reality); that is, they can’t display virtual objects that appear to be part of the real world. Instead, they can only display what I called HUDSpace in the last post – heads-up-display-type information that doesn’t try to seem to be part of the real world (Terminator vision, for example). No one particularly misses AR, because all the functionality of a smartphone is there in a more accessible form, and that’s enough to make the glasses incredibly useful.

But then someone comes out with a special edition of their HUDSpace glasses; the special part is that if you put a marker card down on a convenient surface, you can play virtual card and board games on it, either by yourself or with friends. This is a reasonably popular novelty; then the offerings expand to include anything you can play on a table – strategy games, RTSes, arena-type 2D arcade games extruded into 3D, and new games unique to AR – and someone comes out with a version of the glasses that doesn’t need markers, so you can play games in boring meetings, and suddenly everyone wants one. The race is on, and soon there’s room-scale AR, followed by steady progress on the long march toward general, walk-around AR.

And that’s how I think it’s most likely AR will come into our daily lives.

Last time, I described how my original thinking that AR was likely to be the next great platform shift had evolved to consider the possibility that VR (virtual reality) might be equally important, at least in the short and medium term, and far more tractable today, so perhaps it would make sense to pursue VR as well right now. (See the last post for definitions of AR, VR, and other terms I’ll use in this post.) Then I made the case for VR as the most promising target in the near future. I personally think that case is pretty compelling.

This time I’ll make a case for AR as more likely to succeed even in the short term (last time I explained why I think it’s the most important long-term goal), and I think that case is pretty compelling too. The truth is, given infinite resources, I’d want to pursue both as hard as possible; one doesn’t preclude the other, and both could pan out in a big way. But resources (especially time) are finite, alas, and choices have to be made, so a lot of thought has gone into choosing where to focus, and these two posts recount some of that thinking.

Of course, I hope we get this right, but as Yogi Berra put it: “It’s tough to make predictions, especially about the future.” All we can do is make our best assessment, start doing experiments, and see where that leads, constantly reevaluating and course correcting as needed (and it will be needed!). So I’m by no means laying out a roadmap of the future; this post and the last are just two possible ways wearable computing might unfold.

The first step in assessing whether to focus on AR or VR is to figure out how each is most likely to succeed, and then to compare the strengths and weaknesses of each of those probable paths. The likely path for VR is obvious, and already in motion: The Oculus Rift will come out, running ports of existing PC games. If it’s even moderately successful, games will be written specifically for VR, Oculus will improve the hardware and competitors will emerge, VR will likely spread to mobile and consoles, and the boom will be on.

The path AR might take to success is less clear, because there are many types of AR – tabletop, room-scale, and walk-around – and several platforms it could emerge on – PC, mobile, and console. Also, as the scenario I sketched out at the start illustrates, AR could evolve from HUDSpace. So let’s look in more detail at that scenario, and then examine why I think it might be more promising than other paths for AR and VR.

In my scenario, AR isn’t even part of the picture at first; see-through glasses emerge, but wearable computing develops along the Google Glass path, supporting only display of information that doesn’t appear to be part of the real world, rather than true AR. To be clear, it’s quite possible that Google Glass won’t be see-through, but will just provide an opaque information display above and to the side of your normal line of sight. However, I think see-through glasses have much more potential, if only because they’ll have much more screen real estate, won’t block your view, allow for in-place annotation of the real world, and will be more comfortable to look at. That’s a good thing for my scenario, since see-through potentially leads to AR, while an opaque display out of the line of sight doesn’t.

Having information available everywhere, all the time will be tremendously valuable, and HUDSpace glasses will probably become widely used; in fact, you could make a strong argument that people who wear them will seem to be smarter than everyone else, because they will have faster access to information. Think of all the times you’ve hauled out your phone in a conversation to look something up, and now imagine you can do that without having to visibly query anything; you’ll just seem smarter. (Obviously I could be wrong in assuming that HUDSpace glasses will be widely used – it may turn out that people hate having information fed to them through glasses – but certainly there’s a strong argument to be made that better access to information is likely to be compelling, and since the rest of this scenario depends on it, I’ll just take it as a given.)

You may well wonder why these glasses wouldn’t have AR capabilities – after all, even cellphones can do AR today, right? Here I need to draw a distinction between true AR and cellphone AR. Cellphone AR, although interesting, is at best a distant cousin to true AR, for one key reason: cellphone AR doesn’t have to fool your entire visual perception system into thinking virtual images exist in the real world. By this I’m referring not to photorealistic rendering (the eye and brain are quite tolerant of cruder rendering), but rather to the requirement that virtual images appear to be solid, crisp, and in exactly the right place relative to the real world at all times as your head moves. The tolerance of the human visual system for discrepancies in those areas when viewing 3D virtual images that are supposed to appear to be part of the real world – that is, true AR – is astonishingly low; violate exceedingly tight parameters (for example, something on the order of 20 ms for latency), and virtual objects simply won’t seem like they’re part of the world around you. With cellphone AR, you’re just looking at a 2D picture, like a TV, and in that circumstance there are all sorts of automatic reflexes and visual processing that don’t kick in, which greatly relaxes hardware requirements.

The visual system’s low tolerance for mismatches between the virtual and real worlds means that the hardware required to make true AR work well is significantly more demanding – and expensive – than the hardware needed for HUDSpace. This is particularly the case for general, walk-around AR, which has to constantly cope with new, wildly varying settings and lighting, but it’s true even for room-scale and tabletop AR, primarily due to the requirements for display technology and tracking of the real world. At some point I’ll post about those areas, but for now, trust me, it’s a lot easier to build glasses that display HUD information, or at most images that are loosely related to the real world (like floating signs in the general direction of restaurants) than it is to build one that displays virtual images that fool your visual system into thinking they exist in the real world.

Given that true AR is hard, expensive, and not required, HUDSpace glasses will initially almost certainly not support true AR. Interestingly, because they’ll almost certainly be see-through, HUDSpace glasses won’t even support cellphone AR well,.

So in this scenario, a few years from now we’re all wearing HUDSpace glasses and using them to do what we do now with a smartphone, but more effectively, because the glasses give us access to information all the time, and privately. They’ll also do things that a smartphone isn’t good at, such as popping up the names of people you encounter, which you can’t politely use your smartphone to do. The obvious difference from a smartphone is that the glasses won’t have a capacitive touchscreen, and honestly I don’t know what the input method will be, but there are several plausible answers, so I’ll assume that’ll work out and skip over it for now. Several large companies are making HUDSpace glasses, and the competition is as fierce as it is in smartphones today. All kinds of great apps are being written for the glasses, including HUDSpace versions of existing casual and location-based games, but there are no true AR apps, because the hardware doesn’t support them.

As I described in the opening, it’s at this point that someone will probably put a camera on their glasses that’s good enough for tabletop AR, probably with the help of a fiducial (a marker designed to be tracked by a camera) placed where you want the AR to appear. Add good tracking code, and you’ll be able to play any tabletop game anyone cares to write. The glasses will be networked, so you’ll be able to play any card or board game you can think of, and you’ll be able to do that either with someone sitting at the table with you or with anyone on the Internet. Better tracking hardware and software will eliminate the need for fiducials, and the Tetris or Angry Birds of tabletop AR will appear, sparking a rapidly escalating AR arms race, similar to what happened with 3D accelerators and 3D games. AR will expand to room scale, which will involve group games, of course, and a general expansion of current console gaming possibilities, but also non-game applications like construction kits (living room Minecraft- and Lego-type applications), virtual toys, and virtual pets, and at that point there will be a critical mass of AR users, hardware, and software that makes it economically and technically feasible to start chipping away at walk-around AR. It’ll probably take a decade or two, or even more, before truly general AR exists, but it’s easy to see how an accelerating curve heading in that direction could spring from the first wearable glasses that provide a good-enough tabletop AR experience.

There are several reasons I think evolving from HUDSpace is a more likely way for AR to come into broad use than emerging as a fully-formed product on its own.

The first thing you’ll notice is that my favored scenario doesn’t involve walk-around AR at all for a long time. That’s a huge plus; even though I think walk-around AR is the end point and hugely valuable, it’s very hard to get to in any near-term timeframe. One problem with a lot of potential technological innovations is that they require abandoning existing systems and making a wholesale jump to a new system, and it’s hard to make all the parts of those sorts of transitions happen successfully at the same time. That’s certainly true of walk-around AR, which would require display, image-generation, and tracking technology that doesn’t exist today, all packaged in a form factor similar to bulky sunglasses, running on a power budget that far exceeds what’s now possible in a mobile device, along with completely new types of applications, as I discussed in the last point. Honestly, though, I used walk-around AR as a strawman in that post; it’s clear that it’s a long way away from being good enough to be a product, so it served as a useful counterpoint to illustrate the advantages of VR. Constrained AR, both room-scale and tabletop, lies somewhere between walk-around AR and VR, and is much closer than walk-around AR to being ready for broad use, although not as close as VR. Room-scale AR has many of the same technical challenges as walk-around AR, although to a lesser degree; tracking, for example, is difficult, but there are potentially workable, albeit currently expensive, solutions. Tabletop AR, on the other hand, is relatively tractable, although not quite to VR’s level; the problem with tabletop AR is primarily that because it’s so limited, it’s simply not as compelling or novel as room-scale or walk-around AR.

AR that emerges in stages from HUDSpace glasses, on the other hand, doesn’t require any great leaps; each step is an incremental one that stands on its own. Solving those problems separately and incrementally is far more realistic, especially assuming the preexistence of a HUDSpace business that’s big enough to justify the R&D AR will need. As a starting point, tabletop AR that evolves from HUDSpace glasses involves tracking that’s doable today, optics and image projectors that will be a manageable step from HUDSpace, power and processing technologies that will be largely driven by phones, tablets, and HUDSpace glasses, and initial software that’s familiar, including at least the tabletop games I listed in the introduction.

In short, the technological path from HUDSpace glasses to HUDSpace-plus-tabletop-AR glasses seems realistic, while going from nothing directly to walk-around or even room-scale AR seems like a big stretch. That’s true not only technically, but also in a business sense, because HUDSpace-plus-tabletop-AR doesn’t require AR to justify the cost of the hardware by itself; in contrast, standalone AR systems would be in direct competition with consoles and dedicated gaming devices, with all the costs and risks that involves.

Consider two products that support AR. The first product is a special edition of a widely-used pair of HUDSpace glasses that is normally sold for $199; the special edition sells for $299 because it has cameras and more powerful processors that let it support tabletop AR gaming. The second product is a pair of AR glasses designed specifically for living-room use; it supports room-scale AR games that you can play on your own or with friends, and costs $299, plus $199 for each additional pair of glasses.

Even though the pure AR glasses are more powerful and would support a wider variety of novel experiences for the same total price, it’s hard to see how they could be successful unless the experience was truly awesome. At $299 and up, this would be going directly against existing consoles, and it’s hard to make the first games for a whole new type of gaming be killer apps that it’s worth buying the whole system for, because it takes time to figure out what unique experiences new hardware makes possible. Getting developers to devote effort to support a new, unproven platform is hard as well – it obviously can be done, but it’s a major undertaking. Also, the up-front expenditures and risk would be relatively large, since this would be a new type of product that at least overlaps with the existing console space. In short, it would require a console-scale effort, with all the risk a new console with new technology involves. A tabletop AR product would be less of a step into the unknown, and could be somewhat less expensive – but at the same time it would be more limited and less novel than room-scale AR, so there’s still the question of whether it’d be compelling enough to justify the purchase of a complete system. I’d love to be wrong – It’d be great if a standalone tabletop or room-scale AR system could be successful on its own merits. It just seems like they would have to overcome considerably greater market and technical challenges than evolution from HUDSpace glasses.

On the other hand, I have no problem imagining that a lot of people who are buying the HUDSpace glasses anyway – which they will be, because they’re very useful – would spend $100 to upgrade to make them more fun to use. The key here is that AR itself doesn’t have to justify the cost of the system, just the much smaller upgrade cost. You might say that’s not fair, that it’s not as powerful a system, but that’s the point – in the beefed-up HUDSpace case, AR doesn’t have to be compelling enough to justify the purchase of the glasses in the first place. If you want to convince people to buy a whole new system to put in their living room, or to buy a dedicated AR system for tabletop gaming, you have to get over the barrier of convincing them that they want to own yet another gaming device. If, on the other hand, you want to sell people established HUDSpace glasses with tabletop AR capability, they’ve already decided to make a purchase, and it’s just a question of whether they want to buy a cool and not very expensive option; in fact, far from being a barrier to purchase, the AR option makes the purchase of HUDSpace glasses more attractive.

Better yet, if you want to play a multiplayer game with someone else, they’re likely to have their own glasses, since HUDSpace glasses will probably be widely used, so there’s no incremental cost for multiplayer. The network effect from widespread adoption based on HUDSpace is a huge advantage for beefed-up HUDSpace glasses.

The bottom line is that the HUDSpace-plus-tabletop-AR scenario is a pull model, with the right incentives; a lot of the hardware and a sizeable market get developed for HUDSpace independent of AR, and AR then serves as an enhancement to help sell HUDSpace glasses into that existing market. In contrast, any scenario involving a standalone AR product is a push model, where a market for a new type of relatively expensive product has to be created and developed rapidly, in competition with existing consoles. It could happen, but it seems less likely to succeed.

Now that you know how I think AR is most likely to emerge, and that it will likely be constrained to tabletop and possibly room-scale AR for quite a while, we can return to our original question, which is whether it makes sense to pursue AR only, or a mix of AR and VR, especially in the near- and medium-term. Last time I discussed why VR was interesting; now it’s time to talk about why AR might be more interesting.

I will first note again that last time I compared VR to walk-around AR, and that that was a strawman argument. I don’t think there’s any world in which true walk-around AR is feasible in any way in the next five years. As I discussed above, the challenges that constrained AR – room-scale and tabletop – face are similar to but far less daunting than walk-around AR, and constrained AR is probably doable to at least some extent in the next five years, a somewhat but not greatly longer timeframe than VR, so the question is which makes more sense to pursue.

First off, technically VR is easier to implement with existing and near-term technology; that’s just a fact, as evidenced by the Oculus Rift. The Rift definitely has some rough edges to smooth out, but there are ways to address those, and I expect Oculus to ship a credible product at a consumer price; in contrast, as of this writing, I have been unable to obtain a pair of AR glasses capable of being a successful consumer product. The core issue generally has to do with the great difficulty of making good enough see-through optics in glasses that with acceptable form factor and weight. However, I know of several approaches in development, any of which would be sufficient if all the kinks were ironed out, and it seems probable that this will be solved relatively soon, so it’s a disadvantage for AR, but not a decisive one.

VR is also more immersive in several ways: field of view, blocking of real-world stimuli, and full control over the color and intensity of every pixel , which can make for deeper, more compelling experiences, but there are downsides as well. Immersion may not be good for extended use, either because it induces unpleasant sensory overload or simply because it makes people sick. AR provides anchoring to the real world, and that helps a lot; I personally get simulator sickness quite easily with existing VR systems, but rarely have that problem with AR. I’m confident that AR will be easier for most people to use for long periods than VR.

Another advantage that comes with being less immersive is awareness of the real world around you, and that’s a big one.

For starters, being not-blind means that you can reach for your coffee or soda, find the keyboard and mouse and controller, answer the phone, and see if someone’s come into the room. This is such a big deal that I believe VR will not be widely adopted until VR headsets appear that make it possible to be not-blind instantly, most likely by being able to switch the display over to the feed from a camera on the headset with a touch of a button, but also possibly with a small picture-in-picture feed from the camera while otherwise immersed in VR, or with a display that can become transparent instantly.

Being not-blind also means that you can give only a part of your attention to AR. For example, you could have an in-progress game of chess sitting on a corner of your desk; you’d notice it every so often, but you wouldn’t have to be focused on it all the time. This lets you use AR a lot of the time, in a variety of situations. In contrast, when you’re doing something in VR, it’s the only thing you can be doing, which considerably limits the possibilities. Being not-blind also means that you can be mobile while using AR, even if only to move around a table for a better view, while VR pretty much requires you to be immobile, further limiting the possibilities.

Most important, being able to see the real world means that you can have far more social AR experiences with other people than you can with VR. Sitting around the table with your family playing a board game, sitting on the couch with a friend seeing how high you can build a tower together, or having a quadracopter dogfight are all appealing in very different ways than isolated VR experiences, and given how intensely social humans are, those are ways that are arguably more compelling. In this respect, AR experiences will be more complex and unique than VR experiences, since they will incorporate both the real world and that most unpredictable and creative of factors, other people, and consequently have greater potential.

Finally, constrained AR is on the path to walk-around AR, and walk-around AR is where I think we all end up eventually.

At long last, to quote the renowned technology sage Meat Loaf: “What’s it gonna be, boy?” Unfortunately, after our long and interesting journey through possible futures, I’m not going to give you the crisp, decisive answer you (and I) would like, because there are two time frames and two scopes at work here.

There’s no way it makes sense to simply abandon AR for VR. Interaction with the real world and especially with other people is why AR is the right target in the long run; we live our lives in the real world and in the company of other people, and eventually AR will be woven deeply into our lives. In the medium term, I believe AR will likely emerge from HUDSpace roughly along the lines of the scenario above; another possibility is that a console manufacturer will decide to make room-scale AR a key feature, as hinted at by the purported leak of Microsoft’s Project Fortaleza a few months ago. All this makes it highly likely that work on tabletop and room-scale AR now will bear fruit in the future; it might be a little early right now to be working on that, but the problems are challenging and will take time to solve, so it makes sense to investigate them now.

In the near term, though, VR hardware will be shipping, and because the requirements are more limited, it should improve more rapidly than AR hardware. Also, it’s easier to adapt existing AAA titles to VR, and while VR won’t really take off until there are great games that built around what VR can do, AAA titles should get VR off the ground and attract a hard-core gaming audience. And a lot of the work done on VR will benefit AR as well.

So my personal opinion (which is not necessarily Valve’s) is that it makes sense to do VR now, and push it forward as quickly as possible, but at the same time to continue research into the problems unique to AR, with an eye to tilting more and more toward AR over time as it matures. As I said, it’s not the definitive answer we’d all like, but it’s where my thinking has led me. However, I’ve encountered intelligent opinions from one end of the spectrum to the other, and I look forward to continuing the discussion in the comments.

The Sunday New York Times business section has a nice piece about Valve that talks about our wearable effort, among other things, and includes a few quotes from me.

I’ve seen some speculation online that one of the pictures that accompanies the article, featuring Gordon Stoll wearing a headset, depicts a prototype of a Valve HMD. That might be more plausible if the headset didn’t clearly have the letters “NVIS” on it. It’s an off-the-shelf NVIS ST-50 see-through HMD, with a couple of mounts added and a camera mounted in one of them. It’s a useful platform to experiment on, but it’s definitely not a prototype. It’s also not a video-passthrough AR HMD, as some have guessed; it’s see-through, and the camera is used only for tracking.

Almost exactly twenty years ago, my family and I were living near Burlington, Vermont, and I was working remotely for a small graphics software company in California. They were great people to work for, but I had the sense that their business wasn’t doing well, and living as I was far from potential employers, I had to be proactive in anticipating problems. So I cast about for other work; two different lines of inquiry led to Microsoft, and I ended up as a contractor working on the VGA driver for the first version of Windows NT, which was a little more than a year from being released.

After a couple of months of that, I was offered the opportunity to interview for a full-time position. That was a harder decision than you might think, because if I got the job, we would have to pick up and move across the country. We liked Vermont; it was a good place to raise children, it was beautiful, my wife was working on a master’s degree and putting down roots, and it was a low-pressure lifestyle. I’d have more job security at Microsoft, and my work would be interesting, but we’d be giving up a lot.

In the end, I decided to interview, and when I was offered a job as the NT graphics driver lead, I took it. A few months later, the dev lead for GDI retired, and I ended up in that position for the first two versions of NT. Working on NT was a great experience on the whole. For one thing, Dave Cutler gave me a whole new perspective on what it meant to write good software, and what it took to do that. For another, I helped bring an operating system into existence that directly benefited a lot of people. In his book Flow, Mihaly Csikszentmihalyi describes work satisfaction as a function of having challenging but achievable goals combined with a sense that the work is worthwhile but not overwhelmingly so, and my work on NT fell right in that sweet spot.

How did it all work out in the end? Well, I’m not at Microsoft anymore, but we’re still content living in the Seattle area 20 years later, so the move was a good one. Equally important, prior to NT I had worked mostly on low-impact projects for small companies; starting with NT, I’ve gotten to work on interesting stuff that really matters. So things worked out well as far as I’m concerned.

I’d say it worked out well for Microsoft as well, since I played a significant role in getting GDI and the graphics drivers done and shipped, and NT became a huge success. (To be clear, I was just one programmer among many excellent ones on the NT team, and I’d love to work with most of them again. In fact, I’m working with two of them now at Valve; if you were part of that NT team and you’re ready for a new challenge in an amazing environment, drop me a line.)

But here’s the funny part: I completely bombed the first of the five interviews in my interview loop. The interviewer asked me Windows debugger questions; alas, I didn’t know much about Windows, let alone Windows debuggers, back then. I could not possibly have done worse. If they had continued with the usual interview process, there’s no way I would have gotten the job, because I just didn’t have the kind of experience Microsoft looked for in their standard interview loop. However, the next interviewer, Darryl Havens, said, “Okay, that was a waste of time. What do we have to do to get you here?”, and that set the tone for the rest of the day. (Thank you, Darryl!) Darryl knew that the graphics drivers had been badly neglected until I started working on them, and NT couldn’t ship until they were solid; hiring me was the fastest way to fix that.

If Microsoft had stuck to its interview process, I wouldn’t have been hired, and that would not have been good for either Microsoft or me. One implication of this that has stuck with me through the years is that it’s a bad idea to get too attached to a particular way of thinking about or doing anything. The world is complicated and constantly evolving, so it’s essential to constantly reexamine your plans, decisions, processes, assumptions, and mental models to see if they’re still tracking reality.

My first post on this blog talked about why augmented reality (AR) could well be the next great platform shift, and I still think that’s likely to be true. However, as I’ve worked on AR, I’ve been checking my assumptions, and as part of that process I’ve been thinking about whether a drive straight for AR or a path that includes VR as well – especially in the near term – makes more sense. There are good arguments for both sides, and it’s been an interesting exercise in visualizing the future. In this post, I’ll follow one line of thought that argues for an increased emphasis on VR; in the next post, I’ll follow another that concludes that AR should remain the dominant area for R&D, even in the immediate future. I don’t yet know what the correct choice is, so don’t expect any profound conclusions at the end, but the thought processes are interesting in their own right, and provide some insight into how the future of wearable computing could evolve.

As you read, please keep in mind that I’m not saying this is how it will be, but rather here’s a way it could be. The point is not to wrap things up with a neat bit of prophecy, since I don’t know what the future will hold, but rather to get you thinking, and to start a discussion that I’m looking forward to continuing in the comments.

Before I begin, I’d like to make it clear that this post and the next reflect my thinking, not Valve’s, and don’t represent a product or strategy announcement in any way. They’re just thought experiments on my part, trying to catch a glimpse of what promises to be a really interesting future.

If you’re not familiar with VR and AR, VR is the one where you sit down, put on a headset, and find yourself completely immersed in a virtual world like Snow Crash’s Metaverse or Ready Player One’s OASIS (and if you haven’t read Ready Player One, run don’t walk; it’s a great read, especially if you grew up in the 80’s, but even if not – I didn’t, and I still loved it). AR is the one where you put on glasses and walk around, and find that the real world is still there, but modified to a lesser or greater extent, as in Rainbow’s End’s belief circles or the Rivet Couture virtual society of “To Hie from Far Cilenia.”

So with VR, you might take a seat at your computer, put on your VR headset, and find yourself in Middle Earth or a starship or a Team Fortress 2 level. With AR, as you walk down the (real) street wearing your AR glasses you might find that there are (virtual) aliens shooting at you, or that when you encounter (real) members of your Belief Circle they’re wearing (virtual) medieval costumes and glowing faintly, or, to continue the TF2 analogy, that everyone you see is wearing virtual hats.

The sort of AR I just described, which is what I’m going to talk about in this post, is unconstrained AR – what I call walk-around AR, the kind that works wherever you go. That’s certainly the long-term goal, because it’s a platform shift, but for the next few years it’s something of a strawman, because there are a lot of challenging technical issues to be ironed out before it’s ready for prime time. In contrast, highly constrained AR, for example tabletop or room-scale AR, is considerably more feasible than walk-around AR right now, and certainly has some potentially interesting uses. However, it’s obviously not as generally useful as walk-around AR, is less immersive than VR, and is currently farther from a consumer-ready product than VR, with less capable, more expensive hardware. Nonetheless, constrained AR is a strong counter-argument to VR’s near-term advantage, and will feature prominently in the next post.

There’s also a third sort of wearable display technology, which I’ll call HUDSpace, based on the display of 2D information on see-through glasses, much like having a phone or tablet in view at all times; this is the direction Google appears to be going in with Project Glass. I include in this category very lightweight AR such as having people’s names floating over their heads, arrows to guide you turn by turn to your destination, and information popping up when you’re near points of interest. There’s a great deal of value to this, and it’s clearly going to happen, but it’s considerably less technologically demanding than AR or VR, has little opportunity for deep entertainment experiences, seems largely like an extension of the smartphones we have today rather than a genuinely new platform, and is just way less cool to me, so I’m going to focus on AR and VR.

So if AR is where we’re all headed, why is VR worth bothering with? Two reasons: in the long run, VR-like experiences may be how we use our spiffy AR glasses much of the time, and in the short run, VR is poised to take off well before AR.

Right now, VR is much closer to becoming a consumer product than AR. Perhaps the biggest reason for this is that VR hardware is more capable and easier to make right now. The Oculus Rift, which is intended to ship at a consumer price, has a 90-degree horizontal field of view; in contrast, I’ve never heard of see-through AR glasses with anything like that field of view at any price, and while they may exist, it’s hard to see how they could be made at consumer prices with anything like current technology. (Video-passthrough AR glasses could of course have the same field of view as the Rift, since all that would be required would be to add a camera, but I don’t think video-passthrough AR will be good enough for a number of years, for reasons discussed here.) Also, because VR is used in a fixed location, it can be tethered, sweeping away a host of hard power problems that walk-around AR has to deal with, and enabling the use of far more powerful CPUs and GPUs. Alternatively, VR headsets can be designed to run for just an hour or two between recharges; in contrast, AR has to have the same order of battery life as a phone or tablet. Furthermore, because VR is restricted to one location, it’s much easier to develop tracking technology for. And since you’re not going to wear a VR head mounted display in public, or walk around with it, it doesn’t have to be as stylish, and while it still has to be light and comfortable, it is considerably less constrained than AR glasses that have to look like fat sunglasses. Finally, VR can use existing controllers initially; you’ll be able to play VR games with standard game pads, for example, although I think new VR input will have to evolve quickly in order for VR to really reach its potential. In contrast, the input scheme for AR is an open question.

In terms of hardware problems to be solved, VR is closely related to AR, and in many cases figuring something out in VR’s more tractable space will help in AR as well. In this respect, resources devoted to VR R&D aren’t subtracted from AR efforts; in fact, this may be the most effective way to make progress on technology related to AR, because VR hardware can be made fully functional and iterated on much more rapidly than AR at this point.

This is particularly true because a VR marketplace appears to be emerging as I write this, in the form of the Oculus Rift and support for it in Doom 3: BFG Edition, Hawken, and other games, while AR is still some distance from viable products. It’s far easier to push technology forward when there are real customers to provide feedback, real products to provide incentive for better, cheaper components, and real revenue to spur competition, and VR will likely have all those long before AR does.

VR is more approachable on the software side as well. New experiences often evolve from existing experiences; it’s hard to make a complete break with the past in every respect, if only because your audience will be confused, and also because it’s hard for developers to solve multiple problems in a new space simultaneously. There’s a direct path to at least some interesting VR experiences; PC and console games like first-person shooters and flight, space, and car sims are designed for immersion, and should seem like they’re on steroids in VR. It’s even more obvious what interesting HUDspace experiences are; a few are listed above. However, it’s not at all clear what will constitute compelling walk-around AR experiences. I have no doubt that they exist, but they’re unknown right now. (It’s a lot clearer what might be interesting for constrained AR, and we’ll look at the implications of that in the next post.)

So VR looks pretty good in the short run; how about after that? Even though I think it’s likely that in the long run (defined as five to ten years) AR will have a more radical effect on our lives, it’s possible that VR-like experiences will be where we will spend more of our time once we have really good AR glasses. The key is that AR glasses will be able to get darker or lighter on demand, because that’s necessary in order to work well in both dimly lit rooms and bright sunlight. That means they’ll be able to become almost completely dark at any time – and when they do, they’ll effectively be VR glasses. So your AR glasses will be able to provide both AR and VR experiences.

That’s interesting because VR experiences are richer in important ways. VR is more immersive, and that’s a big plus for many types of games. VR also has better contrast, since it doesn’t have to compete with photons from the outside world, so virtual images will look better. Because VR doesn’t have to interact with the real world, it doesn’t suffer from any of the inconsistencies that inevitably arise in AR; for example, lighting and shadowing in VR can be completely consistent. VR also avoids all the work that’s required in AR to figure out what real-world objects are in the field of view at any time, and to calculate how virtual and real images interact. Another point in VR’s favor is that it has no equivalent to the per-pixel opaquing limitation of AR, so VR software has complete control over the image that reaches the eye. Furthermore, small amounts of latency and tracking error may be more acceptable in VR, because the virtual images don’t have to match the real scene; since we’re not going to get to zero latency or perfect tracking anytime soon, that’s potentially a significant plus. (However, it’s also entirely possible that small amounts of latency and tracking error could cause simulator sickness under conditions of full immersion; this is one of many areas that we’re all going to learn a lot more about in the next few years.)

So AR is the only way to go when you want the virtual and real worlds to interact, but VR and VR-like experiences seem best for purely virtual experiences. (Here, “VR-like” means AR when it dynamically becomes opaque enough so that the virtual world is visually dominant.)

And it’s arguable that you spend most of your time in experiences that are more virtual than real (or at least that I do).

You’re probably thinking that you don’t spend any significant amount of time in virtual experiences, but consider: as you read this, you’re looking at a screen. Imagine you’re doing it on a head-mounted display, and you’ll see that it maps better to VR than to AR. Sure, you could have the text floating in your field of view while still seeing the real world, but why? It seems far more useful to just look at a virtual screen in VR, since all that’s of interest is the text. You could have lots of virtual screens up in 3-space around you, and you could have information presented in all sorts of other ways as well.

Similarly, the real world often doesn’t play an important role in watching TV or movies, or playing video games; certainly it does when you’re with friends, but when you’re alone, the real world doesn’t particularly enhance the experience. And if you ask yourself what percentage of your waking time you spend looking at a screen by yourself, you’ll find it’s a majority if you’re anything like me. So that’s why I say that VR-like experiences may be where we’ll spend a lot of our time once we have good AR glasses; until that time, this argues that VR by itself is interesting.

This is not to say being able to see the real world at the same time as the virtual world doesn’t have benefits; I’ll discuss that aspect in the next post. One thing that absolutely has to be figured out for VR is how to become not-blind instantly, for example by touching a control on the glasses that switches to a camera view; being unable to see without taking the HMD off just isn’t going to be acceptable in a consumer device.

Finally, there’s a wild card that could change the long-term balance between AR and VR dramatically. My thinking to date has assumed that AR will be a major platform shift that fundamentally changes the way we interact with computers, while VR won’t, except to the extent that VR-like experiences are part of the AR future. However, it’s possible that that VR will be a major platform shift all on its own; we could all end up spending our time sitting in our living rooms wearing VR headsets and haptics, while the traditional movie, TV, and videogame businesses wither. (In fact, I’d be surprised if that wasn’t the case someday, but I think it’ll be a long while before that happens.) We all know what that would imply, since we’ve all watched Star Trek – that way lies the Holodeck. If that happens, VR is more than interesting; it’s a big part of the future.

All of which implies that VR and VR-like experiences seem likely to be important in the long run.

None of the foregoing says that standalone VR is going to be more important or successful than AR in the next five to ten years, although that could happen. AR is most likely going to change the way we interact with the world, much as PCs and smartphones did, long before VR makes it to the Holodeck. However, it seems likely that VR is much closer to being deliverable in a truly workable form than walk-around AR, and it also seems likely that VR-like experiences will be an important part of the ultimate AR future. Given which, there’s a strong case to be made that while the long-term goal is to produce superb, do-everything AR glasses, VR and VR-like experiences are worth pursuing as well, both in the near term and down the road.

I should have posted this sooner, but it’s been a little crazy. It was a blast getting up on the stage with John and Palmer and talking about VR, but it was more as well. As I said during the panel, it felt like this might be one of those seminal moments when the world changes, the point at which a new technology that will change our lives started down the runway for takeoff. Of course, it’s entirely possible that that won’t happen, but it feels like the pieces are falling into place: affordable, wide-field-of-view, lightweight HMDs that can deliver a great experience; inexpensive tracking (cameras, gyros, accelerometers, magnetometers); and, critically, an existing software ecosystem – first-person shooters – that can readily move to VR (although that’s just a start; many other experiences more uniquely suited to VR will emerge once VR is established as a viable consumer technology). VR can only take off if all three pieces are working well, and we’re getting close on all three fronts. I don’t think we’re quite there yet, but the remaining issues seem solvable with time and attention, and once they’re solved, we may be off on a long, transformative journey. Where that ends, I have no idea, but I’m looking forward to the ride – and I think it might have started at QuakeCon.