The year is 2015. Wearable glasses have taken off, and they’re game-changers every bit as much as smartphones were, because these descendants of Google Glass give you access to information everywhere, all the time. You’re wearing these glasses, I’m wearing these glasses, all the early adopters are wearing them. We use them to make and receive phone calls, send and receive texts, do instant messaging, do email, get directions and route guidance, browse the Web, and do pretty much everything we do with smartphones today.
However, these glasses don’t support true AR (augmented reality); that is, they can’t display virtual objects that appear to be part of the real world. Instead, they can only display what I called HUDSpace in the last post – heads-up-display-type information that doesn’t try to seem to be part of the real world (Terminator vision, for example). No one particularly misses AR, because all the functionality of a smartphone is there in a more accessible form, and that’s enough to make the glasses incredibly useful.
But then someone comes out with a special edition of their HUDSpace glasses; the special part is that if you put a marker card down on a convenient surface, you can play virtual card and board games on it, either by yourself or with friends. This is a reasonably popular novelty; then the offerings expand to include anything you can play on a table – strategy games, RTSes, arena-type 2D arcade games extruded into 3D, and new games unique to AR – and someone comes out with a version of the glasses that doesn’t need markers, so you can play games in boring meetings, and suddenly everyone wants one. The race is on, and soon there’s room-scale AR, followed by steady progress on the long march toward general, walk-around AR.
And that’s how I think it’s most likely AR will come into our daily lives.
Last time, I described how my original thinking that AR was likely to be the next great platform shift had evolved to consider the possibility that VR (virtual reality) might be equally important, at least in the short and medium term, and far more tractable today, so perhaps it would make sense to pursue VR as well right now. (See the last post for definitions of AR, VR, and other terms I’ll use in this post.) Then I made the case for VR as the most promising target in the near future. I personally think that case is pretty compelling.
This time I’ll make a case for AR as more likely to succeed even in the short term (last time I explained why I think it’s the most important long-term goal), and I think that case is pretty compelling too. The truth is, given infinite resources, I’d want to pursue both as hard as possible; one doesn’t preclude the other, and both could pan out in a big way. But resources (especially time) are finite, alas, and choices have to be made, so a lot of thought has gone into choosing where to focus, and these two posts recount some of that thinking.
Of course, I hope we get this right, but as Yogi Berra put it: “It’s tough to make predictions, especially about the future.” All we can do is make our best assessment, start doing experiments, and see where that leads, constantly reevaluating and course correcting as needed (and it will be needed!). So I’m by no means laying out a roadmap of the future; this post and the last are just two possible ways wearable computing might unfold.
The first step in assessing whether to focus on AR or VR is to figure out how each is most likely to succeed, and then to compare the strengths and weaknesses of each of those probable paths. The likely path for VR is obvious, and already in motion: The Oculus Rift will come out, running ports of existing PC games. If it’s even moderately successful, games will be written specifically for VR, Oculus will improve the hardware and competitors will emerge, VR will likely spread to mobile and consoles, and the boom will be on.
The path AR might take to success is less clear, because there are many types of AR – tabletop, room-scale, and walk-around – and several platforms it could emerge on – PC, mobile, and console. Also, as the scenario I sketched out at the start illustrates, AR could evolve from HUDSpace. So let’s look in more detail at that scenario, and then examine why I think it might be more promising than other paths for AR and VR.
In my scenario, AR isn’t even part of the picture at first; see-through glasses emerge, but wearable computing develops along the Google Glass path, supporting only display of information that doesn’t appear to be part of the real world, rather than true AR. To be clear, it’s quite possible that Google Glass won’t be see-through, but will just provide an opaque information display above and to the side of your normal line of sight. However, I think see-through glasses have much more potential, if only because they’ll have much more screen real estate, won’t block your view, allow for in-place annotation of the real world, and will be more comfortable to look at. That’s a good thing for my scenario, since see-through potentially leads to AR, while an opaque display out of the line of sight doesn’t.
Having information available everywhere, all the time will be tremendously valuable, and HUDSpace glasses will probably become widely used; in fact, you could make a strong argument that people who wear them will seem to be smarter than everyone else, because they will have faster access to information. Think of all the times you’ve hauled out your phone in a conversation to look something up, and now imagine you can do that without having to visibly query anything; you’ll just seem smarter. (Obviously I could be wrong in assuming that HUDSpace glasses will be widely used – it may turn out that people hate having information fed to them through glasses – but certainly there’s a strong argument to be made that better access to information is likely to be compelling, and since the rest of this scenario depends on it, I’ll just take it as a given.)
You may well wonder why these glasses wouldn’t have AR capabilities – after all, even cellphones can do AR today, right? Here I need to draw a distinction between true AR and cellphone AR. Cellphone AR, although interesting, is at best a distant cousin to true AR, for one key reason: cellphone AR doesn’t have to fool your entire visual perception system into thinking virtual images exist in the real world. By this I’m referring not to photorealistic rendering (the eye and brain are quite tolerant of cruder rendering), but rather to the requirement that virtual images appear to be solid, crisp, and in exactly the right place relative to the real world at all times as your head moves. The tolerance of the human visual system for discrepancies in those areas when viewing 3D virtual images that are supposed to appear to be part of the real world – that is, true AR – is astonishingly low; violate exceedingly tight parameters (for example, something on the order of 20 ms for latency), and virtual objects simply won’t seem like they’re part of the world around you. With cellphone AR, you’re just looking at a 2D picture, like a TV, and in that circumstance there are all sorts of automatic reflexes and visual processing that don’t kick in, which greatly relaxes hardware requirements.
The visual system’s low tolerance for mismatches between the virtual and real worlds means that the hardware required to make true AR work well is significantly more demanding – and expensive – than the hardware needed for HUDSpace. This is particularly the case for general, walk-around AR, which has to constantly cope with new, wildly varying settings and lighting, but it’s true even for room-scale and tabletop AR, primarily due to the requirements for display technology and tracking of the real world. At some point I’ll post about those areas, but for now, trust me, it’s a lot easier to build glasses that display HUD information, or at most images that are loosely related to the real world (like floating signs in the general direction of restaurants) than it is to build one that displays virtual images that fool your visual system into thinking they exist in the real world.
Given that true AR is hard, expensive, and not required, HUDSpace glasses will initially almost certainly not support true AR. Interestingly, because they’ll almost certainly be see-through, HUDSpace glasses won’t even support cellphone AR well,.
So in this scenario, a few years from now we’re all wearing HUDSpace glasses and using them to do what we do now with a smartphone, but more effectively, because the glasses give us access to information all the time, and privately. They’ll also do things that a smartphone isn’t good at, such as popping up the names of people you encounter, which you can’t politely use your smartphone to do. The obvious difference from a smartphone is that the glasses won’t have a capacitive touchscreen, and honestly I don’t know what the input method will be, but there are several plausible answers, so I’ll assume that’ll work out and skip over it for now. Several large companies are making HUDSpace glasses, and the competition is as fierce as it is in smartphones today. All kinds of great apps are being written for the glasses, including HUDSpace versions of existing casual and location-based games, but there are no true AR apps, because the hardware doesn’t support them.
As I described in the opening, it’s at this point that someone will probably put a camera on their glasses that’s good enough for tabletop AR, probably with the help of a fiducial (a marker designed to be tracked by a camera) placed where you want the AR to appear. Add good tracking code, and you’ll be able to play any tabletop game anyone cares to write. The glasses will be networked, so you’ll be able to play any card or board game you can think of, and you’ll be able to do that either with someone sitting at the table with you or with anyone on the Internet. Better tracking hardware and software will eliminate the need for fiducials, and the Tetris or Angry Birds of tabletop AR will appear, sparking a rapidly escalating AR arms race, similar to what happened with 3D accelerators and 3D games. AR will expand to room scale, which will involve group games, of course, and a general expansion of current console gaming possibilities, but also non-game applications like construction kits (living room Minecraft- and Lego-type applications), virtual toys, and virtual pets, and at that point there will be a critical mass of AR users, hardware, and software that makes it economically and technically feasible to start chipping away at walk-around AR. It’ll probably take a decade or two, or even more, before truly general AR exists, but it’s easy to see how an accelerating curve heading in that direction could spring from the first wearable glasses that provide a good-enough tabletop AR experience.
There are several reasons I think evolving from HUDSpace is a more likely way for AR to come into broad use than emerging as a fully-formed product on its own.
The first thing you’ll notice is that my favored scenario doesn’t involve walk-around AR at all for a long time. That’s a huge plus; even though I think walk-around AR is the end point and hugely valuable, it’s very hard to get to in any near-term timeframe. One problem with a lot of potential technological innovations is that they require abandoning existing systems and making a wholesale jump to a new system, and it’s hard to make all the parts of those sorts of transitions happen successfully at the same time. That’s certainly true of walk-around AR, which would require display, image-generation, and tracking technology that doesn’t exist today, all packaged in a form factor similar to bulky sunglasses, running on a power budget that far exceeds what’s now possible in a mobile device, along with completely new types of applications, as I discussed in the last point. Honestly, though, I used walk-around AR as a strawman in that post; it’s clear that it’s a long way away from being good enough to be a product, so it served as a useful counterpoint to illustrate the advantages of VR. Constrained AR, both room-scale and tabletop, lies somewhere between walk-around AR and VR, and is much closer than walk-around AR to being ready for broad use, although not as close as VR. Room-scale AR has many of the same technical challenges as walk-around AR, although to a lesser degree; tracking, for example, is difficult, but there are potentially workable, albeit currently expensive, solutions. Tabletop AR, on the other hand, is relatively tractable, although not quite to VR’s level; the problem with tabletop AR is primarily that because it’s so limited, it’s simply not as compelling or novel as room-scale or walk-around AR.
AR that emerges in stages from HUDSpace glasses, on the other hand, doesn’t require any great leaps; each step is an incremental one that stands on its own. Solving those problems separately and incrementally is far more realistic, especially assuming the preexistence of a HUDSpace business that’s big enough to justify the R&D AR will need. As a starting point, tabletop AR that evolves from HUDSpace glasses involves tracking that’s doable today, optics and image projectors that will be a manageable step from HUDSpace, power and processing technologies that will be largely driven by phones, tablets, and HUDSpace glasses, and initial software that’s familiar, including at least the tabletop games I listed in the introduction.
In short, the technological path from HUDSpace glasses to HUDSpace-plus-tabletop-AR glasses seems realistic, while going from nothing directly to walk-around or even room-scale AR seems like a big stretch. That’s true not only technically, but also in a business sense, because HUDSpace-plus-tabletop-AR doesn’t require AR to justify the cost of the hardware by itself; in contrast, standalone AR systems would be in direct competition with consoles and dedicated gaming devices, with all the costs and risks that involves.
Consider two products that support AR. The first product is a special edition of a widely-used pair of HUDSpace glasses that is normally sold for $199; the special edition sells for $299 because it has cameras and more powerful processors that let it support tabletop AR gaming. The second product is a pair of AR glasses designed specifically for living-room use; it supports room-scale AR games that you can play on your own or with friends, and costs $299, plus $199 for each additional pair of glasses.
Even though the pure AR glasses are more powerful and would support a wider variety of novel experiences for the same total price, it’s hard to see how they could be successful unless the experience was truly awesome. At $299 and up, this would be going directly against existing consoles, and it’s hard to make the first games for a whole new type of gaming be killer apps that it’s worth buying the whole system for, because it takes time to figure out what unique experiences new hardware makes possible. Getting developers to devote effort to support a new, unproven platform is hard as well – it obviously can be done, but it’s a major undertaking. Also, the up-front expenditures and risk would be relatively large, since this would be a new type of product that at least overlaps with the existing console space. In short, it would require a console-scale effort, with all the risk a new console with new technology involves. A tabletop AR product would be less of a step into the unknown, and could be somewhat less expensive – but at the same time it would be more limited and less novel than room-scale AR, so there’s still the question of whether it’d be compelling enough to justify the purchase of a complete system. I’d love to be wrong – It’d be great if a standalone tabletop or room-scale AR system could be successful on its own merits. It just seems like they would have to overcome considerably greater market and technical challenges than evolution from HUDSpace glasses.
On the other hand, I have no problem imagining that a lot of people who are buying the HUDSpace glasses anyway – which they will be, because they’re very useful – would spend $100 to upgrade to make them more fun to use. The key here is that AR itself doesn’t have to justify the cost of the system, just the much smaller upgrade cost. You might say that’s not fair, that it’s not as powerful a system, but that’s the point – in the beefed-up HUDSpace case, AR doesn’t have to be compelling enough to justify the purchase of the glasses in the first place. If you want to convince people to buy a whole new system to put in their living room, or to buy a dedicated AR system for tabletop gaming, you have to get over the barrier of convincing them that they want to own yet another gaming device. If, on the other hand, you want to sell people established HUDSpace glasses with tabletop AR capability, they’ve already decided to make a purchase, and it’s just a question of whether they want to buy a cool and not very expensive option; in fact, far from being a barrier to purchase, the AR option makes the purchase of HUDSpace glasses more attractive.
Better yet, if you want to play a multiplayer game with someone else, they’re likely to have their own glasses, since HUDSpace glasses will probably be widely used, so there’s no incremental cost for multiplayer. The network effect from widespread adoption based on HUDSpace is a huge advantage for beefed-up HUDSpace glasses.
The bottom line is that the HUDSpace-plus-tabletop-AR scenario is a pull model, with the right incentives; a lot of the hardware and a sizeable market get developed for HUDSpace independent of AR, and AR then serves as an enhancement to help sell HUDSpace glasses into that existing market. In contrast, any scenario involving a standalone AR product is a push model, where a market for a new type of relatively expensive product has to be created and developed rapidly, in competition with existing consoles. It could happen, but it seems less likely to succeed.
Now that you know how I think AR is most likely to emerge, and that it will likely be constrained to tabletop and possibly room-scale AR for quite a while, we can return to our original question, which is whether it makes sense to pursue AR only, or a mix of AR and VR, especially in the near- and medium-term. Last time I discussed why VR was interesting; now it’s time to talk about why AR might be more interesting.
I will first note again that last time I compared VR to walk-around AR, and that that was a strawman argument. I don’t think there’s any world in which true walk-around AR is feasible in any way in the next five years. As I discussed above, the challenges that constrained AR – room-scale and tabletop – face are similar to but far less daunting than walk-around AR, and constrained AR is probably doable to at least some extent in the next five years, a somewhat but not greatly longer timeframe than VR, so the question is which makes more sense to pursue.
First off, technically VR is easier to implement with existing and near-term technology; that’s just a fact, as evidenced by the Oculus Rift. The Rift definitely has some rough edges to smooth out, but there are ways to address those, and I expect Oculus to ship a credible product at a consumer price; in contrast, as of this writing, I have been unable to obtain a pair of AR glasses capable of being a successful consumer product. The core issue generally has to do with the great difficulty of making good enough see-through optics in glasses that with acceptable form factor and weight. However, I know of several approaches in development, any of which would be sufficient if all the kinks were ironed out, and it seems probable that this will be solved relatively soon, so it’s a disadvantage for AR, but not a decisive one.
VR is also more immersive in several ways: field of view, blocking of real-world stimuli, and full control over the color and intensity of every pixel , which can make for deeper, more compelling experiences, but there are downsides as well. Immersion may not be good for extended use, either because it induces unpleasant sensory overload or simply because it makes people sick. AR provides anchoring to the real world, and that helps a lot; I personally get simulator sickness quite easily with existing VR systems, but rarely have that problem with AR. I’m confident that AR will be easier for most people to use for long periods than VR.
Another advantage that comes with being less immersive is awareness of the real world around you, and that’s a big one.
For starters, being not-blind means that you can reach for your coffee or soda, find the keyboard and mouse and controller, answer the phone, and see if someone’s come into the room. This is such a big deal that I believe VR will not be widely adopted until VR headsets appear that make it possible to be not-blind instantly, most likely by being able to switch the display over to the feed from a camera on the headset with a touch of a button, but also possibly with a small picture-in-picture feed from the camera while otherwise immersed in VR, or with a display that can become transparent instantly.
Being not-blind also means that you can give only a part of your attention to AR. For example, you could have an in-progress game of chess sitting on a corner of your desk; you’d notice it every so often, but you wouldn’t have to be focused on it all the time. This lets you use AR a lot of the time, in a variety of situations. In contrast, when you’re doing something in VR, it’s the only thing you can be doing, which considerably limits the possibilities. Being not-blind also means that you can be mobile while using AR, even if only to move around a table for a better view, while VR pretty much requires you to be immobile, further limiting the possibilities.
Most important, being able to see the real world means that you can have far more social AR experiences with other people than you can with VR. Sitting around the table with your family playing a board game, sitting on the couch with a friend seeing how high you can build a tower together, or having a quadracopter dogfight are all appealing in very different ways than isolated VR experiences, and given how intensely social humans are, those are ways that are arguably more compelling. In this respect, AR experiences will be more complex and unique than VR experiences, since they will incorporate both the real world and that most unpredictable and creative of factors, other people, and consequently have greater potential.
Finally, constrained AR is on the path to walk-around AR, and walk-around AR is where I think we all end up eventually.
At long last, to quote the renowned technology sage Meat Loaf: “What’s it gonna be, boy?” Unfortunately, after our long and interesting journey through possible futures, I’m not going to give you the crisp, decisive answer you (and I) would like, because there are two time frames and two scopes at work here.
There’s no way it makes sense to simply abandon AR for VR. Interaction with the real world and especially with other people is why AR is the right target in the long run; we live our lives in the real world and in the company of other people, and eventually AR will be woven deeply into our lives. In the medium term, I believe AR will likely emerge from HUDSpace roughly along the lines of the scenario above; another possibility is that a console manufacturer will decide to make room-scale AR a key feature, as hinted at by the purported leak of Microsoft’s Project Fortaleza a few months ago. All this makes it highly likely that work on tabletop and room-scale AR now will bear fruit in the future; it might be a little early right now to be working on that, but the problems are challenging and will take time to solve, so it makes sense to investigate them now.
In the near term, though, VR hardware will be shipping, and because the requirements are more limited, it should improve more rapidly than AR hardware. Also, it’s easier to adapt existing AAA titles to VR, and while VR won’t really take off until there are great games that built around what VR can do, AAA titles should get VR off the ground and attract a hard-core gaming audience. And a lot of the work done on VR will benefit AR as well.
So my personal opinion (which is not necessarily Valve’s) is that it makes sense to do VR now, and push it forward as quickly as possible, but at the same time to continue research into the problems unique to AR, with an eye to tilting more and more toward AR over time as it matures. As I said, it’s not the definitive answer we’d all like, but it’s where my thinking has led me. However, I’ve encountered intelligent opinions from one end of the spectrum to the other, and I look forward to continuing the discussion in the comments.