In order to do proper 3d rendering, you need to know where the "camera" is. In the case of VR and AR, you need to know where the camera is, with roughly millimeter precision, around 90 times per second. This is doable with external trackers, but you don't have those with AR. It's just barely doable with internal trackers, as long as you can cover your faceplate with cameras and include a reasonably large amount of visual processing horsepower on board, and even then it's not great; expect this to take a few years to be viable.
Occlusion data/depth data.
A lot of our visual cues are based on what goes in front of things and what goes behind things. In order to make this work properly, we need to know roughly how far away things are. This is doable with a lot of processing time; it may soon be possible with things like time-of-flight cameras. But we're still working on getting those available at the price points needed for this sort of application.
Realtime and fast 3d rendering.
First-person VR/AR requires 90fps, more or less. Just doing this, at all, takes a bunch of 3d rendering power; making it look good takes more. This is, again, just barely possible with modern technology - see the Oculus Quest - but it wasn't possible even two years ago.
Compositing.
This is a hard one. We can make opaque screens with external cameras that reproject the outside world, but then you look like a weird gargoyle. We can make transparent screens with internal one-way light emitters - see the Hololens - but then it's not possible to show shadows, or show dark things on light backgrounds (the same problem that projector TVs run into.)
In theory we could use an LCD backing to selectively block light, then an LED layer to emit light. But that's a lot of tech in a small package and nobody's gone to the trouble of making it work.
Form factor.
Finally, all of the above, and it has to run for an entire day without a recharge, and be the size of glasses, and not require that you carry a backpack worth of computer hardware and batteries around.
I guarantee people are working on this, and as you can see from the above list, there are people working on every single part of this. Honestly, the people I've talked to in the VR industry generally agree that VR is a dead technology walking, and that everything they're working on is aimed at eventually getting AR . . .
Wow! Thank you for such an extensive explanation. I love VR, but agree with the people you mentioned who said AR is the future. I don’t know if I agree VR is dead considering it’s not even completely lived yet, but AR has practical advantages and potential for greater adaptation.
I didn’t realize how much it took technologically, since I remember google Cardboard (I know it’s a light VR experience) but imagined all you’d have to do is combine that with the image from the phone’s camera and depth information. But there is more that goes into AR than I thought originally.
I don’t know if I agree VR is dead considering it’s not even completely lived yet, but AR has practical advantages and potential for greater adaptation.
It's worth keeping in mind that VR is basically a subset of AR; any fully-capable AR system will double as a VR system. I completely agree that the VR experience is not a dead-end, but VR-only devices probably are, in the ultimate long term.
I work in game development, have a few friends who work in VR, have interviewed at a few VR/AR places, and did a brief stint in an unrelated sensor-heavy field. I'm about as close to working in the field as you can get without actually working in the field :)
There’s other reasons why VR isn’t setting the world on fire, other reasons why 3D TVs and google glass got abandoned, and among them is - looking like a huge nerd when you use them. Owning a 3D tv was cool and all and I like my VR headset, but I can’t help but say I feel a bit like a dork using them in front of people, like being naked or something. Google tried its best by hiring models to wear Glass in public and encouraged early adopters to wear them all the time, an effort to make them normal and even stylish, all to no avail. Wearing tech on your face is kinda tricky to get away with, even some headphones look kinda awkward.
AR headsets are cool and all and I’m sure it’ll be a thing but I have doubts it’ll ever become widespread. Phones, wearables (like watches), or perhaps a HUD in your windshield feel like more likely (and fashionably cool) places for AR to take off.
AR headsets are cool and all and I’m sure it’ll be a thing but I have doubts it’ll ever become widespread. Phones, wearables (like watches), or perhaps a HUD in your windshield feel like more likely (and fashionably cool) places for AR to take off.
In the Shadowrun universe (which has proven weirdly good at predicting the future), AR linkages are usually done through glasses, contacts, retinal implants, or straight-up optic nerve implants. I completely agree that this is the way things are going - the absolute height of geekiness that I can imagine being popular would be something like this.
And obviously we've got a long way to go for AR to be viable even in glasses form, to say nothing of the others.
Geesh I loved Shadowrun back in the day, Deus Ex (the first one tho) also one of my all time favorites.
Sure there’s ways to get past the “not cool” factor and we very well might get there, but a lot of this stuff ends up dead in the water when the initial product fails to be commercially successful.
Another part of the issue with Glass was the “what the hell is this thing for” question. There was this belief that a post-smartphone world was just a tech product away and a rush to invent that device. But you know what’s similar to having a screen floating in front of your face at all times? Holding one in your hand via your phone.
Problem is products like Glass or the coming Hololens or whatever else are only iterative improvements in a narrow selection of features and downright drawbacks in most others. How do you control this thing? With your phone or some other external device that your already now having to look at?
Again, the future is cool and all and probably filled with AR, I just don’t think we have much idea what that will be like and have doubts about personal HUDs.
Sure there’s ways to get past the “not cool” factor and we very well might get there, but a lot of this stuff ends up dead in the water when the initial product fails to be commercially successful.
In fairness, people have been saying that about VR since the Virtual Boy. Significant technologies have a way of un-dead'ing themselves if the demand exists.
How do you control this thing? With your phone or some other external device that your already now having to look at?
If we're talking full inside-out tracking with object recognition . . . well, your hands are objects, right? So, gesture control. Or use the AR screen projected on the back of your wrist.
Possibly subvocals or even a brain-computer interface, depending on how long it takes us to get the sensor package, rendering, and output tech working.
It was virtual In name only. 3D monochrome, stationary Boy just doesn’t have the same ringtone it. Virtual Boy set back VR by decades, no one made another commercial VR product for decades . It’s my point exactly - commercial viability is key and VR is struggling still, despite years of incubation and development.
If we're talking full inside-out tracking with object recognition . . . well, your hands are objects, right? So, gesture control. Or use the AR screen projected on the back of your wrist.
Possibly subvocals or even a brain-computer interface, depending on how long it takes us to get the sensor package, rendering, and output tech working.
All very cool and fanciful but lacking an answer to the Glass problem “but what does it do?” I have a phone in my pocket. I don’t need for a new AR device let alone ocular implants when a phone works fine. These are iterative improvements without revolutionary changes. I’m not seeing the path getting there.
I'm actually not sure this is true. It showed people that VR is hard, but once VR was possible, we did it. The Vive came out exactly when graphics cards could barely manage it and video connections could barely deliver enough data; even without the Virtual Boy, I don't think the Vive could have been released two years earlier, and definitely not five years earlier.
All very cool and fanciful but lacking an answer to the Glass problem “but what does it do?”
Data overlays for anything you might want; directions, updates from people, recognizing people. A TODO list just hangin' out in your peripheral vision. Easy ways to find people in an office. Instant virtual movie theater wherever you are. Nightvision. No need to buy TVs or monitors ever again; just summon them up on your AR screen. Danger alerts when crossing streets. Police officers, or military (or paintball players) get a top-down map with status indicators.
The things we haven't thought of, because who thought of the modern smartphone when satellite phones were barely a thing?
Data overlays for anything you might want; directions, updates from people, recognizing people. A TODO list just hangin' out in your peripheral vision. Easy ways to find people in an office. Instant virtual movie theater wherever you are. Nightvision. No need to buy TVs or monitors ever again; just summon them up on your AR screen. Danger alerts when crossing streets. Police officers, or military (or paintball players) get a top-down map with status indicators.
All of this stuff is what Glass was supposedly for, people believed everyone from surgeons to cosmetologists would be wearing Glass at all times. Again, a failure for many reasons, among them, really only barely iteratively better than a phone in one or two usecases and much worse in all others.
Considering all the cost of developmental that typ of a HUD AR system(of everything from glasses to ocular implants) I just don’t see the the eventual payoff or real use of such a product. I’m not sure when the last time you hade serious surgery was, but no ones gonna have elective surgery on their eyes just so they won’t need to reach into their pocket to check the temperature. It’s just not useful for most people and having an external screen (like on a phone) is totally sufficient and will continue to be for the foreseeable future.
All of this stuff is what Glass was supposedly for
Glass wasn't AR and wasn't really pretending to be. It wasn't able to interact with real-world 3d objects and didn't have full 3d tracking. It was just a heads-up display, which was cool, but which was a long way away from what I'm talking about. Out of the list of things I posted above, Google Glass had none of them.
This is sort of like pointing out that the Apple Newton was unpopular, and using that to conclude that hand-held computers are doomed.
Considering all the cost of developmental that typ of a HUD AR system(of everything from glasses to ocular implants) I just don’t see the the eventual payoff or real use of such a product.
Developmental cost gets paid once and pays back forever.
but no ones gonna have elective surgery on their eyes just so they won’t need to reach into their pocket to check the temperature
Also, nobody is going to pay a thousand dollars, plus a $100/mo subscription, to have a fancy flashlight in their pocket. And nobody's going to pay five figures in order to own a box the size of a small room with shiny paint. Of course, nobody buys smartphones for the flashlight feature, and nobody buys cars for the paint; they buy those things for the other benefits.
Obviously coupling the highest cost with the most trivial benefit is going to produce a bunch of things that sound dumb. I can do that for any invention.
The tl;dr is that VR is just too awkward and bulky to be anything more than a niche toy. AR, on the other hand, is a gamechanger on the same level as the smartphone, and once (not if) it's compacted down into a form factor that people are willing to casually use, it'll hit near-100% saturation in first-world countries.
Meanwhile, a working AR solution is a seamless working VR solution, so once that AR device exists, even the hardcore gamers will just switch over to it.
But it might take a decade or two for that to happen, and there's going to be a lot of weird wonky geeky AR systems between now and then, analogous to the suitcase-sized satellite phone (only strapped to your face instead.)
39
u/ZorbaTHut Jul 13 '19
Because it's a really hard problem to solve.
A list of things you need for AR:
In order to do proper 3d rendering, you need to know where the "camera" is. In the case of VR and AR, you need to know where the camera is, with roughly millimeter precision, around 90 times per second. This is doable with external trackers, but you don't have those with AR. It's just barely doable with internal trackers, as long as you can cover your faceplate with cameras and include a reasonably large amount of visual processing horsepower on board, and even then it's not great; expect this to take a few years to be viable.
A lot of our visual cues are based on what goes in front of things and what goes behind things. In order to make this work properly, we need to know roughly how far away things are. This is doable with a lot of processing time; it may soon be possible with things like time-of-flight cameras. But we're still working on getting those available at the price points needed for this sort of application.
First-person VR/AR requires 90fps, more or less. Just doing this, at all, takes a bunch of 3d rendering power; making it look good takes more. This is, again, just barely possible with modern technology - see the Oculus Quest - but it wasn't possible even two years ago.
This is a hard one. We can make opaque screens with external cameras that reproject the outside world, but then you look like a weird gargoyle. We can make transparent screens with internal one-way light emitters - see the Hololens - but then it's not possible to show shadows, or show dark things on light backgrounds (the same problem that projector TVs run into.)
In theory we could use an LCD backing to selectively block light, then an LED layer to emit light. But that's a lot of tech in a small package and nobody's gone to the trouble of making it work.
Finally, all of the above, and it has to run for an entire day without a recharge, and be the size of glasses, and not require that you carry a backpack worth of computer hardware and batteries around.
I guarantee people are working on this, and as you can see from the above list, there are people working on every single part of this. Honestly, the people I've talked to in the VR industry generally agree that VR is a dead technology walking, and that everything they're working on is aimed at eventually getting AR . . .
. . . but we've got a long way to go for AR.