Unmatched Fidelity, Incredibly Immersive: The Future of XR with Timoni West, Head of AR/VR at Unity

Strange but unexpected What used to be happening is happening in the world of VR (virtual reality) and AR (augmented reality).

Like any sci-fi technology, early adopters hype VR and AR as if they belonged to a dream world. Neither of these are currently mainstream, or even popular, but they are essential to the imaginary world that someday someone will bring to life. And now that the Metaverse has joined that world, tech evangelists gleefully say that everyone must wear a headset as their ticket to that fantasy future.

Among the people at the San Francisco headquarters of Unity Technology, who tells WIRED that most XRs are currently made using Unity, the vice president of the augmented and virtual reality division Few people know more about the state of VR and AR than a certain Timoni West. She believes that in the future, almost every creative tool will involve XR as a building block. But at the same time, he keeps his feet firmly planted on the ground. Otherwise, he will not be able to act as an unofficial leader of XR.

Related article: Design a world without gravity: Startups leading the mirror world (3) Unity Technologies

WIRED sat down with West to sift through reality and fantasy to find out what exactly XR can do. Surprisingly, a lot seems to hinge on the ability to collect large amounts of data. Below is the interview. Slightly modified for clarity and volume.

WIRED: Let's start with a sort of ontological question. Tech companies are asserting that there will be one or more Metaverses inside VR or AR, and the idea that people will live or go there is growing. Do you see VR and AR as tools? Or rather the destination?

Timoni West: Good question. I don't think it's either. I see XR as one of many possible mediums. For example, we developed an AR mobile app [beta] that scans a space and converts it to a gray box for object placement, auto-tagging, and more. In other words, we are using AR for what it does best. We will use VR in areas where VR is good. For example, attending something, meeting someone, sculpting, and so on, which is inherently required to be 3D.

VIDEO BY UNITY

──In other words, it is a medium for you. It's been an interesting observation so far, and tech companies seem to waver over whether they should see VR and AR as a tool or a place. It seems undecided whether it's a way to do or access something, or a place to immerse yourself.

Don't you think it's crazy? My thoughts are different, and I think it's a combination of both. Just like in cyberspace. Remember when people used to say, "Let's go to cyberspace and surf the web"? We still use similar words. Go to the website and visit the homepage. It's like traveling.

But do you really travel? That's not true. Perhaps it is human nature to think so. Think of spending time as a journey. When we talk about time, when we travel through time, we use the language of travel. So if there are languages ​​that don't use [such] travel metaphors, I'm very interested in what other metaphors they use.

──I can't even imagine! But if you think about it, the level of immersion in VR and AR is already on the rise, and it seems like we're always connected or always there. Today's AR incorporates a lot of GPS functions, especially in mobile AR such as "Pokemon GO". Is this one of the points that support "being" in AR or "existing" in the Metaverse?

That's right. Have you seen Unity's game engine?

──Yes.

Then you should know that there is a window there, and beyond the window is an infinitely visible plane and a drab blue sky. Personally, I think it would be nice if it was more interesting, but anyway, if you catch someone's communication signal and those people walk around, you will also see them move on the Unity scene. You can see it. And let's say they take their phones and scan the place. Then the scanned object will appear in the Unity scene. If we track people, we humans will be there too.

It's kind of like calling Player 1 and Player 2 in the real world. If you are tracked, you have a digital presence. But that's not presence in the sense of VR. If the computer thinks you're there and interacting, then we're in the metagame. Or vice versa.

──Hearing that story reminds me of an immersive theater using AR. For example, a few years ago I tried out an AR murder mystery type game at the Game Developers Conference in San Francisco. There, various hints were displayed on the screen by scanning an object on the real-world set with a smartphone. It kind of felt like we were playing the game, we were in the game, and we were the game.

That's right. That intrigues me too. It's probably the same reason why I love theme parks. Suddenly you are thrown into a new place. There are its own rules, and you can interact with various characters. Every detail is well thought out, and it goes without saying that it has a lot in common with VR. Theme parks extend your reality through design, but now they also use digital technology. Have you been to Harry Potter World?

──No.

That's great. You actually have a magic wand and you can do many things with it. Magic spells appear when you wave your wand. They are all a blend of digital and reality.

──Yes, I would say it has become a really big thing. Theme parks can now literally track us and input us, our datasets, as avatars into the park's digital simulation. Interesting story. This story reminded me of a controversy that happened a few years ago. I think Disney World has filed a patent for technology that uses shoes to track the movements of visitors. The purpose was to collect data such as popular rides and the travel routes of guests in the park. Have you heard this story?

It would be strange if a popular shoe brand tried to do that, but I think the idea makes sense. Now there is a Disney app that you can download and play games. Not only ordinary games, but also AR games and quiz games are included. And once you reach a certain grade, no matter which ride you go on in the park, the app will detect your whereabouts and give you benefits and so on.

Someday, the app will know when you go to the cafeteria and the robot will talk to you about your achievements. Please try to imagine. You go home and put on your VR headset. And when you visit the VR version of Disney World, the characters speak to you more than in real life. The biggest issue is scale.

──That story reminded me of Niantic, the creator of Pokémon Go. Niantic also digitally mapped the player's area at a very detailed level and incorporated player feedback into the game.

Yeah, that's how Niantic got more pedestrian information than even other big companies have. Niantic has made people walk while other companies only have autonomous vehicles.

──Is collecting data to create a virtual world a philosophy that you should adopt and design?

Yes, that's right. Niantic is one of our favorites and we really appreciate it. I think it's really amazing that you've collected a completely new data set on a scale that no one else has. Thanks to Niantic building tools in Unity as part of Lightship [the underlying platform for Pokémon GO], other people can build third-party games there as well.

Related article: "Pokemon GO" hasn't disappeared - it's still very popular and makes a lot of money, approaching the secret

Indoor matters have a bit of a thorny issue of privacy. But if you want an experience that's really personal and understands your personality, you're going to need an indoor level of detail as well. And there are a lot of people working on ways to protect privacy by processing data in real time so that it is never stored or backtracked.

Other companies also got the trigger system to get more players into the game and better data from it, which in turn got more people playing the game and therefore better quality. I would like to be able to collect data.

──Yes, in a way, it's world-building.

Think of BlizzCon. Possible use cases include going to BlizzCon and creating your own AR world while you are there, or cooperating with others to create the world.

──It would be great if we could see the avatars of “World of Warcraft” overlapping the bodies of the people attending BlizzCon. This is really science fiction.

No, it's not sci-fi anymore! Indeed, at this point in time it is possible.

──Really?

All you need is body tracking. See... the U1 chip in the new iPhones is directional. It's so accurate that the chip knows where you are. There are basically two imperatives, one of which is body tracking. This already exists. Another thing you need is Me who knows exactly who you are. The latter will probably require a device handshake. Maybe a bracelet or some kind of electronic ID would work. Or, if you prefer boring ideas, you can use t-shirt patterns.

──Is it like a QR code?

That's right.

──I think another interesting direction AR could go is unlocking the parts of people who are divided online. For example, the avatars that everyone uses in each game, such as Fortnite skins.

I totally agree with you, and I think it's a really interesting topic. I won't name names here, but my recent experience in VR required a very realistic avatar. I have never used such avatars in Altspace or VRChat. In VRChat, I'm butter with skinny short legs. She's so cute. At Altspace, I have pink skin and white hair. It looks like a robot.

[Recent experience in VR] It was very unsettling to be there with myself, really bored, dressed in clothes that I would never wear in real life. This is the first time I've felt this bad about an avatar, and it got me thinking about how or in what way people would express themselves if it was actually an option. I was.

──Options are important. I play a lot of games, so I'm always impressed by how immersive games like Roblox, for example, are. I think we've learned a lot about immersion in the last five years or so. It's about how much people care about graphic fidelity, how much they care about movement and the precision of movement. What kind of knowledge have you acquired so far? Are there any popular beliefs that have been proven wrong?

High fidelity doesn't seem to matter much. In fact, the feeling, the feeling is more important than the visual.

──Is it about latency or resolution?

That's right. But I think it goes a little deeper than that. As a matter of fact, headsets are slow, so you can't use ultra-realistic visuals in VR, even if you wanted to. There is no way to fill this small gap. Even movies haven't gotten that far yet.

On the other hand, when I try VR that tries to imitate reality as much as possible, I find everything very uncomfortable and oppressive. At the very least, I would like the roof to be removed and a dolphin or something to swim in the background. Why would a VR world need a roof? Ultimately, what matters is whether the mechanics of the world are interesting. And this is where Roblox is so interesting.

Related article: “Roblox”, which takes pride in being the leader of the Metaverse, builds a co-experience ahead of games and entertainment

Roblox has a structure, and people who make games gather there. And inside Roblox, the mechanics change depending on the game. So does Minecraft. There are solid rules and people literally spend millions of hours in it. It's hard to beat when it comes to fidelity, but it's still incredibly immersive.

──However, VR seems to be at a crossroads now. People seem to want such high fidelity. When thinking about the next generation of technology, the next step in immersion, people look to fidelity.

Isn't it a learning problem? If you want it, try it. Try a super-realistic game, experience the possibilities, and then decide if you like it. That said, you are correct, and on the other hand people still don't know anything. By the way, do you know [Second Life creator] Philip Rosedale?

──Yeah, I just interviewed him for the Metaverse a few weeks ago.

Great. My team works with High Fidelity, which Rosedale started, and is involved in the "Amphitheatre." A 2D spatial audio experience. We were wondering why the Rosedales chose such a realistic space. If you can do anything, why not do something better? Rosedale then replied:

Related article: From the partnership between Niantic and the theater company of the experience-based show "Sleep No More", the appearance of the "next AR experience" can be seen

When they launched Second Life, they expected everyone to be hooked right away. However, people started recreating their own homes in Florida. We all started with what we already knew and what we were used to. As time went on, I became more and more immersed in the game, and I think I needed a realistic foundation to start with. I think it's the same mentality that people today want high fidelity in VR. As you familiarize yourself with the medium, you'll begin to see the real advantages of VR as well.

--In my opinion, there are clearly more skeptics about the idea that XR will go mainstream in the future. A lot of that is probably because we're still not quite comfortable with the idea of ​​XR at this point. But what about 10 years from now? Do you think VR will become mainstream and spread like PlayStation, for example? Or do you think it's a fair comparison in the first place?

I actually think it will go mainstream. I would like to warn you, though. Perhaps VR headsets will become more ubiquitous than expected and everyone will own them. But I think it will be completely different from how we interact with mobile phones.

We never let go of our phones. But what about traditional telephones in 1984? At the time, most US households would have owned a phone. So, since then, have you never let go of your phone? That's not true. Telephones were ubiquitous and the market was saturated, but they were used for one specific purpose. It's for calling.

Related article: Which headset should you choose to experience VR? Buying guide for beginners

Nowadays, telephones have become multi-devices packed with all sorts of functions, including televisions. If you only use it to make phone calls, can you say you use it all the time? I can't say.

Some say headsets will become ubiquitous and replace every screen, but I don't think so. Screens are a wonderful piece of technology. I see computers as tools. In my case, I use it when I write a lot or when I need to read a lot of text. Because that's what computers are designed for.

Computers have existed since the 1930s and have been continuously improved for this particular purpose. Along the same lines, VR headsets will become commonplace in every home, but only in areas where VR excels, such as gaming and social experiences. Conversely, I don't think it's going to be a tool that I use all the time.