How much is VR actually being used in society?

As you know, VR is frequently used in the entertainment space, such as at amusement parks and in computer games. However, in many cases, VR is also used for simulations and training. For railway companies, overhead wire replacement drills cannot be done at actual train sites because mistakes would affect the whole facility, so VR is used to conduct the drills instead. VR is also effective for previewing designs. For example, building a mockup of a car takes a lot of effort. However, using VR can greatly reduce that effort. Color changes and other modifications can be made in an instant, and how it feels to get into the car can be conveyed to the user in a way that seems real.

The feeling of experiencing VR for yourself is different from just looking at a picture. VR technology accurately recreates real-world information and sends that input to your sensory organs. What should be done so that VR is perceived as real?

We recognize real-world information through our sensory organs - our eyes, ears, and hands - so first, it is essential to properly recreate the information, then accurately send it to our sensory organs. In other words, if we could input the same information from the real world into our sensory organs as-is, it would be possible for us to realize the ultimate VR, which would be indistinguishable from the real world. However, that’s still difficult for us to do. In the first place, you know that you’re having a VR experience when you put something on, so we can’t call this the ultimate VR. Recently, movie effects have gotten so good you can’t tell what’s CG and what’s real, so you could say that the reproducibility of information is almost 100%. However, you have to wear a head-mounted display (HMD) to make it feel real. With current technology, if you try to show a three-dimensional image in space without wearing a device, you’ll need to prepare special equipment and project the image using fog and other effects.

Out of the five senses, hearing is where we’ve made the most progress, followed by sight. As long as we faithfully record audio, we can reproduce it at the same level. Even without headphones, we have the technology available to line up speakers to reproduce audio, and it’s mostly impossible to distinguish between real-life audio and recorded audio. Next is touch. To a certain extent, we can recreate tactile sensations, the feeling of force when we touch an object, in specific circumstances by having someone hold a device. However, recreating all sorts of textures, such as making something feel rough or slippery, is difficult to do with one device. In addition, one difficult aspect of the sense of touch, which isn’t an issue with sight or hearing, is that depending on how someone touches or holds an object, the way it feels will change.

There are significant obstacles to affecting smell and taste in VR, since chemicals must be pumped into the nose and mouth. We don't judge something using only one of our senses. In particular, our senses of smell and taste work in concert. There have been experiments where even students studying winemaking in France have mistaken white wine with added food coloring for red wine. We could actually say that it’s possible to deceive people through sight, even if a taste hasn’t been perfectly recreated. That means it’s possible to manipulate hard-to-control senses by using easy-to-control senses. It’s necessary to mix multiple senses together in many cases. To reproduce the feeling of eating a fancy steak in VR, we have to make sure that sight, hearing, touch, smell, and taste all support each other. Well, even if we aren’t able to create an experience indistinguishable from the real thing, if the user is able to enjoy eating inexpensive meat in VR, then I count that as a win.

In any event, the brain has to interpret the informational input from the sensory organs. You need to multiply the quality of the information you are given, the quality of the means of transmission, and your own experience to determine whether it is real or not. In the first place, to decide whether or not something is real, you need to know the real thing. To perceive VR as real, experience becomes key. As an example, let's have you try a virtual instant coffee-making experience. First, I’ll put on the HMD and show you an example in practice mode. I’ll project the image I’m seeing on the screen. The cup and the coffee are hidden under the shelf, so try to take them out and put the coffee in the cup within the time limit.

First, I have to wear the HMD. I’m looking at a different room. I can push the start button with my finger. If I check the shelf… I found the coffee container, so I’ll put it on the table. I found the cup, so I’ll put that on there too… huh? The coffee container disappeared! Is it on the shelf? I’ll check again… Now the cup’s gone! While I was doing this, time ran out. What happened?

This is pretending to be a game about pouring coffee, but actually, it’s a VR experience that simulates dementia. I wanted you to experience a situation where there’s a discrepancy between your thoughts and your actual environment, since that’s what confuses people with dementia. An able-bodied person can’t alter their own memory, but they can in a virtual environment. Recreating dementia is an initiative that I’ve been working on with the Sanda City Council of Social Welfare in Hyogo Prefecture. I’m also working on recreating visual impairment together with the Hyogo College of Medicine. In addition to being useful to making diagnoses, externalizing something that was only in the patient's mind will let us experience what they are feeling. One of the viewpoints in my research is using VR as a tool to understand other people. VR allows us to experience the lives of people other than ourselves.

We can experience things we normally couldn’t do, and those experiences lead to understanding. If living in the real world becomes just one option, there should be a lot of people who will be able to live better lives. Going forward, if VR technology continues to develop, it seems that the way we comprehend reality will also change. What do you think?

In the first place, it’s difficult to come up with a definition of ‘reality.’ It’s true that a real world does objectively exist. However, I think reality for individuals is something that they subjectively reconstruct in their minds, based on the information received by their brains. In other words, the things in this room are my entire reality right now. If the outside of this building were to be painted blue this very instant, I wouldn’t know at this point. That’s outside my reality. I think no matter how much society changes in the future, in the end, reality is something perceived by each person. As long as humans remain in this form, that fact won’t change.

Some people actually feel that the online world is reality, more so than the real world. For example, if there’s a child who isn’t attending school, there aren’t many alternatives for them in the real world. However, if there’s another virtual world where they can study, I think it’s fine for them to study there. Furthermore, VR isn’t just one world; it can offer plenty of worlds, tailored to different people.　With learning centered around worlds like the Metaverse, we cannot guarantee that the development of social values, which is one of the functions of a school, will be achieved, and there is no way to conduct experiments. However, I think it’s good to have it as an option. One of the goals of VR research is to eliminate boundaries between virtual and actual reality. However, I think it’s more important to bring value that cannot be achieved in the real world. If it becomes possible for us to see what was once invisible and transcend time and space, if several Metaverses are created where various value systems coexist, and if society becomes able to recognize each of them, then perhaps it will be easier for more people to live their lives.

It will be easier for people if they think that they have more options for a truly prosperous life. Even if we get stuck in the future, if we consider that our lives don’t have to just stick to the real world, we can overcome that sense of limitation. Thank you for sharing your valuable story and experiences!