Here is an overview of recent research and development status. The keywords are vTuber, motion capture, live streaming, and 3D virtual space.

VTuber and motion capture

VTuber, a virtual YouTuber where virtual characters, not living humans, appear on YouTube and talk and dance by sensing human movements with cameras and sensors (motion capture) and reflecting those movements in 2- and 3-dimensional CG characters on the screen, have been gaining popularity since around the summer of 2017. Similar technologies have long been used in film production sites, etc., but in order to use them, expensive and large equipment had to be used. A recent characteristic is that general users can also do it at home by using motion capture suits that cost hundreds of thousands of yen and head-mounted displays (HMD) + sensors that cost tens of thousands of yen. Since VR HMDs are often used for this sensing, I think the fact that engineers in the virtual reality industry are actively participating in technology development and content production is also one of the factors that is accelerating the trend.

As we have been working on sports VR until now, we are proceeding with research and development starting with ballet dances to see if it is possible to measure the movements of professionals such as athletes. In the image above, the ballerina is asked to attach a motion tracking sensor called Perception Neuron to measure their movements.

Fusion with communication systems

At the site of video production using VTuber, the characters on the screen are moved while sensing the performer in a studio where motion capture has been realized. Then, by recording that screen, the general flow is to convert the CG characters moving and speaking into video content. It is interesting as a new method for video production as it is, but since we have been working on VR live streaming for a long time, we have been proceeding with research and development to live stream motion data and three-dimensional data. When you want to deliver live images of people or characters to remote locations, the data capacity increases if the video itself is present, but if the receiving side has CG data of the character beforehand, motion data is sent, and if the receiving side moves the character, live streaming with movement and expression becomes possible. Specifically, two-way communication can be realized by using a low-latency protocol such as WebRTC, and large-scale distribution to a large number of people is also possible if a live streaming protocol such as RTMP/RTSP is used.

This workUnity-chan license termsIt is provided under

In the screenshot image of the prototype above, three virtual idols are singing in a virtual concert hall. Viewers will also be able to enter such a space as avatars and share time and space.

The virtual entertainment era

Until now, most entertainments such as sports, plays, and concerts have provided user experiences where people gather in real spaces such as stadiums and theaters and share fun. However, in real-space entertainment, there were also weaknesses, such as restrictions on the number of participants due to space capacity, and not necessarily easy to participate due to geographical reasons. Although it is a live experience, in virtual entertainment performed in a virtual space, there are no spatial restrictions (although there is server-like capacity instead), and it can be accessed from anywhere on the planet. In the case of sports, the player's location information and movements are sensed and turned into avatars; in the case of theater, the performers' dances and songs can be captured and made into a virtual play, and such viewing targets can be shared with other users while communicating in a virtual space and in verbal and nonverbal communication.

In addition to the live-action virtual reality technology based on 360-degree videos that we have cultivated so far, we are also developing computer graphics (CG) -based virtual reality systems using real-time streaming technology. Please feel free to contact us.