It was sold by HPHP Reverb G2 Omnicept EditionNow that it has arrived, I checked the operation of this VR headset, such as detecting pupils, line of sight, heart rate, and mouth movements, which are particularly characteristic.

As system development using VR headsets (HMD) evolves, the information acquisition needs of users using HMDs are increasing. After creating a website, I would like to know what kind of impressions and emotions users who experience the application through HMD enjoy the content, just like using an analysis tool such as Google Analytics for user analysis visiting that web. For example, the following specific use cases can be considered.

1. When experiencing entertainment content, at what time and where do users enjoy it, and do they think it's boring?
2. Are the trainees focused or nervous when using the training VR app? Are they taking a good look at what they need to see?
3. When using medical applications, can't doctors obtain patients' biological information?

Conventional VR HMD can track the user's head and hand movements, but no further biological information can be acquired. However, if you have the HP Reverb G2 Omnicept Edition, which has been sold in Japan since the end of 2021, it is possible to detect pupil/gaze/heart rate/mouth movements, and it is becoming possible to estimate the user's emotions, concentration, mental and physical conditions, etc. from these data.

In this article, a record of testing the acquisition of various biological information that can be obtained as an initial experiment using the HP Reverb G2 Omnicept Edition is described.

Heart Rate (BPM: Beats Per Minute): Heart rate captured by the heart rate sensor

A heart rate sensor is attached to the part of this HMD that hits the forehead. With this heart rate sensor, the user's heart rate can be acquired in real time. I was able to confirm that my heart rate actually increased by moving my arms and doing squats while wearing the HMD.

Face Camera & Mouth Tracking: Sensing mouth movements using a face camera

A face camera is attached to the part below the nose and in front of the mouth of this HMD. Since this face camera is an infrared camera, it has video output in R (red) color out of RGB, and it cannot be used to analyze skin color around the mouth, and it is assumed that it is mainly used for mouth movements. For example, it is possible to lip-sync the movements of an avatar's mouth according to the actual performer (user wearing an HMD).

Eye Opening/Closing & Eye Direction: Blink and gaze detection with an eye tracking camera

Eye tracking sensors are attached around the lenses of both eyes, and they can detect the user's blink and gaze (whether they are looking up, down, left, left, or right). It seems that this eye tracking and the next pupil measurement sensor use Tobii's technology, and there was the following explanation on Tobii's website.

(Tobii website(Quote) A reflective point of light is generated on the cornea, and the image is taken with a camera. The reflective points of light on the cornea and the pupil are identified from the images of the eyeball taken. The direction of the eyeball is calculated based on light reflection points and other geometric features.

Eye Pupil: pupil size measurement with pupil measurement sensors

It is possible to measure pupil size using an eye tracking sensor/pupil measurement sensor. The pupil is the blackest part of the center of the eye, and changes in pupil size are used to adjust the amount of light applied to the retina. (According to Wikipedia) The pupil diameter varies between 2 mm and 8 mm. In bright places, pupil miosis occurs, and pupil diameter becomes smaller. In the dark, mydriasis occurs, and pupil diameter increases. This movement is similar to the aperture of a camera. In fact, it was confirmed that the size of the pupils changed by creating a bright space and a dark space in the sample application.

Eye Exam: Vision Test

I think this is a bonus function, but I was able to test my eyesight as well. It is an application that turns completely dark one eye at a time, and the letter C, which is common in eyesight tests, changes direction up, down, left, and right, and responds to the one that is open. According to one theory, does using an HMD improve eyesight? There also seems to be a theory, so if you measure eyesight with this function before and after using the HMD, interesting results may be obtained.

Eye Vignette (Field of View): limited viewing angle

One of the root problems of VR is that using HMD can cause VR sickness. However, research has also shown that VR sickness can be prevented to some extent by limiting viewing angles. In the process of developing and providing an application, it would be possible to provide such a viewing angle limit function to users who are prone to VR sickness.

As described above, I tried various biological information acquisition of the HP Reverb G2 Omnicept Edition as a test. In the future, I would like to add functions such as those described above for various entertainment, training, and medical applications related to our company, or consider creating new applications that can be made possible by acquiring these biological information.