3d face reconstruction for VR HMDs in eye region

Abstract

The study of 3D reconstruction using video or photos has been a big issue since the past. As machine learning advances facial wrinkles can be reconstructed in detail and facial expressions can be reconstructed in real-time, but there are no paper about the contents that reconstruct the eyes part intensively. In this pa-per, I introduce a new method for reconstructing the eye area and a new method for real-time 3d reconstruc-tion using only the head-mounted displays(HMDs) without complicated hardware device environment. For the reconstruction, neural networks were constructed using machine learning and lightest neural networks were used to operate in real-time. Using HMD, collect dataset for learning and make ground truth reconstruction object manually. Then I make our own dataset for eye region reconstruction. This neural net-work model enables 3d reconstruction with only one image and allows create expressions such as surprised and frowning to express more diverse expressions while the previous 3d reconstruction models only can realize the expression of closing or opening eyes.