A Deep Learning Technique as a Sensor Fusion for Enhancing the Position in a Virtual Reality Micro-Environment

Abstract

Most virtual reality (VR) applications use a commercial controller for interaction. However, a typical virtual reality controller (VRC) lacks positional precision and accu-racy in millimeter-scale scenarios. This lack of precision and accuracy is caused by built-in sensors drift. Therefore, the tracking performance of a VRC needs to be enhanced for millimeter-scale scenarios. Herein, we introduce a novel way of enhancing the tracking performance of a commercial VRC in a millimeter-scale environment using a deep learning (DL) al-gorithm. Specifically, we use a long short-term memory (LSTM) model trained with data collected from a linear motor, an IMU sensor, and a VRC. We integrate the virtual environment developed in Unity software with the LSTM model running in Python. We designed three experimental conditions: the VRC, Kalman filter (KF), and LSTM modes. Furthermore, we evaluate tracking performances in the three conditions and two other experimental scenarios, namely stationary and dynamic. In the stationary experimental scenario, the system is left motionless for 10 s. By contrast, in the dynamic experimental scenarios, the linear stage moves the system by 12 mm along the X, Y, and Z axes. The experimental results indicate that the deep learning model outperforms the standard controllers positional performance by 85.69 % and 92.14 % in static and dynamic situations, respectively. © 2022 IEEE.