A study on human-robot coordination in augmented reality assisted hand tracking system

Abstract

Augmented reality environments allow users to interact naturally with 3D objects, including robots. Many robots have been used in the automated sector for painting, picking, packing, and palletizing tasks. The Baxter robot is an example of an industrial robot ideal for research and education. Baxter robots can offer multiple benefits compared with regular robots. In this study, we designed an augmented reality system that makes users intuitively interact in 3D environments by using a Leap Motion controller as a hand tracker and performing a basic human-robot coordination task with the Baxter robot. And the Baxter robot with a stereo camera is connected to a Linux computer, which was programmed with python language. The augmented reality world was programmed in the Unity software. The human robot-coordination task consisted of an augmented reality alignment. We asked the subject to wear the head-mounted display and move the hands. Every hand motion was translated into the robot’s limb motion in real-time. The subject and robot had to align two augmented reality markers. Here are two different experimental conditions: visual information activated and deactivated. The subject performed three trials under each condition. The experimental results showed that the subject under the visual information activated mode improved the average time by 70.63 %. © 2022 SPIE