A study on human-robot coordination in augmented reality assisted hand tracking system

John P.; Myungho L.; Cheol S.

doi:10.1117/12.2614986

A study on human-robot coordination in augmented reality assisted hand tracking system

John P., Myungho L., Cheol S.

Department of Robotics and Mechatronics Engineering

Daegu Gyeongbuk Institute of Science and Technology

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

Augmented reality environments allow users to interact naturally with 3D objects, including robots. Many robots have been used in the automated sector for painting, picking, packing, and palletizing tasks. The Baxter robot is an example of an industrial robot ideal for research and education. Baxter robots can offer multiple benefits compared with regular robots. In this study, we designed an augmented reality system that makes users intuitively interact in 3D environments by using a Leap Motion controller as a hand tracker and performing a basic human-robot coordination task with the Baxter robot. And the Baxter robot with a stereo camera is connected to a Linux computer, which was programmed with python language. The augmented reality world was programmed in the Unity software. The human robot-coordination task consisted of an augmented reality alignment. We asked the subject to wear the head-mounted display and move the hands. Every hand motion was translated into the robot's limb motion in real-time. The subject and robot had to align two augmented reality markers. Here are two different experimental conditions: visual information activated and deactivated. The subject performed three trials under each condition. The experimental results showed that the subject under the visual information activated mode improved the average time by 70.63 %.

Original language	English
Title of host publication	Optical Architectures for Displays and Sensing in Augmented, Virtual, and Mixed Reality (AR, VR, MR) III
Editors	Bernard C. Kress, Christophe Peroz
Publisher	SPIE
ISBN (Electronic)	9781510647329
DOIs	https://doi.org/10.1117/12.2614986
State	Published - 2022
Event	Optical Architectures for Displays and Sensing in Augmented, Virtual, and Mixed Reality (AR, VR, MR) III 2022 - San Francisco, United States Duration: 23 Jan 2022 → 25 Jan 2022

Publication series

Name	Progress in Biomedical Optics and Imaging - Proceedings of SPIE
Volume	11931
ISSN (Print)	1605-7422

Conference

Conference	Optical Architectures for Displays and Sensing in Augmented, Virtual, and Mixed Reality (AR, VR, MR) III 2022
Country/Territory	United States
City	San Francisco
Period	23/01/22 → 25/01/22

Bibliographical note

Publisher Copyright:
© 2022 SPIE.

Keywords

Augmented reality
hand tacking
human-robot coordination

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

Access to Document

10.1117/12.2614986

Cite this

P., J., L., M., & S., C. (2022). A study on human-robot coordination in augmented reality assisted hand tracking system. In B. C. Kress, & C. Peroz (Eds.), Optical Architectures for Displays and Sensing in Augmented, Virtual, and Mixed Reality (AR, VR, MR) III Article 119310K (Progress in Biomedical Optics and Imaging - Proceedings of SPIE; Vol. 11931). SPIE. https://doi.org/10.1117/12.2614986

@inproceedings{2c12e49895614df58ad43a1244e0d2e5,

title = "A study on human-robot coordination in augmented reality assisted hand tracking system",

abstract = "Augmented reality environments allow users to interact naturally with 3D objects, including robots. Many robots have been used in the automated sector for painting, picking, packing, and palletizing tasks. The Baxter robot is an example of an industrial robot ideal for research and education. Baxter robots can offer multiple benefits compared with regular robots. In this study, we designed an augmented reality system that makes users intuitively interact in 3D environments by using a Leap Motion controller as a hand tracker and performing a basic human-robot coordination task with the Baxter robot. And the Baxter robot with a stereo camera is connected to a Linux computer, which was programmed with python language. The augmented reality world was programmed in the Unity software. The human robot-coordination task consisted of an augmented reality alignment. We asked the subject to wear the head-mounted display and move the hands. Every hand motion was translated into the robot's limb motion in real-time. The subject and robot had to align two augmented reality markers. Here are two different experimental conditions: visual information activated and deactivated. The subject performed three trials under each condition. The experimental results showed that the subject under the visual information activated mode improved the average time by 70.63 \%.",

keywords = "Augmented reality, hand tacking, human-robot coordination",

author = "John P. and Myungho L. and Cheol S.",

note = "Publisher Copyright: {\textcopyright} 2022 SPIE.; Optical Architectures for Displays and Sensing in Augmented, Virtual, and Mixed Reality (AR, VR, MR) III 2022 ; Conference date: 23-01-2022 Through 25-01-2022",

year = "2022",

doi = "10.1117/12.2614986",

language = "English",

series = "Progress in Biomedical Optics and Imaging - Proceedings of SPIE",

publisher = "SPIE",

editor = "Kress, \{Bernard C.\} and Christophe Peroz",

booktitle = "Optical Architectures for Displays and Sensing in Augmented, Virtual, and Mixed Reality (AR, VR, MR) III",

}

P., J, L., M & S., C 2022, A study on human-robot coordination in augmented reality assisted hand tracking system. in BC Kress & C Peroz (eds), Optical Architectures for Displays and Sensing in Augmented, Virtual, and Mixed Reality (AR, VR, MR) III., 119310K, Progress in Biomedical Optics and Imaging - Proceedings of SPIE, vol. 11931, SPIE, Optical Architectures for Displays and Sensing in Augmented, Virtual, and Mixed Reality (AR, VR, MR) III 2022, San Francisco, United States, 23/01/22. https://doi.org/10.1117/12.2614986

A study on human-robot coordination in augmented reality assisted hand tracking system. / P., John; L., Myungho; S., Cheol.
Optical Architectures for Displays and Sensing in Augmented, Virtual, and Mixed Reality (AR, VR, MR) III. ed. / Bernard C. Kress; Christophe Peroz. SPIE, 2022. 119310K (Progress in Biomedical Optics and Imaging - Proceedings of SPIE; Vol. 11931).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - A study on human-robot coordination in augmented reality assisted hand tracking system

AU - P., John

AU - L., Myungho

AU - S., Cheol

PY - 2022

Y1 - 2022

N2 - Augmented reality environments allow users to interact naturally with 3D objects, including robots. Many robots have been used in the automated sector for painting, picking, packing, and palletizing tasks. The Baxter robot is an example of an industrial robot ideal for research and education. Baxter robots can offer multiple benefits compared with regular robots. In this study, we designed an augmented reality system that makes users intuitively interact in 3D environments by using a Leap Motion controller as a hand tracker and performing a basic human-robot coordination task with the Baxter robot. And the Baxter robot with a stereo camera is connected to a Linux computer, which was programmed with python language. The augmented reality world was programmed in the Unity software. The human robot-coordination task consisted of an augmented reality alignment. We asked the subject to wear the head-mounted display and move the hands. Every hand motion was translated into the robot's limb motion in real-time. The subject and robot had to align two augmented reality markers. Here are two different experimental conditions: visual information activated and deactivated. The subject performed three trials under each condition. The experimental results showed that the subject under the visual information activated mode improved the average time by 70.63 %.

AB - Augmented reality environments allow users to interact naturally with 3D objects, including robots. Many robots have been used in the automated sector for painting, picking, packing, and palletizing tasks. The Baxter robot is an example of an industrial robot ideal for research and education. Baxter robots can offer multiple benefits compared with regular robots. In this study, we designed an augmented reality system that makes users intuitively interact in 3D environments by using a Leap Motion controller as a hand tracker and performing a basic human-robot coordination task with the Baxter robot. And the Baxter robot with a stereo camera is connected to a Linux computer, which was programmed with python language. The augmented reality world was programmed in the Unity software. The human robot-coordination task consisted of an augmented reality alignment. We asked the subject to wear the head-mounted display and move the hands. Every hand motion was translated into the robot's limb motion in real-time. The subject and robot had to align two augmented reality markers. Here are two different experimental conditions: visual information activated and deactivated. The subject performed three trials under each condition. The experimental results showed that the subject under the visual information activated mode improved the average time by 70.63 %.

KW - Augmented reality

KW - hand tacking

KW - human-robot coordination

UR - https://www.scopus.com/pages/publications/85131226014

U2 - 10.1117/12.2614986

DO - 10.1117/12.2614986

M3 - Conference contribution

AN - SCOPUS:85131226014

T3 - Progress in Biomedical Optics and Imaging - Proceedings of SPIE

BT - Optical Architectures for Displays and Sensing in Augmented, Virtual, and Mixed Reality (AR, VR, MR) III

A2 - Kress, Bernard C.

A2 - Peroz, Christophe

PB - SPIE

T2 - Optical Architectures for Displays and Sensing in Augmented, Virtual, and Mixed Reality (AR, VR, MR) III 2022

Y2 - 23 January 2022 through 25 January 2022

ER -

A study on human-robot coordination in augmented reality assisted hand tracking system

Abstract

Publication series

Conference

Bibliographical note

Keywords

UN SDGs

Access to Document

Other files and links

Fingerprint

Cite this