Design and Preliminary Evaluation of an Augmented Reality Interface Control System for a Robotic Arm

David P. McMullen, Matthew S. Fifer, Kapil D. Katyal, Robert Armiger, Guy Hotson, James D. Beaty, Albert Chi, Daniel B. Drachman, Brock A. Wester

Research output: Contribution to journalArticlepeer-review

3 Scopus citations


Despite advances in the capabilities of robotic limbs, their clinical use by patients with motor disabilities is limited because of inadequate levels of user control. Our Johns Hopkins University Applied Physics Laboratory (APL) team and collaborators designed an augmented reality (AR) control interface that accepts multiple levels of user inputs to a robotic limb using noninvasive eye tracking technology to enhance user control. Our system enables either direct control over 3-D endpoint, gripper orientation, and aperture or supervisory control over several common tasks leveraging computer vision and intelligent route-planning algorithms. This system enables automation of several high-frequency movements (e.g., grabbing an object) that are typically time consuming and require high degrees of precision. Supervisory control can increase movement accuracy and robustness while decreasing the demands on user inputs. We conducted a pilot study in which three subjects with Duchenne muscular dystrophy completed a pick-and-place motor task with the AR interface using both traditional direct and newer supervisory control strategies. The pilot study demonstrated the effectiveness of AR interfaces and the utility of supervisory control for reducing completion time and cognitive burden for certain necessary, repeatable prosthetic control tasks. Future goals include generalizing the supervisory control modes to a wider variety of objects and activities of daily living and integrating the capability into wearable headsets with mixed reality capabilities.

Original languageEnglish (US)
Pages (from-to)220-230
Number of pages11
JournalJohns Hopkins APL Technical Digest (Applied Physics Laboratory)
Issue number3
StatePublished - 2020

ASJC Scopus subject areas

  • General Engineering
  • General Physics and Astronomy


Dive into the research topics of 'Design and Preliminary Evaluation of an Augmented Reality Interface Control System for a Robotic Arm'. Together they form a unique fingerprint.

Cite this