HARMONIE: A multimodal control framework for human assistive robotics

Kapil D. Katyal, Matthew S. Johannes, Timothy G. McGee, Andrew J. Harris, Robert S. Armiger, Alex H. Firpi, David McMullen, Guy Hotson, Matthew S. Fifer, Nathan E. Crone, R. Jacob Vogelstein, Brock A. Wester

Research output: Chapter in Book/Report/Conference proceedingConference contribution

12 Scopus citations


Effective user control of highly dexterous and robotic assistive devices requires intuitive and natural modalities. Although surgically implanted brain-computer interfaces (BCIs) strive to achieve this, a number of non-invasive engineering solutions may provide a quicker path to patient use by eliminating surgical implantation. We present the development of a semi-autonomous control system that utilizes computer vision, prosthesis feedback, effector centric device control, smooth movement trajectories, and appropriate hand conformations to interact with objects of interest. Users can direct a prosthetic limb through an intuitive graphical user interface to complete multi-stage tasks using patient appropriate combinations of control inputs such as eye tracking, conventional prosthetic controls/joysticks, surface electromyography (sEMG) signals, and neural interfaces (ECoG, EEG). Aligned with activities of daily living (ADL), these tasks include directing the prosthetic to specific locations or objects, grasping of objects by modulating hand conformation, and action upon grasped objects such as self-feeding. This Hybrid Augmented Reality Multimodal Operation Neural Integration Environment (HARMONIE) semi-autonomous control system lowers the user's cognitive load, leaving the bulk of command and control of the device to the computer. This flexible and intuitive control system could serve patient populations ranging from wheelchair-bound quadriplegics to upper-limb amputees.

Original languageEnglish (US)
Title of host publication2013 6th International IEEE EMBS Conference on Neural Engineering, NER 2013
Number of pages5
StatePublished - Dec 1 2013
Event2013 6th International IEEE EMBS Conference on Neural Engineering, NER 2013 - San Diego, CA, United States
Duration: Nov 6 2013Nov 8 2013

Publication series

NameInternational IEEE/EMBS Conference on Neural Engineering, NER
ISSN (Print)1948-3546
ISSN (Electronic)1948-3554


Other2013 6th International IEEE EMBS Conference on Neural Engineering, NER 2013
Country/TerritoryUnited States
CitySan Diego, CA


  • Assistive robotics
  • Brain-computer interface
  • Brain-machine interface
  • Computer vision
  • Hybrid BCI/BMI
  • Intelligent robotics
  • Modular prosthetic limb
  • Neural prosthetic system
  • Prosthetics
  • Robotic limb
  • Semi-autonomous

ASJC Scopus subject areas

  • Artificial Intelligence
  • Mechanical Engineering


Dive into the research topics of 'HARMONIE: A multimodal control framework for human assistive robotics'. Together they form a unique fingerprint.

Cite this