Multimodal Fusion of Brain Imaging Data: A Key to Finding the Missing Link(s) in Complex Mental Illness

Vince D. Calhoun, Jing Sui

Research output: Contribution to journalReview articlepeer-review

156 Scopus citations


It is becoming increasingly clear that combining multimodal brain imaging data provides more information for individual subjects by exploiting the rich multimodal information that exists. However, the number of studies that do true multimodal fusion (i.e., capitalizing on joint information among modalities) is still remarkably small given the known benefits. In part, this is because multimodal studies require broader expertise in collecting, analyzing, and interpreting the results than do unimodal studies. In this article, we start by introducing the basic reasons why multimodal data fusion is important and what it can do and, importantly, how it can help us avoid wrong conclusions and help compensate for imperfect brain imaging studies. We also discuss the challenges that need to be confronted for such approaches to be more widely applied by the community. We then provide a review of the diverse studies that have used multimodal data fusion (primarily focused on psychosis) as well as provide an introduction to some of the existing analytic approaches. Finally, we discuss some up-and-coming approaches to multimodal fusion including deep learning and multimodal classification that show considerable promise. Our conclusion is that multimodal data fusion is rapidly growing, but it is still underutilized. The complexity of the human brain coupled with the incomplete measurement provided by existing imaging technology makes multimodal fusion essential to mitigate misdirection and hopefully provide a key to finding the missing link(s) in complex mental illness.

Original languageEnglish (US)
Pages (from-to)230-244
Number of pages15
JournalBiological Psychiatry: Cognitive Neuroscience and Neuroimaging
Issue number3
StatePublished - May 1 2016


  • Brain function
  • Connectivity
  • Data fusion
  • Independent component analysis
  • Psychosis
  • Schizophrenia

ASJC Scopus subject areas

  • Radiology Nuclear Medicine and imaging
  • Cognitive Neuroscience
  • Clinical Neurology
  • Biological Psychiatry


Dive into the research topics of 'Multimodal Fusion of Brain Imaging Data: A Key to Finding the Missing Link(s) in Complex Mental Illness'. Together they form a unique fingerprint.

Cite this