Multimodal Fusion of Brain Imaging Data: A Key to Finding the Missing Link(s) in Complex Mental Illness

It is becoming increasingly clear that combining multimodal brain imaging data provides more information for individual subjects by exploiting the rich multimodal information that exists. However, the number of studies that do true multimodal fusion (i.e., capitalizing on joint information among modalities) is still remarkably small given the known benefits. In part, this is because multimodal studies require broader expertise in collecting, analyzing, and interpreting the results than do unimodal studies. In this article, we start by introducing the basic reasons why multimodal data fusion is important and what it can do and, importantly, how it can help us avoid wrong conclusions and help compensate for imperfect brain imaging studies. We also discuss the challenges that need to be confronted for such approaches to be more widely applied by the community. We then provide a review of the diverse studies that have used multimodal data fusion (primarily focused on psychosis) as well as provide an introduction to some of the existing analytic approaches. Finally, we discuss some up-and-coming approaches to multimodal fusion including deep learning and multimodal classification that show considerable promise. Our conclusion is that multimodal data fusion is rapidly growing, but it is still underutilized. The complexity of the human brain coupled with the incomplete measurement provided by existing imaging technology makes multimodal fusion essential to mitigate misdirection and hopefully provide a key to finding the missing link(s) in complex mental illness.