Multimodal fusion of structural and functional brain imaging data

Recent years have witnessed a rapid growth of interest in moving functional magnetic resonance imaging (fMRI) beyond simple scan-length averages and into approaches that can integrate structural MRI measures and capture rich multimodal interactions. It is becoming increasingly clear that multimodal fusion is able to provide more information for individual subjects by exploiting covariation between modalities, rather an analysis of each modality alone. Multimodal fusion is a more complicated endeavor that must be approached carefully and efficient methods should be developed to draw generalized and valid conclusions out of high dimensional data with a limited number of subjects, such as patients with brain disorders. Numerous research efforts have been reported in the field based on various statistical models, including independent component analysis (ICA), canonical correlation analysis (CCA), and partial least squares (PLS). In this chapter, we survey a number of methods previously shown in multimodal fusion reports, performed with or without prior information, and with their possible strengths and limitations addressed. To examine the function-structure associations of the brain in a more comprehensive and integrated manner, we also reviewed a number of multimodal studies that combined fMRI and structural (sMRI and/or diffusion tensor MRI) measures, which could reveal important brain alterations that may not be fully detected by employing separate analysis of individual modalities, and also enable us to identify potential brain illness biomarkers.