G-MIND: An end-to-end multimodal imaging-genetics framework for biomarker identification and disease classification

Sayan Ghosal, Qiang Chen, Giulio Pergola, Aaron L. Goldman, William Ulrich, Karen F. Berman, Giuseppe Blasi, Leonardo Fazio, Antonio Rampino, Alessandro Bertolino, Daniel R. Weinberger, Venkata S. Mattay, Archana Venkataraman

Research output: Chapter in Book/Report/Conference proceedingConference contribution


We propose a novel deep neural network architecture to integrate imaging and genetics data, as guided by diagnosis, that provides interpretable biomarkers. Our model consists of an encoder, a decoder and a classifier. The encoder learns a non-linear subspace shared between the input data modalities. The classifier and the decoder act as regularizers to ensure that the low-dimensional encoding captures predictive differences between patients and controls. We use a learnable dropout layer to extract interpretable biomarkers from the data, and our unique training strategy can easily accommodate missing data modalities across subjects. We have evaluated our model on a population study of schizophrenia that includes two functional MRI (fMRI) paradigms and Single Nucleotide Polymorphism (SNP) data. Using 10-fold cross validation, we demonstrate that our model achieves better classification accuracy than baseline methods, and that this performance generalizes to a second dataset collected at a different site. In an exploratory analysis we further show that the biomarkers identified by our model are closely associated with the well-documented deficits in schizophrenia.

Original languageEnglish (US)
Title of host publicationMedical Imaging 2021
Subtitle of host publicationImage Processing
EditorsIvana Isgum, Bennett A. Landman
ISBN (Electronic)9781510640214
StatePublished - 2021
Externally publishedYes
EventMedical Imaging 2021: Image Processing - Virtual, Online, United States
Duration: Feb 15 2021Feb 19 2021

Publication series

NameProgress in Biomedical Optics and Imaging - Proceedings of SPIE
ISSN (Print)1605-7422


ConferenceMedical Imaging 2021: Image Processing
Country/TerritoryUnited States
CityVirtual, Online


  • Deep Neural Networks
  • Imaging-Genetics
  • Learnable Dropout
  • Schizophrenia

ASJC Scopus subject areas

  • Electronic, Optical and Magnetic Materials
  • Atomic and Molecular Physics, and Optics
  • Biomaterials
  • Radiology Nuclear Medicine and imaging


Dive into the research topics of 'G-MIND: An end-to-end multimodal imaging-genetics framework for biomarker identification and disease classification'. Together they form a unique fingerprint.

Cite this