Deep Stroop: Integrating eye tracking and speech processing to characterize people with neurodegenerative disorders while performing neuropsychological tests

Neurodegenerative diseases (NDs) can be difficult to precisely characterize and monitor as they present complex and overlapping signs despite affecting different neural circuits. Neuropsychological tests are important tools for assessing signs, but only considering patient-generated output can limit insight. Here, we present an improvement to the neuropsychological test evaluation paradigm by deeply characterizing patient interaction and behavior during tests based on multiple perspectives alongside typically evaluated output by performing multi-modal analysis of eye movement and speech data. Using the well-known Stroop Test, we compare behaviors of healthy controls to patients with Alzheimer's Disease (AD), Mild Cognitive Impairment, Parkinson's Disease (PD), and secondary Parkinsonism. We maximize accessibility and reproducibility by automatically extracting metrics, including eye motor behavior, speech patterns, and multimodal interplay, with almost no human input required. We find many metrics including increased horizontal saccade distances sensitive to all NDs, delayed task initiation in AD, response error patterns and blinking patterns that differ between AD and PD. Our metrics show both significantly different distributions between disease groups and simultaneous correlation with the MoCA and MDS-UPDRS-III clinical rating scales. Our findings show the utility of incorporating several perspectives into one output representation, as our metric breadth creates unique sign profiles that quantify and visualize a patient's dysfunction. These metrics provide much better sign characterization between diseases and correlation with disease severity than traditional Stroop measures. This methodology offers the potential to expand its application to other traditional neuropsychological tests, shifting the paradigm in diagnostic precision for NDs and advancing patient care.