Explainable medical imaging AI needs human-centered design: guidelines and evidence from a systematic review

Haomin Chen, Catalina Gomez, Chien Ming Huang, Mathias Unberath

Research output: Contribution to journalReview articlepeer-review


Transparency in Machine Learning (ML), often also referred to as interpretability or explainability, attempts to reveal the working mechanisms of complex models. From a human-centered design perspective, transparency is not a property of the ML model but an affordance, i.e., a relationship between algorithm and users. Thus, prototyping and user evaluations are critical to attaining solutions that afford transparency. Following human-centered design principles in highly specialized and high stakes domains, such as medical image analysis, is challenging due to the limited access to end users and the knowledge imbalance between those users and ML designers. To investigate the state of transparent ML in medical image analysis, we conducted a systematic review of the literature from 2012 to 2021 in PubMed, EMBASE, and Compendex databases. We identified 2508 records and 68 articles met the inclusion criteria. Current techniques in transparent ML are dominated by computational feasibility and barely consider end users, e.g. clinical stakeholders. Despite the different roles and knowledge of ML developers and end users, no study reported formative user research to inform the design and development of transparent ML models. Only a few studies validated transparency claims through empirical user evaluations. These shortcomings put contemporary research on transparent ML at risk of being incomprehensible to users, and thus, clinically irrelevant. To alleviate these shortcomings in forthcoming research, we introduce the INTRPRT guideline, a design directive for transparent ML systems in medical image analysis. The INTRPRT guideline suggests human-centered design principles, recommending formative user research as the first step to understand user needs and domain requirements. Following these guidelines increases the likelihood that the algorithms afford transparency and enable stakeholders to capitalize on the benefits of transparent ML.

Original languageEnglish (US)
Article number156
Journalnpj Digital Medicine
Issue number1
StatePublished - Dec 2022
Externally publishedYes

ASJC Scopus subject areas

  • Health Information Management
  • Health Informatics
  • Medicine (miscellaneous)
  • Computer Science Applications


Dive into the research topics of 'Explainable medical imaging AI needs human-centered design: guidelines and evidence from a systematic review'. Together they form a unique fingerprint.

Cite this