Optimal observer framework and categorization observer framework for three-class ROC analysis

ROC analysis has been an important tool for system evaluation and optimization in medical imaging. Despite its success in evaluating binary classification tasks, ROC analysis does not provide a direct way for evaluating performance on classification tasks that involve more than two diagnostic alternatives. We have previously developed a three-class ROC analysis method that provides a practical way to evaluate three-class task performance. Based on two-class ROC analysis and the proposed three-class ROC analysis method, this work proposes two frameworks, the optimal observer framework and the categorization observer framework, for three-class ROC analysis. The optimal observer framework seeks three-class decision rules and decision variables based on a formal decision strategy; it provides a ROC surface for system comparison on the basis of optimal performance with respect to this strategy. A categorization procedure is the generalization to 3-D of a 2-alternative forced choice procedure and is an important concept in the categorization observer framework. The categorization observer framework seeks three-class decision rules, decision variables and ROC surface such that task performance as measured by volume under the ROC surface (VUS) and the percent correct on the categorization procedure are equal. We then show that how our previously-proposed three-class ROC method fits into both frameworks.