Foibles, follies, and fusion: Web-based collaboration for medical image labeling

Bennett A. Landman, Andrew J. Asman, Andrew G. Scoggins, John A. Bogovic, Joshua A. Stein, Jerry L. Prince

Research output: Contribution to journalArticlepeer-review

8 Scopus citations


Labels that identify specific anatomical and functional structures within medical images are essential to the characterization of the relationship between structure and function in many scientific and clinical studies. Automated methods that allow for high throughput have not yet been developed for all anatomical targets or validated for exceptional anatomies, and manual labeling remains the gold standard in many cases. However, manual placement of labels within a large image volume such as that obtained using magnetic resonance imaging (MRI) is exceptionally challenging, resource intensive, and fraught with intra- and inter-rater variability. The use of statistical methods to combine labels produced by multiple raters has grown significantly in popularity, in part, because it is thought that by estimating and accounting for rater reliability estimates of the true labels will be more accurate. This paper demonstrates the performance of a class of these statistical label combination methodologies using real-world data contributed by minimally trained human raters. The consistency of the statistical estimates, the accuracy compared to the individual observations, and the variability of both the estimates and the individual observations with respect to the number of labels are presented. It is demonstrated that statistical fusion successfully combines label information using data from online (Internet-based) collaborations among minimally trained raters. This first successful demonstration of a statistically based approach using minimally trained raters opens numerous possibilities for very large scale efforts in collaboration. Extension and generalization of these technologies for new applications will certainly present fascinating areas for continuing research.

Original languageEnglish (US)
Pages (from-to)530-539
Number of pages10
Issue number1
StatePublished - Jan 2 2012


  • Delineation
  • Label fusion
  • Labeling
  • Minimal training
  • Parcellation

ASJC Scopus subject areas

  • Neurology
  • Cognitive Neuroscience


Dive into the research topics of 'Foibles, follies, and fusion: Web-based collaboration for medical image labeling'. Together they form a unique fingerprint.

Cite this