A comparison of resampling schemes for estimating model observer performance with small ensembles

Fatma E.A. Elshahaby, Abhinav K. Jha, Michael Ghaly, Eric C. Frey

Research output: Contribution to journalArticlepeer-review

2 Scopus citations


In objective assessment of image quality, an ensemble of images is used to compute the 1st and 2nd order statistics of the data. Often, only a finite number of images is available, leading to the issue of statistical variability in numerical observer performance. Resampling-based strategies can help overcome this issue. In this paper, we compared different combinations of resampling schemes (the leave-one-out (LOO) and the half-train/half-test (HT/HT)) and model observers (the conventional channelized Hotelling observer (CHO), channelized linear discriminant (CLD) and channelized quadratic discriminant). Observer performance was quantified by the area under the ROC curve (AUC). For a binary classification task and for each observer, the AUC value for an ensemble size of 2000 samples per class served as a gold standard for that observer. Results indicated that each observer yielded a different performance depending on the ensemble size and the resampling scheme. For a small ensemble size, the combination [CHO, HT/HT] had more accurate rankings than the combination [CHO, LOO]. Using the LOO scheme, the CLD and CHO had similar performance for large ensembles. However, the CLD outperformed the CHO and gave more accurate rankings for smaller ensembles. As the ensemble size decreased, the performance of the [CHO, LOO] combination seriously deteriorated as opposed to the [CLD, LOO] combination. Thus, it might be desirable to use the CLD with the LOO scheme when smaller ensemble size is available.

Original languageEnglish (US)
Pages (from-to)7300-7320
Number of pages21
JournalPhysics in medicine and biology
Issue number18
StatePublished - Aug 22 2017


  • Hotelling observe
  • channelized model observers
  • leave-one-out resampling scheme
  • rank correlation coefficient

ASJC Scopus subject areas

  • Radiological and Ultrasound Technology
  • Radiology Nuclear Medicine and imaging


Dive into the research topics of 'A comparison of resampling schemes for estimating model observer performance with small ensembles'. Together they form a unique fingerprint.

Cite this