Testing for improvement in prediction model performance

Margaret Sullivan Pepe, Kathleen F. Kerr, Gary Longton, Zheyu Wang

Research output: Contribution to journalArticlepeer-review

136 Scopus citations


Authors have proposed new methodology in recent years for evaluating the improvement in prediction performance gained by adding a new predictor, Y, to a risk model containing a set of baseline predictors, X, for a binary outcome D. We prove theoretically that null hypotheses concerning no improvement in performance are equivalent to the simple null hypothesis that Y is not a risk factor when controlling for X, H0: P(D=1|X,Y)=P(D=1|X). Therefore, testing for improvement in prediction performance is redundant if Y has already been shown to be a risk factor. We also investigate properties of tests through simulation studies, focusing on the change in the area under the ROC curve (AUC). An unexpected finding is that standard testing procedures that do not adjust for variability in estimated regression coefficients are extremely conservative. This may explain why the AUC is widely considered insensitive to improvements in prediction performance and suggests that the problem of insensitivity has to do with use of invalid procedures for inference rather than with the measure itself. To avoid redundant testing and use of potentially problematic methods for inference, we recommend that hypothesis testing for no improvement be limited to evaluation of Y as a risk factor, for which methods are well developed and widely available. Analyses of measures of prediction performance should focus on estimation rather than on testing for no improvement in performance.

Original languageEnglish (US)
Pages (from-to)1467-1482
Number of pages16
JournalStatistics in Medicine
Issue number9
StatePublished - Apr 30 2013
Externally publishedYes


  • Biomarker
  • Logistic regression
  • Receiver operating characteristic curve
  • Risk factors
  • Risk reclassification

ASJC Scopus subject areas

  • Epidemiology
  • Statistics and Probability


Dive into the research topics of 'Testing for improvement in prediction model performance'. Together they form a unique fingerprint.

Cite this