Empirical performance of cross-validation with oracle methods in a genomics context

Josue G. Martinez, Raymond J. Carroll, Samuel Müller, Joshua N. Sampson, Nilanjan Chatterjee

Research output: Contribution to journalArticlepeer-review

15 Scopus citations


When employing model selection methods with oracle properties such as the smoothly clipped absolute deviation (SCAD) and the Adaptive Lasso, it is typical to estimate the smoothing parameter by m-fold cross-validation, for example, m = 10. In problems where the true regression function is sparse and the signals large, such cross-validation typically works well. However, in regression modeling of genomic studies involving Single Nucleotide Polymorphisms (SNP), the true regression functions, while thought to be sparse, do not have large signals. We demonstrate empirically that in such problems, the number of selected variables using SCAD and the Adaptive Lasso, with 10-fold cross-validation, is a random variable that has considerable and surprising variation. Similar remarks apply to nonoracle methods such as the Lasso. Our study strongly questions the suitability of performing only a single run of m-fold crossvalidation with any oracle method, and not just the SCAD and Adaptive Lasso.

Original languageEnglish (US)
Pages (from-to)223-228
Number of pages6
JournalAmerican Statistician
Issue number4
StatePublished - Nov 2011
Externally publishedYes


  • Adaptive lasso
  • Lasso
  • Model selection
  • Oracle estimation

ASJC Scopus subject areas

  • Statistics and Probability
  • General Mathematics
  • Statistics, Probability and Uncertainty


Dive into the research topics of 'Empirical performance of cross-validation with oracle methods in a genomics context'. Together they form a unique fingerprint.

Cite this