Prediction via orthogonalized model mixing

Merlise Clyde, Heather Desimone, Giovanni Parmigiani

Research output: Contribution to journalArticlepeer-review

72 Scopus citations


We introduce an approach and algorithms for model mixing in large prediction problems with correlated predictors. We focus on the choice of predictors in linear models, and mix over possible subsets of candidate predictors. Our approach is based on expressing the space of models in terms of an orthogonalization of the design matrix. Advantages are both statistical and computational. Statistically, orthogonalization often leads to a reduction in the number of competing models by eliminating correlations. Computationally, large model spaces cannot be enumerated; recent approaches are based on sampling models with high posterior probability via Markov chains. Based on orthogonalization of the space of candidate predictors, we can approximate the posterior probabilities of models by products of predictor-specific terms. This leads to an importance sampling function for sampling directly from the joint distribution over the model space, without resorting to Markov chains. Compared to the latter, orthogonalized model mixing by importance sampling is faster in sampling models and is also more efficient in finding models that contribute significantly to the prediction. Further advantages are in the speed of convergence and the availability of more reliable convergence diagnostic tools. We illustrate these in practice, using a data set on prediction of crime rates. The model space is small enough so that enumeration of all models is available for comparison and convergence checks. Also, we demonstrate the feasibility of orthogonalized model mixing in a large-size problem, which is very difficult to attack by other methods. The data set is from a designed experiment dealing with predicting protein activity under different storage conditions. The model space is large (the rank of the design matrix is 88) and very difficult to explore if expressed in terms of the original variables. We obtain prediction intervals and a probability distribution of the setting that produces the highest response.

Original languageEnglish (US)
Pages (from-to)1197-1208
Number of pages12
JournalJournal of the American Statistical Association
Issue number435
StatePublished - Sep 1996
Externally publishedYes


  • Bayesian linear models
  • Importance sampling
  • Model uncertainty
  • Variable selection

ASJC Scopus subject areas

  • General Mathematics
  • Statistics and Probability


Dive into the research topics of 'Prediction via orthogonalized model mixing'. Together they form a unique fingerprint.

Cite this