Comparing the performance of statistical methods that generalize effect estimates from randomized controlled trials to much larger target populations

Ian Schmid, Kara E. Rudolph, Trang Quynh Nguyen, Hwanhee Hong, Marissa J. Seamans, Benjamin Ackerman, Elizabeth A. Stuart

Research output: Contribution to journalArticlepeer-review

Abstract

Policymakers use results from randomized controlled trials to inform decisions about whether to implement treatments in target populations. Various methods—including inverse probability weighting, outcome modeling, and Targeted Maximum Likelihood Estimation—that use baseline data available in both the trial and target population have been proposed to generalize the trial treatment effect estimate to the target population. Often the target population is significantly larger than the trial sample, which can cause estimation challenges. We conduct simulations to compare the performance of these methods in this setting. We vary the size of the target population, the proportion of the target population selected into the trial, and the complexity of the true selection and outcome models. All methods performed poorly when the trial size was only 2% of the target population size or the target population included only 1,000 units. When the target population or the proportion of units selected into the trial was larger, some methods, such as outcome modeling using Bayesian Additive Regression Trees, performed well. We caution against generalizing using these existing approaches when the target population is much larger than the trial sample and advocate future research strives to improve methods for generalizing to large target populations.

Original languageEnglish (US)
Pages (from-to)4326-4348
Number of pages23
JournalCommunications in Statistics: Simulation and Computation
Volume51
Issue number8
DOIs
StatePublished - 2022

Keywords

  • Causal inference
  • External validity
  • Generalizability

ASJC Scopus subject areas

  • Statistics and Probability
  • Modeling and Simulation

Fingerprint

Dive into the research topics of 'Comparing the performance of statistical methods that generalize effect estimates from randomized controlled trials to much larger target populations'. Together they form a unique fingerprint.

Cite this