TY - JOUR
T1 - Covariate Balance for Observational Effectiveness Studies
T2 - A Comparison of Matching and Weighting
AU - Kush, Joseph M.
AU - Pas, Elise T.
AU - Musci, Rashelle J.
AU - Bradshaw, Catherine P.
N1 - Funding Information:
The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through Grants R305H150027 and R305A150221 to the University of Virginia. The opinions expressed are those of the authors and do not represent views of the Institute or the U.S. Department of Education. We thank the Maryland PBIS Management Team, which includes the Maryland State Department of Education, Sheppard Pratt Health System, and the 24 local school districts. We would also like to give special thanks to Drs. Ji Hoon Ryoo and Elizabeth Stuart for providing feedback on the article and for their methodological consultation.
Publisher Copyright:
© 2022 Taylor & Francis Group, LLC.
PY - 2023
Y1 - 2023
N2 - Propensity score matching and weighting methods are often used in observational effectiveness studies to reduce imbalance between treated and untreated groups on a set of potential confounders. However, much of the prior methodological literature on matching and weighting has yet to examine performance for scenarios with a majority of treated units, as is often encountered with programs and interventions that have been widely disseminated or “scaled-up.” Using a series of Monte Carlo simulations, we compare the performance of k:1 matching with replacement and weighting methods with respect to covariate balance, bias, and mean squared error. Results indicate that the accuracy of all methods declined as treatment prevalence increased. While weighting produced the largest reduction in covariate imbalance, 1:1 matching with replacement provided the most unbiased treatment effect estimates. An applied example using empirical school-level data is provided to further illustrate the application and interpretation of these methods to a real-world scale-up effort. We conclude by considering the implications of propensity score methods for observational effectiveness studies with a particular focus on educational research.
AB - Propensity score matching and weighting methods are often used in observational effectiveness studies to reduce imbalance between treated and untreated groups on a set of potential confounders. However, much of the prior methodological literature on matching and weighting has yet to examine performance for scenarios with a majority of treated units, as is often encountered with programs and interventions that have been widely disseminated or “scaled-up.” Using a series of Monte Carlo simulations, we compare the performance of k:1 matching with replacement and weighting methods with respect to covariate balance, bias, and mean squared error. Results indicate that the accuracy of all methods declined as treatment prevalence increased. While weighting produced the largest reduction in covariate imbalance, 1:1 matching with replacement provided the most unbiased treatment effect estimates. An applied example using empirical school-level data is provided to further illustrate the application and interpretation of these methods to a real-world scale-up effort. We conclude by considering the implications of propensity score methods for observational effectiveness studies with a particular focus on educational research.
KW - Propensity scores
KW - matching
KW - treatment prevalence
KW - weighting
UR - http://www.scopus.com/inward/record.url?scp=85137772105&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85137772105&partnerID=8YFLogxK
U2 - 10.1080/19345747.2022.2110545
DO - 10.1080/19345747.2022.2110545
M3 - Article
AN - SCOPUS:85137772105
SN - 1934-5747
VL - 16
SP - 189
EP - 212
JO - Journal of Research on Educational Effectiveness
JF - Journal of Research on Educational Effectiveness
IS - 2
ER -