Improving inter-rater reliability of the enhancing assessment of common therapeutic factors (ENACT) measure through training of raters

Background. As evidence supports task-shifting approaches to reduce the global mental health treatment gap, counselor competency evaluation measures are critical to ensure evidence-based therapies are administered with quality and fidelity. Objective. This article describes a training technique for evaluating lay counselors’ competency for mental health lay practitioners without rating scale experience. Methods: Mental health practitioners were trained to give the Enhancing Assessment of Common Therapeutic Factors (ENACT) test to assess counselor proficiency in delivering the Common Elements Treatment Approach (CETA) in-person and over the phone using standardized video and audio recordings. A two-day in-person training was followed by a one-day remote training session. Training includes a review of item scales through didactic instructions, active learning by witnessing and scoring role-plays, peer interactions, and trainer observation and feedback. The trainees rated video and audio recordings, and ICC values were calculated. Results: The training technique presented in this research helped achieve high counselor competency scores among lay providers with no prior experience using rating scales. ICC rated both trainings satis-factory to exceptional (ICC:.71-.89). Conclusions. Raters with no past experience with rating scales can achieve high consistency when rating counselor competency through training. Effective rater training should include didactic learning, practi-cal learning with trainer observation and feedback, and video and audio recordings to assess consistency.