Gastrointestinal endoscopy competency assessment tool: Reliability and validity evidence

Background Rigorously developed and validated direct observational assessment tools are required to support competency-based colonoscopy training to facilitate skill acquisition, optimize learning, and ensure readiness for unsupervised practice. Objective To examine reliability and validity evidence of the Gastrointestinal Endoscopy Competency Assessment Tool (GiECAT) for colonoscopy for use within the clinical setting. Design Prospective, observational, multicenter validation study. Sixty-one endoscopists performing 116 colonoscopies were assessed using the GiECAT, which consists of a 7-item global rating scale (GRS) and 19-item checklist (CL). A second rater assessed procedures to determine interrater reliability by using intraclass correlation coefficients (ICCs). Endoscopists' first and second procedure scores were compared to determine test-retest reliability by using ICCs. Discriminative validity was examined by comparing novice, intermediate, and experienced endoscopists' scores. Concurrent validity was measured by correlating scores with colonoscopy experience, cecal and terminal ileal intubation rates, and physician global assessment. Setting A total of 116 colonoscopies performed by 33 novice (<50 previous procedures), 18 intermediate (50-500 previous procedures), and 10 experienced (>1000 previous procedures) endoscopists from 6 Canadian hospitals. Main Outcome Measurements Interrater and test-retest reliability, discriminative, and concurrent validity. Results Interrater reliability was high (total: ICC = 0.85; GRS: ICC = 0.85; CL: ICC = 0.81). Test-retest reliability was excellent (total: ICC = 0.91; GRS: ICC = 0.93; CL: ICC = 0.80). Significant differences in GiECAT scores among novice, intermediate, and experienced endoscopists were noted (P <.001). There was a significant positive correlation (P <.001) between scores and number of previous colonoscopies (total: ρ = 0.78, GRS: ρ = 0.80, CL: Spearman's ρ = 0.71); cecal intubation rate (total: ρ = 0.81, GRS: Spearman's ρ = 0.82, CL: Spearman's ρ = 0.75); ileal intubation rate (total: Spearman's ρ = 0.82, GRS: Spearman's ρ = 0.82, CL: Spearman's ρ = 0.77); and physician global assessment (total: Spearman's ρ = 0.90, GRS: Spearman's ρ = 0.94, CL: Spearman's ρ = 0.77). Limitations Nonblinded assessments. Conclusion This study provides evidence supporting the reliability and validity of the GiECAT for use in assessing the performance of live colonoscopies in the clinical setting.