Longitudinal evaluation of interobserver and intraobserver agreement of cervical intraepithelial neoplasia diagnosis among an experienced panel of gynecologic pathologists

Histologic diagnoses of cervical intraepithelial neoplasia grades 2 and 3 (CIN 2/3) are the key end points in clinical trials that evaluate the efficacy of a prophylactic quadrivalent human papillomavirus vaccine against cervical cancer. Adjudication of end points uses a panel of 4 pathologists. Quality control slides (n=185) from a nonclinical trial study with preestablished gold standard CIN diagnoses were used to characterize the panel's agreement on CIN diagnoses and monitor performance longitudinally. At 3-month intervals over 2 years, 1 of 6 different batches of quality control slides (n=30-31) was included with clinical trial slides for independent review by each of the 4 panelists. Unweighted kappas (κ) were estimated within each panelist pair by dichotomizing the diagnoses as CIN+ versus non-CIN+ (including normal, unsatisfactory, and atypical immature metaplasia) or CIN 2/3+ versus non-CIN 2/3+ (including normal, unsatisfactory, atypical immature metaplasia, and CIN 1). Quadratic weighted κ was calculated within each panelist pair using 4 diagnostic categories: normal, CIN 1, CIN 2, and CIN 3 or worse. Substantial interobserver agreement was observed (weighted κ=0.765 to 0.865). Agreement with weighted κ=0.779 to 0.887 was observed between the individual panelists and the gold standard, which is almost perfect agreement by Landis-defined categories. Intraobserver agreement was very high (weighted κ=0.756 to 0.883). Some fluctuation in intraobserver and interobserver agreement was observed over the study period but there was no decreasing time trend. These data indicate that the interpretation of histologic end points used in the quadrivalent vaccine clinical trial program is highly valid and reliable.