Correcting for Rater Effects in Operating Room Surgical Skills Assessment

Ryan Chou, Hajira Naz, Kofi D.O. Boahene, Jessica H. Maxwell, John R. Wanamaker, Patrick J Byrne, Ira D. Papel, Theda C. Kontis, Gregory D. Hager, Lisa E. Ishii, Sonya Malekzadeh, S. Swaroop Vedula, Masaru Ishii

Research output: Contribution to journalArticlepeer-review


Objective: To estimate and adjust for rater effects in operating room surgical skills assessment performed using a structured rating scale for nasal septoplasty. Methods: We analyzed survey responses from attending surgeons (raters) who supervised residents and fellows (trainees) performing nasal septoplasty in a prospective cohort study. We fit a structural equation model with the rubric item scores regressed on a latent component of skill and then fit a second model including the rating surgeon as a random effect to model a rater-effects-adjusted latent surgical skill. We validated this model against conventional measures including the level of expertise and post-graduation year (PGY) commensurate with the trainee's performance, the actual PGY of the trainee, and whether the surgical goals were achieved. Results: Our dataset included 188 assessments by 7 raters and 41 trainees. The model with one latent construct for surgical skill and the rater as a random effect was the best. Rubric scores depended on how severe or lenient the rater was, sometimes almost as much as they depended on trainee skill. Rater-adjusted latent skill scores increased with attending-estimated skill levels and PGY of trainees, increased with the actual PGY, and appeared constant over different levels of achievement of surgical goals. Conclusion: Our work provides a method to obtain rater effect adjusted surgical skill assessments in the operating room using structured rating scales. Our method allows for the creation of standardized (i.e., rater-effects-adjusted) quantitative surgical skill benchmarks using national-level databases on trainee assessments. Level of Evidence: N/A Laryngoscope, 2024.

Original languageEnglish (US)
StateAccepted/In press - 2024


  • rater bias
  • rater effect
  • septoplasty
  • SGAT
  • surgical skill assessment

ASJC Scopus subject areas

  • Otorhinolaryngology


Dive into the research topics of 'Correcting for Rater Effects in Operating Room Surgical Skills Assessment'. Together they form a unique fingerprint.

Cite this