Development and pilot-testing of a feasible, reliable, and valid operative competency assessment tool forendoscopicsinus surgery

Sandra Y. Lin, Kulsoom Laeeq, Masaru Ishii, Jean Kim, Andrew P. Lane, Douglas Reh, Nasir I. Bhatti

Research output: Contribution to journalArticlepeer-review

54 Scopus citations


Background: Otolaryngology residency programs are required by the Accreditation Council for Graduate Medical Education to evaluate residents' operative competency. Many such tools based on the model of objective structured assessment of technical skills (OSATSs) have been developed in other surgical specialties, but no such instruments exist for otolaryngologic procedures except for tonsillectomy. Endoscopic sinus surgery (ESS) is among the most common rhinologic procedure and lends itself to objective evaluation of operative competency. The purpose of this study was to develop and test a new tool for ESS, focusing on feasibility, content and construct validity, and interrater agreement that can be used for such assessment in the operating room and the cadaver dissection course. Methods: Faculty input via the modified Delphi technique helped develop the content of a new OSATS-based instrument. The instrument underwent serial improvements based on 3 years of endoscopic sinus surgery (ESS) cadaver courses. All evaluations were used to calculate construct validity while paired observations were used to determine interrater agreement. Regional and national faculty input was incorporated for increasing generalizability. Internal consistency was calculated using Cronbach's alpha. Results: A total of 51 assessments were completed for 28 residents who were evaluated by 15 faculty members as they performed ESS on cadavers over a period of 3 years. A high degree of internal consistency (0.99) and feasibility was noted for the instrument, which took 7 minutes to complete. The interrater agreement improved with focused faculty development for the 3rd year of the course. Conclusion: Our results and experience suggest that a feasible, reliable, and valid instrument for objective evaluation of operative competency can be developed for ESS. Further experience at other otolaryngology programs and efforts focused on faculty development will be needed to enhance faculty buy-in. The instrument can be used for formative and summative feedback as well as for identifying residents needing remediation.

Original languageEnglish (US)
Pages (from-to)354-359
Number of pages6
JournalAmerican Journal of Rhinology and Allergy
Issue number3
StatePublished - 2009


  • Assessment
  • Competency
  • Construct validity
  • Endoscopic sinus surgery (ESS)
  • Feedback
  • Formative
  • Objective
  • Objective structural assessment of technical skills (OSATS)
  • Operative
  • Otolaryngology
  • Residency
  • Rhinology
  • Summative

ASJC Scopus subject areas

  • Immunology and Allergy
  • Otorhinolaryngology


Dive into the research topics of 'Development and pilot-testing of a feasible, reliable, and valid operative competency assessment tool forendoscopicsinus surgery'. Together they form a unique fingerprint.

Cite this