Evaluation of a simpler tool to assess nontechnical skills during simulated critical events

Scott C. Watkins, David A. Roberts, John R. Boulet, Matthew D. Mcevoy, Matthew B. Weinger

Research output: Contribution to journalArticlepeer-review

15 Scopus citations


Introduction Management of critical events requires teams to employ nontechnical skills (NTS), such as teamwork, communication, decision making, and vigilance. We sought to estimate the reliability and provide evidence for the validity of the ratings gathered using a new tool for assessing the NTS of anesthesia providers, the behaviorally anchored rating scale (BARS), and compare its scores with those of an established NTS tool, the Anaesthetists' Nontechnical Skills (ANTS) scale. Methods Six previously trained raters (4 novices and 2 experts) reviewed and scored 18 recorded simulated pediatric crisis management scenarios using a modified ANTS and a BARS tool. Pearson correlation coefficients were calculated separately for the novice and expert raters, by scenario, and overall. Results The intrarater reliability of the ANTS total score was 0.73 (expert, 0.57; novice, 0.84); for the BARS tool, it was 0.80 (expert, 0.79; novice, 0.81). The average interrater reliability of BARS scores (0.58) was better than ANTS scores (0.37), and the interrater reliabilities of scores from novices (0.69 BARS and 0.52 ANTS) were better than those obtained from experts (0.47 BARS and 0.21 ANTS) for both scoring instruments. The Pearson correlation between the ANTS and BARS total scores was 0.74. Conclusions Overall, reliability estimates were better for the BARS scores than the ANTS scores. For both measures, the intrarater and interrater reliability was better for novices compared with domain experts, suggesting that properly trained novices can reliably assess the NTS of anesthesia providers managing a simulated critical event. There was substantial correlation between the 2 scoring instruments, suggesting that the tools measured similar constructs. The BARS tool can be an alternative to the ANTS scale for the formative assessment of NTS of anesthesia providers.

Original languageEnglish (US)
Pages (from-to)69-75
Number of pages7
JournalSimulation in Healthcare
Issue number2
StatePublished - Apr 1 2017
Externally publishedYes


  • Clinical competency
  • Competency-based education
  • Educational measurement

ASJC Scopus subject areas

  • Epidemiology
  • Medicine (miscellaneous)
  • Education
  • Modeling and Simulation


Dive into the research topics of 'Evaluation of a simpler tool to assess nontechnical skills during simulated critical events'. Together they form a unique fingerprint.

Cite this