Inter-Rater Agreement of Intensivists Evaluating the Goal Concordance of Preference-Sensitive ICU Interventions

Alison E. Turnbull, Sarina K. Sahetya, Elizabeth Colantuoni, Josephine Kweku, Roozbeh Nikooie, J. Randall Curtis

Research output: Contribution to journalArticlepeer-review

5 Scopus citations


Context: Goal-concordant care has been identified as an important outcome of advance care planning and shared decision-making initiatives. However, validated methods for measuring goal concordance are needed. Objectives: To estimate the inter-rater reliability of senior critical care fellows rating the goal concordance of preference-sensitive interventions performed in intensive care units (ICUs) while considering patient-specific circumstances as described in a previously proposed methodology. Methods: We identified ICU patients receiving preference-sensitive interventions in three adult ICUs at Johns Hopkins Hospital. A simulated cohort was created by randomly assigning each patient one of 10 sets of goals and preferences about limiting life support. Critical care fellows then independently reviewed patient charts and answered two questions: 1) Is this patient's goal achievable? and 2) Will performing this intervention help achieve the patient's goal? When the answer to both questions was yes, the intervention was rated as goal concordant. Inter-rater agreement was summarized by estimating intraclass correlation coefficient using mixed-effects models. Results: Six raters reviewed the charts of 201 patients. Interventions were rated as goal concordant 22%–92% of the time depending on the patient's goal-limitation combination. Percent agreement between pairs of raters ranged from 59% to 86%. The intraclass correlation coefficient for ratings of goal concordance was 0.50 (95% CI 0.31–0.69) and was robust to patient age, gender, ICU, severity of illness, and lengths of stay. Conclusion: Inter-rater agreement between intensivists using a standardized methodology to evaluate the goal concordance of preference-sensitive ICU interventions was moderate. Further testing is needed before this methodology can be recommended as a clinical research outcome.

Original languageEnglish (US)
Pages (from-to)406-413.e3
JournalJournal of Pain and Symptom Management
Issue number3
StatePublished - Sep 2018


  • Goal-concordant care
  • critical care outcomes
  • critical illness
  • interobserver variability
  • outcome measures
  • patient-centered care
  • psychometrics

ASJC Scopus subject areas

  • General Nursing
  • Clinical Neurology
  • Anesthesiology and Pain Medicine


Dive into the research topics of 'Inter-Rater Agreement of Intensivists Evaluating the Goal Concordance of Preference-Sensitive ICU Interventions'. Together they form a unique fingerprint.

Cite this