A comparison of expert and nonexpert raters using the consensual assessment technique

James C. Kaufman; John Baer; Jason C. Cole; Janel D. Sexton

doi:10.1080/10400410802059929

A comparison of expert and nonexpert raters using the consensual assessment technique

James C. Kaufman, John Baer, Jason C. Cole, Janel D. Sexton

Research output: Contribution to journal › Article › peer-review

187 Scopus citations

Abstract

The Consensual Assessment Technique (CAT) is one of the most highly regarded assessment tools in creativity, but it is often difficult and/or expensive to assemble the teams of experts required by the CAT. Some researchers have tried using nonexpert raters in their place, but the validity of replacing experts with nonexperts has not been adequately tested. Expert (n = 10) and nonexpert (n = 106) creativity ratings of 205 poems were compared and found to be quite different, making the simple replacement of experts by nonexpert raters suspect. Nonexpert raters' judgments of creativity were inconsistent (showing low interrater reliability) and did not match those of the expert raters. Implications are discussed, including the appropriate selection of expert raters for different kinds of creativity assessment.

Original language	English (US)
Pages (from-to)	171-178
Number of pages	8
Journal	Creativity Research Journal
Volume	20
Issue number	2
DOIs	https://doi.org/10.1080/10400410802059929
State	Published - Apr 2008
Externally published	Yes

ASJC Scopus subject areas

Visual Arts and Performing Arts
Developmental and Educational Psychology
Psychology (miscellaneous)

Access to Document

10.1080/10400410802059929

Cite this

@article{e5e2953ee649406cbd038bbf5fb6e429,

title = "A comparison of expert and nonexpert raters using the consensual assessment technique",

abstract = "The Consensual Assessment Technique (CAT) is one of the most highly regarded assessment tools in creativity, but it is often difficult and/or expensive to assemble the teams of experts required by the CAT. Some researchers have tried using nonexpert raters in their place, but the validity of replacing experts with nonexperts has not been adequately tested. Expert (n = 10) and nonexpert (n = 106) creativity ratings of 205 poems were compared and found to be quite different, making the simple replacement of experts by nonexpert raters suspect. Nonexpert raters' judgments of creativity were inconsistent (showing low interrater reliability) and did not match those of the expert raters. Implications are discussed, including the appropriate selection of expert raters for different kinds of creativity assessment.",

author = "Kaufman, {James C.} and John Baer and Cole, {Jason C.} and Sexton, {Janel D.}",

year = "2008",

month = apr,

doi = "10.1080/10400410802059929",

language = "English (US)",

volume = "20",

pages = "171--178",

journal = "Creativity Research Journal",

issn = "1040-0419",

publisher = "Routledge",

number = "2",

}

TY - JOUR

T1 - A comparison of expert and nonexpert raters using the consensual assessment technique

AU - Kaufman, James C.

AU - Baer, John

AU - Cole, Jason C.

AU - Sexton, Janel D.

PY - 2008/4

Y1 - 2008/4

N2 - The Consensual Assessment Technique (CAT) is one of the most highly regarded assessment tools in creativity, but it is often difficult and/or expensive to assemble the teams of experts required by the CAT. Some researchers have tried using nonexpert raters in their place, but the validity of replacing experts with nonexperts has not been adequately tested. Expert (n = 10) and nonexpert (n = 106) creativity ratings of 205 poems were compared and found to be quite different, making the simple replacement of experts by nonexpert raters suspect. Nonexpert raters' judgments of creativity were inconsistent (showing low interrater reliability) and did not match those of the expert raters. Implications are discussed, including the appropriate selection of expert raters for different kinds of creativity assessment.

AB - The Consensual Assessment Technique (CAT) is one of the most highly regarded assessment tools in creativity, but it is often difficult and/or expensive to assemble the teams of experts required by the CAT. Some researchers have tried using nonexpert raters in their place, but the validity of replacing experts with nonexperts has not been adequately tested. Expert (n = 10) and nonexpert (n = 106) creativity ratings of 205 poems were compared and found to be quite different, making the simple replacement of experts by nonexpert raters suspect. Nonexpert raters' judgments of creativity were inconsistent (showing low interrater reliability) and did not match those of the expert raters. Implications are discussed, including the appropriate selection of expert raters for different kinds of creativity assessment.

UR - http://www.scopus.com/inward/record.url?scp=45849121773&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=45849121773&partnerID=8YFLogxK

U2 - 10.1080/10400410802059929

DO - 10.1080/10400410802059929

M3 - Article

AN - SCOPUS:45849121773

SN - 1040-0419

VL - 20

SP - 171

EP - 178

JO - Creativity Research Journal

JF - Creativity Research Journal

IS - 2

ER -

A comparison of expert and nonexpert raters using the consensual assessment technique

Abstract

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this