TY - JOUR
T1 - A tool for automatic scoring of spelling performance
AU - Themistocleous, Charalambos
AU - Neophytou, Kyriaki
AU - Rapp, Brenda
AU - Tsapkini, Kyrana
N1 - Funding Information:
This study was supported by the Science of Learning Institute at Johns Hopkins University and National Institute on Deafness and Other Communication Disorders Grant R01 DC014475 (to Kyrana Tsapkini).
Publisher Copyright:
© 2020 The Authors.
PY - 2020/12
Y1 - 2020/12
N2 - Purpose: The evaluation of spelling performance in aphasia reveals deficits in written language and can facilitate the design of targeted writing treatments. Nevertheless, manual scoring of spelling performance is time-consuming, laborious, and error prone. We propose a novel method based on the use of distance metrics to automatically score spelling. This study compares six automatic distance metrics to identify the metric that best corresponds to the gold standard— manual scoring—using data from manually obtained spelling scores from individuals with primary progressive aphasia. Method: Three thousand five hundred forty word and nonword spelling productions from 42 individuals with primary progressive aphasia were scored manually. The gold standard—the manual scores—were compared to scores from six automated distance metrics: sequence matcher ratio, Damerau–Levenshtein distance, normalized Damerau– Levenshtein distance, Jaccard distance, Masi distance, and Jaro–Winkler similarity distance. We evaluated each distance metric based on its correlation with the manual spelling score. Results: All automatic distance scores had high correlation with the manual method for both words and nonwords. The normalized Damerau–Levenshtein distance provided the highest correlation with the manual scoring for both words (rs = .99) and nonwords (rs = .95). Conclusions: The high correlation between the automated and manual methods suggests that automatic spelling scoring constitutes a quick and objective approach that can reliably substitute the existing manual and time-consuming spelling scoring process, an important asset for both researchers and clinicians.
AB - Purpose: The evaluation of spelling performance in aphasia reveals deficits in written language and can facilitate the design of targeted writing treatments. Nevertheless, manual scoring of spelling performance is time-consuming, laborious, and error prone. We propose a novel method based on the use of distance metrics to automatically score spelling. This study compares six automatic distance metrics to identify the metric that best corresponds to the gold standard— manual scoring—using data from manually obtained spelling scores from individuals with primary progressive aphasia. Method: Three thousand five hundred forty word and nonword spelling productions from 42 individuals with primary progressive aphasia were scored manually. The gold standard—the manual scores—were compared to scores from six automated distance metrics: sequence matcher ratio, Damerau–Levenshtein distance, normalized Damerau– Levenshtein distance, Jaccard distance, Masi distance, and Jaro–Winkler similarity distance. We evaluated each distance metric based on its correlation with the manual spelling score. Results: All automatic distance scores had high correlation with the manual method for both words and nonwords. The normalized Damerau–Levenshtein distance provided the highest correlation with the manual scoring for both words (rs = .99) and nonwords (rs = .95). Conclusions: The high correlation between the automated and manual methods suggests that automatic spelling scoring constitutes a quick and objective approach that can reliably substitute the existing manual and time-consuming spelling scoring process, an important asset for both researchers and clinicians.
UR - http://www.scopus.com/inward/record.url?scp=85097809335&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85097809335&partnerID=8YFLogxK
U2 - 10.1044/2020_JSLHR-20-00177
DO - 10.1044/2020_JSLHR-20-00177
M3 - Article
C2 - 33151810
AN - SCOPUS:85097809335
SN - 1092-4388
VL - 63
SP - 4179
EP - 4192
JO - Journal of Speech and Hearing Disorders
JF - Journal of Speech and Hearing Disorders
IS - 12
ER -