TY - JOUR
T1 - Optimising mHealth helpdesk responsiveness in South Africa
T2 - Towards automated message triage
AU - Engelhard, Matthew
AU - Copley, Charles
AU - Watson, Jacqui
AU - Pillay, Yogan
AU - Barron, Peter
AU - Lefevre, Amnesty E
PY - 2018/1/1
Y1 - 2018/1/1
N2 - In South Africa, a national-level helpdesk was established in August 2014 as a social accountability mechanism for improving governance, allowing recipients of public sector services to send complaints, compliments and questions directly to a team of National Department of Health (NDoH) staff members via text message. As demand increases, mechanisms to streamline and improve the helpdesk must be explored. This work aims to evaluate the need for and feasibility of automated message triage to improve helpdesk responsiveness to high-priority messages. Drawing from 65 768 messages submitted between October 2016 and July 2017, the quality of helpdesk message handling was evaluated via detailed inspection of (1) a random sample of 481 messages and (2) messages reporting mistreatment of women, as identified using expert-curated keywords. Automated triage was explored by training a naive Bayes classifier to replicate message labels assigned by NDoH staff. Classifier performance was evaluated on 12 526 messages withheld from the training set. 90 of 481 (18.7%) NDoH responses were scored as suboptimal or incorrect, with median response time of 4.0 hours. 32 reports of facility-based mistreatment and 39 of partner and family violence were identified; NDoH response time and appropriateness for these messages were not superior to the random sample (P>0.05). The naive Bayes classifier had average accuracy of 85.4%, with ≥98% specificity for infrequently appearing (<50%) labels. These results show that helpdesk handling of mistreatment of women could be improved. Keyword matching and naive Bayes effectively identified uncommon messages of interest and could support automated triage to improve handling of high-priority messages.
AB - In South Africa, a national-level helpdesk was established in August 2014 as a social accountability mechanism for improving governance, allowing recipients of public sector services to send complaints, compliments and questions directly to a team of National Department of Health (NDoH) staff members via text message. As demand increases, mechanisms to streamline and improve the helpdesk must be explored. This work aims to evaluate the need for and feasibility of automated message triage to improve helpdesk responsiveness to high-priority messages. Drawing from 65 768 messages submitted between October 2016 and July 2017, the quality of helpdesk message handling was evaluated via detailed inspection of (1) a random sample of 481 messages and (2) messages reporting mistreatment of women, as identified using expert-curated keywords. Automated triage was explored by training a naive Bayes classifier to replicate message labels assigned by NDoH staff. Classifier performance was evaluated on 12 526 messages withheld from the training set. 90 of 481 (18.7%) NDoH responses were scored as suboptimal or incorrect, with median response time of 4.0 hours. 32 reports of facility-based mistreatment and 39 of partner and family violence were identified; NDoH response time and appropriateness for these messages were not superior to the random sample (P>0.05). The naive Bayes classifier had average accuracy of 85.4%, with ≥98% specificity for infrequently appearing (<50%) labels. These results show that helpdesk handling of mistreatment of women could be improved. Keyword matching and naive Bayes effectively identified uncommon messages of interest and could support automated triage to improve handling of high-priority messages.
UR - http://www.scopus.com/inward/record.url?scp=85061813504&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85061813504&partnerID=8YFLogxK
U2 - 10.1136/bmjgh-2017-000567
DO - 10.1136/bmjgh-2017-000567
M3 - Article
C2 - 29713508
AN - SCOPUS:85061813504
SN - 2059-7908
VL - 3
JO - BMJ Global Health
JF - BMJ Global Health
M1 - e000567
ER -