TY - JOUR
T1 - Improving the Identification of Diabetic Retinopathy and Related Conditions in the Electronic Health Record Using Natural Language Processing Methods
AU - Harrigian, Keith
AU - Tran, Diep
AU - Tang, Tina
AU - Gonzales, Anthony
AU - Nagy, Paul
AU - Kharrazi, Hadi
AU - Dredze, Mark
AU - Cai, Cindy X.
N1 - Publisher Copyright:
© 2024 American Academy of Ophthalmology
PY - 2024/11/1
Y1 - 2024/11/1
N2 - Purpose: To compare the performance of 3 phenotyping methods in identifying diabetic retinopathy (DR) and related clinical conditions. Design: Three phenotyping methods were used to identify clinical conditions including unspecified DR, nonproliferative DR (NPDR) (mild, moderate, severe), consolidated NPDR (unspecified DR or any NPDR), proliferative DR, diabetic macular edema (DME), vitreous hemorrhage, retinal detachment (RD) (tractional RD or combined tractional and rhegmatogenous RD), and neovascular glaucoma (NVG). The first method used only International Classification of Diseases, 10th Revision (ICD-10) diagnosis codes (ICD-10 Lookup System). The next 2 methods used a Bidirectional Encoder Representations from Transformers with a dense Multilayer Perceptron output layer natural language processing (NLP) framework. The NLP framework was applied either to free-text of provider notes (Text-Only NLP System) or both free-text and ICD-10 diagnosis codes (Text-and-International Classification of Diseases [ICD] NLP System). Subjects: Adults ≥18 years with diabetes mellitus seen at the Wilmer Eye Institute. Methods: We compared the performance of the 3 phenotyping methods in identifying the DR related conditions with gold standard chart review. We also compared the estimated disease prevalence using each method. Main Outcome Measures: Performance of each method was reported as the macro F1 score. The agreement between the methods was calculated using the kappa statistic. Prevalence estimates were also calculated for each method. Results: A total of 91 097 patients and 692 486 office visits were included in the study. Compared with the gold standard, the Text-and-ICD NLP System had the highest F1 score for most clinical conditions (range 0.39–0.64). The agreement between the ICD-10 Lookup System and Text-Only NLP System varied (kappa of 0.21–0.81). The prevalence of DR and related conditions ranged from 1.1% for NVG to 17.9% for DME (using the Text-and-ICD NLP System). Conclusions: The prevalence of DR and related conditions varied significantly depending on the methodology of identifying cases. The best performing phenotyping method was the Text-and-ICD NLP System that used information in both diagnosis codes as well as free-text notes. Financial Disclosures: Proprietary or commercial disclosure may be found in the Footnotes and Disclosures at the end of this article.
AB - Purpose: To compare the performance of 3 phenotyping methods in identifying diabetic retinopathy (DR) and related clinical conditions. Design: Three phenotyping methods were used to identify clinical conditions including unspecified DR, nonproliferative DR (NPDR) (mild, moderate, severe), consolidated NPDR (unspecified DR or any NPDR), proliferative DR, diabetic macular edema (DME), vitreous hemorrhage, retinal detachment (RD) (tractional RD or combined tractional and rhegmatogenous RD), and neovascular glaucoma (NVG). The first method used only International Classification of Diseases, 10th Revision (ICD-10) diagnosis codes (ICD-10 Lookup System). The next 2 methods used a Bidirectional Encoder Representations from Transformers with a dense Multilayer Perceptron output layer natural language processing (NLP) framework. The NLP framework was applied either to free-text of provider notes (Text-Only NLP System) or both free-text and ICD-10 diagnosis codes (Text-and-International Classification of Diseases [ICD] NLP System). Subjects: Adults ≥18 years with diabetes mellitus seen at the Wilmer Eye Institute. Methods: We compared the performance of the 3 phenotyping methods in identifying the DR related conditions with gold standard chart review. We also compared the estimated disease prevalence using each method. Main Outcome Measures: Performance of each method was reported as the macro F1 score. The agreement between the methods was calculated using the kappa statistic. Prevalence estimates were also calculated for each method. Results: A total of 91 097 patients and 692 486 office visits were included in the study. Compared with the gold standard, the Text-and-ICD NLP System had the highest F1 score for most clinical conditions (range 0.39–0.64). The agreement between the ICD-10 Lookup System and Text-Only NLP System varied (kappa of 0.21–0.81). The prevalence of DR and related conditions ranged from 1.1% for NVG to 17.9% for DME (using the Text-and-ICD NLP System). Conclusions: The prevalence of DR and related conditions varied significantly depending on the methodology of identifying cases. The best performing phenotyping method was the Text-and-ICD NLP System that used information in both diagnosis codes as well as free-text notes. Financial Disclosures: Proprietary or commercial disclosure may be found in the Footnotes and Disclosures at the end of this article.
KW - Clinical free-text notes
KW - Diabetic retinopathy
KW - Electronic health record
KW - Natural language processing
KW - Prevalence
UR - http://www.scopus.com/inward/record.url?scp=85201261073&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85201261073&partnerID=8YFLogxK
U2 - 10.1016/j.xops.2024.100578
DO - 10.1016/j.xops.2024.100578
M3 - Article
C2 - 39253550
AN - SCOPUS:85201261073
SN - 2666-9145
VL - 4
JO - Ophthalmology Science
JF - Ophthalmology Science
IS - 6
M1 - 100578
ER -