TY - JOUR
T1 - Accuracy and reliability of eye-based vs quadrant-based diagnosis of plus disease in retinopathy of prematurity
AU - Imaging and Informatics in Retinopathy of Prematurity (i-ROP) Research Consortium
AU - Kim, Sang Jin
AU - Peter Campbell, J.
AU - Kalpathy-Cramer, Jayashree
AU - Ostmo, Susan
AU - Jonas, Karyn E.
AU - Choi, Dongseok
AU - Paul Chan, R. V.
AU - Chiang, Michael F.
AU - Sonmez, Kemal
AU - Horowitz, Jason
AU - Coki, Osode
AU - Eccles, Cheryl Ann
AU - Sarna, Leora
AU - Orlin, Anton
AU - Berrocal, Audina
AU - Negron, Catherin
AU - Denser, Kimberly
AU - Cumming, Kristi
AU - Osentoski, Tammy
AU - Check, Tammy
AU - Zajechowski, Mary
AU - Lee, Thomas
AU - Kruger, Evan
AU - McGovern, Kathryn
AU - Simmons, Charles
AU - Murthy, Raghu
AU - Galvis, Sharon
AU - Rotter, Jerome
AU - Chen, Ida
AU - Li, Xiaohui
AU - Taylor, Kent
AU - Roll, Kaye
AU - Erdogmus, Deniz
AU - Ioannidis, Stratis
AU - Martinez-Castellanos, Maria Ana
AU - Salinas-Longoria, Samantha
AU - Romero, Rafael
AU - Arriola, Andrea
AU - Olguin-Manriquez, Francisco
AU - Meraz-Gutierrez, Miroslava
AU - Dulanto-Reinoso, Carlos M.
AU - Montero-Mendoza, Cristina
N1 - Publisher Copyright:
© 2018 American Medical Association. All rights reserved.
PY - 2018/6
Y1 - 2018/6
N2 - IMPORTANCE Presence of plus disease in retinopathy of prematurity is the most critical element in identifying treatment-requiring disease. However, there is significant variability in plus disease diagnosis. In particular, plus disease has been defined as 2 or more quadrants of vascular abnormality, and it is not clear whether it is more reliably and accurately diagnosed by eye-based assessment of overall retinal appearance or by quadrant-based assessment combining grades of 4 individual quadrants. OBJECTIVE To compare eye-based vs quadrant-based diagnosis of plus disease and to provide insight for ophthalmologists about the diagnostic process. DESIGN, SETTING, AND PARTICIPANTS In this multicenter cohort study, we developed a database of 197 wide-angle retinal images from 141 preterm infants from neonatal intensive care units at 9 academic institutions (enrolled from July 2011 to December 2016). Each image was assigned a reference standard diagnosis based on consensus image-based and clinical diagnosis. Data analysis was performed from February 2017 to September 2017. INTERVENTIONS Six graders independently diagnosed each of the 4 quadrants (cropped images) of the 197 eyes (quadrant-based diagnosis) as well as the entire image (eye-based diagnosis). Images were displayed individually, in random order. Quadrant-based diagnosis of plus disease was made when 2 or more quadrants were diagnosed as indicating plus disease by combining grades of individual quadrants post hoc. MAIN OUTCOMES AND MEASURES Intragrader and intergrader reliability (absolute agreement and ? statistic) and accuracy compared with the reference standard diagnosis. RESULTS Of the 141 included preterm infants, 65 (46.1%) were female and 116 (82.3%) white, and the mean (SD) gestational age was 27.0 (2.6) weeks. There was variable agreement between eye-based and quadrant-based diagnosis among the 6 graders (Cohen ? range, 0.32-0.75). Four graders showed underdiagnosis of plus disease with quadrant-based diagnosis compared with eye-based diagnosis (by McNemar test). Intergrader agreement of quadrant-based diagnosis was lower than that of eye-based diagnosis (Fleiss ?, 0.75 [95% CI, 0.71-0.78] vs 0.55 [95% CI, 0.51-0.59]). The accuracy of eye-based diagnosis compared with the reference standard diagnosis was substantial to near-perfect, whereas that of quadrant-based plus disease diagnosis was only moderate to substantial for each grader. CONCLUSIONS AND RELEVANCE Graders had lower reliability and accuracy using quadrant-based diagnosis combining grades of individual quadrants than with eye-based diagnosis, suggesting that eye-based diagnosis has advantages over quadrant-based diagnosis. This has implications for more precise definitions of plus disease regarding the criterion of 2 or more quadrants, clinical care, computer-based image analysis, and education for all ophthalmologists who manage retinopathy of prematurity.
AB - IMPORTANCE Presence of plus disease in retinopathy of prematurity is the most critical element in identifying treatment-requiring disease. However, there is significant variability in plus disease diagnosis. In particular, plus disease has been defined as 2 or more quadrants of vascular abnormality, and it is not clear whether it is more reliably and accurately diagnosed by eye-based assessment of overall retinal appearance or by quadrant-based assessment combining grades of 4 individual quadrants. OBJECTIVE To compare eye-based vs quadrant-based diagnosis of plus disease and to provide insight for ophthalmologists about the diagnostic process. DESIGN, SETTING, AND PARTICIPANTS In this multicenter cohort study, we developed a database of 197 wide-angle retinal images from 141 preterm infants from neonatal intensive care units at 9 academic institutions (enrolled from July 2011 to December 2016). Each image was assigned a reference standard diagnosis based on consensus image-based and clinical diagnosis. Data analysis was performed from February 2017 to September 2017. INTERVENTIONS Six graders independently diagnosed each of the 4 quadrants (cropped images) of the 197 eyes (quadrant-based diagnosis) as well as the entire image (eye-based diagnosis). Images were displayed individually, in random order. Quadrant-based diagnosis of plus disease was made when 2 or more quadrants were diagnosed as indicating plus disease by combining grades of individual quadrants post hoc. MAIN OUTCOMES AND MEASURES Intragrader and intergrader reliability (absolute agreement and ? statistic) and accuracy compared with the reference standard diagnosis. RESULTS Of the 141 included preterm infants, 65 (46.1%) were female and 116 (82.3%) white, and the mean (SD) gestational age was 27.0 (2.6) weeks. There was variable agreement between eye-based and quadrant-based diagnosis among the 6 graders (Cohen ? range, 0.32-0.75). Four graders showed underdiagnosis of plus disease with quadrant-based diagnosis compared with eye-based diagnosis (by McNemar test). Intergrader agreement of quadrant-based diagnosis was lower than that of eye-based diagnosis (Fleiss ?, 0.75 [95% CI, 0.71-0.78] vs 0.55 [95% CI, 0.51-0.59]). The accuracy of eye-based diagnosis compared with the reference standard diagnosis was substantial to near-perfect, whereas that of quadrant-based plus disease diagnosis was only moderate to substantial for each grader. CONCLUSIONS AND RELEVANCE Graders had lower reliability and accuracy using quadrant-based diagnosis combining grades of individual quadrants than with eye-based diagnosis, suggesting that eye-based diagnosis has advantages over quadrant-based diagnosis. This has implications for more precise definitions of plus disease regarding the criterion of 2 or more quadrants, clinical care, computer-based image analysis, and education for all ophthalmologists who manage retinopathy of prematurity.
UR - http://www.scopus.com/inward/record.url?scp=85048787689&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85048787689&partnerID=8YFLogxK
U2 - 10.1001/jamaophthalmol.2018.1195
DO - 10.1001/jamaophthalmol.2018.1195
M3 - Article
C2 - 29710185
AN - SCOPUS:85048787689
SN - 2168-6165
VL - 136
SP - 648
EP - 655
JO - JAMA ophthalmology
JF - JAMA ophthalmology
IS - 6
ER -