Analysis of underlying causes of inter-expert disagreement in reti-nopathy of prematurity diagnosis: Application of machine learning principles

E. Ataer-Cansizoglu, J. Kalpathy-Cramer, S. You, K. Keck, D. Erdogmus, M. F. Chiang

Research output: Contribution to journalArticlepeer-review

20 Scopus citations


Objective: Inter-expert variability in imagebased clinical diagnosis has been demonstrated in many diseases including retinopa -thy of prematurity (ROP), which is a disease affecting low birth weight infants and is a major cause of childhood blindness. In order to better understand the underlying causes of variability among experts, we propose a method to quantify the variability of expert decisions and analyze the relationship between expert diagnoses and features computed from the images. Identification of these features is relevant for development of computer-based decision support systems and educational systems in ROP, and these methods may be applicable to other diseases where inter-expert variability is observed.

Methods: The experiments were carried out on a dataset of 34 retinal images, each with diagnoses provided independently by 22 experts. Analysis was performed using concepts of Mutual Information (MI) and Kernel Density Estimation. A large set of structural features (a total of 66) were extracted from retinal images. Feature selection was utilized to identify the most important features that correlated to actual clinical decisions by the 22 study experts.

Results: The results demonstrate that a group of observers (17 among 22) decide consistently with each other. Mean and second central moment of arteriolar tortuosity is among the reasons of disagreement between this group and the rest of the observers, meaning that the group of experts consider amount of tortuosity as well as the variation of tortuosity in the image.

Conclusion: Given a set of image-based features, the proposed analysis method can identify critical image-based features that lead to expert agreement and disagreement in diagnosis of ROP. Although tree-based features and various statistics such as central moment are not popular in the literature, our results suggest that they are important for diagnosis.

Original languageEnglish (US)
Pages (from-to)93-102
Number of pages10
JournalMethods of Information in Medicine
Issue number1
StatePublished - 2015


  • Feature selection
  • Inter-expert disagreement
  • Kernel density estimation
  • Retinopathy of prematurity

ASJC Scopus subject areas

  • Health Informatics
  • Advanced and Specialized Nursing
  • Health Information Management


Dive into the research topics of 'Analysis of underlying causes of inter-expert disagreement in reti-nopathy of prematurity diagnosis: Application of machine learning principles'. Together they form a unique fingerprint.

Cite this